Kafka Streams
Stream Processing Library for Apache Kafka
@jaceklaskowski
/
StackOverflow
/
GitHub
/
LinkedIn
The "Internals" Books:
books.japila.pl
## Agenda 1. [Kafka Streams](#/intro) 1. [High-Level Streams DSL](#/high-level-streams-dsl) 1. [Low-Level Processor API](#/low-level-processor-api)
## Kafka Streams 1. **Kafka Streams** is a client library for **stream processing applications** that process data stored in Kafka 1. Uses Kafka Producer and Consumer APIs 1. Stream processing primitives * High-level **Streams DSL** * Low-level **Processor API** 1. Supports **Exactly-Once Processing** * Records are processed once and only once 1. Supports **Fault-Tolerant Local State** for stateful operations * e.g. windowed joins and aggregations 1. [Home Page](https://kafka.apache.org/documentation/streams/)
## High-Level Streams DSL <small>(1 of 2)</small> 1. **Streams DSL** is a high-level API built on top of the Processor API 1. Built-in abstractions for streams and tables * **KStream** * **KTable** * **GlobalKTable** 1. Declarative, functional programming style * Stateless transformations (e.g. **map** and **filter**) * Stateful transformations (e.g. **count**, **reduce**, **join**, and windowing) 1. Recommended for most developers, esp. beginners
## High-Level Streams DSL <small>(2 of 2)</small> 1. Typical development flow 1. Read input streams (from Kafka topics) 1. Apply transformations 1. Write output streams (to Kafka topics) 1. Interactive queries (e.g. via a REST API) 1. Learn more in [The Internals of Kafka Streams](https://books.japila.pl/kafka-streams-internals/kstream/)
## Low-Level Processor API 1. Stream **Processor API** is a low-level API to compose topologies made up of custom processors * Node in a processor topology for a single processing step 1. [Processor<KIn,VIn,KOut,VOut>](https://kafka.apache.org/33/javadoc/org/apache/kafka/streams/processor/api/Processor.html) * (KIn,VIn) ==> Processor ==> (KOut,VOut) 1. Allows for stateless and stateful operations * Arbitrary stream processors * Connect processors with state stores 1. Combining the convenience of Streams DSL with the power and flexibility of Processor API
## Recap 1. [Kafka Streams](#/intro) 1. [High-Level Streams DSL](#/high-level-streams-dsl) 1. [Low-Level Processor API](#/low-level-processor-api)
# Questions? * Read [The Internals of Apache Kafka](https://books.japila.pl/kafka-internals/) * Read [The Internals of Kafka Streams](https://books.japila.pl/kafka-streams-internals) * Read [The Internals of ksqlDB](https://books.japila.pl/ksqldb-internals/) * Follow [@jaceklaskowski](https://twitter.com/jaceklaskowski) on twitter (DMs open) * Upvote [my questions and answers on StackOverflow](http://stackoverflow.com/users/1305344/jacek-laskowski) * Contact me at **jacek@japila.pl**
