I'll start with the premise that Kafka is the ideal backplane for reliable capture and organization of data streams for downstream consumption. Then, I'll present microservice applications implemented using Akka Streams and Kafka Streams on top of Kafka. The goal is to understand the relative strengths and weaknesses of these toolkits for building Kafka-based streaming applications. I'll also compare and contrast them to Spark Streaming and Flink, to understand when those tools are better choices. Briefly, Akka Streams and Kafka Streams are best for data-centric microservices, while Spark Streaming and Flink are best for richer analytics over large volume streams where scalability through "automatic" partitioning is required.
SHARE THIS TALK
Lightbend, VP of Rocket Surgery
Dean Wampler, Ph.D., is the VP of Fast Data Engineering at Lightbend. He leads the development of Lightbend Fast Data Platform, a distribution of scalable, distributed stream processing tools including Spark, Flink, Kafka, and Akka, with machine learning and management tools. Dean is the author of several books, a frequent conference speaker and organizer, and he helps run several Meetups in Chicago.