Building and deploying distributed streaming applications is a complicated, error-prone process. Most streaming applications use several frameworks, such as Kafka, Spark, and Akka. How you integrate components of these applications is usually an exercise left to development teams: How do you enable multiple teams to work on parts of the streaming application and then compose these segments? How do you simplify or automate boilerplate, such as serialization between components and URL management? How do devops manage deployment and lifecycle management? How do you expose you expose streaming pipelines and integrate with a microservices-based application? In this talk, we show how we are thinking about these problems and more, and demo tooling we are working on to solve these problems.
SHARE THIS TALK
Lightbend, Senior Product Director
Craig is a product manager with over thirty years experience in the software industry. Over the years, Craig's focus has been on building highly-performant, highly scalable solutions. He is skilled in taking new product ideas from the conceptual phase through implementation and translating business objectives into concrete products. Experienced at all levels of product management, including market and requirements analysis, pricing and marketing strategies, partner selection, and technology selection. Proven ability to work with partners, manage multiple-site teams, and represent company objectives in industry associations.
Lightbend, Senior SW Engineer
I'm a hands-on technical leader. While keeping strong bonds with the software stack and technical architecture, I guide and coach individuals to ensure that the team adapts and grows to excel on the expected goals.
My current professional interest is in the area of distributed and scalable stream processing, in particular using an integrated open-source based stack: "Get the data flowing, the value collectors going and the storage to scale."
Co-author of the book: Stream Processing with Apache Spark, by O'Reilly.