kafka

How to assign partitions to your consumers?

Introduction to Kafka assignors

Integrating with Kafka usually starts with subscribing your consumers to Kafka with default configuration and it just works. However, after working a while with Kafka you will encounter various "weird" behaviors originating from rebalancing settings such as chosen assignor implementation. Let's see how different assignors work and what are their pros and cons.

Posted by Łukasz Chrząszcz on Friday, January 7, 2022

How to test the application's integration with Kafka?

Example of testing application's integration with Kafka for both sunny-day scenarios and failure cases.

After introducing Kafka to your application you want to test it properly. For sure you want to write some integration tests with real Kafka working underneath. What is more, you probably want to test not only sunny-day scenarios but failure cases as well. How to achieve that? Let us discover how Testcontainers and Toxiproxy fit in with Kafka in your application's integration tests!

Posted by Łukasz Chrząszcz on Sunday, September 6, 2020

How do transactions work in Apache Kafka?

Happy path of an application performing "consume, transform, produce" in a single transaction.

Do you need an exactly-once guarantee in Apache Kafka? Transactions enable you to achieve it, but how does it work under-the-hood? Let's skim through the happy path in the code and find out.

Posted by Łukasz Chrząszcz on Sunday, December 1, 2019

How to count events in Kafka Streams?

Stores and aggregation in Kafka Streams

Kafka Streams enables you to use stateful processing via Stores. Let's go through a simple example of order-counting app and discover how Stores fit in the stream processing world

Posted by Łukasz Chrząszcz on Saturday, September 28, 2019

How to use Kafka Streams to process events from one topic to another?

Introduction to Kafka Streams

Kafka Streams is a high-level client library for interacting with Kafka. How does it help to process events from topic to topic? What are the capabilities of this library? Join me and let's find out

Posted by Łukasz Chrząszcz on Tuesday, August 6, 2019

What happens when a new consumer joins the group in Kafka?

Rebalancing in Apache Kafka

Ever wondered how Kafka manages rebalancing? How are partitions assigned? Let's find out!

Posted by Łukasz Chrząszcz on Sunday, July 14, 2019

What does the heartbeat thread do in Kafka Consumer?

Behavior of Kafka consumer's background thread

We will dig into the heartbeat thread code to discover what it does and how it communicates with the rest of the consumer.

Posted by Łukasz Chrząszcz on Saturday, June 22, 2019

What happens when you call poll on Kafka Consumer?

Internals of Kafka consumer initialization and first fetch

We will create a new consumer, subscribe to a topic and execute first poll, describing what happens step by step.

Posted by Łukasz Chrząszcz on Sunday, June 16, 2019

Kafka 101

Introduction to Apache Kafka

Let's explore Kafka basics. We will discover what it does, what problem does it solve, and we will explore simple message passing from producer to consumer.

Posted by Łukasz Chrząszcz on Sunday, May 26, 2019

Łukasz Chrząszcz