https://chrzaszcz.dev/tags/kafka/ Recent content in kafka on Łukasz Chrząszcz Hugo -- gohugo.io en-us Fri, 07 Jan 2022 00:00:00 +0000 https://chrzaszcz.dev/2021/09/kafka-assignors/ Fri, 07 Jan 2022 00:00:00 +0000 https://chrzaszcz.dev/2021/09/kafka-assignors/ Using Kafka is usually fairly simple. You just create consumers, subscribe them to the topic of your choice, and your messages are being automagically consumed. Usually that is all you need to be “successful” with Kafka. Unfortunately, sometimes things do not go as planned, and you end up with your throughput looking like in one of the diagrams below. Although it is never an easy answer what is causing such behavior, nevertheless it might be the case your consumers are suffering from using the wrong assignor. https://chrzaszcz.dev/2020/08/kafka-testing/ Sun, 06 Sep 2020 00:00:00 +0000 https://chrzaszcz.dev/2020/08/kafka-testing/ Every application needs proper tests. We can write them on many levels ranging from unit tests, through integration tests to performance tests. When it comes to integrating our applications with Kafka it is easy to make mistakes and introduce bugs in the code, so it is crucial to write proper tests checking this integration. To write true integration tests you need a real-life Kafka working somewhere to test how your application cooperates with it. https://chrzaszcz.dev/2019/12/kafka-transactions/ Sun, 01 Dec 2019 00:00:00 +0000 https://chrzaszcz.dev/2019/12/kafka-transactions/ When you first encounter Kafka, you will face either at-least-once, or at-most-once guarantees, depending on when you commit your offsets. This is sufficient for many or even most use cases. However, if you are processing events from one topic to another, you might worry about the duplication of your messages on the target topic, or lost events on the source topic. A solution to that problem is a transaction. What can we expect from Kafka’s transactions? https://chrzaszcz.dev/2019/09/kafka-streams-store-basics/ Sat, 28 Sep 2019 00:00:00 +0000 https://chrzaszcz.dev/2019/09/kafka-streams-store-basics/ Stateful? In the previous post, we have discussed how to define topologies in Kafka Streams to apply our processing logic to every record and send it to another topic. Examples we have run through, revolved around stateless processing, but moving forward, what if we want to aggregate some events? What if we want to calculate the average value of users’ orders in the last hour or calculate how many times did users order pizza in our pizzeria? https://chrzaszcz.dev/2019/08/kafka-streams/ Tue, 06 Aug 2019 00:00:00 +0000 https://chrzaszcz.dev/2019/08/kafka-streams/ What is Kafka Streams? We have already discussed a lot about producers and consumers. We know that we can take a consumer, poll for some messages, transform them, and finally publish the result on another topic. As simple as it sounds it might get boring to write yet another thread that does the fetching, processing, error handling, and again and again. Not only it is cumbersome, but also error-prone. Is there a better way? https://chrzaszcz.dev/2019/06/kafka-rebalancing/ Sun, 14 Jul 2019 00:00:00 +0000 https://chrzaszcz.dev/2019/06/kafka-rebalancing/ What is a rebalancing? As you know, all messages on a topic are spread out among the members of consumer group. Every consumer has its set of partitions assigned exclusively to it and rebalancing is all about maintaining all partitions assigned to active consumers. When one consumer dies Kafka needs to reassign orphaned partitions to the rest of the consumers. Similarly, when a new consumer joins the group Kafka needs to free up some partitions and assign them to the new consumers (if it can). https://chrzaszcz.dev/2019/06/kafka-heartbeat-thread/ Sat, 22 Jun 2019 00:00:00 +0000 https://chrzaszcz.dev/2019/06/kafka-heartbeat-thread/ Recap In the previous post we’ve discussed how does Kafka Consumer work underneath. Now we know what are the very first steps to fetch records. Remember what I’ve intentionally omitted from previous post? That’s right! Heartbeat thread. We will investigate what it does exactly and how it works in general. Samples and setup If you want to run some samples on debug to discover heartbeat code by yourself, please by all means use my simple project on this GitHub repo. https://chrzaszcz.dev/2019/06/16/kafka-consumer-poll/ Sun, 16 Jun 2019 00:00:00 +0000 https://chrzaszcz.dev/2019/06/16/kafka-consumer-poll/ Recap In the previous post we’ve discussed what Kafka is and how to interact with it. We explored how consumers subscribe to the topic and consume messages from it. We know that consumers form a group called consumer group and that Kafka split messages among members of the consumer group. That’s of course after the initialization is finished, but what exactly is done in the background when you create a new consumer and call the very first poll? https://chrzaszcz.dev/2019/05/26/kafka-101/ Sun, 26 May 2019 00:00:00 +0000 https://chrzaszcz.dev/2019/05/26/kafka-101/ What is Kafka? Kafka is a popular stream-processing platform. What does it mean? You can think of it as a service that takes messages from one place and put it in another. It allows you to construct asynchronous processing of events, create pub-sub mechanism or distribute work evenly among worker services - the are many use cases for it. Whatever your use case is, you need to know a few basic ideas about Kafka.