With 5 demos, this article helps readers to understand fundamental concepts in Kafka
There are some key terms using in Kafka:
- Topic: Kafka messages are organized into topics. To consume messages, consumers need to specify the topic they want to read from. Same for producers when writing messages
- Partition: each topic has 1 or more paritions. Based on the key of the message, Kafka decides which partition the message is located. Also, with proper configuration, Kafka can replicate partitions to multiple machines to improve read performance and data recovery ability
We need to have the following items ready to be able to run through the demos:
- JDK 11+
- Follow step 1 and 2 to install and test Kafka on your local
- Install Docker on your local. Lately we will use this to setup a Kafka cluster on Docker which is convenient for testing
- Clone the repository kafka-plain-java to prepare for experiments
There are some important files in this repository:
- demo.jar: runnable jar file. Some commands we will use in demos:
| Command | Description |
|---|---|
| java -jar demo.jar --mode c --id "Consumer 2" | to start a consumer with id Consumer 2. Using --mode p to start a producer |
| java -jar demo.jar --mode c --topic topic-demo3 --bootstrap-server localhost:29092 | to start a consumer, connect to localhost:29092 and consume messages in topic topic-demo3 |
- RunnableConsumer.java: to consume messages when running with
--mode c - RunnableProducer.java: to produce messages when running with
--mode p
- Demo 1: multi-consumer instances with 1 topic and multi-partition: to understand how Kafka messsages are distributed to partition and how they're consumed if there're multiple consumers
- Demo 2: Kafka cluster: to answer 2 questions: 1/ how to setup a Kafka cluster locally? 2/ in a cluster, producer sends messages to 1 broker, can consumers read these messages from another one?
- Demo 3: testing replication: to test the partition replication in a Kafka cluster
- Demo 4: Kafka cluster in Docker
- Demo 5: data persistence