Duration: 30 mins
In this exercise you’ll run multiple instances of OrderConsumerApp (from 05_kafka_consumer.md) in the same consumer group and watch partitions get assigned, rebalanced and reclaimed.
Start a Kafka cluster (01_kafka_docker.md)
Recreate topic orders with 3 partitions (delete and re-create if it has a different partition count already)
docker exec -it kafka bash
cd /opt/kafka
./bin/kafka-topics.sh --bootstrap-server :9092 --create --topic t1 --partitions 3
Start OrderConsumerApp once — call this instance c1 — with
group.id=CG1. It should be assigned all 3 partitions.
./bin/kafka-console-consumer.sh --bootstrap-server :9092 --topic t1 --command-property group.id=CG1
Run OrderProducerApp to send a batch of keyed records. Confirm c1 receives all of them.
kcat -P -b localhost:9092 -t t1 -p 0 -K :
(-K is for a key-value separator)
Start a second instance of OrderConsumerApp — c2 — same
group.id=CG1. Watch the logs (ConsumerRebalanceListener events, or
just the change in which partitions each instance’s poll returns
records for): the 3 partitions should now be split across c1 and
c2.
kcat -P -b 0.0.0.0:9092 -t t1 -p 2 -K :
Send another batch of keyed records and confirm messages are split between c1 and c2 roughly by partition, not by round-robin per-message.
Start a third instance — c3 — same group. Send more records. With 3 partitions and 3 consumers, each should now own exactly one.
Start a fourth instance — c4 — same group, and send more records. Predict first: does c4 receive anything? Confirm: a consumer group can have at most as many active consumers as there are partitions — extra consumers sit idle.
Kill c2 (Ctrl+C) and send another batch. Observe which of the
remaining consumers picks up the orphaned partition(s), and how long
the rebalance takes to happen (governed by session.timeout.ms /
heartbeat.interval.ms for the abrupt-kill case).
Discuss: what’s the practical implication for scaling a consumer application horizontally? (Partition count is a hard ceiling on effective parallelism per consumer group.)