Kafka Learning Roadmap

What is Kafka and when should I use it?

Kafka is a distributed event streaming platform — a durable log of ordered messages that multiple consumers can read independently. Use Kafka when: (1) you need to decouple microservices with events, (2) you need to process data in real time (analytics, fraud detection), (3) you need an audit log, or (4) you need to fan out events to multiple consumers. Don't use Kafka for simple request-response communication — REST or gRPC is better there.

Kafka vs RabbitMQ — which should I use?

RabbitMQ for traditional message queuing — task queues, work distribution, simple pub/sub. Messages are consumed and deleted. Kafka for event streaming — messages are retained as a durable log, multiple consumer groups can read independently, and you can replay historical events. Kafka is more complex but more powerful. Most modern architectures at scale use Kafka; RabbitMQ is fine for simpler messaging needs.

What is a consumer group?

A consumer group is a set of consumer instances that coordinate to consume a topic. Each partition in a topic is assigned to exactly one consumer in the group — this enables parallel processing. Multiple consumer groups can consume the same topic independently, each getting their own copy of every message. This is Kafka's key advantage over traditional queues: one topic can serve multiple independent services simultaneously.

What does Kafka guarantee about message delivery?

By default, Kafka guarantees at-least-once delivery — messages won't be lost, but might be delivered twice if a consumer crashes after processing but before committing. Enable idempotent producers (enable.idempotence=true) and use exactly-once semantics (transactions) for at-most-once guarantees. For most use cases, at-least-once with idempotent consumers (checking if you've processed a message ID before) is the right approach.

What is Kafka Streams and when should I use it?

Kafka Streams is a Java library for building real-time streaming applications on top of Kafka — filtering, transforming, aggregating, and joining streams without a separate cluster (unlike Spark or Flink). Use it when you need stateful stream processing (windowed aggregations, joins) and you're already in the Java/JVM ecosystem. For Python or Node.js streaming, use Faust or the Kafka Streams REST Proxy.