# Kafka Explained Like You're 5: Events, Partitions, and Consumer Groups # Kafka Explained Like You're 5: Events, Partitions, and Consumer Groups > **TL;DR:** Apache Kafka is a distributed event streaming platform that lets systems communicate asynchronously at massive scale. This article builds up the full mental model — from why events exist, through topics, partitions, and consumer groups — with real architecture examples and honest trade-off analysis. No prior Kafka experience needed. ## Introduction How does a large application process millions of events every day? That's not a rhetorical question. It's the exact problem that LinkedIn's engineering team faced in 2010. Their infrastructure was generating enormous volumes of data — user activity, page views, job searches, connection requests — and their systems couldn't keep up. The traditional patterns they relied on were buckling under the weight. Out of that pain came Apache Kafka, which is now the backbone of real-time data pipelines at companies like Uber, Netflix, Airbnb, Twitter, and thousands of others. This article is for backend developers, software architects, and technical leads who want to understand Kafka deeply — not just the API surface, but the *reasoning* behind every design decision. We'll skip the boilerplate setup tutorial and go straight into architecture thinking. Assumed knowledge: basic familiarity with distributed systems concepts (databases, APIs, microservices). No prior Kafka experience required. --- ## Background: Why Systems Need Event Streaming ### The Traditional Request-Response Model In a classic web application, services talk to each other through synchronous HTTP calls. Service A needs something from Service B, so it sends a request and waits for a response. ```mermaid sequenceDiagram participant OrderService participant InventoryService participant PaymentService participant NotificationService OrderService->>InventoryService: Reserve item (HTTP) InventoryService-->>OrderService: OK OrderService->>PaymentService: Charge card (HTTP) PaymentService-->>OrderService: OK OrderService->>NotificationService: Send email (HTTP) NotificationService-->>OrderService: OK ``` This works fine when you have a handful of services and modest traffic. The moment your system grows, the cracks show fast: **Tight coupling.** Every service needs to know the exact address, API contract, and availability of every other service it depends on. Change one service's API and you potentially break five downstream callers. **Cascading failures.** If the Notification Service goes down, the Order Service's HTTP call fails. Now your order flow is broken — not because of a payment or inventory problem, but because of a *notification* problem. Services fail together. **No natural backpressure.** If 10,000 orders come in simultaneously, every service in the chain gets hammered at the same time. There's no buffer. Each service must scale to handle the peak load — even though most of the time that capacity sits idle. **No replay.** Once the HTTP response is returned, it's gone. If the Notification Service was down and missed 5,000 emails, those events are lost forever. ### What Event-Driven Architecture Changes Instead of services calling each other directly, they produce and consume **events**. An event is an immutable record of something that happened: "Order 4821 was placed at 14:32:09 UTC by user 99123." That event is written to a durable log. Any service that cares about it can read from that log — now, or an hour from now, or next week. This shift decouples producers from consumers. The Order Service doesn't need to know that a Notification Service exists. It just says "an order happened" and moves on. The Notification Service subscribes to order events on its own schedule. Real-world systems built on this model: - **Uber** uses Kafka to process real-time location events from millions of drivers and riders. - **Netflix** uses it to ingest 500 billion events per day for analytics and recommendations. - **Robinhood** uses it for trade execution and fraud detection pipelines. --- ## What Is Kafka? Apache Kafka is a **distributed event streaming platform**. At its core, it is a durable, ordered, high-throughput log that producers write to and consumers read from. It is **not** just a message queue, though it shares some superficial similarities. The critical distinction: traditional message queues delete messages after they're consumed. Kafka retains events for a configurable period (days, weeks, forever) and lets multiple independent consumers read the same data. ![High-level Kafka architecture diagram showing producers writing to a Kafka cluster and multiple consumer groups reading independently](https://i.ibb.co/B22sbvww/9318f920c137.png) Kafka's design priorities, ranked: 1. **Durability** — events are written to disk and replicated across multiple machines. 2. **Throughput** — millions of events per second on commodity hardware. 3. **Scalability** — horizontal scaling through partitioning. 4. **Fault tolerance** — replication means no single machine failure loses data. --- ## Understanding Events An **event** is a record of something that happened in your system. It has three components: | Component | Description | Example | |-----------|-------------|-------| | **Key** | Identifies what the event is about | `user_id: 99123` | | **Value** | The payload — what happened | `{ "action": "signup", "email": "..." }` | | **Timestamp** | When it happened | `2024-01-15T14:32:09Z` | | **Headers** | Optional metadata | `{ "source": "web", "region": "us-east-1" }` | Events are **immutable**. Once written, they cannot be changed. This is a feature, not a limitation. It gives you a reliable audit trail and lets any number of systems reconstruct state by replaying history. Here's what a real event payload might look like for an e-commerce order: ```json { "event_type": "order.created", "event_id": "evt_8a3f9c21-4b2e-4d8a-b3e1-9f2a1c7d4e5f", "timestamp": "2024-01-15T14:32:09.847Z", "version": "2.1", "data": { "order_id": "ord_4821", "user_id": "usr_99123", "items": [ { "sku": "LAPTOP-PRO-15", "quantity": 1, "unit_price_cents": 149900 }, { "sku": "MOUSE-WIRELESS", "quantity": 2, "unit_price_cents": 2999 } ], "total_cents": 155898, "currency": "USD", "shipping_address": { "city": "San Francisco", "state": "CA", "country": "US" } } } ``` **Producers** are the services that write events. **Consumers** are the services that read and act on them. The key insight: a producer never knows who its consumers are. It just appends to the log. Typical lifecycle for the `order.created` event above: 1. **OrderService** produces the event to Kafka. 2. **InventoryService** consumes it → reserves the stock. 3. **PaymentService** consumes it → initiates charge. 4. **NotificationService** consumes it → queues a confirmation email. 5. **AnalyticsService** consumes it → updates real-time revenue dashboards. 6. **FraudDetectionService** consumes it → runs risk scoring. All six consumers read the *same* event independently. None of them need to coordinate with each other. --- ## Topics: Organizing the Event Stream A **topic** is a named, ordered log of events. Think of it as a category or a channel. All events of the same type go into the same topic. ``` orders → all order lifecycle events payments → payment initiated, completed, failed user-activity → clicks, searches, profile updates inventory-updates → stock levels, restocks, reservations notifications → emails, push notifications, SMS ``` Topics let you separate concerns. The Payment Service subscribes to `orders` but not to `user-activity`. The Analytics Service might subscribe to all of them. > **Tip:** Topic naming conventions matter at scale. Use a consistent schema like `{domain}.{entity}.{event-type}` — e.g., `ecommerce.order.created`, `ecommerce.payment.completed`. This makes topics discoverable and makes access control policies easier to write. A topic can have any number of producers writing to it and any number of consumers reading from it. The log is append-only: new events are always added to the end. --- ## Partitions: The Engine of Kafka's Scalability Here's where Kafka gets interesting. A single topic can handle tens of thousands of events per second — but what if you need to handle *millions*? You need to parallelize. Kafka solves this with **partitions**. A topic is split into multiple partitions, and each partition is an independent ordered log. Producers write events to specific partitions. Consumers read from specific partitions. ![Diagram showing how a Kafka topic is split into partitions and how events are distributed across them based on key hashing](https://i.ibb.co/TBmPQxsQ/a8acfb3c6d26.png) ### How Partitions Work When a producer sends an event, Kafka uses the **event key** to determine which partition it goes to — typically by hashing the key and taking the modulus of the partition count: ```python def get_partition(event_key: str, num_partitions: int) -> int: """ Simplified partition assignment. Real Kafka uses Murmur2 hashing under the hood. """ key_bytes = event_key.encode('utf-8') hash_value = hash(key_bytes) # Kafka uses murmur2 return abs(hash_value) % num_partitions # Example: topic has 6 partitions num_partitions = 6 orders = [ ("ord_4821", "user_99123"), ("ord_4822", "user_44201"), ("ord_4823", "user_99123"), # Same user as ord_4821 ("ord_4824", "user_78902"), ] for order_id, user_id in orders: partition = get_partition(user_id, num_partitions) print(f"Order {order_id} (user: {user_id}) → Partition {partition}") # Output (illustrative, hash values are deterministic): # Order ord_4821 (user: user_99123) → Partition 3 # Order ord_4822 (user: user_44201) → Partition 1 # Order ord_4823 (user: user_99123) → Partition 3 ← same user, same partition # Order ord_4824 (user: user_78902) → Partition 5 ``` Notice that `user_99123` always goes to Partition 3. This is the key guarantee: **all events with the same key are ordered within a single partition**. For our order system, keying by `user_id` means all orders from a given user land in the same partition, in the exact order they were placed. The fulfillment service reading that partition will always process a user's orders in sequence — which matters if order N+1 should only ship after order N is confirmed. ### Partition Count and Throughput If a single partition can handle 10 MB/s of writes, then 12 partitions give you 120 MB/s. Scaling is almost purely additive. This is why Kafka can sustain millions of events per second on commodity hardware. > **Warning:** Choosing partition count is a consequential decision. You can increase partitions later, but doing so changes the key-to-partition mapping, breaking ordering guarantees for existing keys. Start with a thoughtful estimate: `desired_throughput / single_partition_throughput`, rounded up to a power of two. --- ## Why Kafka Is Fast: Sequential Writes and the Append-Only Log Kafka's throughput is not magic — it's a consequence of how it writes data. Random disk writes (jumping between locations on disk) are slow, even on SSDs. Sequential disk writes (appending to the end of a file) are orders of magnitude faster — often approaching the raw I/O bandwidth of the hardware. Kafka's log is **append-only**. Events are never updated or randomly inserted. Every write goes to the end of the current partition segment. This means: - **Disk throughput is maximized** — the OS can optimize sequential writes aggressively. - **Page cache works efficiently** — the OS keeps recently written data in memory. Hot reads are served from RAM, not disk. - **No write amplification** — unlike B-tree databases that must rebalance on every write. ```bash # Kafka partition storage on disk (simplified) /kafka/data/orders-0/ 00000000000000000000.log # Segment file: raw event bytes, append-only 00000000000000000000.index # Offset index for fast seeks 00000000000000000000.timeindex # Timestamp index 00000000000001048576.log # Next segment after 1GB 00000000000001048576.index ``` Each `.log` file is a raw byte sequence of event records. Kafka doesn't parse them — it just writes bytes and reads bytes. The consumer is responsible for deserialization. This design lets Kafka achieve **p99 write latencies under 5ms** and sustain **millions of messages per second** on a three-node cluster with modest hardware. --- ## Consumer Groups: The Most Misunderstood Concept in Kafka This is the concept that trips up almost everyone learning Kafka, so let's build the mental model carefully. ### The Problem Consumer Groups Solve Imagine your `orders` topic has 6 partitions. Millions of orders are flowing in. You have a single `InventoryService` instance reading all 6 partitions. The service is getting overwhelmed — it can't keep up. Naive solution: run two instances of `InventoryService`. But now both instances read from all 6 partitions — you've doubled your processing, which means every order gets processed *twice*. The inventory gets deducted twice. That's worse. You need a way for multiple service instances to divide the work without duplicating it. **Consumer groups** are Kafka's solution. ### How Consumer Groups Work A **consumer group** is a set of consumer instances that cooperate to consume a topic. Kafka assigns each partition to exactly one consumer within the group at any given time. ``` Topic: orders (6 partitions) Consumer Group: inventory-service ├── Instance A → reads Partitions 0, 1 ├── Instance B → reads Partitions 2, 3 └── Instance C → reads Partitions 4, 5 ``` Each event is processed by exactly one instance. Work is distributed. Throughput scales linearly with the number of instances (up to the number of partitions). ```python # Pseudocode illustrating consumer group semantics # (Illustrative, not Kafka's actual client API) class ConsumerGroupMember: def __init__(self, group_id: str, instance_id: str): self.group_id = group_id self.instance_id = instance_id self.assigned_partitions = [] # Set by Kafka's Group Coordinator def process_events(self): """ Each member only reads from its assigned partitions. Kafka's Group Coordinator ensures no two members in the same group are assigned the same partition. """ for partition in self.assigned_partitions: events = self.poll_partition(partition) for event in events: self.handle(event) self.commit_offset(partition, event.offset) def handle(self, event: dict): if event["event_type"] == "order.created": sku = event["data"]["items"][0]["sku"] quantity = event["data"]["items"][0]["quantity"] print(f"[{self.instance_id}] Reserving {quantity}x {sku}") # Call inventory database... # Three instances, same group instance_a = ConsumerGroupMember("inventory-service", "inv-A") instance_b = ConsumerGroupMember("inventory-service", "inv-B") instance_c = ConsumerGroupMember("inventory-service", "inv-C") # Kafka assigns partitions: # inv-A → [0, 1], inv-B → [2, 3], inv-C → [4, 5] ``` ### Multiple Independent Consumer Groups Here's the second key insight: **different consumer groups each maintain their own offset position**. Two groups reading the same topic are completely independent. ``` Topic: orders (6 partitions) Consumer Group: inventory-service └── 3 instances → reads all 6 partitions (2 each) Consumer Group: notification-service └── 2 instances → reads all 6 partitions (3 each) Consumer Group: analytics-pipeline └── 6 instances → reads all 6 partitions (1 each) ``` All three groups process every order event. None of them interfere with each other. This is the fundamental difference from a traditional queue, where consuming an event removes it. ![Diagram showing how multiple consumer groups independently consume the same Kafka topic partitions without interfering with each other](https://i.ibb.co/BHtTC707/f145d6460bcc.png) ### Rebalancing When a consumer instance joins or leaves a group (deployment, crash, scale-out), Kafka triggers a **rebalance** — it redistributes partition assignments across the remaining members. This is automatic and transparent to the application. ``` Before crash: inv-A → [0, 1] inv-B → [2, 3] inv-C → [4, 5] ← crashes After rebalance: inv-A → [0, 1, 4] inv-B → [2, 3, 5] ``` During rebalancing, consumption pauses briefly. This is the cost of automatic fault tolerance. In practice, with `PartitionAssignor` strategies like **Sticky Assignor**, Kafka minimizes unnecessary reassignments to reduce this pause. > **Note:** The rule of thumb: the maximum parallelism for a consumer group equals the number of partitions. If you have 6 partitions and 8 consumer instances, 2 instances will be idle — Kafka cannot assign a partition to more than one consumer in a group simultaneously. --- ## Kafka Architecture: The Full Picture Now that we have all the pieces, here's how they fit together: ```mermaid flowchart LR subgraph Producers P1[OrderService] P2[PaymentService] P3[UserService] end subgraph Kafka Cluster B1[Broker 1] B2[Broker 2] B3[Broker 3] ZK[(ZooKeeper / KRaft)] B1 --- ZK B2 --- ZK B3 --- ZK end subgraph Topics & Partitions T1["orders\nP0 P1 P2 P3 P4 P5"] T2["payments\nP0 P1 P2"] end subgraph Consumer Groups CG1["inventory-service\n(3 instances)"] CG2["notification-service\n(2 instances)"] CG3["analytics-pipeline\n(6 instances)"] end P1 --> B1 P2 --> B2 P3 --> B3 B1 --> T1 B2 --> T1 B3 --> T2 T1 --> CG1 T1 --> CG2 T1 --> CG3 T2 --> CG2 ``` **Brokers** are the Kafka servers. Each broker stores a subset of partitions. A topic's partitions are spread across brokers for load distribution and fault tolerance. Each partition has a **leader** (handles all reads/writes) and zero or more **replicas** on other brokers. Kafka's coordination layer (historically ZooKeeper, now moving to KRaft — Kafka's built-in Raft implementation) manages leader election, group membership, and cluster metadata. --- ## Kafka vs. Traditional Message Queues | Feature | Traditional Queue (RabbitMQ, SQS) | Kafka | |--------|-----------------------------------|-------| | **Message retention** | Deleted after consumption | Retained for configurable period | | **Multiple consumers** | Competing consumers share messages | Each consumer group gets all messages | | **Ordering** | Per-queue, generally FIFO | Per-partition, strict | | **Replayability** | No (message is gone) | Yes (seek to any offset) | | **Throughput** | Moderate (tens of thousands/sec) | Very high (millions/sec) | | **Delivery semantics** | At-least-once, at-most-once | At-least-once, exactly-once (with config) | | **Use case fit** | Task queues, work distribution | Event logs, analytics, audit trails | | **Operational complexity** | Lower | Higher | Traditional queues are the right choice for simple job queues — sending an email, resizing an image, running a background task. The message is a *command* with a single intended recipient. Kafka is the right choice when events need to be consumed by multiple independent systems, when replay is needed, or when throughput is in the millions-per-second range. --- ## Real-World Applications of Kafka ### Food Delivery Platform (e.g., DoorDash) When a customer places an order: - `OrderService` produces `order.placed` to the `orders` topic. - `DispatchService` consumes it → assigns a nearby driver. - `RestaurantService` consumes it → notifies the kitchen. - `EtaService` consumes it → begins calculating delivery windows. - `AnalyticsService` consumes it → updates real-time order volume dashboards. Driver location updates (every 4 seconds for every active driver) flow through a `location-updates` topic. At 100,000 active drivers, that's 25,000 events per second on that topic alone — trivial for Kafka, catastrophic for a database. ### E-Commerce Recommendation Engine Every product view, search query, cart addition, and purchase produces an event to `user-activity`. A recommendation service consumes this stream, updates a user behavior model in near real-time, and serves personalized recommendations with sub-100ms latency. ### Payment Processing Payment events flow through Kafka with strict ordering (keyed by `account_id`) to ensure balance updates are applied in sequence. The fraud detection system independently consumes the same stream, running ML scoring without blocking the payment flow. ### Analytics Pipelines Kafka's retention makes it a natural landing zone for event data before it flows into a data warehouse. Rather than batch-loading CSV files daily, raw events stream directly into Snowflake, BigQuery, or ClickHouse via Kafka Connect — enabling analytics on data that's minutes old instead of hours old. --- ## Trade-offs and Pitfalls ### The Good - **Decoupling** — producers and consumers evolve independently. - **Resilience** — a consumer being down doesn't affect producers or other consumers. Events wait in the log. - **Scalability** — add partitions and consumer instances to handle more load. - **Auditability** — the log is a complete history. You can always answer "what happened and when." ### The Hard Parts **Operational complexity.** Kafka clusters require care: monitoring consumer lag, tuning JVM settings, managing partition rebalancing, handling disk capacity. This is why managed Kafka (Confluent Cloud, AWS MSK, Aiven) has become popular. **Eventual consistency.** Because consumers process events asynchronously, the system is eventually consistent by design. If you read the database immediately after placing an order, inventory might not be updated yet. This requires careful API design and user experience considerations. **Schema evolution.** Events are immutable, but your event schema will change over time. Without a schema registry (e.g., Confluent Schema Registry with Avro or Protobuf), you'll eventually publish a breaking schema change that crashes your consumers. **Exactly-once semantics.** Kafka supports exactly-once delivery (idempotent producers + transactional consumers), but it requires explicit configuration and adds overhead. Most systems accept at-least-once delivery and make consumers idempotent instead. | Risk | Mitigation | |------|------------| | Consumer lag growing | Scale out consumer instances; optimize processing logic | | Partition skew | Choose partition keys with high cardinality and uniform distribution | | Schema breaking changes | Use Confluent Schema Registry with compatibility rules | | Disk saturation | Set `retention.ms` and `retention.bytes` appropriately per topic | | Rebalance storms | Use `session.timeout.ms` and `heartbeat.interval.ms` conservatively | --- ## Best Practices 1. **Key events by the entity they describe**, not by random IDs. Orders keyed by `user_id` or `order_id` — whichever ordering matters most to your consumers. 2. **Use a schema registry.** Raw JSON is fine in development. In production, Avro or Protobuf with schema compatibility enforcement prevents breaking changes from reaching consumers silently. 3. **Monitor consumer lag obsessively.** Lag is the number of events a consumer group is behind the latest offset. A lag that's growing means your consumers can't keep up — a leading indicator of an incident, not a lagging one. 4. **Design consumers to be idempotent.** At-least-once delivery means your consumer might process the same event twice (e.g., after a rebalance). If processing the event twice causes side effects (double-charging a card), your consumer has a bug. Use idempotency keys. 5. **Start with more partitions than you think you need.** Reducing partitions is not supported without recreating the topic. Starting with 12–24 partitions for a high-traffic topic gives you room to scale consumers without topology changes. 6. **Separate topics by retention requirements.** A topic holding raw click events (retain 7 days) should not be the same topic holding financial transaction events (retain 7 years for compliance). 7. **Use consumer group IDs that are specific and human-readable.** `notification-service-email-worker` is better than `consumer-group-1`. You will thank yourself during an incident. --- ## Conclusion Kafka's design can be understood completely from first principles: - Systems grow beyond what synchronous request-response can handle. - Events are an immutable record of what happened — durable, replayable, and decoupled from how they're processed. - Topics organize events by type. Partitions split topics to enable parallel processing at scale. - Consumer groups let multiple instances of a service divide partition assignments, achieving throughput that scales linearly — while separate groups ensure independent services each get a complete copy of the stream. - Sequential, append-only writes on disk give Kafka its extraordinary throughput at remarkably low latency. The next concrete step: deploy a local Kafka cluster with Docker Compose, create a topic with 6 partitions, write a producer that generates realistic JSON events keyed by a realistic entity ID, and run two consumer instances in the same group. Watch the partition assignments in real time. That hands-on experience makes the mental model click in a way no article fully can. --- ## Further Reading - **[Apache Kafka Documentation](https://kafka.apache.org/documentation/)** — The official docs are genuinely good, especially the Design section which covers the storage model in depth. - **"Designing Data-Intensive Applications" by Martin Kleppmann** — Chapter 11 covers stream processing and Kafka's design philosophy better than almost anything else written. - **[Confluent Developer — Kafka Tutorials](https://developer.confluent.io/tutorials/)** — Hands-on tutorials covering producers, consumers, Kafka Streams, and ksqlDB with real code. - **"The Log: What every software engineer should know about real-time data's unifying abstraction" by Jay Kreps** — The original essay by one of Kafka's creators. Foundational reading. - **[Kafka: The Definitive Guide (O'Reilly, 2nd Ed.)](https://www.oreilly.com/library/view/kafka-the-definitive/9781492043072/)** — The most comprehensive book on Kafka for practitioners.