Introduction

Apache Kafka is an open source message management service which aim is managing data communication between producers and consumers.

It’s even driven, which basically means we can set listeners (consumers) that will react when a particular event is triggered (by a producer).

All this makes it pretty handy for projects with a microservice architecture, but can be used for many other things, ie: events that trigger emails, logging, monitoring activity,…

This would be the simplified flow:

CLUSTER IS MADE OF BROKERS AND MANAGED BY ZOOKEEPER

A Kafka broker (aka Kafka server aka Kafka node) has the topics where producers will write to, and consumers will read from.

All brokers together make a cluster.

A cluster has always a broker that acts as the controller, which will assign partitions and monitor (other) broker failures.

Within a cluster, brokers talk to each other. To do that they use an external service called Zookeeper. Zookeeper will do other things as well, like be in charge of choosing (and changing when needed) a broker controller, management of topics and partitions metadata, metadata storage, balance of quotas for clients (both producers or consumers),…

But for this introduction we won’t talk much about Zookeeper, as its does a silent job and we don’t need to interact with it.

** Kafka seems to be getting rid of Zookeeper, but it still should not be used in production environments.

TOPICS

Events (or messages) are organised in topics. They live in brokers and are divided in “partitions”.

A topic can have 1 or more partitions.

A partition acts as a commit log.

Kafka will ONLY guarantee ordered messages within a partition. So you can’t expect messages delivered by (for example) partition 1 and partition 2 to be in sync.

CONSUMERS ARE (ALMOST ALWAYS) PART OF CONSUMER GROUPS

Consumers are gathered in consumer groups.

Consumer groups will guarantee that no more than one consumer of that group will receive a message for a certain topic.

That is very useful for a microservice architecture, as we can put all the replicas of a certain microservice in a consumer group. That way we know that a message for a certain topic won’t be consumed by more than one microservice.

TOPICS AND ITS PARTITIONS

EXPLAINING ITS PARTS A LITTLER FURTHER

Messages

Messages are written in random partitions (round robin by default) but we can choose, if we want, which partition to use for a message.

Messages are not written in Kafka 1 by 1. They are written in batches, which is a bunch of messages for the same topic and partition. Can be further controlled with some Kafka setting. Batches can be compressed for efficient data transfer.

Partitions

A partition act as a commit log. Messages only guarantees order within a partition.

When a message is written in Kafka, it’s have an “offset”, depending of the partition it’s in (different partitions won’t necessary have the same

offset).

If message A was written before message B by the same producer in the same partition, Kafka guarantees that offset of B will be bigger that offset of A, so consumers will read A before B.

It’s up to the consumer to remember which offset each partition is at. That way, even if the service goes down, the consumer can keep its position in the queue.

Consumers

Consumers normally belong to a consumer group. It’s common that a consumer group consumes a single topic.

The group makes sure that each partition is only consumed by one member of the consumer group. This way the number of consumers

can scale horizontally (increase in number).

If a consumer fails the rest of the consumers will rebalance the partitions.

Consumers pull messages from topics, the reason Kafka adopted this approach is extensibly explained here: tps://kafka.apache.org/documentation.html#design_pull) in the docs.

The number of consumers for a topic should never be bigger than the number of partitions for that topic. If that was the case, there will be consumers that will remain idle.

In the other hand, having more partitions than consumers shouldn’t be a problem, as Kafka will assign 2 partitions (or more if necessary) to 1 consumer.

Brokers

A partition will “belong” to a broker, but can be replicated in others (and it should), providing redundancy. A common number of replications is 3.

When a partition is replicated in several brokers, there is always one that acts as the leader for that partition.

When the broker leader for a partition fails, some other broker (which has that partition replicated) will become the leader for that partition.

FAQ

• When is a message considered commited?

Messages are considered “committed” when they are written to the broker leader and all the in-sync replicas. Consumers can’t read messages until those are committed. Committed messages won’t be lost as long as there is one replica remains alive.

• Are messages still available once they have been commited?

Messages are not necessarily deleted after they are read. How long they will remain is indicated by the settings (default is 7 days). Kafka's performance is effectively constant with respect to data size, so storing data for a long time is perfectly fine. That is why Kafka might be considered, in a way, a database as well.

• Can we manage authentication and authorization?

Kafka provides optional encryption of messages.

Authentication: So only the clients that are authenticated can connect to Kafka.

Authorization: To be sure that only certain clients can see certain topics.

• Does Kafka guarantees that a message will be delivered effectively once?

No, Kafka does not guarantees this out of the box. There is several approaches that can be taken to achieve the best result. Kafka is highly configurable in this matter. Extensively explained in this article: https://medium.com/@andy.bryant/processing-guarantees-in-kafka-12dd2e30be0e