Kubernetes may not be the first choice for running Apache Kafka clusters due to Kafka’s strong dependency on storage, which can be a challenge when it comes to persistent storage in Kubernetes. Kafka brokers are unique and stateful, so implementing them in Kubernetes requires a special approach. This is where Strimzi, a Kafka operator for Kubernetes curated by Red Hat, comes in. Strimzi helps solve the problems of running Kafka in Kubernetes and allows for the integration of additional Kafka tools. In this article, we will explore the basics of Strimzi, compare it with other Kafka operators, and discuss its key features.
Strimzi is a Kubernetes Operator designed to simplify the deployment of Apache Kafka clusters on cloud native infrastructures. It extends the Kubernetes API by providing resources to manage Kafka clusters, topics, users, MirrorMaker2 instances, and Connect instances. Currently, Strimzi is in the “Sandbox” stage at the Cloud Native Computing Foundation, which means it is an experimental project that has not yet been widely tested in production.
To deploy a 3-broker TLS-encrypted Kafka cluster with Strimzi, you can simply apply a YAML file like the following:
“`html
apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
name: my-cluster
spec:
kafka:
version: 3.2.3
replicas: 3
listeners:
– name: plain
port: 9092
type: internal
tls: false
– name: tls
port: 9093
type: internal
tls: true
config:
offsets.topic.replication.factor: 3
transaction.state.log.replication.factor: 3
transaction.state.log.min.isr: 2
default.replication.factor: 3
min.insync.replicas: 2
inter.broker.protocol.version: “3.2”
storage:
type: jbod
volumes:
– id: 0
type: persistent-claim
size: 100Gi
deleteClaim: false
– id: 1
type: persistent-claim
size: 100Gi
deleteClaim: false
zookeeper:
replicas: 3
storage:
type: persistent-claim
size: 100Gi
deleteClaim: false
entityOperator:
topicOperator: {}
userOperator: {}
“`
An example of a Kafka topic YAML file would be:
“`html
apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaTopic
metadata:
name: my-topic
labels:
strimzi.io/cluster: my-cluster
spec:
partitions: 1
replicas: 1
config:
retention.ms: 7200000
segment.bytes: 1073741824
“`
Strimzi also provides out-of-the-box security features. By default, intra-broker communication is encrypted with TLS, and communication with ZooKeeper is both authenticated and encrypted with mTLS. The ZooKeeper clusters used by the Kafka instances are not exposed outside of the Kubernetes cluster, adding an extra layer of security. While these configurations cannot be overridden, there is a project by scholzj that allows access to ZooKeeper.
Kubernetes provides its own solution for managing distributed stateful applications called StatefulSets. However, StatefulSets have some limitations when it comes to scaling and managing Kafka brokers. To address these limitations, Strimzi introduced its own resource called StrimziPodSets in version 0.29.0. StrimziPodSets provide more flexibility in scaling, per-broker configuration, and pave the way for ZooKeeper-less Kafka in the future.
Deploying Strimzi is made easy with the Quickstart documentation provided by the project. However, there are some additional integrations that are not covered by Strimzi’s documentation. For example, deploying a Kafka UI on top of a Strimzi cluster as a native Kubernetes resource can be useful. There are multiple open source Kafka UI projects available, and the article provides an example of deploying Kafka UI using the Kafka UI project from GitHub.
Another integration that is not currently supported by Strimzi is the deployment of a Schema Registry instance for Kafka clusters running in Kubernetes. The article mentions how the Rubin Observatory Science Quality and Reliability Engineering team worked on the strimzi-registry-operator to address this limitation.
The article also mentions the progress of running Kafka in KRaft mode (ZooKeeper-less). The Apache Kafka team announced that KRaft mode became production ready with the Kafka 3.3 release. Strimzi has also been working on supporting KRaft mode since version 0.29.0, although it is still considered experimental for both Kafka and Strimzi.
Finally, the article briefly touches on the topic of storage in Kubernetes. While Ceph with Rook is a popular choice for provisioning storage on Kubernetes, other solutions such as Longhorn, OpenEBS, Portworx, and Linstor also exist. The article suggests referring to a previous article on Ceph object storage with Rook for more information on storage options.
Overall, the article provides an overview of Strimzi, its features, integrations, and its current stage of development.