I am working with Apache Kafka and looking to start using Strimzi Kafa. But I am having trouble finding if there is a way to make sure replicas are in a separate data centers. I know kafka has strech clusters which has a single cluster that can be across multiple data centers, but Strimzi doesn't support that from what I can tell.
Is there any way to do this with Strimzi Kafka?
It always depends what you mean with different DC. It could mean different Availability Zones in AWS which will be close to each other and have a good latency. Or some DCs on different continents which would be something Apache Kafka cannot handle. Or maybe something in between.
In general, Strimzi lets you deploy a Kafka cluster within a single Kubernetes cluster. But as long as the latency is good enough, it does not really care whether it is all in one DC or in multiple DCs.
Strimzi gives you too tools how to control this:
You can configure the pod scheduling to distribute your Kafka broker pods across any zones of your cluster
It lets you configure the Rack awareness which configured the broker.rack option in Apache Kafka to make sure your replicas are distributed across the racks / zones.
You can also use Cruise Control to automatically reassign any replicas which would not be distributed.
Related
I'm new to kafka.
Kafka is supposed to be used as a distributed service. But the tutorials and blog posts i found online never mention if there is one or several zookeeper nodes.
The tutorials just pop one zookeper instance, and then multiple kafka brokers.
Is it how it is supposed to be done?
Zookeeper is a co-ordination service (in a centralized manner) for distributed systems that is used by clusters for maintenance of distributed system . The distributed synchronization achieved by it via metadata such as configuration information, naming, etc.
In general architectures, Kafka cluster shall be served by 3 ZooKeeper nodes, but if the size of deployment is huge, then it can be ramped up to 5 ZooKeeper nodes but that in turn will add load on the nodes as all nodes try to be in sync as all metadata related activities are handled by ZooKeeper.
Also, it should be noted that as an improvement, the new release of Kafka reduces dependency on ZooKeeper in order to enhance scalability of metadata across, to reduce the complexity in maintaining the meta data with external components and to enhance the recovery from unexpected shutdowns. With new approach, the controller failover is almost instantaneous. This is achieved by Kafka Raft Metadata mode termed as 'KRaft' that will run Kafka without ZooKeeper by merging all the responsibilities handled by ZooKeeper inside a service in the Kafka Cluster itself and operates on event based mechanism that is used in the KRaft protocol.
Tutorials generally keep things nice and simple, so one ZooKeeper (often one Kafka broker too). Useful for getting started; useless for any kind of resilience :)
In practice, you are going to need three ZooKeeper nodes minimum.
If it helps, here is an enterprise reference architecture whitepaper for the deployment of Apache Kafka
Disclaimer: I work for Confluent, who publish the above whitepaper.
I am trying to figure out an appropriate production deployment strategy for an Apache Kafka cluster with High Availability.
I was unable to find a specific documentation which describes such a strategy. So based on the articles I found, I have come up with the following strategy.
3 zookeeper nodes
3 kafka brokers (each having a replica of all the topic partitions that I'm planning to use)
Replication factor of 3 for each Topic
on 3 physical machines (each having a zookeeper node and a broker node)
The reason why I have decided to have a zookeeper node and a broker node on each machine is to avoid a 'brain split' in an event of a network partitioning as described in this question and the accepted answer
I want to know,
Whether there is a adverse performance impact in having both a zookeeper node and a broker node on a single machine? (and whether it would make more sense to go ahead with 6 physical machines by deploying such that each machine would either have a kafka broker or a zookeeper node?)
Whether the deployment strategy I have come with is suitable for a production deployment? (Or how it can be improved?)
Also, if you have come across a guide which recommends a suitable deployment configuration, kindly include its link.
Appreciate any help on this matter. Thanks in advance.
Three is the minimum number of brokers you'll need, but you might want more for additional redundancy and/or capacity
Usually, people deploy their Kafka brokers and Zookeeper nodes on separate hardware.
This Reference Architecture should help you further.
I've been looking for specific information about high availability deployments of Kafka Connect connectors but found nothing.
In my case I have a Mongodb source connector deployed using the Confluent Helm chart. This chart supports setting the number of replicas.
Is setting replicaCount to a value >1 enough or there are other factors to consider (tasks.max, ...)?
If you want highly available workers, then it's pod replicas, yes.
If you want distributed tasks across workers, that's tasks.max; if one worker dies, then tasks get rebalanced
I'm planning to build a Kafka Cluster using two servers, and host Zookeeper on these two servers as well.
The Question is, since Kafka requires Zookeeper to run, what is the best cluster build for zookeeper to implement Kafka Cluster on two servers?
for eg. I'm currently running two zookeepers on both servers and one Kafka on each server, and in the Kafka configuration they point to all Zookeepers.
Is there a better way to do this?
First of all, you don't have to setup Zookeper and Kafka in the same server. One of the roles of Zookeeper is electing controller. (one of the brokers which is responsible for maintaining the leader/follower relationship for all the partitions) For election; majority of Zookeper nodes must be alive. In your case even one Zookeeper instance is down, you cannot select controller. So there is no difference between having one Zookeper or two. That's why it is recommended to have at least 3 nodes in Zookeeper cluster. By this way you can handle failure of one Zookeeper node.
An addition to this, it is highly recommended to have at least three brokers in your Kafka cluster to maintain both consistency and high availability. (link1, link2)
UPDATE:
As long as you are limited to only two servers, then you can consider sacrificing from high availability by set up your broker by setting min.insync.replicas=2 and having topics with replication.factor=2. If HA is more important than data loss, then you can use min.insync.replicas=1 (default) broker config with again topic replication.factor=2. In this circumstance, your options are these IMHO. (Having one or two Zookeepers is not important as I mentioned above)
I am often faced with the same problem as you do #frisky5 where i would like to achieve a "suboptimal" HA system using only 2 nodes, and thus workarounds are always needed with cloud-native frameworks that rely on the assumption that clusters will have lot of nodes available.
That ain't always the case in real life, is it ;) ?
That being said, i see you essentially having 2 options:
Externalize zookeeper configuration on a replicated storage system using 2 nodes (e.g. DRBD)
Replicate Kafka data volumes entirely on the second nodes and use 2 one-node Kafka clusters that you switch on and off depending on who is the current master node.
I would go for the first option. In that case you would have 2 Kafka servers and one zookeeper server whose ip needs to be static (virtual ip). When the zookeeper node goes down, it is restarted one the second node with same VIP, but it needs to access the synchronized data folder.
I am not too familiar with zookeepers internals and i can't tell you whether it will go in conflict when starting up on a data store who "wasn't its own" but i would guess it makes sense for you to test it using a simple rsync setup.
Another way to achieve consensus if you are using a k3s based kubernetes cluster would be to rely on internal k8s distributed consensus mechanics to "tell Kafka" which node is the leader. This works for the postgresoperator by chruncydata because Patroni is cool ( https://patroni.readthedocs.io/en/latest/kubernetes.html ) 😎 but i am not sure if Kafka/zookeeper are that flexible and can communicate with a rest API to set their locks ...
Once you have achieved this intermediate step, then you can use a PostgreSQL db as external source of truth for k3s and then it is as simple as syncing the postgres data folder between the machines (easily done with rsync). The beauty of this approach is that it is way more generic and could be used for other systems too.
Let me know what do you think about these two approaches and whether you manage to setup a test environment. If you do on GitHub i can help you out with implementation
I am trying to replicate data from one to another kafka cluster by using mirror maker . Suppose if master cluster is down, is it possible to automatically send the kafka messages to the second cluster ? And also is it possible to synchronise the cluster 1 with cluster 2 when the cluster 1 is up again automatically with less manual intervention?
any help is highly appreciated .
I think you meant to ask how to maintain copies between Kafka brokers, that together are considered to be a Kafka Cluster.
If that's the case, it's pretty simple, all you have to do is configure a Kafka Cluster and to create a topic with replication factor with size that is equal to the size of the nodes in the Cluster.
For example:
Let's say that we want to have 3 Brokers on our Kafka Cluster, then you'll need to prepare for each broker a different configuration file, then startup them as a cluster, and then create a topic with replication factor of 3.
Kafka will be responsible for maintaining the Fault Tolerance.
For further info on actually do the configuration, watch these videos on youtube:
https://www.youtube.com/channel/UCDLPjuuYHxPbHdN8RXxrGdw