I have a setup for Apache Solr 7.4.0 for which I currently use Zookeeper 3.4.13. From what I have been experiencing and what I have read, an upgrade may be in order.
I had previously installed Zookeeper through the instructions I had gleaned on Stack Overflow.
Could anyone tell me how this may be done, preferably in a step by step fashion?
I can stop Zookeeper and all Solr servers to accomplish this.
Related
we are using Apache kafka 2.0 in our production environment and now we are planning to upgrade the kafka version to 2.6 from 2.0
we are running in three broker based cluster setup
i am having the below questions.
1)is it possible to upgrade the kafka from one version to higher version?
2)while upgrading is there any data loss happen?
3)is it possible to perform while the cluster is running?
4)How to rollback to the down version if something wrong happened?
can you share your valuable thoughts for this question..
it would be helpful to setup..
Yes, upgrades are possible - http://kafka.apache.org/26/documentation.html#upgrade
Data that's already written to the topics shouldn't get lost if you follow the guide. Active clients might experience network exceptions, retries, and potential dropped packets while individual brokers are restarting.
A rolling upgrade is possible to prevent downtime
Depending on the exact version, rollbacks are not possible due to internal log format changes (as indicated in the documentation)
I was wondering if someone has used zookeeper/kafka embeeded within Tomcat. I know that Kafka requires Zookeeper, but does it mean that I have to run Kafka and Zookeeper as separate instances? So far I cannot see any use cases where everything has been bolt in. Could anyone advise?
My question is more around the concept of using zookeeper and kafka as a jar within the same tomcat web application.
Both Kafka and Zookeeper are meant to be used in a stand-alone fashion, run as separate processes.
They should even be on different machines/vms/containers than the tomcat web application.
You also probably want a Zookeeper cluster of 3-5 machines, rather than a single one, at least for production.
Both of them have Java clients though, for you to interact from the web application with them, and those are OK to include.
I'm new in Confluent world, and I know how to start kafka, zookeepers from confluent, but it's not that what I need.
I have already 3 kafka nodes and 2 zookeepers installed by Ambari. Afterwards I downloaded 3.0.0 version of Confluent and now I want to connect Confluent with already running Kafka and zookeeper. I don't want to instance new kafka server or zookeeper server which confluent is giving.
Does anyone has an idea how to accomplish that, what to actually run from Confluent and what to change.
By now I was only chaning files in ./etc/kafka or ./etc/zookeeper which are in Confluent dir. Thank you!
clarify some basics about Confluent and how manage communication between Confluent and Kafka
First things first, there is no single application called "Confluent" that can be started all on its own.
There's is nothing to configure for Kafka or Zookeeper. The Confluent Platform doesn't add anything on top of the existing Apache Kafka you have (presumably, via Hortonworks or Cloudera).
In fact, those companies do add patches to Kafka that would be slightly different than the base Apache versions you would get from Confluent.
That being said, if you read through each of extra services that Confluent provides, you'll notice either a Zookeeper or a Bootstrap server configuration option. Fill out those fields, start the respective services, and you're good to go.
what to actually run from Confluent
Look in the bin directory, you can find all the start scripts. From the comments, looks like you're trying to use Connect Distributed (which should already be installed by any recent Kafka installation, it's not Confluent specific), and Schema Registry. You'll have to be more specific about the errors that you get, but the config files are all in the etc path.
Unless you're using KSQL, REST Proxy or Control Center, there's not much to run because, as mentioned, Kafka Connect is included with the base Apache Kafka project and Hortonworks is maintaining their own Schema Registry project
2 zookeepers installed by Ambari
This is a highly non-recommended setup. Please install an odd number of Zookeepers. 3 or 5, preferably
I have kafka conenct running in Marathon container. If I want to update the connector plugin (jar) I have to upload the new one and then restart the Connect task.
Is it possible to do that without restarting/downtime?
The updated jar for the connector plugin needs to be added to the classpath and then the classloader for the worker needs to pick it up. The best way to do this currently is to take an outage as described here.
Depending on your connector, you might be able to do rolling upgrades, but the generic answer is that if you need to upgrade the connector plugin, you currently have to take an outage.
I'd like to use zookeeper in one of my applications for distributed configuration management. The application is currently running in distributed environment and having to restart nodes for configuration files changes is a headache.
However, we want the zookeeper process to be started from within the application. The point is to reduced startup dependency and reduce operational cost. We've already have startup/shutdown scripts for the application and we need to reduce impact for operations team.
Has any one done something similar? Is this setup recommended or there are better solutions? Any tip or feedback is appreciated.
I have a blog post that describes how to embed Zookeeper in an application. The Zookeeper developers don't recommend it, though, and I would tend to agree now, though I had the same rationale for embedding it that you do - to reduce the number of moving parts.
You want to keep your ZK cluster stable but you will need to restart your app to do code updates, etc, impacting the ZK cluster stability.
Ultimately you will end up using your ZK cluster for multiple apps and those extra moving parts will be amortized over a number of projects.