I'm newbie to Zookeeper.Trying to setup clustering server for zookeeper exhibitor to modify the data. I have tried the server setup with 3 nodes, but data modification not reflected on all the zookeeper.
I refereed the following url & also setup the server in the same way. But no use, some thing i'm missing in that config to run it correctly.
http://zookeeper.apache.org/doc/trunk/zookeeperAdmin.html#sc_zkMulitServerSetup
Exhibitor startup command is:
java -jar exhibitor-war-1.0-jar-with-dependencies.jar -c file --nodemodification true --port 9090
Farther I need to add any other config with this to get my data modification reflect on all the zookeeper.
Advance thanks for ur kind time!
I have similar scenario, and I'm on Windows. So I get some problems due to the fact that Exhibitor is Unix-oriented - it tries to restart zkServer.sh (instead of zkServer.bat). So I:
1. have manually started ZK ensemble (all instances get data modifications from each other).
2. set up Exhibitor above every ZK instance - with single network config file.
Hope it helps. If not, give more details.
Related
I am seeing various different issues while trying to start Kafka services on wsl2. Details/symptoms below:
Confluent Kafka (7.0.0) platform
wsl2 - ubuntu 20.04LTS
When I use the command:
confluent local services start
Typically the system will take a long time and then exit with service failed (e.g. zookeeper, as that is the first service to start).
If I check the logs, it is actually started. So I again type the command and sure enough it immediately says zookeeper up, then proceed to try start kafka, which again after a min will say failed to start (but it really has started).
I suspect after starting the service (which is quite fast), system is not able to communicate back/exit and thus times out, I am not sure where the logs related to this are.
Can see this in the screenshot below
This means to start the whole stack (zookeeper/kafka/schema-registry/kafka-rest/kafka-connect/etc), takes forever, and in between I start getting other errors (sometimes, schema-registry is not able to find the cluster id, sometimes its a log file related error), which means I need to destroy and start again.
I have tried this over a couple of days and cant get this to work. Is confluent kafka that unstable on windows or I am missing some config change.
In terms of setup, I have not done any change in the config and am using the default config/ports.
I wanted to install the Apache Kafka but stuck in installing the Zookeeper
I extracted all the file and created the environment variables as well and now it stopped working after
"Zookeeper audit is disabled."
Now after following through these links,
https://www.programmersought.com/article/22066571206/
https://zookeeper.apache.org/doc/r3.6.2/zookeeperAuditLogs.html
I updated both the files (skServer.cmd and conf/zoo.cfg). Adder a line "-Dzookeeper.audit.enable=true" in skServer.cmd file and "audit.enable=true" in conf/zoo.cfg.
Now the output has changed to "Zookeeper audit is Enabled." but still it doesn't do anything and stops there like before.
Output after running the zkServer command with files edited
Even tried changing one file at a time but the same output and still stops working.
Can anyone help me understand the problem and provide a solution as well?
Thank you so much.
zookeeper-server-start or zkServer.cmd do not return, by design. It's tailing the logs and waiting for a client connection
You must start a second terminal to run the Kafka broker, as mentioned in the Kafka documentation
Kill all java services.
In Linux run killall -9 java
Re re-run the zookeeper and Kafka server.
It works
I am following a quick start guide for getting kafka up and running.
I have zookeeper running but when I try to start kafka server with the following command:
.\bin\windows\kafka-server-start.bat .\config\server.properties
I get the following error
\Teradata\Client\16.20\bin\ was unexpected at this time.
I cant even begin to understand how the two might be related.
Any tip is appreciated. I am very much stuck....
In my case I needed to change the CLASSPATH environment variable, changing from Program Files to Progra~1 and Program Files (x86) to Progra~2
This solved the problem.
This isn't specific to Kafka or Zookeeper, but how they load the Java Home and classpath.
You need to fix your PATH variable in Windows settings to not contain any spaces
looks like the kafka process needs all environment variables in the CLASSPATH to be present. I was getting the same error for Meld which I had deleted but CLASSPATH still have a reference to that.
Removing it from the CLASSPATH worked.
If you are using the Apache Kafka package structure as it is, then fix the path.it will be resolved automatically.
downloaded apache Kafka folder structure kafka_2.11-0.9.0.0 ->bin->windows-> All *.bat files.
kafka_2.11-0.9.0.0->config-> all *.properties files
so to start zookeeper go to the kafka_2.11-0.9.0.0 ->bin->windows->in window address bar type cmd and enter after that enter:
zookeeper-server-start.bat ..\..\config/zookeeper.properties
o/p: zookeeper will start working
To start broker again go to the same folder
kafka_2.11-0.9.0.0 ->bin->windows->in window address bar type cmd and enter after that enter:
kafka-server-start.bat ..\..\config/server.properties
o/p: broker will start working.
Note: you can update your own configuration using this properties file.like change in port etc..
I can't even begin to understand how the two might be related.<br/>
For more information, you can check the blog: Role of Apache ZooKeeper in Kafka — Monitoring & Configuration
The case is: to separate client and broker replication communication + introduce security.
Question is: is it possible to separate the communication with some procedure like rolling restart? Without need to have downtime on the whole cluster.
Configuration as is (simple with one port for everything wihout security):
listeners=PLAINTEXT://server1:9092
Wanted configuration (different ports and some with security, replication on 9094 port):
listener.security.protocol.map=PLAINTEXT:PLAINTEXT,SASLPLAIN:SASL_PLAINTEXT,REPLICATION:SASL_PLAINTEXT
listeners=PLAINTEXT://server1:9092,SASLPLAIN://server1,REPLICATION://server1:9094
inter.broker.listener.name=REPLICATION
sasl.mechanism.inter.broker.protocol=PLAIN
sasl.enabled.mechanisms=PLAIN
authorizer.class.name=kafka.security.auth.SimpleAclAuthorizer
Progress:
Configuration below is well working. But only way, without putting cluster into inconsistent state i know now, is to stop the cluster, introduce new configuration as shown above, and start cluster again. That´s obviously not wanted by the customer.
Gratefull for any thoughts how to proceed without need to stop/start whole cluster.
I managed to proceed from original, one listener configuration, to desired by below steps.
If someone has any idea to ease up the process, please add.
Original config:
listeners=PLAINTEXT://server1:9092
1.Change server.properties and do rolling restart
listener.security.protocol.map=PLAINTEXT:PLAINTEXT,SASLPLAIN:SASL_PLAINTEXT,REPLICATION:SASL_PLAINTEXT
listeners=PLAINTEXT://SERVER1:9092,SASL_PLAINTEXT://SERVER1:9093,REPLICATION://SERVER1:9094
sasl.enabled.mechanisms=PLAIN
Also include jaas config as jvm parameter.
-Djava.security.auth.login.config=/path/to/kafka_server_jaas.conf
2.Modify the server.properties and do rolling restart
listener.security.protocol.map=PLAINTEXT:PLAINTEXT,SASLPLAIN:SASL_PLAINTEXT,REPLICATION:SASL_PLAINTEXT
listeners=PLAINTEXT://SERVER1:9092,SASL_PLAINTEXT://SERVER1:9093,REPLICATION://SERVER1:9094
inter.broker.listener.name=REPLICATION
sasl.mechanism.inter.broker.protocol=PLAIN
sasl.enabled.mechanisms=PLAIN
3.Modify server properties one last time and do third rolling restart
listener.security.protocol.map=PLAINTEXT:PLAINTEXT,SASLPLAIN:SASL_PLAINTEXT,REPLICATION:SASL_PLAINTEXT
listeners=PLAINTEXT://SERVER1:9092,SASL_PLAINTEXT://SERVER1:9093,REPLICATION://SERVER1:9094
inter.broker.listener.name=REPLICATION
sasl.mechanism.inter.broker.protocol=PLAIN
sasl.enabled.mechanisms=PLAIN
authorizer.class.name=kafka.security.auth.SimpleAclAuthorizer
allow.everyone.if.no.acl.found=true
I am a noob in Solr and zookeeper and trying to learn by myself. I understood that zookeeper is a file structure that manages solr cluster and prevents race condition using locks. I didn’t understand what is upconfig and downconfig and when we do that. It would be of great help if someone can give me a clear picture on it. Thanks in advance!
A better and more general description of Zookeeper is an application that provides centralised configuration for distributed systems. So in Solr Cloud, you can have multiple Solr instances across multiple servers acting together as a single cloud. However, if you want to update a collection's configuration, you don't want to have to go to each server and update them all individually. You want only one version of the config which is then used by any collection that needs it. Hence the conf commands.
upconfig uploads a configuration to ZooKeeper, which then ensures that all collections using that configuration (throughout the Cloud, on all the servers) have that specific config. So you only need to upload it once, on one server.
downconfig lets you fetch a configuration from Zookeeper.