I am following a quick start guide for getting kafka up and running.
I have zookeeper running but when I try to start kafka server with the following command:
.\bin\windows\kafka-server-start.bat .\config\server.properties
I get the following error
\Teradata\Client\16.20\bin\ was unexpected at this time.
I cant even begin to understand how the two might be related.
Any tip is appreciated. I am very much stuck....
In my case I needed to change the CLASSPATH environment variable, changing from Program Files to Progra~1 and Program Files (x86) to Progra~2
This solved the problem.
This isn't specific to Kafka or Zookeeper, but how they load the Java Home and classpath.
You need to fix your PATH variable in Windows settings to not contain any spaces
looks like the kafka process needs all environment variables in the CLASSPATH to be present. I was getting the same error for Meld which I had deleted but CLASSPATH still have a reference to that.
Removing it from the CLASSPATH worked.
If you are using the Apache Kafka package structure as it is, then fix the path.it will be resolved automatically.
downloaded apache Kafka folder structure kafka_2.11-0.9.0.0 ->bin->windows-> All *.bat files.
kafka_2.11-0.9.0.0->config-> all *.properties files
so to start zookeeper go to the kafka_2.11-0.9.0.0 ->bin->windows->in window address bar type cmd and enter after that enter:
zookeeper-server-start.bat ..\..\config/zookeeper.properties
o/p: zookeeper will start working
To start broker again go to the same folder
kafka_2.11-0.9.0.0 ->bin->windows->in window address bar type cmd and enter after that enter:
kafka-server-start.bat ..\..\config/server.properties
o/p: broker will start working.
Note: you can update your own configuration using this properties file.like change in port etc..
I can't even begin to understand how the two might be related.<br/>
For more information, you can check the blog: Role of Apache ZooKeeper in Kafka — Monitoring & Configuration
Related
I'm trying to use confluent CLI for the first time. I've successfully installed it and I can run the confluent command. But when I try to start confluent with the command confluent local services start I have this error:
Obviously there is a bad configuration issue with the path to the zookeeper.properties but I can't find how this path is built. It seems that it takes my CONFLUENT_HOME variable and add "etc/kafka/zookeeper.properties" to it but I don't understand why.
Can someone explain what I'm doing wrong?
I wanted to install the Apache Kafka but stuck in installing the Zookeeper
I extracted all the file and created the environment variables as well and now it stopped working after
"Zookeeper audit is disabled."
Now after following through these links,
https://www.programmersought.com/article/22066571206/
https://zookeeper.apache.org/doc/r3.6.2/zookeeperAuditLogs.html
I updated both the files (skServer.cmd and conf/zoo.cfg). Adder a line "-Dzookeeper.audit.enable=true" in skServer.cmd file and "audit.enable=true" in conf/zoo.cfg.
Now the output has changed to "Zookeeper audit is Enabled." but still it doesn't do anything and stops there like before.
Output after running the zkServer command with files edited
Even tried changing one file at a time but the same output and still stops working.
Can anyone help me understand the problem and provide a solution as well?
Thank you so much.
zookeeper-server-start or zkServer.cmd do not return, by design. It's tailing the logs and waiting for a client connection
You must start a second terminal to run the Kafka broker, as mentioned in the Kafka documentation
Kill all java services.
In Linux run killall -9 java
Re re-run the zookeeper and Kafka server.
It works
I built a zookeeper cluster and it runs very well. But I found that the log directory I set in the zoo.cfg seems not working. Below is my config about log directory and snapshots directory.
dataDir=/var/lib/zookeeper
dataLogDir=/var/lib/zookeeper/logs
However, file zookeeper.out is generated in /var/lib/zookeeper rather than the subsidiary log folder /var/lib/zookeeper/logs.
I restarted zookeeper on every server many times, but made no sense.
This happens because zookeeper.out is related to other type of log (application log) instead of the one specified by dataLogDir which relates to transaction log.
dataLogDir
This option will direct the machine to write the transaction log to
the dataLogDir rather than the dataDir. This allows a dedicated log
device to be used, and helps avoid competition between logging and
snaphots.
By checking zkServer.sh you'll see that zookeeper.out is related to _ZOO_DAEMON_OUT which depends on ZOO_LOG_DIR which is set by default by zkEnv.sh. Depending on your environment and zookeeper (ZK) version the zookeeper.out file might land in different places (according to this answer even in the working directory from which ZK is started).
For application logging you'll better configure the log4j.properties file; that's because ZK uses log4j.
Are there any existing tools that help to read the Zookeeper transaction log? By default, it is in binary format and I would like to read it in human readable form.
I don't know if u have solved this question.
Answer:
cd the zookeeper dir.
If u want to read snapshots, use:
java -cp zookeeper-3.4.6.jar:lib/log4j-1.2.16.jar:lib/slf4j-log4j12-1.6.1.jar:lib/slf4j-api-1.6.1.jar org.apache.zookeeper.server.SnapshotFormatter version-2/snapshot.xxx
If u want to read logs, use:
java -cp zookeeper-3.4.6.jar:lib/log4j-1.2.16.jar:lib/slf4j-log4j12-1.6.1.jar:lib/slf4j-api-1.6.1.jar org.apache.zookeeper.server.LogFormatter version-2/log.xxx
You can use something like this
java -cp $ZOOKEEPER_CLASSPATH org.apache.zookeeper.server.LogFormatter [zookeeper log file path]
Building on the previous two answers, using Zookeeper 3.5.6: from the /path/to/zookeeper/lib dir which contains all the ZK and supporting jars run:
java -cp * org.apache.zookeeper.server.LogFormatter /path/to/zookeeper/transaction/logs/version-2/log.xxx
Since zookeeper version 3.6 there are tools to read transaction log and snapshots in zookeeper distribution:
For transaction log:
bin/zkTxnLogToolkit.sh --dump /datalog/version-2/log.f3aa6
For snapshots:
./zkSnapShotToolkit.sh -d /data/zkdata/version-2/snapshot.fa01000186d
See the details in official docs
You can enable ZooKeeper audit logs for ZooKeeper 3.6.o and upwards. To enable audit logs configure audit.enable=true in conf/zoo.cfg.
One thing to keep in mind is that logs from different servers of the same ensemble should be aggregated because each one of them includes operations that where executed only form clients connected to this particular server.
Full information here: https://zookeeper.apache.org/doc/r3.6.1/zookeeperAuditLogs.html
I'm newbie to Zookeeper.Trying to setup clustering server for zookeeper exhibitor to modify the data. I have tried the server setup with 3 nodes, but data modification not reflected on all the zookeeper.
I refereed the following url & also setup the server in the same way. But no use, some thing i'm missing in that config to run it correctly.
http://zookeeper.apache.org/doc/trunk/zookeeperAdmin.html#sc_zkMulitServerSetup
Exhibitor startup command is:
java -jar exhibitor-war-1.0-jar-with-dependencies.jar -c file --nodemodification true --port 9090
Farther I need to add any other config with this to get my data modification reflect on all the zookeeper.
Advance thanks for ur kind time!
I have similar scenario, and I'm on Windows. So I get some problems due to the fact that Exhibitor is Unix-oriented - it tries to restart zkServer.sh (instead of zkServer.bat). So I:
1. have manually started ZK ensemble (all instances get data modifications from each other).
2. set up Exhibitor above every ZK instance - with single network config file.
Hope it helps. If not, give more details.