While running Zookeeper with Kafka on windows 10 I am getting below error :
kafka_2.12-2.4.1>bin\windows\zookeeper-server-start.bat config\zookeeper.properties
The input line is too long.
The syntax of the command is incorrect.
Please advice on how can this be solved.
P.S : I am using JDK - 1.8.0_181.
You need to move your Kafka distribution somewhere else, so you don't have very long path to that directory + bin\windows\...
IMHO, on Windows it's better to use Kafka in the Docker images, than try to run it natively - Windows has a lot of restrictions, compared to Linux/Unix that are primary platforms for running Kafka & other big data applications.
Related
I'm trying to run this command : confluent local services start
I don't know why each time it gives me an error before passing to the next step. So I had to run it again over and over until it passes all the steps.
what is the reason for the error and how to solve the problem?
You need to open the log files to inspect any errors that would be happening.
But, it's possible the services are having a race condition. Schema Registry requires Kafka, REST Proxy and Connect require the Schema Registry... Maybe they are not waiting for the previous components to start.
Or maybe your machine does not have enough resources to start all services. E.g. I believe at least 6GB of RAM are necessary. If you have 8GB on the machine, and Chrome and lots of other services are running, for example, then you wouldn't have 6GB readily available.
I am seeing various different issues while trying to start Kafka services on wsl2. Details/symptoms below:
Confluent Kafka (7.0.0) platform
wsl2 - ubuntu 20.04LTS
When I use the command:
confluent local services start
Typically the system will take a long time and then exit with service failed (e.g. zookeeper, as that is the first service to start).
If I check the logs, it is actually started. So I again type the command and sure enough it immediately says zookeeper up, then proceed to try start kafka, which again after a min will say failed to start (but it really has started).
I suspect after starting the service (which is quite fast), system is not able to communicate back/exit and thus times out, I am not sure where the logs related to this are.
Can see this in the screenshot below
This means to start the whole stack (zookeeper/kafka/schema-registry/kafka-rest/kafka-connect/etc), takes forever, and in between I start getting other errors (sometimes, schema-registry is not able to find the cluster id, sometimes its a log file related error), which means I need to destroy and start again.
I have tried this over a couple of days and cant get this to work. Is confluent kafka that unstable on windows or I am missing some config change.
In terms of setup, I have not done any change in the config and am using the default config/ports.
I wanted to install the Apache Kafka but stuck in installing the Zookeeper
I extracted all the file and created the environment variables as well and now it stopped working after
"Zookeeper audit is disabled."
Now after following through these links,
https://www.programmersought.com/article/22066571206/
https://zookeeper.apache.org/doc/r3.6.2/zookeeperAuditLogs.html
I updated both the files (skServer.cmd and conf/zoo.cfg). Adder a line "-Dzookeeper.audit.enable=true" in skServer.cmd file and "audit.enable=true" in conf/zoo.cfg.
Now the output has changed to "Zookeeper audit is Enabled." but still it doesn't do anything and stops there like before.
Output after running the zkServer command with files edited
Even tried changing one file at a time but the same output and still stops working.
Can anyone help me understand the problem and provide a solution as well?
Thank you so much.
zookeeper-server-start or zkServer.cmd do not return, by design. It's tailing the logs and waiting for a client connection
You must start a second terminal to run the Kafka broker, as mentioned in the Kafka documentation
Kill all java services.
In Linux run killall -9 java
Re re-run the zookeeper and Kafka server.
It works
I am following a quick start guide for getting kafka up and running.
I have zookeeper running but when I try to start kafka server with the following command:
.\bin\windows\kafka-server-start.bat .\config\server.properties
I get the following error
\Teradata\Client\16.20\bin\ was unexpected at this time.
I cant even begin to understand how the two might be related.
Any tip is appreciated. I am very much stuck....
In my case I needed to change the CLASSPATH environment variable, changing from Program Files to Progra~1 and Program Files (x86) to Progra~2
This solved the problem.
This isn't specific to Kafka or Zookeeper, but how they load the Java Home and classpath.
You need to fix your PATH variable in Windows settings to not contain any spaces
looks like the kafka process needs all environment variables in the CLASSPATH to be present. I was getting the same error for Meld which I had deleted but CLASSPATH still have a reference to that.
Removing it from the CLASSPATH worked.
If you are using the Apache Kafka package structure as it is, then fix the path.it will be resolved automatically.
downloaded apache Kafka folder structure kafka_2.11-0.9.0.0 ->bin->windows-> All *.bat files.
kafka_2.11-0.9.0.0->config-> all *.properties files
so to start zookeeper go to the kafka_2.11-0.9.0.0 ->bin->windows->in window address bar type cmd and enter after that enter:
zookeeper-server-start.bat ..\..\config/zookeeper.properties
o/p: zookeeper will start working
To start broker again go to the same folder
kafka_2.11-0.9.0.0 ->bin->windows->in window address bar type cmd and enter after that enter:
kafka-server-start.bat ..\..\config/server.properties
o/p: broker will start working.
Note: you can update your own configuration using this properties file.like change in port etc..
I can't even begin to understand how the two might be related.<br/>
For more information, you can check the blog: Role of Apache ZooKeeper in Kafka — Monitoring & Configuration
If I have one Kafka running on Windows server, and another Kafka running on Linux server, and the last one running on a Unix server, can I use those three as one cluster?
Server A - Windows
Server B - Linux
Server C - Unix
Can I use them for replication? As in --replication-factor 3?
Brokers communicate via well defined APIs that don't depend on the operating system, so in theory that should work. However, I doubt anybody has tried running such a cluster!
It's worth noting that Windows is not an officially supported platform: http://kafka.apache.org/documentation/#os so you may run into a few issues.
I'm not sure why you'd even consider doing this as this sounds like an operation nightmare! For science!