i want to start my very first kafka, but when i tried to run this on my kafka_2.13-2.8.0 directory bin\windows\zookeeper-server-start.bat .. \ .. \config\zookeeper.properties
why it returns \Kafka\kafka_2.13-2.8.0\bin\windows\../ ../config/log4j.properties was unexpected at this time
idk i already followed this tip to install kafka https://www.youtube.com/watch?v=bYVyRh4C94E&t=303s
It's a known error in the Kafka log4j settings, especially if the install path contains spaces or non alphanumeric characters
If you really want to run Kafka on Windows, you should use WSL2 anyway, or Docker. Otherwise, assuming you did get the bat file working, you'd eventually run into other errors that crash the broker
Related
I am new to kafka and following this tutorial https://www.tutorialspoint.com/apache_kafka/apache_kafka_basic_operations.htm
I have installed and configure everything zookeeper, kafka etc and it's working fine. But when I run any command my output terminal window close within second, so I am not able to see output. Is there property to set output window time?
kafka-topics.sh --list --zookeeper localhost:2181
I used this command to list topic but listed topics window closed not given enough time me to see.
Powershell doesn't run .sh scripts. If CMD or Powershell are closing its because you're getting a fatal exception and should really be using bin\windows\kafka-topics.bat, which ran on its own with no arguments should print help text, and not close any window
Also relevant - Run Kafka on WSL2 instead of direct in windows if you want to avoid LogCleaner errors you'll eventually find with file permissions
I also suggest following the official Kafka website documentation since the Zookeeper flag is deprecated
I have installed Zookeeper and Kakfa separately. Have started Zookeeper successfully. When I try to start Kafka on windows using the command,
C:\kafka_2.12-2.3.0\bin\windows>kafka-server-start.bat ../../config/server.properties
I keep getting,
\Novosoft\C2J\Bin\c2jruntime.zip was unexpected at this time.
Not sure what's causing this.
My environment variables had the CLASSPATH variable set to C:\Program Files (x86)\Novosoft\C2J\Bin\c2jruntime.zip.
Maybe Kafka was not liking it. Removed it and worked.
Kafka server started now
To date I have either used an existing professional installation for Hadoop with components running, or, installed Kafka and used the also-supplied Zookeeper in a native VM.
I am trying to get the mapR Community Edition Sandbox to run now.
There is a KAFKA library on mapR, but here is no kafka shown when using jps. Seems odd? I managed to get KAFKA to start once.
There is a Zookeeper service on mapR but it uses port 5181, not 2181.
Kafka uses port 9092.
The log.dirs for kafka was set to /tmp/kafka-logs, I changed that to /opt/kafka-logs
The dataDir was also set to /tmp/zookeeper, I changed that to /opt/zookeeper
I also changed the Zookeeper port to 5181 as that is what mapR uses.
It ran once, and then I re-started and I still get this type of error:
java.io.FileNotFoundException: /tmp/kafka-logs/.lock (Permission denied)
I have done chmod 777 where required I think, but I changed the paths to /opt/... from /tmp. So why is it picking /tmp up again?
I have the impression that it keeps on point to /tmp regardless of the updates to the configurations.
I also see a warning - although I do not think this is an issue:
[2019-01-14 13:26:46,355] WARN No meta.properties file under dir /tmp/kafka-logs/meta.properties (kafka.server.BrokerMetadataCheckpoint)
May be because of the mapR Streams I cannot influence it so as to run natively?
OK, I could delete the question as I solved it, but for those on mapR I deduced:
You need to update the port 2181 to 5181 on server.properties immediately. In this case we integrate with an existing zookeeper instance.
Likewise, update the log.dirs for Kafka from /tmp/kafka-logs asap to /opt/kafka-logs.
Likewise, update the dataDir from /tmp/zookeeper asap to /opt/zookeeper.
Trying to fix latterly otherwise leads to all sorts of issues. I ended up just re-installing and doing it right from scratch.
mapR has a faster version called mapR Streams which implements Kafka. I was not wanting to use that for what I was wanting to do, but mapR Sandbox has a lot of up-to-date items straight out of the box -certainly compared to Cloudera.
I am currently using windows machine and able to read whole file through command prompt using Kafka producer and consumer. I need to only get the recent changes in a file and need to use it in as input for Apache flink. I tried using this link but due to kafka client jar mismatch issue, i was not able to use it.
In my current approach when i call my producer each time it loads the whole file and we need to run it every time to see the changes occurred to file. I thought of using threads and some way of comparing difference in file using java code but is there any of doing only by Kafka.
I had similar problem recently (but in Linux) and solved it following way:
tail -f somefile.log | kafka-console-producer.sh ...
In your case you can try some Windows alternatives to Linux's tail: 13 Ways to Tail a Log File on Windows & Linux
I have a very strange problem when trying to connect locally to Kafka 0.10.0.0 using Python client on CentOS.
My connection options are pretty simple and default:
kafka_consumer = kafka.KafkaConsumer(
bootstrap_servers=['localhost:9092'],
client_id="python-test-consumer"
)
When I manually set listeners option in Kafka's server.properties file like:
listeners=PLAINTEXT://localhost:9092
I get the kafka.errors.NoBrokersAvailable despite the fact that I can still easily connect to Kafka broker server with curl or other linux stuff.
No advertised.listeners or other deprecated advertised options help to solve the problem. Thus, the only state of configuration which is working is one without listeners. What is certainly unacceptable, because we need to setup local cluster somehow.
It seems that solution for this silly problem is simple and is wondering around, but we couldn't figure it ourselves.
This may sound silly, but the exact same problem happened to me because of this:
I upgraded to Kafka 0.10.0.0 via brew (Mac package manager).
Brew then suggests to run like this one-liner:
$ zookeeper-server-start /usr/local/etc/kafka/zookeeper.properties; kafka-server-start /usr/local/etc/kafka/server.properties
Instead of how I executed before:
$ zkServer start
$ kafka-server-start /usr/local/etc/kafka/server.properties
The approach suggested kept throwing those "No Brokers Available" errors in the client. Then I just split the command in two lines:
$ zookeeper-server-start /usr/local/etc/kafka/zookeeper.properties
$ kafka-server-start /usr/local/etc/kafka/server.properties
And everything works like before!
Sorry if this doesn't work for you, but I figured it was worth mentioning.