I have installed Zookeeper and Kakfa separately. Have started Zookeeper successfully. When I try to start Kafka on windows using the command,
C:\kafka_2.12-2.3.0\bin\windows>kafka-server-start.bat ../../config/server.properties
I keep getting,
\Novosoft\C2J\Bin\c2jruntime.zip was unexpected at this time.
Not sure what's causing this.
My environment variables had the CLASSPATH variable set to C:\Program Files (x86)\Novosoft\C2J\Bin\c2jruntime.zip.
Maybe Kafka was not liking it. Removed it and worked.
Kafka server started now
Related
i want to start my very first kafka, but when i tried to run this on my kafka_2.13-2.8.0 directory bin\windows\zookeeper-server-start.bat .. \ .. \config\zookeeper.properties
why it returns \Kafka\kafka_2.13-2.8.0\bin\windows\../ ../config/log4j.properties was unexpected at this time
idk i already followed this tip to install kafka https://www.youtube.com/watch?v=bYVyRh4C94E&t=303s
It's a known error in the Kafka log4j settings, especially if the install path contains spaces or non alphanumeric characters
If you really want to run Kafka on Windows, you should use WSL2 anyway, or Docker. Otherwise, assuming you did get the bat file working, you'd eventually run into other errors that crash the broker
To date I have either used an existing professional installation for Hadoop with components running, or, installed Kafka and used the also-supplied Zookeeper in a native VM.
I am trying to get the mapR Community Edition Sandbox to run now.
There is a KAFKA library on mapR, but here is no kafka shown when using jps. Seems odd? I managed to get KAFKA to start once.
There is a Zookeeper service on mapR but it uses port 5181, not 2181.
Kafka uses port 9092.
The log.dirs for kafka was set to /tmp/kafka-logs, I changed that to /opt/kafka-logs
The dataDir was also set to /tmp/zookeeper, I changed that to /opt/zookeeper
I also changed the Zookeeper port to 5181 as that is what mapR uses.
It ran once, and then I re-started and I still get this type of error:
java.io.FileNotFoundException: /tmp/kafka-logs/.lock (Permission denied)
I have done chmod 777 where required I think, but I changed the paths to /opt/... from /tmp. So why is it picking /tmp up again?
I have the impression that it keeps on point to /tmp regardless of the updates to the configurations.
I also see a warning - although I do not think this is an issue:
[2019-01-14 13:26:46,355] WARN No meta.properties file under dir /tmp/kafka-logs/meta.properties (kafka.server.BrokerMetadataCheckpoint)
May be because of the mapR Streams I cannot influence it so as to run natively?
OK, I could delete the question as I solved it, but for those on mapR I deduced:
You need to update the port 2181 to 5181 on server.properties immediately. In this case we integrate with an existing zookeeper instance.
Likewise, update the log.dirs for Kafka from /tmp/kafka-logs asap to /opt/kafka-logs.
Likewise, update the dataDir from /tmp/zookeeper asap to /opt/zookeeper.
Trying to fix latterly otherwise leads to all sorts of issues. I ended up just re-installing and doing it right from scratch.
mapR has a faster version called mapR Streams which implements Kafka. I was not wanting to use that for what I was wanting to do, but mapR Sandbox has a lot of up-to-date items straight out of the box -certainly compared to Cloudera.
In Lagom Dev Enviornment, after starting Kafka using lagomKafkaStart
sometimes it shows KafkaServer Closed Unexpectedly, after that i need to run clean command to again get it running.
Please suggest is this the expected behaviour.
This can happen if you forcibly shut down sbt and the ZooKeeper data becomes corrupted.
Other than running the clean command, you can manually delete the target/lagom-dynamic-projects/lagom-internal-meta-project-kafka/ directory.
This will clear your local data from Kafka, but not from any other database (Cassandra or RDBMS). If you are using Lagom's message broker API, it will automatically repopulate the Kafka topic from the source database when you restart your service.
I want to install kafka in my centos 6.5 machine. In kafka installation tutorial, I came to know that it needs zookeeper to run. I have already install hbase which also uses zookeeper service internally and zookeeper service only starts when I start hbase service.
So in order to install kafka, do I need install zookeeper separately? Please suggest.
Kafka is designed to use zookeeper by default. If you have already installed zookeeper in your system, you can create a bash script to start the zookeeper whenever you start the kafka. On your zookeeper installation directory there should be zkServer.sh start (to start zookeeper) and in kafka installation directory kafka-server-start.sh (to start the kafka).
Kafka architecture works best with distributed platform, if you are experimenting with sudo cluster, you can look for alternative message brokers like HiveMQ or RabbitMQ.
You can look further discussions at: Kafka: Is Zookeeper a must?
Installing the zookeeper cluster is the best practice. You can use it for hbase and kafka.(just define the different root dir in zk)
I am new to kafka and am still learning the basics of the same. I want to create a durable topic which is preserved even after the zoopkeeper and/or kafka server shutdown.
What I notice it this - I have a zookeeper and kafka server running on my local macbook. When I shutdown the zookeeper server and again bring it up quickly I can see the previously created topics. But If I restart the system and then restart the zookeeper server - I dont see the topic that I had created earlier.
I am running kafka_2.9.2-0.8.1.1 on my local system.
It happens because /tmp is getting cleaned after reboot resulting in loss of your data.
To fix this modify your Zookeeper dataDir property (in config/zookeeper.properties) and Kafka log.dirs (in config/server.properties) to be somewhere NOT in /tmp.