ERROR Error while creating ephemeral at /brokers/ids/0, node already exists and owner '72067757872119809' does not match current session '72067836689711106' (kafka.zk.KafkaZkClient$CheckedEphemeral) 2021-05-05 02:19:44.796 [INF] [Kafka] [2021-05-05 02:19:44,786] ERROR [KafkaServer id=0] Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)
The error is saying a broker already is running with id=0, or that Zookeeper is corrupt because a broker did not previously cleanly shut down...
In the later case, you can attempt to use zookeeper-shell to rmr /brokers/ids/0, however, this might have more unintended consequences than preforming a restart of Zookeeper as well as the brokers
The only solution which works here is restart the broker and then restart the server.
Restart the zookeeper and the broker is fixed the issue for me.
If you using the docker-compose, you can restart simply by using
docker-compose restart
Related
I am a beginner and I have to use Kafka for data transfer into/from Hadoop FS (or any other application, not just through put or copyFromLocal commands),kafka needs zookeeper as well, I enabled Zooekeeper audit logging but I still get errors.
For Kafka, when I want to start it:
JMX_PORT=8004 bin/kafka-server-start.sh config/server.properties
I get the error:
[2022-02-16 13:56:45,939] INFO shutting down (kafka.server.KafkaServer)
[2022-02-16 13:56:46,114] INFO App info kafka.server for 0 unregistered (org.apache.kafka.common.utils.AppInfoParser)
[2022-02-16 13:56:46,133] INFO shut down completed (kafka.server.KafkaServer)
[2022-02-16 13:56:46,133] ERROR Exiting Kafka. (kafka.Kafka$)
[2022-02-16 13:56:46,165] INFO shutting down (kafka.server.KafkaServer)
And when I want to start Zookeeper using the command:
bin/zookeeper-server-start.sh config/zookeeper.properties
I get the following (and it gets stuck on it):
[2022-02-16 14:03:13,954] INFO zookeeper.request_throttler.shutdownTimeout = 10000 (org.apache.zookeeper.server.RequestThrottler)
[2022-02-16 14:03:13,955] INFO PrepRequestProcessor (sid:0) started, reconfigEnabled=false (org.apache.zookeeper.server.PrepRequestProcessor)
[2022-02-16 14:03:14,136] INFO Using checkIntervalMs=60000 maxPerMinute=10000 maxNeverUsedIntervalMs=0 (org.apache.zookeeper.server.ContainerManager)
[2022-02-16 14:03:14,138] INFO ZooKeeper audit is disabled. (org.apache.zookeeper.audit.ZKAuditProvider)
Does anyone know how to work this out? I enabled audit logging but still. Same problem.
Zookeeper CLI isn't "stuck"; it's waiting for connections.
Open a new terminal and start Kafka
Alternatively, you could use Docker Compose / Kubernetes, if you think your host / local JVM is causing issues
I am currently using the confluent platform community license. I started Zookeeper, Kafka and the schema-registry - all are used in local mode. However when starting the schema-registry for the first time, 50 messages are sent and stored inside the __consumer_offset topic (__consumer_offsets-0 to __consumer_offsets-49). Those messages are stored in the kafka-logs and when I am trying to start the services again, it fails. To be more precise: Zookeeper works but Kafka fails with the error:
"ERROR Shutdown broker because all log dirs have failed".
As suggested in some other posts I deleted the log.dirs directory referenced in the zookeeper.properties file and the log.dirs directory referenced in the server.properties file. After doing this I can start kafka again without any error - but the 50 messages are stored in __consumer_offset again when starting the schema-registry and after stopping kafka and trying to start kafka again it fails with the same error.
Any help is greatly appreciated. :)
EDIT:
Above that error theres another error saying:
"ERROR Failed to clean up log for _schemas-0 in dir /mnt/c/Users/Username/Desktop/Big_Data/confluent-6.0.0/kafka-logs due to IOException (kafka.server.LogDirFailureChannel) java.io.IOException: Invalid argument"
and also two warnings:
"WARN [ReplicaManager broker=0] Stopping serving replicas in dir /mnt/c/Users/Username/Desktop/Big_Data/confluent-6.0.0/kafka-logs (kafka.server.ReplicaManager)"
and
"WARN [ReplicaManager broker=0] Broker 0 stopped fetcher for partitions __consumer_offsets-22, ... (all of the 50 offsets are then listed)"
I'm trying to run Kafka. I'm able to start up zookeeper but when I start the Kafka server using
.\bin\windows\kafka-server-start.bat .\config\server.properties
I get the following error:
WARN [Controller id=0, targetBrokerId=0] Connection to node 0 could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
Might anyone know how to resolve this issue? Thanks in advance.
I had this error yesterday and just resolved it this morning. For others who run into a similar issue, my fix was in server.properties to change
listeners=PLAINTEXT://:9092 to listeners=PLAINTEXT://127.0.0.1:9092
I got this error ERROR Disk error while locking directory while trying to start kafka-server-start.sh config/server.properties
[2020-05-21 23:44:11,323] INFO [ThrottledChannelReaper-Fetch]: Starting (kafka.server.ClientQuotaManager$ThrottledChannelReaper)
[2020-05-21 23:44:11,323] INFO [ThrottledChannelReaper-Produce]: Starting (kafka.server.ClientQuotaManager$ThrottledChannelReaper)
[2020-05-21 23:44:11,324] INFO [ThrottledChannelReaper-Request]: Starting (kafka.server.ClientQuotaManager$ThrottledChannelReaper)
[2020-05-21 23:44:11,340] ERROR Disk error while locking directory /opt/kafka2.13/kafka_2.13-2.5.0/data_log_tu_tao/kafka_data (kafka.server.LogDirFailureChannel)
java.nio.file.AccessDeniedException: /opt/kafka2.13/kafka_2.13-2.5.0/data_log_tu_tao/kafka_data/.lock
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:84)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
at sun.nio.fs.UnixFileSystemProvider.newFileChannel(UnixFileSystemProvider.java:177)
at java.nio.channels.FileChannel.open(FileChannel.java:287)
at java.nio.channels.FileChannel.open(FileChannel.java:335)
at kafka.utils.FileLock.<init>(FileLock.scala:31)
at kafka.log.LogManager.$anonfun$lockLogDirs$1(LogManager.scala:235)
at scala.collection.StrictOptimizedIterableOps.flatMap(StrictOptimizedIterableOps.scala:118)
at scala.collection.StrictOptimizedIterableOps.flatMap$(StrictOptimizedIterableOps.scala:105)
at scala.collection.mutable.ArraySeq.flatMap(ArraySeq.scala:38)
at kafka.log.LogManager.lockLogDirs(LogManager.scala:233)
at kafka.log.LogManager.<init>(LogManager.scala:104)
at kafka.log.LogManager$.apply(LogManager.scala:1084)
at kafka.server.KafkaServer.startup(KafkaServer.scala:253)
at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:44)
at kafka.Kafka$.main(Kafka.scala:82)
at kafka.Kafka.main(Kafka.scala)
[2020-05-21 23:44:11,344] ERROR [KafkaServer id=0] Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)
java.nio.file.AccessDeniedException: /opt/kafka2.13/kafka_2.13-2.5.0/data_log_tu_tao/kafka_data/recovery-point-offset-checkpoint
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:84)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
at sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:214)
at java.nio.file.Files.newByteChannel(Files.java:361)
at java.nio.file.Files.createFile(Files.java:632)
at kafka.server.checkpoints.CheckpointFile.<init>(CheckpointFile.scala:45)
at kafka.server.checkpoints.OffsetCheckpointFile.<init>(OffsetCheckpointFile.scala:57)
at kafka.log.LogManager.$anonfun$recoveryPointCheckpoints$1(LogManager.scala:106)
at scala.collection.StrictOptimizedIterableOps.map(StrictOptimizedIterableOps.scala:100)
at scala.collection.StrictOptimizedIterableOps.map$(StrictOptimizedIterableOps.scala:87)
at scala.collection.mutable.ArraySeq.map(ArraySeq.scala:38)
at kafka.log.LogManager.<init>(LogManager.scala:105)
at kafka.log.LogManager$.apply(LogManager.scala:1084)
at kafka.server.KafkaServer.startup(KafkaServer.scala:253)
at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:44)
at kafka.Kafka$.main(Kafka.scala:82)
at kafka.Kafka.main(Kafka.scala)
This is known issue with Kafka distribution for windows. Refer: https://issues.apache.org/jira/browse/KAFKA-13391
Either use Kafka 2.8 (kafka_2.12-2.8.1.tgz) or wait for Kafka 3.0.1 or Kafka 3.1.0
For people using Kafka on windows and having a related error to
java.nio.file.AccessDeniedException:
This is a common error when log retention happens.
Kafka doesn’t have good support for windows filesystem.
You can use WSL2 or Docker to work around these limitations
Try to use kafka_2.12-2.8.1.tgz. This resolved the issue for me.
As the error states, the user that starts the Kafka Server process does not have access to your log.dirs:
[2020-05-21 23:44:11,344] ERROR [KafkaServer id=0] Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)
java.nio.file.AccessDeniedException: /opt/kafka2.13/kafka_2.13-2.5.0/data_log_tu_tao/kafka_data/recovery-point-offset-checkpoint
You can either:
Change log.dirs (Make sure NOT to use /tmp/)
Or grant read/write access for /opt/kafka2.13/kafka_2.13-2.5.0/data_log_tu_tao/kafka_data/
If none of the above options works for you, then probably it might be worth checking if the directory actually exists. If not, simply create it by running
mkdir -p /opt/kafka2.13/kafka_2.13-2.5.0/data_log_tu_tao/kafka_data
As a side note, I wouldn't say that /opt/ is the best place to store data.
I am trying to start up Zookeeper via the CLI with the command:
bin/zookeeper-server-start.sh ../config/zookeeper.properties
And it hums along for a second with what all seems to be correct until it says this:
INFO binding to port 0.0.0.0/0.0.0.0:2181 (org.apache.zookeeper.server.NIOServerCnxnFactory)
and then the below loops indefinitely until I exit:
[2018-08-10 15:07:48,223] INFO Accepted socket connection from /172.31.39.32:46374 (org.apache.zookeeper.server.NIOServerCnxnFactory)
[2018-08-10 15:07:48,228] WARN Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running (org.apache.zookeeper.server.NIOServerCnxn)
[2018-08-10 15:07:48,228] INFO Closed socket connection for client /172.31.39.32:46374 (no session established for client) (org.apache.zookeeper.server.NIOServerCnxn)
This is a single server and I believe a single node test server, so there isn't a quorum or other pieces running. My zookeeper config is basic, it only contains this:
dataDir=/tmp/zookeeper
clientPort=2181
maxClientCnxns=0
The weird thing is, my zookeeper had been running fine, and I had made NO changes to the config. Pulled it down to try to fix something else to do a quick restart on the zookeeper, and it won't budge. I've checked, and nothing else is running on port 2181.
I see this question has been asked several times with no answers, any ideas?
This might be happening because of some corruption in zookeeper data. You should not set dataDir to /tmp/*. If your computer purges some data of /tmp, it will be difficult for zookeeper to restore the state upon restart. If you check the zookeeper logs, you should see some kind of exception there.
Since you mentioned this zookeeper instance is for test purpose only. You should set
dataDir to anything but /tmp and try restart.