When i am using confluent ksql to create a stream and after resetting the offeset to earliest and trying to query the stream i am seeing the Kafka server crashing. I am using windows 10 and i have tried both Ubuntu and Debian as WSL.
I have tried to clear the log folder sudo rm -fr /tmp/confl* and tried to restart the confluent platform using confluent local start but kafka is not getting started.
Below is the error i am seeing in the confluent local log kafka
INFO [Transaction State Manager 0]: Loading transaction metadata from __transaction_state-8 at epoch 0 (kafka.coordinator.transaction.TransactionStateManager)
[2020-06-26 11:27:26,208] **ERROR Error while renaming dir for _confluent-ksql-default_transient_1143297338875599674_1593157827320-Aggregate-Aggregate-Materialize-changelog-0 in log dir /tmp/confluent.HBnj6u7x/kafka/data (kafka.server.LogDirFailureChannel)
java.nio.file.AccessDeniedException: /tmp/confluent.HBnj6u7x/kafka/data/_confluent-ksql-default_transient_1143297338875599674_1593157827320-Aggregate-Aggregate-Materialize-changelog-0 -> /tmp/confluent.HBnj6u7x/kafka/data/_confluent-ksql-default_transient_1143297338875599674_1593157827320-Aggregate-Aggregate-Materialize-changelog-0.355fe6c61afa41609e74e252e3dbac92-delete**
[2020-06-26 11:27:26,287] WARN Stopping serving logs in dir /tmp/confluent.HBnj6u7x/kafka/data (kafka.log.LogManager)
[2020-06-26 11:27:26,292] **ERROR Shutdown broker because all log dirs in /tmp/confluent.HBnj6u7x/kafka/data have failed (kafka.log.LogManager)**
[2020-06-26 11:27:26,294] INFO [Transaction State Manager 0]: Completed loading transaction metadata from __transaction_state-44 for coordinator epoch 0 (kafka.coordinator.transaction.TransactionStateManager)
[2020-06-26 11:27:26,295] INFO [Transaction State Manager 0]: Loading transaction metadata from __transaction_state-27 at epoch 0 (kafka.coordinator.transaction.TransactionStateManager)
The java.nio.file.AccessDeniedException error message tells me that something is off regarding KSQL being able to read/write files in the filesystem provided by WSL. This might be related to a common problem of WSL not providing the proper Linux permissions on the filesystem. A possible fix is describe on this blog.
Regardless, I would encourage you to try using ksqlDB (the community version of Confluent KSQL) that provides ready-to-be-used scripts based on Docker so you can abstract away these filesystem problems and jump straight to the coding part.
ksqlDB Quickstart
Related
I am a beginner and I have to use Kafka for data transfer into/from Hadoop FS (or any other application, not just through put or copyFromLocal commands),kafka needs zookeeper as well, I enabled Zooekeeper audit logging but I still get errors.
For Kafka, when I want to start it:
JMX_PORT=8004 bin/kafka-server-start.sh config/server.properties
I get the error:
[2022-02-16 13:56:45,939] INFO shutting down (kafka.server.KafkaServer)
[2022-02-16 13:56:46,114] INFO App info kafka.server for 0 unregistered (org.apache.kafka.common.utils.AppInfoParser)
[2022-02-16 13:56:46,133] INFO shut down completed (kafka.server.KafkaServer)
[2022-02-16 13:56:46,133] ERROR Exiting Kafka. (kafka.Kafka$)
[2022-02-16 13:56:46,165] INFO shutting down (kafka.server.KafkaServer)
And when I want to start Zookeeper using the command:
bin/zookeeper-server-start.sh config/zookeeper.properties
I get the following (and it gets stuck on it):
[2022-02-16 14:03:13,954] INFO zookeeper.request_throttler.shutdownTimeout = 10000 (org.apache.zookeeper.server.RequestThrottler)
[2022-02-16 14:03:13,955] INFO PrepRequestProcessor (sid:0) started, reconfigEnabled=false (org.apache.zookeeper.server.PrepRequestProcessor)
[2022-02-16 14:03:14,136] INFO Using checkIntervalMs=60000 maxPerMinute=10000 maxNeverUsedIntervalMs=0 (org.apache.zookeeper.server.ContainerManager)
[2022-02-16 14:03:14,138] INFO ZooKeeper audit is disabled. (org.apache.zookeeper.audit.ZKAuditProvider)
Does anyone know how to work this out? I enabled audit logging but still. Same problem.
Zookeeper CLI isn't "stuck"; it's waiting for connections.
Open a new terminal and start Kafka
Alternatively, you could use Docker Compose / Kubernetes, if you think your host / local JVM is causing issues
ERROR Error while creating ephemeral at /brokers/ids/0, node already exists and owner '72067757872119809' does not match current session '72067836689711106' (kafka.zk.KafkaZkClient$CheckedEphemeral) 2021-05-05 02:19:44.796 [INF] [Kafka] [2021-05-05 02:19:44,786] ERROR [KafkaServer id=0] Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)
The error is saying a broker already is running with id=0, or that Zookeeper is corrupt because a broker did not previously cleanly shut down...
In the later case, you can attempt to use zookeeper-shell to rmr /brokers/ids/0, however, this might have more unintended consequences than preforming a restart of Zookeeper as well as the brokers
The only solution which works here is restart the broker and then restart the server.
Restart the zookeeper and the broker is fixed the issue for me.
If you using the docker-compose, you can restart simply by using
docker-compose restart
I am trying to learn kafka and I had the below error:
[2021-01-21 13:46:43,247] WARN [ReplicaManager broker=0] Broker 0 stopped fetcher for partitions
__consumer_offsets-22,first_topic-2,__consumer_offsets-37,first_topic-0,__consumer_offsets-
38,__consumer_offsets-13,twitter_tweets-5,__consumer_offsets-30,twitter_tweets-3,__consumer_offsets-
8,__consumer_offsets-21,__consumer_offsets-4,__consumer_offsets-27,__consumer_offsets-
7,__consumer_offsets-9,__consumer_offsets-46,new_topic-0,__consumer_offsets-25,__consumer_offsets-
35,twitter_tweets-0,__consumer_offsets-41,__consumer_offsets-33,__consumer_offsets-
23,__consumer_offsets-49,__consumer_offsets-47,__consumer_offsets-16,__consumer_offsets-
32,__consumer_offsets-40 and stopped moving logs for partitions because they are in the failed log
directory C:\kafka_2.13-2.6.0\data\kafka. (kafka.server.ReplicaManager)
[2021-01-21 13:46:43,252] WARN Stopping serving logs in dir C:\kafka_2.13-2.6.0\data\kafka
(kafka.log.LogManager)
[2021-01-21 13:46:43,254] ERROR Shutdown broker because all log dirs in C:\kafka_2.13-2.6.0\data\kafka have failed (kafka.log.LogManager)
This happens every time I run a command for example:
bin\windows\kafka-topics.bat --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic streams-plaintext-input.
When I delete all the offsets in /data folder then everything run smoothly. Is this happening because of the 7 day existing period that Kafka has?
Main issue is that Kafka depends on POSIX filesystem semantics that don't work well on windows.
Kafka uses specific features of POSIX to achieve high performance, so emulations—which happen on WSL 1—are insufficient. For example, the broker will crash when it rolls a segment file
This appears to be the error you're mentioning about the segment retention
If you want to use Kafka on windows, WSL2 is the suggested solution.
https://www.confluent.io/blog/set-up-and-run-kafka-on-windows-linux-wsl-2/
Also note: --zookeeper flag is deprecated
I am currently using the confluent platform community license. I started Zookeeper, Kafka and the schema-registry - all are used in local mode. However when starting the schema-registry for the first time, 50 messages are sent and stored inside the __consumer_offset topic (__consumer_offsets-0 to __consumer_offsets-49). Those messages are stored in the kafka-logs and when I am trying to start the services again, it fails. To be more precise: Zookeeper works but Kafka fails with the error:
"ERROR Shutdown broker because all log dirs have failed".
As suggested in some other posts I deleted the log.dirs directory referenced in the zookeeper.properties file and the log.dirs directory referenced in the server.properties file. After doing this I can start kafka again without any error - but the 50 messages are stored in __consumer_offset again when starting the schema-registry and after stopping kafka and trying to start kafka again it fails with the same error.
Any help is greatly appreciated. :)
EDIT:
Above that error theres another error saying:
"ERROR Failed to clean up log for _schemas-0 in dir /mnt/c/Users/Username/Desktop/Big_Data/confluent-6.0.0/kafka-logs due to IOException (kafka.server.LogDirFailureChannel) java.io.IOException: Invalid argument"
and also two warnings:
"WARN [ReplicaManager broker=0] Stopping serving replicas in dir /mnt/c/Users/Username/Desktop/Big_Data/confluent-6.0.0/kafka-logs (kafka.server.ReplicaManager)"
and
"WARN [ReplicaManager broker=0] Broker 0 stopped fetcher for partitions __consumer_offsets-22, ... (all of the 50 offsets are then listed)"
I was running Kafka with 2 borker for cluster.
But I keep getting the WARN log.
I checked all my systems and there was no host using IP 10.8.7.1.
By the way, there was more IPs looks like from zookeeper or broker ?
If I shotdown on of Kafka, the WARNING log will be less
I am not familiar with Kafka and zookeeper, just getting starting and study
Any ideas?
Kafka version: 1.0.1
WARN log similar as below(get this kind of log about 10 secs),
[2018-04-19 09:13:08,342] WARN [SocketServer brokerId=0] Unexpected error from /10.8.7.1; closing connection (org.apache.kafka.common.network.Selector)
org.apache.kafka.common.network.InvalidReceiveException: Invalid receive (size = 369295616 larger than 104857600)
at org.apache.kafka.common.network.NetworkReceive.readFromReadableChannel(NetworkReceive.java:132)
at org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:93)
at org.apache.kafka.common.network.KafkaChannel.receive(KafkaChannel.java:235)
at org.apache.kafka.common.network.KafkaChannel.read(KafkaChannel.java:196)
at org.apache.kafka.common.network.Selector.attemptRead(Selector.java:545)
at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:483)
at org.apache.kafka.common.network.Selector.poll(Selector.java:412)
at kafka.network.Processor.poll(SocketServer.scala:551)
at kafka.network.Processor.run(SocketServer.scala:468)
at java.lang.Thread.run(Thread.java:748)
One possible cause is that a Kafka producer on 10.8.7.1 is attempting to stream 0.369 GB of data in a batch instead of streaming. You may have to trace down the kafka producer and see whats going.
Hope this helps.