I have an issue, starting my zkServer is working well but when i do
sudo bin/zkCli.sh
It displayed the follow for hours:
Connecting to localhost:2181
Welcome to ZooKeeper!
JLine support is enabled
[zk: localhost:2181(CONNECTING) 0]
if i tried
ls /
it will show (i don't think that's the issue as it is not connected):
Exception in thread "main"
org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /
at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1212)
at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1241)
at org.apache.zookeeper.ZooKeeperMain.processZKCmd(ZooKeeperMain.java:725)
at org.apache.zookeeper.ZooKeeperMain.processCmd(ZooKeeperMain.java:599)
at org.apache.zookeeper.ZooKeeperMain.run(ZooKeeperMain.java:362)
at org.apache.zookeeper.ZooKeeperMain.main(ZooKeeperMain.java:290)
I faced a similar issue. For me the zookeeper hadn't started.
Before my fix, when I did zkcli -server localhost:2181, I got
Connecting to localhost:2181
Welcome to ZooKeeper!
JLine support is enabled
[zk: localhost:2181(CONNECTING) 0]
like you. Notice the "CONNECTING" here. Doing ls / here gave error and exited zkcli:
Exception in thread "main" org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /
at org.apache.zookeeper.KeeperException.create(KeeperException.java:102)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:54)
at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1541)
at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1569)
at org.apache.zookeeper.ZooKeeperMain.processZKCmd(ZooKeeperMain.java:732)
at org.apache.zookeeper.ZooKeeperMain.processCmd(ZooKeeperMain.java:600)
at org.apache.zookeeper.ZooKeeperMain.executeLine(ZooKeeperMain.java:372)
at org.apache.zookeeper.ZooKeeperMain.run(ZooKeeperMain.java:332)
at org.apache.zookeeper.ZooKeeperMain.main(ZooKeeperMain.java:291)
The fix is is to start your zookeeper using zkServer start
On running this, you get:
ZooKeeper JMX enabled by default
Using config: /usr/local/etc/zookeeper/zoo.cfg
Starting zookeeper ... STARTED
After this when you do zkcli -server localhost:2181, you get:
Connecting to localhost:2181
Welcome to ZooKeeper!
JLine support is enabled
WATCHER::
WatchedEvent state:SyncConnected type:None path:null
[zk: localhost:2181(CONNECTED) 0] ls /
[cf, zookeeper]
[zk: localhost:2181(CONNECTED) 1]
Notice the 'CONNECTED' here. I had created /cf after successful connection.
Hope this solution helps.
For mac users, your zkServer is in usr/local/Cellar/zookeeper/3.4.13/bin
Related
Just installed and started zookeeper on windows, try to connect using:
$ bin/zkCli.sh -server 127.0.0.1:in/zkCli.sh -server 127.0.0.1:2181
and server starts correctly as below, but then if I type help and click 'enter', it just keep showing new lines? I am supposed to see some information, right?
Connecting to 127.0.0.1:2181
...
Welcome to ZooKeeper!
...
JLine support is enabled
2022-03-08 15:04:47,230 [myid:127.0.0.1:2181] - INFO [main-SendThread(127.0.0.1:2181):o.a.z.ClientCnxn$SendThread#1444] - Session establishment complete on server 127.0.0.1/127.0.0.1:2181, session id = 0x100475ac9070009, negotiated timeout = 30000
WATCHER::
WatchedEvent state:SyncConnected type:None path:null
[zk: 127.0.0.1:2181(CONNECTED) 0] help
(enter just create more new blank line here...)
I am a beginner and I have to use Kafka for data transfer into/from Hadoop FS (or any other application, not just through put or copyFromLocal commands),kafka needs zookeeper as well, I enabled Zooekeeper audit logging but I still get errors.
For Kafka, when I want to start it:
JMX_PORT=8004 bin/kafka-server-start.sh config/server.properties
I get the error:
[2022-02-16 13:56:45,939] INFO shutting down (kafka.server.KafkaServer)
[2022-02-16 13:56:46,114] INFO App info kafka.server for 0 unregistered (org.apache.kafka.common.utils.AppInfoParser)
[2022-02-16 13:56:46,133] INFO shut down completed (kafka.server.KafkaServer)
[2022-02-16 13:56:46,133] ERROR Exiting Kafka. (kafka.Kafka$)
[2022-02-16 13:56:46,165] INFO shutting down (kafka.server.KafkaServer)
And when I want to start Zookeeper using the command:
bin/zookeeper-server-start.sh config/zookeeper.properties
I get the following (and it gets stuck on it):
[2022-02-16 14:03:13,954] INFO zookeeper.request_throttler.shutdownTimeout = 10000 (org.apache.zookeeper.server.RequestThrottler)
[2022-02-16 14:03:13,955] INFO PrepRequestProcessor (sid:0) started, reconfigEnabled=false (org.apache.zookeeper.server.PrepRequestProcessor)
[2022-02-16 14:03:14,136] INFO Using checkIntervalMs=60000 maxPerMinute=10000 maxNeverUsedIntervalMs=0 (org.apache.zookeeper.server.ContainerManager)
[2022-02-16 14:03:14,138] INFO ZooKeeper audit is disabled. (org.apache.zookeeper.audit.ZKAuditProvider)
Does anyone know how to work this out? I enabled audit logging but still. Same problem.
Zookeeper CLI isn't "stuck"; it's waiting for connections.
Open a new terminal and start Kafka
Alternatively, you could use Docker Compose / Kubernetes, if you think your host / local JVM is causing issues
I'm trying to setup my kafka cluster to accept SASL_SSL / SCRAM authentication.
First of all, everything is working actually and clients connect using SASL_SSL / GSSAPI. My Zk servers are configured also for SASL authentication and TLS.
I'm using confluent docker images for Kafka and Zk:
confluentinc/cp-kafka:6.0.1
confluentinc/cp-zookeeper:5.5.3-3
So I just modified my setup to allow SCRAM-SHA-512 in Kafka: KAFKA_SASL_ENABLED_MECHANISMS=GSSAPI,SCRAM-SHA-512
Following these instructions I now want to add the users in Zk and this is where the problems start (from the Kafka node):
[root#kafka1 [RCI] ~]# /usr/bin/podman exec kafka kafka-configs --zk-tls-config-file /etc/kafka/secrets/zk-ssl.properties --zookeeper Zk:3181 --alter --entity-type topics --entity-name test_jerome --add-config 'retention.ms=1'
Warning: --zookeeper is deprecated and will be removed in a future version of Kafka.
Use --bootstrap-server instead to specify a broker to connect to.
[2021-01-21 17:44:14,807] WARN zookeeper.ssl.keyStore.location not specified (org.apache.zookeeper.common.X509Util)
Error while executing config command with args '--zk-tls-config-file /etc/kafka/secrets/zk-ssl.properties --zookeeper Zk:3181 --alter --entity-type topics --entity-name test_jerome --add-config retention.ms=1'
org.apache.zookeeper.KeeperException$InvalidACLException: KeeperErrorCode = InvalidACL for /config/changes
at org.apache.zookeeper.KeeperException.create(KeeperException.java:128)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:54)
at kafka.zookeeper.AsyncResponse.maybeThrow(ZooKeeperClient.scala:564)
at kafka.zk.KafkaZkClient.createRecursive(KafkaZkClient.scala:1646)
at kafka.zk.KafkaZkClient.makeSurePersistentPathExists(KafkaZkClient.scala:1568)
at kafka.zk.KafkaZkClient.createConfigChangeNotification(KafkaZkClient.scala:395)
at kafka.zk.AdminZkClient.changeEntityConfig(AdminZkClient.scala:385)
at kafka.zk.AdminZkClient.changeTopicConfig(AdminZkClient.scala:342)
at kafka.zk.AdminZkClient.changeConfigs(AdminZkClient.scala:278)
at kafka.admin.ConfigCommand$.alterConfigWithZk(ConfigCommand.scala:167)
at kafka.admin.ConfigCommand$.processCommandWithZk(ConfigCommand.scala:118)
at kafka.admin.ConfigCommand$.main(ConfigCommand.scala:92)
at kafka.admin.ConfigCommand.main(ConfigCommand.scala)
Error: non zero exit code: 1: OCI runtime error
ZK logs are not really helpful:
[2021-01-21 17:58:08,333] INFO Successfully authenticated client: authenticationID=admin; authorizationID=admin. (org.apache.zookeeper.server.auth.SaslServerCallbackHandler)
[2021-01-21 17:58:08,333] INFO Successfully authenticated client: authenticationID=admin; authorizationID=admin. (org.apache.zookeeper.server.auth.SaslServerCallbackHandler)
[2021-01-21 17:58:08,333] INFO Setting authorizedID: admin (org.apache.zookeeper.server.auth.SaslServerCallbackHandler)
[2021-01-21 17:58:08,333] INFO Setting authorizedID: admin (org.apache.zookeeper.server.auth.SaslServerCallbackHandler)
[2021-01-21 17:58:08,334] INFO adding SASL authorization for authorizationID: admin (org.apache.zookeeper.server.ZooKeeperServer)
[2021-01-21 17:58:08,334] INFO adding SASL authorization for authorizationID: admin (org.apache.zookeeper.server.ZooKeeperServer)
As you can see, the SASL authentication works and the admin user is well identified.
If I login into the Zk shell (from the Zk node), you cann see that the acls are fully open:
getAcl /config/users
'world,'anyone
: cdrwa
getAcl /config
'world,'anyone
: cdrwa
getAcl /
'world,'anyone
: cdrwa
If I create the directory inside the Zk shell (from the Zk node) it works:
create /config/users/topicctl
Created /config/users/topicctl
There are no logs on the Zk server when I do this as I do not authenticate.
I now spend the afternoon on this problem without any progress.
What could be the problem please ?
I finally found my issue thanks to this post: Kafka not starting up if zookeeper.set.acl is set to true
I just added this info to the KAFKA_OPTS env variable: "-Dzookeeper.kerberos.removeHostFromPrincipal=true -Dzookeeper.kerberos.removeRealmFromPrincipal=true -Dzookeeper.authProvider.1=org.apache.zookeeper.server.auth.SASLAuthenticationProvider -Dzookeeper.requireClientAuthScheme=sasl"
My main question is now: why sasl authentication was working before that !
I have an environment problem.
I want using zookeeper and Kafka cluster to solve my problem.
My zookeeper version is 3.4.12 and Kafka is 2.12-2.1.0
I also change the zoo.cfg in zookeeper.
dataDir=D:/WEBSOCKET/zookeeper-3.4.12/data
and server.properties in kafka.
log.dirs=D:/WEBSOCKET/kafka_2.12-2.1.0/logs
I see all the tutorial and do it the exact same way.
And also usgin kafka open zookeeper.
this is my command:
1) open zookeeper (zkServer.cmd)
2) in kafka
.\bin\windows\kafka-server-start.bat .\config\server.properties
3) create topic
.\bin\windows\kafka-topics.bat --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic hello
4) create a producer
.\bin\windows\kafka-console-producer.bat --bootstrap-server localhost:2181 --topic hello
5) create a consumer
.\bin\windows\kafka-console-consumer.bat --bootstrap-server localhost:2181 --topic hello
when I get in step 5, I always fail.
zookeeper will give me a lot of console like :
WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn#383] - Exception causing close of session 0x0: null
INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory#215] - Accepted socket connection from /127.0.0.1:55192
and
2019-01-08 17:05:24,822 [myid:] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn#1040] - Closed socket connection for client /127.0.0.1:50874 (no session established for client)
2019-01-08 17:05:25,783 [myid:] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory#215] - Accepted socket connection from /127.0.0.1:56089
I don't know how to fix it. I google for two days...
when I open my Kafka with step 2, my zookeeper some times doesn't hvae any response or shows me this:
[ProcessThread(sid:0 cport:2181)::PrepRequestProcessor#596] - Got user-level KeeperException when processing sessionid:0x1000058f8960000 type:multi cxid:0x36 zxid:0x69 txntype:-1 reqpath:n/a aborting remaining multi ops. Error Path:/admin/preferred_replica_election Error:KeeperErrorCode = NoNode for /admin/preferred_replica_election
I also google this, but not helpful.
I set this in kfaka before:
advertised.host.name = localhost
listeners=PLAINTEXT://127.0.0.1:9092
my host have set
127.0.0.1 localhost
please help me create local server I want to coding my project..
thank you read all.
Producers and consumers need to use port 9092 (Kafka)
You are seeing Zookeeper logs and errors because you are trying to use bootstrap-server or broker-list and port 2181 (Zookeeper)
Check again the quickstart guide
bin/zookeeper-shell.sh config/zookeeper.properties
Connecting to config/zookeeper.properties
Exception in thread "main" java.net.UnknownHostException: config: Name or service not known
To run the zookeeper shell you need to provide the machine on which it is running as parameter, not the properties file:
./bin/zookeeper-shell.sh localhost:2181