I want to fetch the list of broker ids in a cluster using kubectl exec command.
I am able to run the commands from inside the pod and fetch the list of broker ids, however I need to find the list without having to go inside.
I am using kafka helm charts from incubator and Kubernetes distribution which comes along with docker for Mac.
kubectl exec hissing-warthog-kafka-1 -- /usr/bin/zookeeper-shell hissing-warthog-zookeeper:2181 <<< "ls /brokers/ids"
Expected result:
Welcome to ZooKeeper!
JLine support is enabled
WATCHER::
WatchedEvent state:SyncConnected type:None path:null
[zk: hissing-warthog-zookeeper:2181(CONNECTED) 0] ls /brokers/ids
[0, 1, 2]
Actual result:
Connecting to hissing-warthog-zookeeper:2181
Welcome to ZooKeeper!
JLine support is enabled
WATCHER::
WatchedEvent state:SyncConnected type:None path:null
[zk: hissing-warthog-zookeeper:2181(CONNECTED) 0]
It should work in following way:
kubectl exec hissing-warthog-kafka-1 -- /usr/bin/zookeeper-shell hissing-warthog-zookeeper:2181 -c ls /brokers/ids
Hope this helps.
Related
Just installed and started zookeeper on windows, try to connect using:
$ bin/zkCli.sh -server 127.0.0.1:in/zkCli.sh -server 127.0.0.1:2181
and server starts correctly as below, but then if I type help and click 'enter', it just keep showing new lines? I am supposed to see some information, right?
Connecting to 127.0.0.1:2181
...
Welcome to ZooKeeper!
...
JLine support is enabled
2022-03-08 15:04:47,230 [myid:127.0.0.1:2181] - INFO [main-SendThread(127.0.0.1:2181):o.a.z.ClientCnxn$SendThread#1444] - Session establishment complete on server 127.0.0.1/127.0.0.1:2181, session id = 0x100475ac9070009, negotiated timeout = 30000
WATCHER::
WatchedEvent state:SyncConnected type:None path:null
[zk: 127.0.0.1:2181(CONNECTED) 0] help
(enter just create more new blank line here...)
I'm trying to setup my kafka cluster to accept SASL_SSL / SCRAM authentication.
First of all, everything is working actually and clients connect using SASL_SSL / GSSAPI. My Zk servers are configured also for SASL authentication and TLS.
I'm using confluent docker images for Kafka and Zk:
confluentinc/cp-kafka:6.0.1
confluentinc/cp-zookeeper:5.5.3-3
So I just modified my setup to allow SCRAM-SHA-512 in Kafka: KAFKA_SASL_ENABLED_MECHANISMS=GSSAPI,SCRAM-SHA-512
Following these instructions I now want to add the users in Zk and this is where the problems start (from the Kafka node):
[root#kafka1 [RCI] ~]# /usr/bin/podman exec kafka kafka-configs --zk-tls-config-file /etc/kafka/secrets/zk-ssl.properties --zookeeper Zk:3181 --alter --entity-type topics --entity-name test_jerome --add-config 'retention.ms=1'
Warning: --zookeeper is deprecated and will be removed in a future version of Kafka.
Use --bootstrap-server instead to specify a broker to connect to.
[2021-01-21 17:44:14,807] WARN zookeeper.ssl.keyStore.location not specified (org.apache.zookeeper.common.X509Util)
Error while executing config command with args '--zk-tls-config-file /etc/kafka/secrets/zk-ssl.properties --zookeeper Zk:3181 --alter --entity-type topics --entity-name test_jerome --add-config retention.ms=1'
org.apache.zookeeper.KeeperException$InvalidACLException: KeeperErrorCode = InvalidACL for /config/changes
at org.apache.zookeeper.KeeperException.create(KeeperException.java:128)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:54)
at kafka.zookeeper.AsyncResponse.maybeThrow(ZooKeeperClient.scala:564)
at kafka.zk.KafkaZkClient.createRecursive(KafkaZkClient.scala:1646)
at kafka.zk.KafkaZkClient.makeSurePersistentPathExists(KafkaZkClient.scala:1568)
at kafka.zk.KafkaZkClient.createConfigChangeNotification(KafkaZkClient.scala:395)
at kafka.zk.AdminZkClient.changeEntityConfig(AdminZkClient.scala:385)
at kafka.zk.AdminZkClient.changeTopicConfig(AdminZkClient.scala:342)
at kafka.zk.AdminZkClient.changeConfigs(AdminZkClient.scala:278)
at kafka.admin.ConfigCommand$.alterConfigWithZk(ConfigCommand.scala:167)
at kafka.admin.ConfigCommand$.processCommandWithZk(ConfigCommand.scala:118)
at kafka.admin.ConfigCommand$.main(ConfigCommand.scala:92)
at kafka.admin.ConfigCommand.main(ConfigCommand.scala)
Error: non zero exit code: 1: OCI runtime error
ZK logs are not really helpful:
[2021-01-21 17:58:08,333] INFO Successfully authenticated client: authenticationID=admin; authorizationID=admin. (org.apache.zookeeper.server.auth.SaslServerCallbackHandler)
[2021-01-21 17:58:08,333] INFO Successfully authenticated client: authenticationID=admin; authorizationID=admin. (org.apache.zookeeper.server.auth.SaslServerCallbackHandler)
[2021-01-21 17:58:08,333] INFO Setting authorizedID: admin (org.apache.zookeeper.server.auth.SaslServerCallbackHandler)
[2021-01-21 17:58:08,333] INFO Setting authorizedID: admin (org.apache.zookeeper.server.auth.SaslServerCallbackHandler)
[2021-01-21 17:58:08,334] INFO adding SASL authorization for authorizationID: admin (org.apache.zookeeper.server.ZooKeeperServer)
[2021-01-21 17:58:08,334] INFO adding SASL authorization for authorizationID: admin (org.apache.zookeeper.server.ZooKeeperServer)
As you can see, the SASL authentication works and the admin user is well identified.
If I login into the Zk shell (from the Zk node), you cann see that the acls are fully open:
getAcl /config/users
'world,'anyone
: cdrwa
getAcl /config
'world,'anyone
: cdrwa
getAcl /
'world,'anyone
: cdrwa
If I create the directory inside the Zk shell (from the Zk node) it works:
create /config/users/topicctl
Created /config/users/topicctl
There are no logs on the Zk server when I do this as I do not authenticate.
I now spend the afternoon on this problem without any progress.
What could be the problem please ?
I finally found my issue thanks to this post: Kafka not starting up if zookeeper.set.acl is set to true
I just added this info to the KAFKA_OPTS env variable: "-Dzookeeper.kerberos.removeHostFromPrincipal=true -Dzookeeper.kerberos.removeRealmFromPrincipal=true -Dzookeeper.authProvider.1=org.apache.zookeeper.server.auth.SASLAuthenticationProvider -Dzookeeper.requireClientAuthScheme=sasl"
My main question is now: why sasl authentication was working before that !
Is there a command to show the details of Kafka server or the status of Kafka server? (I am not trying to find out if the kafka server is running.)
I can only find information on topic, partition, producer, and consumer CLI commands.
If you are looking for the Kafka cluster broker status, you can use zookeeper cli to find the details for each broker as given below:
ls /brokers/ids returns the list of active brokers IDs on the cluster.
get /brokers/ids/<id> returns the details of the broker with the given ID.
Example :
kafka_2.12-1.1.1 % ./bin/zookeeper-shell.sh localhost:2181 ls /brokers/ids
Connecting to localhost:2181
WATCHER::
WatchedEvent state:SyncConnected type:None path:null
[0]
kafka_2.12-1.1.1 % ./bin/zookeeper-shell.sh localhost:2181 get /brokers/ids/0
Connecting to localhost:2181
WATCHER::
WatchedEvent state:SyncConnected type:None path:null
{"listener_security_protocol_map":{"PLAINTEXT":"PLAINTEXT"},"endpoints":["PLAINTEXT://localhost:9092"],"jmx_port":-1,"host":"localhost","timestamp":"1558428038778","port":9092,"version":4}
cZxid = 0x116
ctime = Tue May 21 08:40:38 UTC 2019
mZxid = 0x116
mtime = Tue May 21 08:40:38 UTC 2019
pZxid = 0x116
cversion = 0
dataVersion = 0
aclVersion = 0
ephemeralOwner = 0x16ad9830f16000b
dataLength = 188
numChildren = 0
You can put these steps in some shell script to get the details for all brokers.
You can activate JMX metrics by setting environment variable JMX_PORT.
$ export JMX_PORT=9010
$ ./bin/kafka-server-start.sh ./config/server.properties
Then, you can use jconsole or Java Mission Control to display cluster metrics.
I have created a znode using:
zookeeper-0:/opt/zookeeper/bin # ./zkCli.sh create /mynode content digest:user:pass:cdrwa
How to access the znode using the zkCli.sh utility now?
zookeeper-0:/opt/zookeeper/bin # ./zkCli.sh get /mynode
Connecting to localhost:2181
WATCHER::
WatchedEvent state:SyncConnected type:None path:null
Authentication is not valid : /mynode
zookeeper-0:/opt/zookeeper/bin #
The getAcl is showing the following:
zookeeper-0:/opt/zookeeper/bin # ./zkCli.sh getAcl /mynode
Connecting to localhost:2181
WATCHER::
WatchedEvent state:SyncConnected type:None path:null
'digest,'user:pass
: cdrwa
zookeeper-0:/opt/zookeeper/bin #
You need to create the digest ACL using the hashed password.
ZooKeeper Programmer's Guide
digest uses a username:password string to generate MD5 hash which is then used as an ACL ID identity. Authentication is done by sending the username:password in clear text. When used in the ACL the expression will be the username:base64 encoded SHA1 password digest.
Generate the hashed password
$ java -cp "./zookeeper-3.4.13.jar:./lib/slf4j-api-1.7.25.jar" \
org.apache.zookeeper.server.auth.DigestAuthenticationProvider user:pass
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
user:pass->user:smGaoVKd/cQkjm7b88GyorAUz20=
Create a node using the hashed password
[zk: zookeeper(CONNECTED) 0] create /mynode content digest:user:smGaoVKd/cQkjm7b88GyorAUz20=:cdrwa
Created /mynode
Accessing the protected node
[zk: zookeeper(CONNECTED) 1] get /mynode
Authentication is not valid : /mynode
[zk: zookeeper(CONNECTED) 2] addauth digest user:pass
[zk: zookeeper(CONNECTED) 3] get /mynode
content
cZxid = 0x14
ctime = Wed Sep 12 19:37:48 GMT 2018
mZxid = 0x14
mtime = Wed Sep 12 19:37:48 GMT 2018
pZxid = 0x14
cversion = 0
dataVersion = 0
aclVersion = 0
ephemeralOwner = 0x0
dataLength = 7
numChildren = 0
If you look at the content of the zkcli.sh script, you will see a commented out block showing how to configure an environment variable with credentials:
#SOLR_ZK_CREDS_AND_ACLS="-DzkACLProvider=org.apache.solr.common.cloud.VMParamsAllAndReadonlyDigestZkACLProvider \
# -DzkCredentialsProvider=org.apache.solr.common.cloud.VMParamsSingleSetCredentialsDigestZkCredentialsProvider \
# -DzkDigestUsername=admin-user -DzkDigestPassword=CHANGEME-ADMIN-PASSWORD \
# -DzkDigestReadonlyUsername=readonly-user -DzkDigestReadonlyPassword=CHANGEME-READONLY-PASSWORD"
You can configure the environment variable SOLR_ZK_CREDS_AND_ACLS on your local system with the correct credentials following this template and then the zkcli.sh script will use them when communicating with ZooKeeper.
I have an issue, starting my zkServer is working well but when i do
sudo bin/zkCli.sh
It displayed the follow for hours:
Connecting to localhost:2181
Welcome to ZooKeeper!
JLine support is enabled
[zk: localhost:2181(CONNECTING) 0]
if i tried
ls /
it will show (i don't think that's the issue as it is not connected):
Exception in thread "main"
org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /
at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1212)
at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1241)
at org.apache.zookeeper.ZooKeeperMain.processZKCmd(ZooKeeperMain.java:725)
at org.apache.zookeeper.ZooKeeperMain.processCmd(ZooKeeperMain.java:599)
at org.apache.zookeeper.ZooKeeperMain.run(ZooKeeperMain.java:362)
at org.apache.zookeeper.ZooKeeperMain.main(ZooKeeperMain.java:290)
I faced a similar issue. For me the zookeeper hadn't started.
Before my fix, when I did zkcli -server localhost:2181, I got
Connecting to localhost:2181
Welcome to ZooKeeper!
JLine support is enabled
[zk: localhost:2181(CONNECTING) 0]
like you. Notice the "CONNECTING" here. Doing ls / here gave error and exited zkcli:
Exception in thread "main" org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /
at org.apache.zookeeper.KeeperException.create(KeeperException.java:102)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:54)
at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1541)
at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1569)
at org.apache.zookeeper.ZooKeeperMain.processZKCmd(ZooKeeperMain.java:732)
at org.apache.zookeeper.ZooKeeperMain.processCmd(ZooKeeperMain.java:600)
at org.apache.zookeeper.ZooKeeperMain.executeLine(ZooKeeperMain.java:372)
at org.apache.zookeeper.ZooKeeperMain.run(ZooKeeperMain.java:332)
at org.apache.zookeeper.ZooKeeperMain.main(ZooKeeperMain.java:291)
The fix is is to start your zookeeper using zkServer start
On running this, you get:
ZooKeeper JMX enabled by default
Using config: /usr/local/etc/zookeeper/zoo.cfg
Starting zookeeper ... STARTED
After this when you do zkcli -server localhost:2181, you get:
Connecting to localhost:2181
Welcome to ZooKeeper!
JLine support is enabled
WATCHER::
WatchedEvent state:SyncConnected type:None path:null
[zk: localhost:2181(CONNECTED) 0] ls /
[cf, zookeeper]
[zk: localhost:2181(CONNECTED) 1]
Notice the 'CONNECTED' here. I had created /cf after successful connection.
Hope this solution helps.
For mac users, your zkServer is in usr/local/Cellar/zookeeper/3.4.13/bin