The following error is being prompted when it is tried to add a new cluster in 'CMAK' in the K8s cluster.
Yikes! KeeperErrorCode = Unimplemented for /kafka-manager/mutex Try again.
My cluster configurations are as follows,
zookeeper: wurstmeister/zookeeper
kafka-manager: kafkamanager/kafka-manager:3.0.0.4
kafka: wurstmeister/kafka:2.12-2.4.1
I could resolve it by following the steps.
Connect to the 'zookeeper' container in k8s
k exec -it podid -- bash
Connect with zookeeper cli,
./bin/zkCli.sh
Make sure that it has created the 'kafka-manager' path already. if it does not exist, then try to create a cluster in 'kafka-manager' first.
ls /kafka-manager
Hit the following commands to create subsequent paths,
create /kafka-manager/mutex ""
create /kafka-manager/mutex/locks ""
create /kafka-manager/mutex/leases ""
Now try to create the cluster again.
The output would be like this,
WATCHER::
WatchedEvent state:SyncConnected type:None path:null
[zk: localhost:2181(CONNECTED) 0] ls /kafka-manager
[configs, deleteClusters, clusters]
[zk: localhost:2181(CONNECTED) 1] create /kafka-manager/mutex ""
Created /kafka-manager/mutex
[zk: localhost:2181(CONNECTED) 2] create /kafka-manager/mutex/locks ""
Created /kafka-manager/mutex/locks
[zk: localhost:2181(CONNECTED) 3] create /kafka-manager/mutex/leases ""
Created /kafka-manager/mutex/leases
[zk: localhost:2181(CONNECTED) 4]
The original answer is mentioned here,
https://github.com/yahoo/CMAK/issues/731#issuecomment-643880544
Related
I'm trying to remove one of the five servers using the reconfig command but get KeeperErrorCode = MarshallingError.
Here's the cluster information and the error message:
[zk: [ClientIP](CONNECTED) 2] get /zookeeper/config
server.0=[ip0]:port1:port2:participant
server.1=[ip1]:port1:port2:participant
server.2=[ip2]:port1:port2:participant
server.3=[ip3]:port1:port2:participant
server.4=[ip4]:port1:port2:participant
version=200000000
[zk: [ClientIP](CONNECTED) 3] reconfig -remove server.2=[ip2]:port1:port2:participant
KeeperErrorCode = MarshallingError
I'm not sure why this error occurred and how to solve it. How can I remove one of the servers?
You have 2 problems.
You don't need to specify server IPs on remove:
reconfig [-s] [-v version] [[-file path] | [-members serverID=host:port1:port2;port3[,...]]] | [-add serverId=host:port1:port2;port3[,...]] [-remove serverId[,...]*]
only set the serverId
Example of good command:
[shahar.l]# /opt/kafka/bin/zookeeper-shell.sh localhost:2181 reconfig -remove 5
Connecting to localhost:2181
WATCHER::
WatchedEvent state:SyncConnected type:None path:null
Committed new configuration:
server.4=IP:2888:3888:participant;0.0.0.0:2181
server.6=IP:2888:3888:participant;0.0.0.0:2181
server.7=IP:2888:3888:participant;0.0.0.0:2181
version=1700007e9d
I have very simple setup. I am trying to start ZooKeeper (apache-zookeeper-3.6.1-bin) on two machines. I get following error when i do zookeeper status
/cygdrive/c/ZooKeeper/apache-zookeeper-3.6.1-bin/apache-zookeeper-3.6.1-bin
$ bin/zkServer.sh restart
ZooKeeper JMX enabled by default
Using config: C:\ZooKeeper\apache-zookeeper-3.6.1-bin\apache-zookeeper-3.6.1-bin\conf\zoo.cfg
ZooKeeper JMX enabled by default
Using config: C:\ZooKeeper\apache-zookeeper-3.6.1-bin\apache-zookeeper-3.6.1-bin\conf\zoo.cfg
Stopping zookeeper ... STOPPED
ZooKeeper JMX enabled by default
Using config: C:\ZooKeeper\apache-zookeeper-3.6.1-bin\apache-zookeeper-3.6.1-bin\conf\zoo.cfg
Starting zookeeper ... STARTED
kalsa#CO01EAP00000027 /cygdrive/c/ZooKeeper/apache-zookeeper-3.6.1-bin/apache-zookeeper-3.6.1-bin
$ bin/zkServer.sh status
ZooKeeper JMX enabled by default
Using config: C:\ZooKeeper\apache-zookeeper-3.6.1-bin\apache-zookeeper-3.6.1-bin\conf\zoo.cfg
cat: '/tmp/zookeeper/'$'\r''/myid': No such file or directory
clientPort not found and myid could not be determined. Terminating.
My Zoo.cfg
tickTime=5000
dataDir=/tmp/zookeeper/
clientPort=2181
initLimit=5
syncLimit=2
server.1=XYZ:2888:3888
server.2=ABC:2888:3888
I have proper IPs in place of XYZ and ABC.
I have created myid file created as well. Can someone let me know if i am missing anything obvious
enter image description here
The owner of datadir should be the zookeeper user. If not, change the owner.
Steps:
Started Zookeeper instance: bin/zookeeper-server-start.sh
config/zookeeper.properties :SUCCESS
Started Broker 0 :
bin/kafka-server-start.sh config/server-0.properties SUCCESS
Started Broker 1 :
bin/kafka-server-start.sh config/server-1.properties FAILED
ERROR: a.common.InconsistentBrokerIdException: Configured broker.id 1
doesn't match stored broker.id 0 in meta.properties. If you moved your
data, make sure your configured broker.id matches. If you intend to
create a new broker, you should remove all data in your
Edit config/server-1.properties and replace the existing config values as follows:
broker.id=1
port=9093
log.dir=/tmp/kafka-logs-1
Also make sure that there are no multiple references of log.dir in the property file which might overwrite the previous value.
I am running a Kafka / Zk cluster in AWS MSK, and am connecting schema registry to it.
If I specify the broker list in kafkastore.bootstrap.servers, it connects fine (using protocol SSL). However, if I remove bootstrap server config and switch to a Zookeeper query for broker discovery using kafkastore.connection.url, then I am unable to connect to the cluster. It is because of what Zookeeper is returning from broker queries and how S.R. uses it:
$ ./bin/zkCli.sh -server z-1.cluster.address:2181
> ls /brokers/ids
[1, 2]
> get /brokers/ids/1
{
"listener_security_protocol_map": {
"CLIENT_SECURE": "SSL",
"REPLICATION": "PLAINTEXT",
"REPLICATION_SECURE": "SSL"
},
"endpoints": [
"CLIENT_SECURE://b-1.cluster.address:9094",
"REPLICATION://b-1-internal.cluster.address:9093",
"REPLICATION_SECURE://b-1-internal.cluster.address:9095"
],
....
}
I am guessing that by specifying SSL protocol, it resolves the map back and then connects to both CLIENT_SECURE and REPLICATION_SECURE endpoints. However, all -internal.cluster.address endpoints are inaccessible.
It is possible to configure S.R to only use CLIENT_SECURE endpoints and not try connect to REPLICATION_SECURE ones?
It is related to https://github.com/confluentinc/schema-registry/issues/1126
I would like to be able to programmatically (or use a a shell script) to list all the zookeeper bootstrap servers for each of my environments.
If i just know one of the bootstrap servers can I get a list of all of them?
All active brokers are registered under /brokers/ids/[brokerId], you can query ZNode information via any ZK client(for example org.I0Itec.zkclient). All you need is zkQuorum address.
zkClient = new ZkClient(properties.getProperty("zkQuorum"), zkSessionTimeout, zkConnectionTimeout,
ZKStringSerializer$.MODULE$);
Kafka data structures in Zookeeper
Not sure if this helps directly, but I used the following to find a bootstrap server from a Kafka zookeeper:
zkCli.sh -server host:port
[zkCli prompt] ls /kafka
[..., brokers, ...]
[zkCli prompt] ls /kafka/brokers
[1, 2, 3, 4, 5]
[zkCli prompt] ls /kafka/brokers/1
[1]
[zkCli prompt] get /kafka/brokers/1
>> broker info<<