we have zookeeper cluster with 3 nodes
when we perform the following commands
echo stat | nc zookeeper_server01 2181 | grep Mode
echo stat | nc zookeeper_server02 2181 | grep Mode
echo stat | nc zookeeper_server03 2181 | grep Mode
we saw that zookeeper_server03 is the leader and other are the Mode: follower
but we noticed that every couple min the state is change and indeed after 4 min zookeeper_server01 became the leader and other are Mode: follower
again after 6 min zookeeper_server02 became a leader and so on
my Question is - dose this strange behavior is normal ?
I want to say that production Kafka cluster is using this zookeeper servers so , we are worry about that
Related
I want to use JMX metrics to collect stats on topic size from my 6 broker Kafka cluster.
I have created a test topic and sent 1000 test messages to it.
Now I am using the JMX to to look at the topic.
Broker 6 shows all the partitions and the 1000 messages, broker 4 only shows partition 2!!
$ kubectl exec -it kafka-broker4-0 -c kafka-broker -- bash -c "/kafka_2.13-2.8.1/bin/kafka-run-class.sh kafka.tools.JmxTool --jmx-url service:jmx:rmi:///jndi/rmi://localhost:9581/jmxrmi --one-time true --report-format tsv" | grep greg-test | grep Offset
kafka.cluster:type=Partition,name=LastStableOffsetLag,topic=greg-test,partition=2:Value 0
kafka.log:type=Log,name=LogEndOffset,topic=greg-test,partition=2:Value 341
kafka.log:type=Log,name=LogStartOffset,topic=greg-test,partition=2:Value 0
$ kubectl exec -it kafka-broker6-0 -c kafka-broker -- bash -c "/kafka_2.13-2.8.1/bin/kafka-run-class.sh kafka.tools.JmxTool --jmx-url service:jmx:rmi:///jndi/rmi://localhost:9581/jmxrmi --one-time true --report-format tsv" | grep greg-test | grep Offset
kafka.cluster:type=Partition,name=LastStableOffsetLag,topic=greg-test,partition=0:Value 0
kafka.cluster:type=Partition,name=LastStableOffsetLag,topic=greg-test,partition=1:Value 0
kafka.cluster:type=Partition,name=LastStableOffsetLag,topic=greg-test,partition=2:Value 0
kafka.log:type=Log,name=LogEndOffset,topic=greg-test,partition=0:Value 348
kafka.log:type=Log,name=LogEndOffset,topic=greg-test,partition=1:Value 311
kafka.log:type=Log,name=LogEndOffset,topic=greg-test,partition=2:Value 341
kafka.log:type=Log,name=LogStartOffset,topic=greg-test,partition=0:Value 0
kafka.log:type=Log,name=LogStartOffset,topic=greg-test,partition=1:Value 0
kafka.log:type=Log,name=LogStartOffset,topic=greg-test,partition=2:Value 0
I am using PostgreSQL 14 and ubuntu as my OS.
I have done the database replication in synchronous mode
Now I want to add another server and the relationship between primary and new_standby node will be asynchronous.
Another problem, how to set application_name for different nodes? I have got same application_name=14/main for my two standby servers
Can anyone help me to solve these issues?
Yes, you can run it simultaneously.
You should have this configuration in your primary node:
listen_addresses = '*'
port = 5432
wal_level = hot_standby
max_wal_senders = 16
wal_keep_segments = 32
synchronous_commit = on
synchronous_standby_names = 'pgsql_0_node_0'
Restart the node to take the changes:
$ systemctl restart postgresql-14
Create the replication role:
$ CREATE ROLE replication_user WITH LOGIN PASSWORD 'PASSWORD' REPLICATION;
And configure this in your standby nodes:
Both:
wal_level = hot_standby
hot_standby = on
Sync:
standby_mode = 'on'
promote_trigger_file='/tmp/failover_5432.trigger'
recovery_target_timeline=latest
primary_conninfo='application_name=pgsql_0_node_0 host=PRIMARY_NODE port=5432 user=replication_user password=PASSWORD'
Replace PRIMARY_NODE, user, and password with the correct values.
Async:
promote_trigger_file='/tmp/failover_5432.trigger'
recovery_target_timeline=latest
primary_conninfo='application_name=pgsql_0_node_1 host=PRIMARY_NODE port=5432 user=replication_user password=PASSWORD'
Replace PRIMARY_NODE, user, and password with the correct values.
Restart the node to take the changes:
$ systemctl restart postgresql-14
Then, you can run this in your Primary node to see the replication nodes:
$ SELECT pid,usename,application_name,state,sync_state FROM pg_stat_replication;
pid | usename | application_name | state | sync_state
-------+------------------+------------------+-----------+------------
10951 | replication_user | pgsql_0_node_1 | streaming | async
10952 | replication_user | pgsql_0_node_0 | streaming | sync
(2 rows)
Kafka was working just fine for me until about 30min ago. I'm using windows and the commands I use are:
c:\kafka>.\bin\windows\zookeeper-server-start.bat .\config\zookeeper.properties
to start up zookeeper and then
c:\kafka>.\bin\windows\kafka-server-start.bat .\config\server.properties
to start up the kafka server.
Right now zookeeper isn't working for some reason. That is when I run the command to start zookeeper, nothing happens, a line is skipped in my terminal and I just have:
C:\Kafka>.\bin\windows\zookeeper-server-start.bat .\config\zookeeper.properties
C:\Kafka>
Has this ever happened to anyone? Might someone know how to resolve this?
(1) kafka version which I am using in production kafka_2.11-2.1.1
(2) OS in which I deployed Kafka
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
Required steps to start your kafka Service :-
Starting zookeeper-server
Start your Zookeeper
[root#ie1 kafka_2.11-2.1.1]# ./bin/zookeeper-server-start.sh
config/zookeeper.properties
check port is running or not ( Default port used by zookeeper :2181 )
linux commnad to check port is LISTEN or NOT
[myhostname#ie1 kafka_2.11-2.1.1]$ netstat -telnup | grep '2181'
(Not all processes could be identified, non-owned process info
will not be shown, you would have to be root to see it all.)
tcp6 0 0 :::2181 :::* LISTEN 0 1205756192 -
if you are getting this it mean all are ok, now start follow further steps.
Starting kafka-server
Start your kafka-server
[root#ie1 kafka_2.11-2.1.1]# ./bin/kafka-server-start.sh
config/server.properties
check port is running or not ( Default port used by zookeeper :9094 )
linux commnad to check port is LISTEN or NOT
[myhostname#ie1 kafka_2.11-2.1.1]$ netstat -telnup | grep '9094'
(Not all processes could be identified, non-owned process info
will not be shown, you would have to be root to see it all.)
tcp6 0 0 :::9094 :::* LISTEN 0 1205756192 -
if you are getting this it mean all are ok, now you are ready to go.
Note:- Both services should be run in background otherwise as you will kill you CMD both services would be down.
for Information Please look into log file `
kafka_2.11-2.1.1/logs/
`
Happy Reading (' ')
Thanks
I am making Zookeeper cluster, and I separately start the Zookeeper in Confluent by using:
./bin/zookeeper-server-start etc/kafka/zookeeper.properties
and I want to get the status of Zookeeper.
I search it online, and all of them is using:
./zkServer.sh status
But I can't find zkServer.sh in Confluent.
I know that I can use ./bin/confluent status to get status. But I want more information about the Zookeeper like follow:
./zkServer.sh status
JMX enabled by default
Using config: /opt/../conf/zoo.cfg
Mode: follower
How can I do that?
You can use the Four Letter Words to get the same information or better instead. The output from stat:
$ echo "stat" | nc <ZOOKEEPER-IP-ADDRESS> 2181
Zookeeper version: 3.4.10
Clients:
/192.168.1.2:49618[1](queued=0,recved=1304,sent=1304)
/192.168.1.3:53484[0](queued=0,recved=1,sent=0)
Latency min/avg/max: 0/0/15
Received: 1330
Sent: 1329
Connections: 2
Outstanding: 0
Zxid: 0x1000001ee
Mode: leader
Node count: 435
The output from conf:
$ echo "conf" | nc <ZOOKEEPER-IP-ADDRESS> 2181
clientPort=2181
dataDir=/var/zookeeper/data
dataLogDir=/var/log/zookeeper
tickTime=2000
maxClientCnxns=0
minSessionTimeout=4000
maxSessionTimeout=40000
serverId=3
initLimit=20
syncLimit=5
electionAlg=3
electionPort=3888
quorumPort=2888
peerType=0
How do I verify that all the nodes in a zookeeper are part of a quorum and are healthy? Manual talks about "ruok" but that doesnt still say if the zookeeper node is part of quorum and is in sync with the rest.
You can use the srvr command documented in The Four Letter Words to get more detailed status information about each ZooKeeper server in the ensemble. See below for sample output from a 3-node cluster, with hosts named ubuntu1, ubuntu2 and ubuntu3.
The Mode field will tell you if that particular server is the leader or a follower. The Zxid field refers to the ZooKeeper cluster's internal transaction ID used for tracking state changes to the tree of znodes. In a healthy cluster, you'll see one leader, multiple followers, and all nodes will generally be close to one another in the zxid value.
> for x in ubuntu1 ubuntu2 ubuntu3; do echo $x; echo srvr|nc $x 2181; echo; done
ubuntu1
Zookeeper version: 3.4.7-1713338, built on 11/09/2015 04:32 GMT
Latency min/avg/max: 3/9/21
Received: 9
Sent: 8
Connections: 1
Outstanding: 0
Zxid: 0x100000004
Mode: follower
Node count: 6
ubuntu2
Zookeeper version: 3.4.7-1713338, built on 11/09/2015 04:32 GMT
Latency min/avg/max: 0/0/0
Received: 2
Sent: 1
Connections: 1
Outstanding: 0
Zxid: 0x100000004
Mode: leader
Node count: 6
ubuntu3
Zookeeper version: 3.4.7-1713338, built on 11/09/2015 04:32 GMT
Latency min/avg/max: 0/0/0
Received: 2
Sent: 1
Connections: 1
Outstanding: 0
Zxid: 0x100000004
Mode: follower
Node count: 6