Is the second component of the list of zookeepers just a sequence numbering or does it have to correspond to the myId of the instance? For instance if I set up a new node with myId=4 and take deprovision the instance with myId=3 of the existing nodes, would my config have to look like the following:
tickTime=2000
dataDir=/usr/src/zookeeper
clientPort=2181
initLimit=10
syncLimit=5
server.1=192.168.1.2:2888:3888
server.2=192.168.1.3:2888:3888
server.4=192.168.1.5:2888:3888
It corresponds to the myId of the instance.
Every machine that is part of the ZooKeeper ensemble should know about
every other machine in the ensemble. You accomplish this with the
series of lines of the form server.id=host:port:port. The parameters
host and port are straightforward. You attribute the server id to each
machine by creating a file named myid, one for each server, which
resides in that server's data directory, specified by the
configuration file parameter dataDir.
Source: http://zookeeper.apache.org/doc/trunk/zookeeperAdmin.html#sc_zkMulitServerSetup
Related
Not able to connect to mongo standalone node experiencing below error.
ERROR [cluster-ClusterId{value='asdfs', description='null'}-x.x.x.x:27017] org.mongodb.driver.cluster - Expecting replica set member, but found a STANDALONE. Removing x.x.x.x:27017 from client view of cluster.
Is it okay to give multiple Ip's in config file while only one mongo node is there?
Is it okay to give multiple Ip's in config file while only one mongo node is there?
Not for standalone connections, no.
Specify the address of your standalone only.
The case is: to separate client and broker replication communication + introduce security.
Question is: is it possible to separate the communication with some procedure like rolling restart? Without need to have downtime on the whole cluster.
Configuration as is (simple with one port for everything wihout security):
listeners=PLAINTEXT://server1:9092
Wanted configuration (different ports and some with security, replication on 9094 port):
listener.security.protocol.map=PLAINTEXT:PLAINTEXT,SASLPLAIN:SASL_PLAINTEXT,REPLICATION:SASL_PLAINTEXT
listeners=PLAINTEXT://server1:9092,SASLPLAIN://server1,REPLICATION://server1:9094
inter.broker.listener.name=REPLICATION
sasl.mechanism.inter.broker.protocol=PLAIN
sasl.enabled.mechanisms=PLAIN
authorizer.class.name=kafka.security.auth.SimpleAclAuthorizer
Progress:
Configuration below is well working. But only way, without putting cluster into inconsistent state i know now, is to stop the cluster, introduce new configuration as shown above, and start cluster again. That´s obviously not wanted by the customer.
Gratefull for any thoughts how to proceed without need to stop/start whole cluster.
I managed to proceed from original, one listener configuration, to desired by below steps.
If someone has any idea to ease up the process, please add.
Original config:
listeners=PLAINTEXT://server1:9092
1.Change server.properties and do rolling restart
listener.security.protocol.map=PLAINTEXT:PLAINTEXT,SASLPLAIN:SASL_PLAINTEXT,REPLICATION:SASL_PLAINTEXT
listeners=PLAINTEXT://SERVER1:9092,SASL_PLAINTEXT://SERVER1:9093,REPLICATION://SERVER1:9094
sasl.enabled.mechanisms=PLAIN
Also include jaas config as jvm parameter.
-Djava.security.auth.login.config=/path/to/kafka_server_jaas.conf
2.Modify the server.properties and do rolling restart
listener.security.protocol.map=PLAINTEXT:PLAINTEXT,SASLPLAIN:SASL_PLAINTEXT,REPLICATION:SASL_PLAINTEXT
listeners=PLAINTEXT://SERVER1:9092,SASL_PLAINTEXT://SERVER1:9093,REPLICATION://SERVER1:9094
inter.broker.listener.name=REPLICATION
sasl.mechanism.inter.broker.protocol=PLAIN
sasl.enabled.mechanisms=PLAIN
3.Modify server properties one last time and do third rolling restart
listener.security.protocol.map=PLAINTEXT:PLAINTEXT,SASLPLAIN:SASL_PLAINTEXT,REPLICATION:SASL_PLAINTEXT
listeners=PLAINTEXT://SERVER1:9092,SASL_PLAINTEXT://SERVER1:9093,REPLICATION://SERVER1:9094
inter.broker.listener.name=REPLICATION
sasl.mechanism.inter.broker.protocol=PLAIN
sasl.enabled.mechanisms=PLAIN
authorizer.class.name=kafka.security.auth.SimpleAclAuthorizer
allow.everyone.if.no.acl.found=true
According to https://zookeeper.apache.org/doc/r3.1.2/zookeeperAdmin.html#sc_zkMulitServerSetup
Cross Machine Requirements For the ZooKeeper service to be active,
there must be a majority of non-failing machines that can communicate
with each other. To create a deployment that can tolerate the failure
of F machines, you should count on deploying 2xF+1 machines. Thus, a
deployment that consists of three machines can handle one failure, and
a deployment of five machines can handle two failures. Note that a
deployment of six machines can only handle two failures since three
machines is not a majority. For this reason, ZooKeeper deployments are
usually made up of an odd number of machines.
To achieve the highest probability of tolerating a failure you should
try to make machine failures independent. For example, if most of the
machines share the same switch, failure of that switch could cause a
correlated failure and bring down the service. The same holds true of
shared power circuits, cooling systems, etc.
My question is:
What should we do after we identified a node failure within Zookeeper cluster to make the cluster 2F+1 again? Do we need to restart all the zookeeper nodes? Also the clients connects to Zookeeper cluster, suppose we used DNS name and the recovered node using same DNS name.
For example:
10.51.22.89 zookeeper1
10.51.22.126 zookeeper2
10.51.23.216 zookeeper3
if 10.51.22.89 dies and we bring up 10.51.22.90 as zookeeper1, and all the nodes can identify this change.
If you connect 10.51.22.90 as zookeeper1 (with the same myid file and configuration as 10.51.22.89 had before) and the data dir is empty, the process will connect to current leader (zookeeper2 or zookeeper3) and copy snapshot of the data. After successful initialization the node will inform rest of the cluster nodes and you have 2F+1 again.
Try this yourself, having tail -f on log files. It won't hurt the cluster and you will learn a lot on zookeeper internals ;-)
I am getting different things from different sources. At first I thought that I could just have one configuration file (on the monitor), with sections for each node (include osd nodes). But when on the new OSD, ceph osd create fails, saying there is no configuration file.
So, how does the configuration structure of ceph work? Further, is the FSID (a UUID) in each configuration file the same?
Yes, every machine planned to deploy osd/mon/mds should have a /etc/ceph/ceph.conf file.
When creating osd instance, it needs to communicate with mon from configure file. The fsid should be same, as osd/mds/mon will compare the fsid when handling internal messages; if they are not the same, the message will be dropped.
I have a zookeeper ensemble with 3 server with ensemble cfg as
ensemble configuration
server.1=zoo01:2888:3888
server.2=zoo02:2888:3888
server.3=zoo03:2888:3888
Note zoo01 is a entry in etc/hosts file.
I have to move zoo01 to a different physical box, but I want to retain it's identity (myid)
Is this possible ? If so, how can I achieve this. I can take a very short down time if required