The case is: to separate client and broker replication communication + introduce security.
Question is: is it possible to separate the communication with some procedure like rolling restart? Without need to have downtime on the whole cluster.
Configuration as is (simple with one port for everything wihout security):
listeners=PLAINTEXT://server1:9092
Wanted configuration (different ports and some with security, replication on 9094 port):
listener.security.protocol.map=PLAINTEXT:PLAINTEXT,SASLPLAIN:SASL_PLAINTEXT,REPLICATION:SASL_PLAINTEXT
listeners=PLAINTEXT://server1:9092,SASLPLAIN://server1,REPLICATION://server1:9094
inter.broker.listener.name=REPLICATION
sasl.mechanism.inter.broker.protocol=PLAIN
sasl.enabled.mechanisms=PLAIN
authorizer.class.name=kafka.security.auth.SimpleAclAuthorizer
Progress:
Configuration below is well working. But only way, without putting cluster into inconsistent state i know now, is to stop the cluster, introduce new configuration as shown above, and start cluster again. That´s obviously not wanted by the customer.
Gratefull for any thoughts how to proceed without need to stop/start whole cluster.
I managed to proceed from original, one listener configuration, to desired by below steps.
If someone has any idea to ease up the process, please add.
Original config:
listeners=PLAINTEXT://server1:9092
1.Change server.properties and do rolling restart
listener.security.protocol.map=PLAINTEXT:PLAINTEXT,SASLPLAIN:SASL_PLAINTEXT,REPLICATION:SASL_PLAINTEXT
listeners=PLAINTEXT://SERVER1:9092,SASL_PLAINTEXT://SERVER1:9093,REPLICATION://SERVER1:9094
sasl.enabled.mechanisms=PLAIN
Also include jaas config as jvm parameter.
-Djava.security.auth.login.config=/path/to/kafka_server_jaas.conf
2.Modify the server.properties and do rolling restart
listener.security.protocol.map=PLAINTEXT:PLAINTEXT,SASLPLAIN:SASL_PLAINTEXT,REPLICATION:SASL_PLAINTEXT
listeners=PLAINTEXT://SERVER1:9092,SASL_PLAINTEXT://SERVER1:9093,REPLICATION://SERVER1:9094
inter.broker.listener.name=REPLICATION
sasl.mechanism.inter.broker.protocol=PLAIN
sasl.enabled.mechanisms=PLAIN
3.Modify server properties one last time and do third rolling restart
listener.security.protocol.map=PLAINTEXT:PLAINTEXT,SASLPLAIN:SASL_PLAINTEXT,REPLICATION:SASL_PLAINTEXT
listeners=PLAINTEXT://SERVER1:9092,SASL_PLAINTEXT://SERVER1:9093,REPLICATION://SERVER1:9094
inter.broker.listener.name=REPLICATION
sasl.mechanism.inter.broker.protocol=PLAIN
sasl.enabled.mechanisms=PLAIN
authorizer.class.name=kafka.security.auth.SimpleAclAuthorizer
allow.everyone.if.no.acl.found=true
Related
I'm looking to set up a writing proxy for our MariaDB database on Kubernetes. The problem we are currently having is that we only have one Write master on our 3 master galera cluster setup. So even though we have ours pods replication properly, if our first node goes down then our other two masters end up failing because they are not able to be written to.
I saw this was a possible option to use either ProxySQL or MaxScale for Write proxying, but I'm not sure if I'm reading their uses properly. Do I have the right idea looking to deploy either of these two applications/services on Kubernetes to fix my problem? Would I be able to write to any of the Masters in the cluster?
MaxScale will handle selecting which server to write to as long as you use the readwritesplit router and the galeramon monitor.
Here's an example configuration for MaxScale that does load balancing of reads but sends writes to one node:
[maxscale]
threads=auto
[node1]
type=server
address=node1-address
port=3306
[node2]
type=server
address=node2-address
port=3306
[node3]
type=server
address=node3-address
port=3306
[Galera-Cluster]
type=monitor
module=galeramon
servers=node1,node2,node3
user=my-user
password=my-password
[RW-Split-Router]
type=service
router=readwritesplit
cluster=Galera-Cluster
user=my-user
password=my-password
[RW-Split-Listener]
type=listener
service=RW-Split-Router
protocol=mariadbclient
port=4006
The reason writes are only done on one node at a time is because doing it on multiple Galera nodes won't improve write performance and it results in conflicts when transactions are committed (applications seem to rarely handle these).
I have a Zookeeper ensemble I'd like to upgrade from 3.3.X to 3.4.X branch. Can this be done on a live cluster? Requires downtime? Are there any instructions to be found on this somewhere?
From what I checked ,
What are the options/process for upgrading ZooKeeper?
There are two primary ways of doing this; 1) full restart or 2) rolling restart.
In the full restart case you can stage your updated code/configuration/etc..., stop all of the servers in the ensemble, switch code/configuration, and restart the ZooKeeper ensemble. If you do this programmatically (scripts typically, ie not by hand) the restart can be done on order of seconds. As a result the clients will lose connectivity to the ZooKeeper cluster during this time, however it looks to the clients just like a network partition. All existing client sessions are maintained and re-established as soon as the ZooKeeper ensemble comes back up. Obviously one drawback to this approach is that if you encounter any issues (it's always a good idea to test/stage these changes on a test harness) the cluster may be down for longer than expected.
The second option, preferable for many users, is to do a "rolling restart". In this case you upgrade one server in the ZooKeeper ensemble at a time; bring down the server, upgrade the code/configuration/etc..., then restart the server. The server will automatically rejoin the quorum, update it's internal state with the current ZK leader, and begin serving client sessions. As a result of doing a rolling restart, rather than a full restart, the administrator can monitor the ensemble as the upgrade progresses, perhaps rolling back if any issues are encountered
I am thinking you need to download the file and extract to a local folder and transfer the configuration file/settings into the new version and have the old server down and run the new version server
https://wiki.apache.org/hadoop/ZooKeeper/FAQ
I wonder about the best strategy with regard to Zookeeper and SolrCloud clusters. Should one Zookeeper cluster be dedicated per SolrCloud cluster or multiple SolrCloud clusters can share one Zookeeper cluster? I guess the former must be a very safe approach but I am wondering if the 2nd option is fine as well.
As far as I know, SolrCloud use Zookeeper to share cluster state (up, down nodes) and to load core shared configurations (solrconfig.xml, schema.xml, etc...) on boot. If you have clients based on SolrJ's CloudSolrServer implementation than they will mostly perform reads of the cluster state.
In this respect, I think it should be fine to share the same ZK ensemble. Many reads and few writes, this is exactly what ZK is designed for.
SolrCloud puts very little load on a ZooKeeper cluster, so if it's purely a performance consideration then there's no problem. It would probably be a waste of resources to have one ZK cluster per SolrCloud if they're all on a local network. Just make sure the ZooKeeper configurations are in separate ZooKeeper paths. For example, using -zkHost :/ for one SolrCloud, and replace "path1" with "path2" for the second one will put the solr files in separate paths within ZooKeeper to ensure they don't conflict.
Note that the ZK cluster should be well-configured and robust, because if it goes down then none of the SolrClouds are going to be able to respond to changes in node availability or state. (If SolrCloud leader is lost, not connectable, or if a node enters recovering state, etc.)
I'm newbie to Zookeeper.Trying to setup clustering server for zookeeper exhibitor to modify the data. I have tried the server setup with 3 nodes, but data modification not reflected on all the zookeeper.
I refereed the following url & also setup the server in the same way. But no use, some thing i'm missing in that config to run it correctly.
http://zookeeper.apache.org/doc/trunk/zookeeperAdmin.html#sc_zkMulitServerSetup
Exhibitor startup command is:
java -jar exhibitor-war-1.0-jar-with-dependencies.jar -c file --nodemodification true --port 9090
Farther I need to add any other config with this to get my data modification reflect on all the zookeeper.
Advance thanks for ur kind time!
I have similar scenario, and I'm on Windows. So I get some problems due to the fact that Exhibitor is Unix-oriented - it tries to restart zkServer.sh (instead of zkServer.bat). So I:
1. have manually started ZK ensemble (all instances get data modifications from each other).
2. set up Exhibitor above every ZK instance - with single network config file.
Hope it helps. If not, give more details.
I'm interested in using Celery for an app I'm working on. It all seems pretty straight forward, but I'm a little confused about what I need to do if I have multiple load balanced application servers. All of the documentation assumes that the broker will be on the same server as the application. Currently, all of my application servers sit behind an Amazon ELB and tasks need to be able to come from any one of them.
This is what I assume I need to do:
Run a broker server on a separate instance
Configure each application instance to connect to that broker server
Each application instance will also be be a celery working (running
celeryd)?
My only beef with that is: What happens if my broker instance dies? Can I run 2 broker instances some how so I'm safe if one goes under?
Any tips or information on what to do in a setup like mine would be greatly appreciated. I'm sure I'm missing something or not understanding something.
For future reference, for those who do prefer to stick with RabbitMQ...
You can create a RabbitMQ cluster from 2 or more instances. Add those instances to your ELB and point your celeryd workers at the ELB. Just make sure you connect the right ports and you should be all set. Don't forget to allow your RabbitMQ machines to talk among themselves to run the cluster. This works very well for me in production.
One exception here: if you need to schedule tasks, you need a celerybeat process. For some reason, I wasn't able to connect the celerybeat to the ELB and had to connect it to one of the instances directly. I opened an issue about it and it is supposed to be resolved (didn't test it yet). Keep in mind that celerybeat by itself can only exist once, so that's already a single point of failure.
You are correct in all points.
How to make reliable broker: make clustered rabbitmq installation, as described here:
http://www.rabbitmq.com/clustering.html
Celery beat also doesn't have to be a single point of failure if you run it on every worker node with:
https://github.com/ybrs/single-beat