I uploaded a configuration folder for Solr core to Apache zookeeper using zkClient.
When I delete a file in my local configuration and update it to Zookeeper again, I can't see the change reflected in Solr admin page.
Could somebody please explain how to update/delete files from zookeeper?
Also where to find the physical files in zookeeper folder?
In order to upload a modified file in zookeeper client, you need to:
remove the old file from Zookeeper and
upload the new one and
restart the Solr nodes (depending on the change, you could reload the collection instead).
For instance if you need to update solrconfig.xml, you can:
a) clear old file on zookeeper (otherwise depending from the client version you'll get an error):
zkcli.sh --zkhost <ZK_HOST>:<ZK_PORT> -cmd clear /configs/<MY_COLLECTION>/solrconfig.xml
b) upload the updated file:
zkcli.sh --zkhost <ZK_HOST>:<ZK_PORT> -cmd putfile /configs/<MY_COLLECTION>/solrconfig.xml /<MY_UPDATED_FILE_LOCAL_FOLDER>/solrconfig.xml
c) Restart the Solr nodes.
I believe your Solr files should be in /configs/<MY_COLLECTION>.
Related
Whenever I'm starting my zookeeper server, I'm getting the below error.
[2020-05-12 08:11:43,510] ERROR Unexpected exception, exiting abnormally (org.apache.zookeeper.server.quorum.QuorumPeerMain)
org.apache.zookeeper.server.persistence.FileTxnSnapLog$SnapDirContentCheckException: Snapshot directory has log files. Check if dataLogDir and dataDir configuration is correct.
at org.apache.zookeeper.server.persistence.FileTxnSnapLog.checkSnapDir(FileTxnSnapLog.java:140)
at org.apache.zookeeper.server.persistence.FileTxnSnapLog.<init>(FileTxnSnapLog.java:109)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:141)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:114)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:81)
Here is what I have in my zookeeper.properties
dataDir=/var/appdata/zookeeper
dataLogDir=/var/applogs/zookeeper
The default setting in zookeeper is that the snapshot and log files are stored in the same directory(the dataDir dir). Now that you have started your cluster the system would have created both the logs and snapshot files in the same directory.
Later on when you decide to have separate folders(maybe to increase efficiency) for data(snapshot) and logs(log) files you would simply added the two properties (dataDir,dataLogDir) and restart your cluster . But this will not work and the server fails during startup saying "Snapshot directory has log files". This is right because your setting clearly demands unique folders for the two(but both the files are present in the same folder already in your case. So u have to migrate now).
Soln:
cut all the log files from the dataDir and paste inside the corresponding version folder(eg version-2) of the dataLogDir. This would help you resume your zookeeper from the previous state(snapshot).
I have a novice question on kafka upgrade.. This is the 1st time I'm upgrading my kafka in Linux.
My current version is "kafka_2.11-1.0.0.tgz".. when I initially setup I had a folder named kafka_2.11-1.0.0.
Now I downloaded a new version "kafka_2.12-2.3.0.tgz". If I extract it is going to create a new folder kafka_2.12-2.3.0 which will result in 2 independent kafka with server.properties.
As per documentation I have to update server.properties with below 2 properties..
inter.broker.protocol.version=2.3
log.message.format.version=2.3
How does this affect if it is going to install in a new directory with new server.properties?
How can I merge server.properties & do the upgrade? Please share if you have documents or steps..
it's fairly simple to upgrade Kafka.
It would have been easier for you to separate config files from binary directories, as a result, from what I understand, your config file remains with the untar package folder.
You can put the config file in /etc/kafka next time you'll package it on your Linux server.
What you can do here is , after untar your kafka_2.12-2.3.0.tgz file, you just copy the former server.properties ( and other config files you might use as well) and replace the one in the 2.3.0 arborescence.
But be careful, for inter.broker.protocol.version=2.3 and log.message.format.version=2.3 parameters, you must first specify the former version for those parameters ( and message.format is not mandatory to change, double check the doc for this one) before doing your rolling restart.
If you are using 1.0 now, just put the following :
inter.broker.protocol.version=1.0 and log.message.format.version=1.0
then restart your brokers one by one (using the new package folder this time)
Then edit them again as follows :
inter.broker.protocol.version=2.3 and log.message.format.version=2.3 and do a second rolling restart.
Then you should be good
More details here :
https://kafka.apache.org/documentation/#upgrade_2_3_0
Yannick
I built a zookeeper cluster and it runs very well. But I found that the log directory I set in the zoo.cfg seems not working. Below is my config about log directory and snapshots directory.
dataDir=/var/lib/zookeeper
dataLogDir=/var/lib/zookeeper/logs
However, file zookeeper.out is generated in /var/lib/zookeeper rather than the subsidiary log folder /var/lib/zookeeper/logs.
I restarted zookeeper on every server many times, but made no sense.
This happens because zookeeper.out is related to other type of log (application log) instead of the one specified by dataLogDir which relates to transaction log.
dataLogDir
This option will direct the machine to write the transaction log to
the dataLogDir rather than the dataDir. This allows a dedicated log
device to be used, and helps avoid competition between logging and
snaphots.
By checking zkServer.sh you'll see that zookeeper.out is related to _ZOO_DAEMON_OUT which depends on ZOO_LOG_DIR which is set by default by zkEnv.sh. Depending on your environment and zookeeper (ZK) version the zookeeper.out file might land in different places (according to this answer even in the working directory from which ZK is started).
For application logging you'll better configure the log4j.properties file; that's because ZK uses log4j.
I am running 3 node zookeeper cluster to process storm and kafka.Zookeeper Data directory eats up all the space in my system.I am not sure how to clean it up.As, I don't want to delete the data entirely because i will lose the state of the processes.I looked into autopurge.purgeInterval in zoo.cfg, but it doesn't work as I expected.
I am using zookeeper 3.4.6
How can I delete the old data without affecting the new ones?
Assuming you have zookeeper installed in /opt/zookeeper-3.4.6, the following will delete all but the last 10 snapshots.
stop zookeeper (/opt/zookeeper-3.4.6/bin/zkServer.sh stop)
cd /opt/zookeeper-3.4.6
./zkCleanup.sh ../data/ -n 10
Probably wise to make a backup of the whole /opt/zookeeper-3.4.6 dir before doing this.
Stop Zookeeper
Go to the bin folder of your Zookeeper
Run ./zkCli.sh
Use ls / to check Zookeeper's content
Identify what you want to delete with the exact path
Delete /znode or path of what you want to delete
I'm newbie to Zookeeper.Trying to setup clustering server for zookeeper exhibitor to modify the data. I have tried the server setup with 3 nodes, but data modification not reflected on all the zookeeper.
I refereed the following url & also setup the server in the same way. But no use, some thing i'm missing in that config to run it correctly.
http://zookeeper.apache.org/doc/trunk/zookeeperAdmin.html#sc_zkMulitServerSetup
Exhibitor startup command is:
java -jar exhibitor-war-1.0-jar-with-dependencies.jar -c file --nodemodification true --port 9090
Farther I need to add any other config with this to get my data modification reflect on all the zookeeper.
Advance thanks for ur kind time!
I have similar scenario, and I'm on Windows. So I get some problems due to the fact that Exhibitor is Unix-oriented - it tries to restart zkServer.sh (instead of zkServer.bat). So I:
1. have manually started ZK ensemble (all instances get data modifications from each other).
2. set up Exhibitor above every ZK instance - with single network config file.
Hope it helps. If not, give more details.