I am currently working with SolrCloud and have set up Zookeeper on a server. To create a collection in SolrCloud, I upload a configset in Zookeeper and use it to create a collection. On the zookeeper server, I use Zookeeper's CLI to view the config files as follows: -
zk#ip-some-ip-some-ip:/opt/zookeeper/bin$ ./zkCli.sh -server 127.0.0.1:9983
/usr/bin/java
Connecting to 127.0.0.1:9983
[zk: 127.0.0.1:9983(CONNECTED) 0]
[zk: 127.0.0.1:9983(CONNECTED) 0] ls /configs
[_default, config_1, config_2]
[zk: 127.0.0.1:9983(CONNECTED) 1]
I tried searching for /configs directory within all zookeeper directories but couldn't find it. Where is it actually present on the system?
I tried searching for the configs directory as follows: -
find /var/zookeeper/ -name "configs"
find /opt/zookeeper/ -name "configs"
but didn't find anything.
I don't have any zookeeper directories anywhere else on the system.
Please let me know if any other information is needed from my end. Thank you for reading.
Zookeeper keeps its data in-memory. From the documentation,
Unlike a typical file system, which is designed for storage, ZooKeeper data is kept in-memory, which means ZooKeeper can acheive high throughput and low latency numbers.
Transaction logs are stored in Zookeeper's data directory
Related
I built a zookeeper cluster and it runs very well. But I found that the log directory I set in the zoo.cfg seems not working. Below is my config about log directory and snapshots directory.
dataDir=/var/lib/zookeeper
dataLogDir=/var/lib/zookeeper/logs
However, file zookeeper.out is generated in /var/lib/zookeeper rather than the subsidiary log folder /var/lib/zookeeper/logs.
I restarted zookeeper on every server many times, but made no sense.
This happens because zookeeper.out is related to other type of log (application log) instead of the one specified by dataLogDir which relates to transaction log.
dataLogDir
This option will direct the machine to write the transaction log to
the dataLogDir rather than the dataDir. This allows a dedicated log
device to be used, and helps avoid competition between logging and
snaphots.
By checking zkServer.sh you'll see that zookeeper.out is related to _ZOO_DAEMON_OUT which depends on ZOO_LOG_DIR which is set by default by zkEnv.sh. Depending on your environment and zookeeper (ZK) version the zookeeper.out file might land in different places (according to this answer even in the working directory from which ZK is started).
For application logging you'll better configure the log4j.properties file; that's because ZK uses log4j.
I want to know if there is a way to walk the Zookeeper's (ZK) in memory database and find if any particular node exists. Something similar to find . -name file inside ZK
I am logged in to ZK using zkCli.
Once you have your ZK cluster running, you can connect to a node and query the cluster.
For example:
$ZK_HOME/bin/zkCli.sh -server localhost
List of nodes:
ls /
List of commands:
?
I uploaded a configuration folder for Solr core to Apache zookeeper using zkClient.
When I delete a file in my local configuration and update it to Zookeeper again, I can't see the change reflected in Solr admin page.
Could somebody please explain how to update/delete files from zookeeper?
Also where to find the physical files in zookeeper folder?
In order to upload a modified file in zookeeper client, you need to:
remove the old file from Zookeeper and
upload the new one and
restart the Solr nodes (depending on the change, you could reload the collection instead).
For instance if you need to update solrconfig.xml, you can:
a) clear old file on zookeeper (otherwise depending from the client version you'll get an error):
zkcli.sh --zkhost <ZK_HOST>:<ZK_PORT> -cmd clear /configs/<MY_COLLECTION>/solrconfig.xml
b) upload the updated file:
zkcli.sh --zkhost <ZK_HOST>:<ZK_PORT> -cmd putfile /configs/<MY_COLLECTION>/solrconfig.xml /<MY_UPDATED_FILE_LOCAL_FOLDER>/solrconfig.xml
c) Restart the Solr nodes.
I believe your Solr files should be in /configs/<MY_COLLECTION>.
I am running 3 node zookeeper cluster to process storm and kafka.Zookeeper Data directory eats up all the space in my system.I am not sure how to clean it up.As, I don't want to delete the data entirely because i will lose the state of the processes.I looked into autopurge.purgeInterval in zoo.cfg, but it doesn't work as I expected.
I am using zookeeper 3.4.6
How can I delete the old data without affecting the new ones?
Assuming you have zookeeper installed in /opt/zookeeper-3.4.6, the following will delete all but the last 10 snapshots.
stop zookeeper (/opt/zookeeper-3.4.6/bin/zkServer.sh stop)
cd /opt/zookeeper-3.4.6
./zkCleanup.sh ../data/ -n 10
Probably wise to make a backup of the whole /opt/zookeeper-3.4.6 dir before doing this.
Stop Zookeeper
Go to the bin folder of your Zookeeper
Run ./zkCli.sh
Use ls / to check Zookeeper's content
Identify what you want to delete with the exact path
Delete /znode or path of what you want to delete
I'm newbie to Zookeeper.Trying to setup clustering server for zookeeper exhibitor to modify the data. I have tried the server setup with 3 nodes, but data modification not reflected on all the zookeeper.
I refereed the following url & also setup the server in the same way. But no use, some thing i'm missing in that config to run it correctly.
http://zookeeper.apache.org/doc/trunk/zookeeperAdmin.html#sc_zkMulitServerSetup
Exhibitor startup command is:
java -jar exhibitor-war-1.0-jar-with-dependencies.jar -c file --nodemodification true --port 9090
Farther I need to add any other config with this to get my data modification reflect on all the zookeeper.
Advance thanks for ur kind time!
I have similar scenario, and I'm on Windows. So I get some problems due to the fact that Exhibitor is Unix-oriented - it tries to restart zkServer.sh (instead of zkServer.bat). So I:
1. have manually started ZK ensemble (all instances get data modifications from each other).
2. set up Exhibitor above every ZK instance - with single network config file.
Hope it helps. If not, give more details.