Hibernate Search +infinispan +jgroups back end slave lock issue - hibernate-search

Question

I am new to hibernate search. We decided to user hibernate search for my application. We choose jgroups as a backend. Here is my configuration file.
<?xml version="1.0" encoding="UTF-8"?>
<infinispan xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="urn:infinispan:config:7.0
http://www.infinispan.org/schemas/infinispan-config-7.0.xsd
urn:infinispan:config:store:jdbc:7.0 http://www.infinispan.org/schemas/infinispan-cachestore-jdbc-config-7.0.xsd"
xmlns="urn:infinispan:config:7.0"
xmlns:jdbc="urn:infinispan:config:store:jdbc:7.0">
<!-- *************************** -->
<!-- System-wide global settings -->
<!-- *************************** -->
<jgroups>
<!-- Note that the JGroups transport uses sensible defaults if no configuration
property is defined. See the JGroupsTransport javadocs for more flags.
jgroups-udp.xml is the default stack bundled in the Infinispan core jar: integration
and tuning are tested by Infinispan. -->
<stack-file name="default-jgroups-tcp" path="proform-jgroups.xml" />
</jgroups>
<cache-container name="HibernateSearch" default-cache="default" statistics="false" shutdown-hook="DONT_REGISTER">
<transport stack="default-jgroups-tcp" cluster="venkatcluster"/>
<!-- Duplicate domains are allowed so that multiple deployments with default configuration
of Hibernate Search applications work - if possible it would be better to use JNDI to share
the CacheManager across applications -->
<jmx duplicate-domains="true" />
<!-- *************************************** -->
<!-- Cache to store Lucene's file metadata -->
<!-- *************************************** -->
<replicated-cache name="LuceneIndexesMetadata" mode="SYNC" remote-timeout="25000">
<transaction mode="NONE"/>
<state-transfer enabled="true" timeout="480000" await-initial-transfer="true" />
<indexing index="NONE" />
<eviction max-entries="-1" strategy="NONE"/>
<expiration max-idle="-1"/>
<persistence passivation="false">
<jdbc:string-keyed-jdbc-store preload="true" fetch-state="true" read-only="false" purge="false">
<property name="key2StringMapper">org.infinispan.lucene.LuceneKey2StringMapper</property>
<jdbc:connection-pool connection-url="jdbc:mysql://localhost:3306/entityindex" driver="com.mysql.jdbc.Driver" password="pf_user1!" username="pf_user"></jdbc:connection-pool>
<jdbc:string-keyed-table drop-on-exit="false" create-on-start="true" prefix="ISPN_STRING_TABLE">
<jdbc:id-column name="ID" type="VARCHAR(255)"/>
<jdbc:data-column name="DATA" type="BLOB"/>
<jdbc:timestamp-column name="TIMESTAMP" type="BIGINT"/>
</jdbc:string-keyed-table>
</jdbc:string-keyed-jdbc-store>
</persistence>
</replicated-cache>
<!-- **************************** -->
<!-- Cache to store Lucene data -->
<!-- **************************** -->
<distributed-cache name="LuceneIndexesData" mode="SYNC" remote-timeout="25000">
<transaction mode="NONE"/>
<state-transfer enabled="true" timeout="480000" await-initial-transfer="true" />
<indexing index="NONE" />
<eviction max-entries="-1" strategy="NONE"/>
<expiration max-idle="-1"/>
<persistence passivation="false">
<jdbc:string-keyed-jdbc-store preload="true" fetch-state="true" read-only="false" purge="false">
<property name="key2StringMapper">org.infinispan.lucene.LuceneKey2StringMapper</property>
<jdbc:connection-pool connection-url="jdbc:mysql://localhost:3306/entityindex" driver="com.mysql.jdbc.Driver" password="pf_user1!" username="pf_user"></jdbc:connection-pool>
<jdbc:string-keyed-table drop-on-exit="false" create-on-start="true" prefix="ISPN_STRING_TABLE">
<jdbc:id-column name="ID" type="VARCHAR(255)"/>
<jdbc:data-column name="DATA" type="BLOB"/>
<jdbc:timestamp-column name="TIMESTAMP" type="BIGINT"/>
</jdbc:string-keyed-table>
</jdbc:string-keyed-jdbc-store>
</persistence>
</distributed-cache>
<!-- ***************************** -->
<!-- Cache to store Lucene locks -->
<!-- ***************************** -->
<replicated-cache name="LuceneIndexesLocking" mode="SYNC" remote-timeout="25000">
<transaction mode="NONE"/>
<state-transfer enabled="true" timeout="480000" await-initial-transfer="true" />
<indexing index="NONE" />
<eviction max-entries="-1" strategy="NONE"/>
<expiration max-idle="-1"/>
<persistence passivation="false">
<jdbc:string-keyed-jdbc-store preload="true" fetch-state="true" read-only="false" purge="false">
<property name="key2StringMapper">org.infinispan.lucene.LuceneKey2StringMapper</property>
<jdbc:connection-pool connection-url="jdbc:mysql://localhost:3306/entityindex" driver="com.mysql.jdbc.Driver" password="pf_user1!" username="pf_user"></jdbc:connection-pool>
<jdbc:string-keyed-table drop-on-exit="false" create-on-start="true" prefix="ISPN_STRING_TABLE">
<jdbc:id-column name="ID" type="VARCHAR(255)"/>
<jdbc:data-column name="DATA" type="BLOB"/>
<jdbc:timestamp-column name="TIMESTAMP" type="BIGINT"/>
</jdbc:string-keyed-table>
</jdbc:string-keyed-jdbc-store>
</persistence>
</replicated-cache>
</cache-container>
This is my jgroups-file:
<config xmlns="urn:org:jgroups"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="urn:org:jgroups
http://www.jgroups.org/schema/JGroups-3.6.xsd">
<TCP bind_addr="${jgroups.tcp.address:127.0.0.1}"
bind_port="${jgroups.tcp.port:7801}"
enable_diagnostics="false"
thread_naming_pattern="pl"
send_buf_size="640k"
sock_conn_timeout="300"
thread_pool.min_threads="${jgroups.thread_pool.min_threads:2}"
thread_pool.max_threads="${jgroups.thread_pool.max_threads:30}"
thread_pool.keep_alive_time="60000"
thread_pool.queue_enabled="false"
internal_thread_pool.min_threads=
"${jgroups.internal_thread_pool.min_threads:5}"
internal_thread_pool.max_threads=
"${jgroups.internal_thread_pool.max_threads:20}"
internal_thread_pool.keep_alive_time="60000"
internal_thread_pool.queue_enabled="true"
internal_thread_pool.queue_max_size="500"
oob_thread_pool.min_threads="${jgroups.oob_thread_pool.min_threads:20}"
oob_thread_pool.max_threads="${jgroups.oob_thread_pool.max_threads:200}"
oob_thread_pool.keep_alive_time="60000"
oob_thread_pool.queue_enabled="false"
/>
<S3_PING access_key=""
secret_access_key=""
location="mybucket"
/>
<MERGE3 min_interval="10000"
max_interval="30000"
/>
<FD_SOCK />
<FD_ALL timeout="60000"
interval="15000"
timeout_check_interval="5000"
/>
<VERIFY_SUSPECT timeout="5000" />
<pbcast.NAKACK2 use_mcast_xmit="false"
xmit_interval="1000"
xmit_table_num_rows="50"
xmit_table_msgs_per_row="1024"
xmit_table_max_compaction_time="30000"
max_msg_batch_size="100"
resend_last_seqno="true"
/>
<UNICAST3 xmit_interval="500"
xmit_table_num_rows="50"
xmit_table_msgs_per_row="1024"
xmit_table_max_compaction_time="30000"
max_msg_batch_size="100"
conn_expiry_timeout="0"
/>
<pbcast.STABLE stability_delay="500"
desired_avg_gossip="5000"
max_bytes="1M"
/>
<pbcast.GMS print_local_addr="false"
join_timeout="15000"
/>
<MFC max_credits="2m"
min_threshold="0.40"
/>
<FRAG2 />
</config>
This is my flush-tcp file:-
<config xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns="urn:org:jgroups"
xsi:schemaLocation="urn:org:jgroups
http://www.jgroups.org/schema/jgroups.xsd">
<TCP bind_port="7801"/>
<S3_PING access_key=""
secret_access_key=""
location=""
/>
<MERGE3/>
<FD_SOCK/>
<FD/>
<VERIFY_SUSPECT/>
<pbcast.NAKACK2 use_mcast_xmit="false"/>
<UNICAST3/>
<pbcast.STABLE/>
<pbcast.GMS/>
<MFC/>
<FRAG2/>
<pbcast.STATE_TRANSFER/>
<pbcast.FLUSH timeout="0"/>
</config>
These are hibernate settings:
propertyMap.put("hibernate.search.default.directory_provider",
"infinispan");
propertyMap.put("hibernate.search.lucene_version",
KeywordUtil.LUCENE_4_10_4);
propertyMap.put("hibernate.search.infinispan.configuration_resourcename",
"hibernate-search-infinispan-config.xml");
propertyMap.put("hibernate.search.default.​worker.execution","sync");
propertyMap.put("hibernate.search.default.​worker.backend","jgroups");
propertyMap.put("hibernate.search.services.jgroups.configurationFile",
"flush-tcp.xml");
propertyMap.put("hibernate.search.default.exclusive_index_use","true");
Initially we start the cluster with one node with the above configuration. Depends on the load we will add nodes to the cluster. This is our architecture.
Assume that 10-00 AM we started the cluster. only node will become master node. and everthing is fine.
10-10 Am we added one more node to the cluster with slight config change. Here is the change
propertyMap.put("hibernate.search.default.exclusive_index_use","false");
I created an index through second node. Then the locking error comes up. here is the error.
org.apache.lucene.store.LockObtainFailedException: Lock obtain timed out:
org.infinispan.lucene.locking.BaseLuceneLock#46578a74
Problem:- In theory second node should become slave and it should never acquire lock on the index. It should indicate the master node to create the index through jgroups channel. But its not happening. Can one of you please help me on this. Our production system is in problem. Please help me on this.

Answer 1

Problem:- In theory second node should become slave and it should
never acquire lock on the index. It should indicate the master node to
create the index through jgroups channel.
There may be two problems here.
1. Using different values for exclusive_index_use
Maybe someone else can confirm, but unless your new node only deals with a completely separate persistent unit with completely different indexes, I doubt it's a good idea to use a different value for exclusive_index_use on different nodes.
exclusive_index_use is not about not acquiring locks, it's more about releasing them as soon as possible (after each changeset). If your other nodes work in exclusive node, they will never release locks, and your new node will time out waiting for the lock.
Also note that disabling exclusive_index_use is a sure way to decrease write performance, because it requires to constantly close and open index writers. Use with caution.
And finally, as you pointed out, only one node should write to the index at any given time (the JGroups master), so disabling exclusive_index_use should not be needed in your case. There must be another problem...
2. Master/slave election
If I remember correctly, the default master/node election strategy elect a new master when you add a new node. Also, we fixed a few bugs related to dynamic master election in the latest Hibernate Search version (not released yet), so you may be affected by one of those.
You can try using the jgroupsMaster backend on your first node and jgroupsSlave on your second node. There won't be any automatic master election anymore, so you'll lose the ability to maintain service when the master node fails, but from what I understand your main concern is scaling up, so it might give you a temporary solution.
On the master node:
propertyMap.put("hibernate.search.default.​worker.backend","jgroupsMaster");
On the slave node:
propertyMap.put("hibernate.search.default.​worker.backend","jgroupsSlave");
WARNING: you will need a full restart! Keeping the current jgroups backend on your master while adding another node with the jgroupsSlave backend will result in trouble!
You may also need some configuration changes regarding your Infinispan directory, but I'm not familiar with this directory. You can check the documentation: https://docs.jboss.org/hibernate/stable/search/reference/en-US/html_single/#jgroups-backend