Web app won't join Infinispan cluster - jboss

Question

I've been playing around with Infinispan lately, having had no previous experience with Infinispan.
I've come across an interesting problem and I wondered if anyone might be able to shed some light on it.
I have a standalone Java application (GridGrabber.jar) that bundles the Infinispan jars and has a class to add/remove and list items from the grid. Within the application I create the CacheManager as follows:
DefaultCacheManager m = new DefaultCacheManager("cluster.xml");
The application runs on the command line with java -jar GridGrabber.jar and works fine, and when I start two instances at the terminal, they both cluster together and I can see the contents of the grid from both nodes:
Sam#macbook $ java -jar GridGrabber.jar -Djava.net.preferIPv4Stack=true -Djgroups.bind_addr=127.0.0.1
Apr 16, 2012 9:50:04 PM org.infinispan.remoting.transport.jgroups.JGroupsTransport start
INFO: ISPN000078: Starting JGroups Channel
Apr 16, 2012 9:50:04 PM org.infinispan.remoting.transport.jgroups.JGroupsTransport buildChannel
INFO: ISPN000088: Unable to use any JGroups configuration mechanisms provided in properties {}. Using default JGroups configuration!
Apr 16, 2012 9:50:05 PM org.jgroups.logging.JDKLogImpl warn
WARNING: receive buffer of socket java.net.DatagramSocket#bc794cc was set to 20MB, but the OS only allocated 65.51KB. This might lead to performance problems. Please set your max receive buffer in the OS correctly (e.g. net.core.rmem_max on Linux)
Apr 16, 2012 9:50:05 PM org.jgroups.logging.JDKLogImpl warn
WARNING: receive buffer of socket java.net.MulticastSocket#731de9b was set to 25MB, but the OS only allocated 65.51KB. This might lead to performance problems. Please set your max receive buffer in the OS correctly (e.g. net.core.rmem_max on Linux)
Apr 16, 2012 9:50:06 PM org.infinispan.remoting.transport.jgroups.JGroupsTransport viewAccepted
INFO: ISPN000094: Received new cluster view: [Samuel-Weavers-MacBook-Pro-62704|1] [Samuel-Weavers-MacBook-Pro-62704, Samuel-Weavers-MacBook-Pro-3360]
Apr 16, 2012 9:50:06 PM org.infinispan.remoting.transport.jgroups.JGroupsTransport startJGroupsChannelIfNeeded
INFO: ISPN000079: Cache local address is Samuel-Weavers-MacBook-Pro-3360, physical addresses are [fe80:0:0:0:e2f8:47ff:fe06:109a%5:51422]
Apr 16, 2012 9:50:06 PM org.jgroups.logging.JDKLogImpl warn
WARNING: Samuel-Weavers-MacBook-Pro-3360: not member of view [Samuel-Weavers-MacBook-Pro-62704|2]; discarding it
Apr 16, 2012 9:50:06 PM org.infinispan.factories.GlobalComponentRegistry start
INFO: ISPN000128: Infinispan version: Infinispan 'Brahma' 5.1.2.FINAL
Apr 16, 2012 9:50:06 PM org.infinispan.jmx.CacheJmxRegistration start
INFO: ISPN000031: MBeans were successfully registered to the platform mbean server.
Apr 16, 2012 9:50:12 PM org.infinispan.remoting.transport.jgroups.JGroupsTransport viewAccepted
INFO: ISPN000093: Received new, MERGED cluster view: MergeView::[Samuel-Weavers-MacBook-Pro-3360|3] [Samuel-Weavers-MacBook-Pro-3360, Samuel-Weavers-MacBook-Pro-62704], subgroups=[Samuel-Weavers-MacBook-Pro-62704|1] [Samuel-Weavers-MacBook-Pro-3360], [Samuel-Weavers-MacBook-Pro-62704|2] [Samuel-Weavers-MacBook-Pro-62704]
Now, here it where it gets interesting. I also have a Web application (servlet + JSP) that it deployed on AS7, and who's aim is to provide a real time update web page of data entering the grid. The servlet calls essentially the same code as the GridGrabber and uses the same cluster.xml.
When it starts up everything deploys ok, but but for some reason it doesn't pick up the other nodes in the Grid although they are both running on the same machine and see each other no problem. Deployment messages are:
17:06:16,889 INFO [org.jboss.modules] JBoss Modules version 1.1.1.GA
17:06:17,107 INFO [org.jboss.msc] JBoss MSC version 1.0.2.GA
17:06:17,155 INFO [org.jboss.as] JBAS015899: JBoss AS 7.1.1.Final "Brontes" starting
17:06:17,972 INFO [org.xnio] XNIO Version 3.0.3.GA
17:06:17,972 INFO [org.jboss.as.server] JBAS015888: Creating http management service using socket-binding (management-http)
17:06:17,981 INFO [org.xnio.nio] XNIO NIO Implementation Version 3.0.3.GA
17:06:17,990 INFO [org.jboss.remoting] JBoss Remoting version 3.2.3.GA
17:06:18,006 INFO [org.jboss.as.logging] JBAS011502: Removing bootstrap log handlers
17:06:18,013 INFO [org.jboss.as.configadmin] (ServerService Thread Pool -- 26) JBAS016200: Activating ConfigAdmin Subsystem
17:06:18,017 INFO [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 31) JBAS010280: Activating Infinispan subsystem.
17:06:18,038 INFO [org.jboss.as.naming] (ServerService Thread Pool -- 38) JBAS011800: Activating Naming Subsystem
17:06:18,062 INFO [org.jboss.as.osgi] (ServerService Thread Pool -- 39) JBAS011940: Activating OSGi Subsystem
17:06:18,070 INFO [org.jboss.as.security] (ServerService Thread Pool -- 44) JBAS013101: Activating Security Subsystem
17:06:18,077 INFO [org.jboss.as.webservices] (ServerService Thread Pool -- 48) JBAS015537: Activating WebServices Extension
17:06:18,103 INFO [org.jboss.as.security] (MSC service thread 1-8) JBAS013100: Current PicketBox version=4.0.7.Final
17:06:18,121 INFO [org.jboss.as.connector] (MSC service thread 1-3) JBAS010408: Starting JCA Subsystem (JBoss IronJacamar 1.0.9.Final)
17:06:18,134 INFO [org.jboss.as.naming] (MSC service thread 1-4) JBAS011802: Starting Naming Service
17:06:18,173 INFO [org.jboss.as.mail.extension] (MSC service thread 1-4) JBAS015400: Bound mail session [java:jboss/mail/Default]
17:06:18,241 INFO [org.jboss.ws.common.management.AbstractServerConfig] (MSC service thread 1-7) JBoss Web Services - Stack CXF Server 4.0.2.GA
17:06:18,290 INFO [org.jboss.as.connector.subsystems.datasources] (ServerService Thread Pool -- 27) JBAS010403: Deploying JDBC-compliant driver class org.h2.Driver (version 1.3)
17:06:18,407 INFO [org.apache.coyote.http11.Http11Protocol] (MSC service thread 1-7) Starting Coyote HTTP/1.1 on http-localhost-127.0.0.1-8080
17:06:18,802 INFO [org.jboss.as.connector.subsystems.datasources] (MSC service thread 1-7) JBAS010400: Bound data source [java:jboss/datasources/ExampleDS]
17:06:18,885 INFO [org.jboss.as.server.deployment.scanner] (MSC service thread 1-7) JBAS015012: Started FileSystemDeploymentService for directory /Users/Sam/Downloads/jboss-as-7.1.1.Final/standalone/deployments
17:06:18,893 INFO [org.jboss.as.server.deployment.scanner] (DeploymentScanner-threads - 1) JBAS015003: Found TwitterWebApp.war in deployment directory. To trigger deployment create a file called TwitterWebApp.war.dodeploy
17:06:18,894 INFO [org.jboss.as.server.deployment.scanner] (DeploymentScanner-threads - 1) JBAS015003: Found SBTwitterStreamGui.war in deployment directory. To trigger deployment create a file called SBTwitterStreamGui.war.dodeploy
17:06:18,920 INFO [org.jboss.as.remoting] (MSC service thread 1-5) JBAS017100: Listening on localhost/127.0.0.1:4447
17:06:18,920 INFO [org.jboss.as.remoting] (MSC service thread 1-2) JBAS017100: Listening on /127.0.0.1:9999
17:06:19,018 INFO [org.jboss.as] (Controller Boot Thread) JBAS015951: Admin console listening on http://127.0.0.1:9990
17:06:19,019 INFO [org.jboss.as] (Controller Boot Thread) JBAS015874: JBoss AS 7.1.1.Final "Brontes" started in 2479ms - Started 133 of 208 services (74 services are passive or on-demand)
17:06:19,031 INFO [org.jboss.as.server.deployment] (MSC service thread 1-5) JBAS015876: Starting deployment of "TwitterWebApp.war"
17:06:19,031 INFO [org.jboss.as.server.deployment] (MSC service thread 1-4) JBAS015876: Starting deployment of "SBTwitterStreamGui.war"
17:06:19,426 INFO [org.jboss.web] (MSC service thread 1-7) JBAS018210: Registering web context: /SBTwitterStreamGui
17:06:20,267 INFO [org.jboss.web] (MSC service thread 1-3) JBAS018210: Registering web context: /TwitterWebApp
17:06:20,313 INFO [org.jboss.as.server] (DeploymentScanner-threads - 2) JBAS018559: Deployed "TwitterWebApp.war"
17:06:20,313 INFO [org.jboss.as.server] (DeploymentScanner-threads - 2) JBAS018559: Deployed "SBTwitterStreamGui.war"
17:06:23,399 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (http-localhost-127.0.0.1-8080-1) ISPN000078: Starting JGroups Channel
17:06:23,400 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (http-localhost-127.0.0.1-8080-1) ISPN000088: Unable to use any JGroups configuration mechanisms provided in properties {}. Using default JGroups configuration!
17:06:23,805 WARN [org.jgroups.protocols.UDP] (http-localhost-127.0.0.1-8080-1) receive buffer of socket java.net.DatagramSocket#4ad6be89 was set to 20MB, but the OS only allocated 65.51KB. This might lead to performance problems. Please set your max receive buffer in the OS correctly (e.g. net.core.rmem_max on Linux)
17:06:23,807 WARN [org.jgroups.protocols.UDP] (http-localhost-127.0.0.1-8080-1) receive buffer of socket java.net.MulticastSocket#7bb28246 was set to 25MB, but the OS only allocated 65.51KB. This might lead to performance problems. Please set your max receive buffer in the OS correctly (e.g. net.core.rmem_max on Linux)
17:06:26,824 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (http-localhost-127.0.0.1-8080-1) ISPN000094: Received new cluster view: [Samuel-Weavers-MacBook-Pro-102|0] [Samuel-Weavers-MacBook-Pro-102]
17:06:26,946 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (http-localhost-127.0.0.1-8080-1) ISPN000079: Cache local address is Samuel-Weavers-MacBook-Pro-102, physical addresses are [10.36.5.166:64952]
17:06:26,955 INFO [org.infinispan.factories.GlobalComponentRegistry] (http-localhost-127.0.0.1-8080-1) ISPN000128: Infinispan version: Infinispan 'Brahma' 5.1.2.FINAL
17:06:26,956 WARN [org.infinispan.config.TimeoutConfigurationValidatingVisitor] (http-localhost-127.0.0.1-8080-1) ISPN000148: Invalid <transport>: distributedSyncTimout value of 240000. It can not be higher than <stateRetrieval>:timeout which is 120000
17:06:27,045 INFO [org.infinispan.jmx.CacheJmxRegistration] (http-localhost-127.0.0.1-8080-1) ISPN000031: MBeans were successfully registered to the platform mbean server.
17:06:27,061 INFO [stdout] (http-localhost-127.0.0.1-8080-1) Cache Size: 0
17:06:27,062 INFO [stdout] (http-localhost-127.0.0.1-8080-1) Cluster Name: demoCluster
17:06:27,063 INFO [stdout] (http-localhost-127.0.0.1-8080-1) Cluster Size: 1
17:06:27,063 INFO [stdout] (http-localhost-127.0.0.1-8080-1) Cluster Members: [Samuel-Weavers-MacBook-Pro-102]
17:06:27,064 INFO [stdout] (http-localhost-127.0.0.1-8080-1) Cache Manager: org.infinispan.manager.DefaultCacheManager#3c818c4#Address:Samuel-Weavers-MacBook-Pro-102
Does anyone know if there is a reason why an the AS7 deployment wouldn't see the other nodes in the grid? It seems to me that it should work as they all are looking for the same cluster name (demoCluster) and they all use the same cluster.xml. The cluster.xml is as follows:
<infinispan
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="urn:infinispan:config:5.1 http://www.infinispan.org/schemas/infinispan-config-5.1.xsd"
xmlns="urn:infinispan:config:5.1">
<global>
<transport clusterName="demoCluster"/>
<globalJmxStatistics enabled="true"/>
</global>
<default>
<jmxStatistics enabled="true"/>
<clustering mode="distribution">
<hash numOwners="2" rehashRpcTimeout="120000"/>
<sync/>
</clustering>
</default>
</infinispan>
One thing I have noticed from the deployment messages is that the web app seems to use an ipv4 address and the jar seems to use an ipv6 address with JGroups but I'm not sure if this would could be the root cause. Any help appreciated.

Answer 1

I seemed to have resolved this issue by doing the following:
I was along the right lines about the JGroups ipv4/ipv6 address, so I forced the standalone application to use IPv4 by giving the following JVM argument: -Djava.net.preferIPv4Stack=true
One thing to note here - I was running the standalone jar from the terminal, and the web app was being deployed onto AS7 from within Eclipse. Passing the JVM arguments when running the jar from the terminal was unsuccessful. However, running the application from inside Eclipse also, and giving it the JVM argument in the Run Configuration "Run" menu -> "Run Configurations" -> Add new under Java application (or edit existing one) -> Arguments -> VM arguments -> paste above preferIPv4stack argument in the box.
Now when running both standalone application and web application from within Eclipse, they should join the same cluster and start sending messages to each other.
On a side note - I had a JGroups error after this saying "Message Dropped - sender not in window". After some further investigation, it turns out that being connected to a (corporate) VPN was causing the messages to be dropped. As soon as I turned VPN off it started working perfectly.
Hopefully this might help someone with the same issue.

Answer 2

If you wanna deploy Infinispan instances in AS7, I'd strongly recommend you follow the guidance here. That way, if you end up using other clustering services in AS7, you won't need to keep a separate communications layer with the burden it has...etc.