JBoss EJB application stops accepting connections - jboss

Question

We are running a web application that is deployed as a single ear file to JBoss 7.1.1 and is using a number of ejb's. We having an issue where everything is working just fine but about twice a day the application stops accepting connections. When this happens the following shows up in the server log:
14:49:00,582 ERROR [org.jboss.remoting.remote.connection] (Remoting "step" read-1) JBREM000200: Remote connection failed: java.io.IOException: Received an invalid message length of 1195986768
The following also appears upstream in the log:
14:33:29,042 INFO [org.jboss.as.naming] (Remoting "step" task-3) JBAS011806: Channel end notification received, closing channel Channel ID 676ac4e7 (inbound) of Remoting connection 611812ed to null
14:33:50,071 ERROR [org.jboss.as.ejb3.remote.protocol.versionone.MethodInvocationMessageHandler] (EJB default - 10) Could not write method invocation result for method due to : org.jboss.remoting3.NotOpenException: Writes closed
at org.jboss.remoting3.remote.RemoteConnectionChannel.openOutboundMessage(RemoteConnectionChannel.java:107) [jboss-remoting-3.2.3.GA.jar:3.2.3.GA]
at org.jboss.remoting3.remote.RemoteConnectionChannel.writeMessage(RemoteConnectionChannel.java:296) [jboss-remoting-3.2.3.GA.jar:3.2.3.GA]
at org.jboss.as.ejb3.remote.protocol.versionone.MethodInvocationMessageHandler.writeMethodInvocationResponse(MethodInvocationMessageHandler.java:330)
at org.jboss.as.ejb3.remote.protocol.versionone.MethodInvocationMessageHandler.access$500(MethodInvocationMessageHandler.java:64)
at org.jboss.as.ejb3.remote.protocol.versionone.MethodInvocationMessageHandler$1.run(MethodInvocationMessageHandler.java:226)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_45]
at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_45]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_45]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_45]
at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_45]
at org.jboss.threads.JBossThread.run(JBossThread.java:122) [jboss-threads-2.0.0.GA.jar:2.0.0.GA]
This is the exception we are seeing on the client side when we try to create an ejb connection
javax.naming.NamingException: Failed to create remoting connection [Root exception is java.lang.RuntimeException: Operation failed with status WAITING]
at org.jboss.naming.remote.client.ClientUtil.namingException(ClientUtil.java:36)
at org.jboss.naming.remote.client.InitialContextFactory.getInitialContext(InitialContextFactory.java:121)
at javax.naming.spi.NamingManager.getInitialContext(NamingManager.java:684)
at javax.naming.InitialContext.getDefaultInitCtx(InitialContext.java:307)
at javax.naming.InitialContext.init(InitialContext.java:242)
at javax.naming.InitialContext.<init>(InitialContext.java:216)
Caused by: java.lang.RuntimeException: Operation failed with status WAITING
at org.jboss.naming.remote.protocol.IoFutureHelper.get(IoFutureHelper.java:89)
at org.jboss.naming.remote.client.NamingStoreCache.getRemoteNamingStore(NamingStoreCache.java:56)
at org.jboss.naming.remote.client.InitialContextFactory.getOrCreateCachedNamingStore(InitialContextFactory.java:166)
at org.jboss.naming.remote.client.InitialContextFactory.getOrCreateNamingStore(InitialContextFactory.java:139)
at org.jboss.naming.remote.client.InitialContextFactory.getInitialContext(InitialContextFactory.java:104)
... 91 more
So far we have been unable to figure out what is causing the application to stop accepting ejb connections.
We have been able to figure out that if we start jboss and remove the application (or try to connect to a fresh JBoss instance where the application has not been deployed) we get the exact same error right down to the number of bytes in the "invalid message length of" error if we try to get an ejb connection.
We can also get a similar error if we try to connect to the application using some other protocol (e.g. if we type http://localhost:4447 into a browser window we see a similar exception but the length of the message is different).
The code that attemts the ejb connection looks like this:
public void initialize(String host, int port, String instance, String user, String password) {
this.host = host;
this.port = port;
this.user = user;
if (instance != null) {
this.instance = instance;
}
final Properties jndiProperties = new Properties();
String providerURL = "remote://" + host + ":" + port;
jndiProperties.put(Context.INITIAL_CONTEXT_FACTORY, org.jboss.naming.remote.client.InitialContextFactory.class.getName());
jndiProperties.put(Context.PROVIDER_URL, providerURL);
jndiProperties.put("jboss.naming.client.ejb.context", true);
jndiProperties.put("jboss.naming.client.connect.options.org.xnio.Options.SASL_POLICY_NOPLAINTEXT", "false");
jndiProperties.put(Context.SECURITY_PRINCIPAL, user);
jndiProperties.put(Context.SECURITY_CREDENTIALS, password);
try {
context = new InitialContext(jndiProperties);
logger.info("JNDI context initialized.");
} catch (NamingException e) {
throw new RuntimeException(e);
}
}
When we open the JBoss admin console it indicates that the application is still deployed and enabled.
What would cause the application to stop accepting connections and given the errors we are seeing?
Any help would be greatly appreciated.
Thanks in advance.