JAX-WS client causes Jetty EofException - sockets

I am using an embedded Jetty server to provide Web Services (using Jetty-JAXWS2-SPI to allow Jetty to serve the requests) and I am seeing the following errors when a connection terminates from my JAX-WS test client (generated by wsimport):
11:32:54.544 [pool-1-thread-5] DEBUG com.mycompany.SomeClass - EOF
org.eclipse.jetty.io.EofException: null
at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:321) ~[jetty-http-7.3.1.v20110307.jar:7.3.1.v20110307]
at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:214) ~[jetty-http-7.3.1.v20110307.jar:7.3.1.v20110307]
at org.eclipse.jetty.server.HttpConnection.handle(HttpConnection.java:411) ~[jetty-server-7.3.1.v20110307.jar:7.3.1.v20110307]
at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:526) ~[jetty-io-7.3.1.v20110307.jar:7.3.1.v20110307]
at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:41) [jetty-io-7.3.1.v20110307.jar:7.3.1.v20110307]
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [na:1.6.0_24]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [na:1.6.0_24]
at java.lang.Thread.run(Thread.java:662) [na:1.6.0_24]
Even though these are not important errors (logged at DEBUG level) and I only see them from my test client, not from soapUI or the 'real' (.NET client), I would rather not see them at all.
I guess the problem is caused by the socket behavior within the JAX-WS client? Is there any way to influence the socket behavior to avoid this issue?
Other information:
Others have reported this issue on the Jetty mailing list, also when providing Web Services using Jetty-JAXWS2-SPI.
I am using JDK 1.6.0_24 and the JAX-WS implementation it contains.
I am using Jetty 7.3.1-20110307

Related

JBoss automatic .war redeployment can't start application properly - Error on Keycloak redirection

A colleague of mine wants to automate their JBoss (JBoss EAP v7.4.0) deployments.
So far the process they followed manually was:
Shutdown Jboss entirely
Copy the new .war file under /jboss-eap-7.4.0/standalone/deployments directory
Restart the server by executing standalone.sh
This process, although a bit 'dump', manages to deploy the application with no issues.
I on the other hand, tried to make use of JBoss's auto-redeployment feature, so I simply pasted the new .war file under the /jboss-eap-7.4.0/standalone/deployments directory.
JBoss indeed picks up the new .war and automatically starts the redeployment process.
From the server.log file, it seems at first that the application deploys successfully.
However when I try to access it from my browser, it crashes and gives a code 500 response (See error logs at the end of the post). The expected behavior would be to redeploy me to keycloak in order to login.
The exact same error occurs if I try to manually undeploy and deploy the .war from the jboss.cli.
Any hints on the issue?
I can't figure out if the error is on the server or the application side.
It seems a bit odd that it should require a full JBoss server restart in order for the application to be properly deployed.
Thank you in advance.
Server logs:
2022-06-14 17:22:25,850 WARN [org.apache.jcp.xml.dsig.internal.dom.DOMReference] (default task-1) The input bytes to the digest operation are null. This may be due to a problem with the Reference URI or its Transforms.
2022-06-14 17:22:25,879 ERROR [io.undertow.request] (default task-1) UT005023: Exception handling request to /myapplication: java.lang.RuntimeException: Could not create authentication request.
at org.keycloak.keycloak-saml-adapter-core#15.0.2//org.keycloak.adapters.saml.AbstractInitiateLogin.challenge(AbstractInitiateLogin.java:64)
at org.keycloak.keycloak-saml-wildfly-elytron-adapter#15.0.2//org.keycloak.adapters.saml.elytron.ElytronHttpFacade.noAuthenticationInProgress(ElytronHttpFacade.java:123)
at org.keycloak.keycloak-saml-wildfly-elytron-adapter#15.0.2//org.keycloak.adapters.saml.elytron.KeycloakHttpServerAuthenticationMechanism.evaluateRequest(KeycloakHttpServerAuthenticationMechanism.java:125)
at org.wildfly.security.elytron-private#1.15.3.Final-redhat-00001//org.wildfly.security.http.util.SetMechanismInformationMechanismFactory$1.evaluateRequest(SetMechanismInformationMechanismFactory.java:119)
at org.wildfly.security.elytron-private#1.15.3.Final-redhat-00001//org.wildfly.security.http.util.SocketAddressCallbackServerMechanismFactory$1.evaluateRequest(SocketAddressCallbackServerMechanismFactory.java:82)
at org.wildfly.security.elytron-private#1.15.3.Final-redhat-00001//org.wildfly.security.auth.server.SecurityIdentityServerMechanismFactory$1.evaluateRequest(SecurityIdentityServerMechanismFactory.java:85)
at org.wildfly.security.elytron-private#1.15.3.Final-redhat-00001//org.wildfly.security.http.HttpAuthenticator$AuthenticationExchange.authenticate(HttpAuthenticator.java:326)
at org.wildfly.security.elytron-private#1.15.3.Final-redhat-00001//org.wildfly.security.http.HttpAuthenticator$AuthenticationExchange.access$800(HttpAuthenticator.java:301)
at org.wildfly.security.elytron-private#1.15.3.Final-redhat-00001//org.wildfly.security.http.HttpAuthenticator.authenticate(HttpAuthenticator.java:94)
at org.wildfly.security.elytron-web.undertow-server#1.9.0.Final-redhat-00001//org.wildfly.elytron.web.undertow.server.SecurityContextImpl.authenticate(SecurityContextImpl.java:99)
at org.wildfly.security.elytron-web.undertow-server-servlet#1.9.0.Final-redhat-00001//org.wildfly.elytron.web.undertow.server.servlet.ServletSecurityContextImpl.authenticate(ServletSecurityContextImpl.java:115)
at io.undertow.servlet#2.2.5.Final-redhat-00001//io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:55)
at io.undertow.core#2.2.5.Final-redhat-00001//io.undertow.server.handlers.DisableCacheHandler.handleRequest(DisableCacheHandler.java:33)
at io.undertow.core#2.2.5.Final-redhat-00001//io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
at io.undertow.core#2.2.5.Final-redhat-00001//io.undertow.security.handlers.AuthenticationConstraintHandler.handleRequest(AuthenticationConstraintHandler.java:53)
at io.undertow.core#2.2.5.Final-redhat-00001//io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46)
at io.undertow.servlet#2.2.5.Final-redhat-00001//io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64)
at io.undertow.servlet#2.2.5.Final-redhat-00001//io.undertow.servlet.handlers.security.ServletSecurityConstraintHandler.handleRequest(ServletSecurityConstraintHandler.java:59)
at io.undertow.core#2.2.5.Final-redhat-00001//io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43)
at org.wildfly.security.elytron-web.undertow-server-servlet#1.9.0.Final-redhat-00001//org.wildfly.elytron.web.undertow.server.servlet.CleanUpHandler.handleRequest(CleanUpHandler.java:38)
at io.undertow.core#2.2.5.Final-redhat-00001//io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
at org.wildfly.extension.undertow#7.4.0.GA-redhat-00005//org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(JACCContextIdHandler.java:61)
at io.undertow.core#2.2.5.Final-redhat-00001//io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
at org.wildfly.extension.undertow#7.4.0.GA-redhat-00005//org.wildfly.extension.undertow.deployment.GlobalRequestControllerHandler.handleRequest(GlobalRequestControllerHandler.java:68)
at io.undertow.servlet#2.2.5.Final-redhat-00001//io.undertow.servlet.handlers.SendErrorPageHandler.handleRequest(SendErrorPageHandler.java:52)
at io.undertow.core#2.2.5.Final-redhat-00001//io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
at io.undertow.servlet#2.2.5.Final-redhat-00001//io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:269)
at io.undertow.servlet#2.2.5.Final-redhat-00001//io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:78)
at io.undertow.servlet#2.2.5.Final-redhat-00001//io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:133)
at io.undertow.servlet#2.2.5.Final-redhat-00001//io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:130)
at io.undertow.servlet#2.2.5.Final-redhat-00001//io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48)
at io.undertow.servlet#2.2.5.Final-redhat-00001//io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43)
at org.wildfly.extension.undertow#7.4.0.GA-redhat-00005//org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1530)
at org.wildfly.extension.undertow#7.4.0.GA-redhat-00005//org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1530)
at org.wildfly.extension.undertow#7.4.0.GA-redhat-00005//org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1530)
at org.wildfly.extension.undertow#7.4.0.GA-redhat-00005//org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1530)
at org.wildfly.extension.undertow#7.4.0.GA-redhat-00005//org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1530)
at io.undertow.servlet#2.2.5.Final-redhat-00001//io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:249)
at io.undertow.servlet#2.2.5.Final-redhat-00001//io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:78)
at io.undertow.servlet#2.2.5.Final-redhat-00001//io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:99)
at io.undertow.core#2.2.5.Final-redhat-00001//io.undertow.server.Connectors.executeRootHandler(Connectors.java:387)
at io.undertow.core#2.2.5.Final-redhat-00001//io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:841)
at org.jboss.threads#2.4.0.Final-redhat-00001//org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
at org.jboss.threads#2.4.0.Final-redhat-00001//org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1990)
at org.jboss.threads#2.4.0.Final-redhat-00001//org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1486)
at org.jboss.threads#2.4.0.Final-redhat-00001//org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1377)
at org.jboss.xnio#3.8.4.Final-redhat-00001//org.xnio.XnioWorker$WorkerThreadFactory$1$1.run(XnioWorker.java:1280)
at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: org.apache.xml.serializer.utils.WrappedRuntimeException: class org.apache.xml.serializer.ToXMLSAXHandler cannot be cast to class org.apache.xml.serializer.SerializationHandler (org.apache.xml.serializer.ToXMLSAXHandler is in unnamed module of loader 'deployment.myapplication.war' #15a25903; org.apache.xml.serializer.SerializationHandler is in unnamed module of loader 'deployment.myapplication.war' #68ae4310)
at deployment.myapplication.war//org.apache.xml.serializer.SerializerFactory.getSerializer(SerializerFactory.java:179)
at deployment.myapplication.war//org.apache.xalan.transformer.TransformerIdentityImpl.createResultContentHandler(TransformerIdentityImpl.java:261)
at deployment.myapplication.war//org.apache.xalan.transformer.TransformerIdentityImpl.transform(TransformerIdentityImpl.java:330)
at org.keycloak.keycloak-saml-core#15.0.2//org.keycloak.saml.common.util.DocumentUtil.getNodeAsString(DocumentUtil.java:228)
at org.keycloak.keycloak-saml-core#15.0.2//org.keycloak.saml.common.util.DocumentUtil.getDocumentAsString(DocumentUtil.java:207)
at org.keycloak.keycloak-saml-core#15.0.2//org.keycloak.saml.BaseSAML2BindingBuilder.getSAMLResponse(BaseSAML2BindingBuilder.java:311)
at org.keycloak.keycloak-saml-core#15.0.2//org.keycloak.saml.BaseSAML2BindingBuilder.buildHtmlPostResponse(BaseSAML2BindingBuilder.java:307)
at org.keycloak.keycloak-saml-core#15.0.2//org.keycloak.saml.BaseSAML2BindingBuilder$BasePostBindingBuilder.getHtmlRequest(BaseSAML2BindingBuilder.java:172)
at org.keycloak.keycloak-saml-adapter-core#15.0.2//org.keycloak.adapters.saml.SamlUtil.sendSaml(SamlUtil.java:45)
at org.keycloak.keycloak-saml-adapter-core#15.0.2//org.keycloak.adapters.saml.profile.AbstractSamlAuthenticationHandler$3.sendAuthnRequest(AbstractSamlAuthenticationHandler.java:622)
at org.keycloak.keycloak-saml-adapter-core#15.0.2//org.keycloak.adapters.saml.AbstractInitiateLogin.challenge(AbstractInitiateLogin.java:61)

java connection refused error with spring boot data geode remote locator

Per my question Apache Geode Web framework I've checked through various spring guides from here and spring data geode samples from here and written a short spring data geode application but it cannot connect to the remote GFSH started Geode locator. The Application class is:
package cm;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.data.gemfire.config.annotation.ClientCacheApplication;
import org.springframework.data.gemfire.config.annotation.ClientCacheApplication.Locator;
import org.springframework.data.gemfire.config.annotation.EnablePdx;
import org.springframework.data.gemfire.repository.config.EnableGemfireRepositories;
#SpringBootApplication
#ClientCacheApplication(name = "CmWeb", locators = #Locator, subscriptionEnabled = true)
#EnableGemfireRepositories(basePackageClasses= {CmRequest.class})
#EnablePdx
public class CmWeb {
public static void main(String[] args) {
SpringApplication.run(CmWeb.class, args);
}
}
and in the resources directory application.properties I've set up the remote locator:
# Configure the client's connection Pool to the servers in the cluster
spring.data.gemfire.pool.locators=1.2.3.4[10334]
Build and run the application and it discovers the locator (which it returns as the server name)
[Timer-DEFAULT-2] o.a.g.c.c.i.AutoConnectionSourceImpl : AutoConnectionSource discovered new locators [UAT:10334]
A couple of seconds later it throws the error:
[Timer-DEFAULT-2] o.a.g.c.c.i.AutoConnectionSourceImpl : locator UAT:10334 is not running.
and
java.net.ConnectException: Connection refused: connect
at java.net.DualStackPlainSocketImpl.waitForConnect(Native Method) ~[na:1.8.0_232]
at java.net.DualStackPlainSocketImpl.socketConnect(DualStackPlainSocketImpl.java:85) ~[na:1.8.0_232]
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[na:1.8.0_232]
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:204) ~[na:1.8.0_232]
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[na:1.8.0_232]
at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:172) ~[na:1.8.0_232]
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[na:1.8.0_232]
at java.net.Socket.connect(Socket.java:607) ~[na:1.8.0_232]
at org.apache.geode.internal.net.SocketCreator.connect(SocketCreator.java:958) ~[geode-core-1.9.2.jar:na]
at org.apache.geode.internal.net.SocketCreator.connect(SocketCreator.java:899) ~[geode-core-1.9.2.jar:na]
at org.apache.geode.internal.net.SocketCreator.connect(SocketCreator.java:888) ~[geode-core-1.9.2.jar:na]
at org.apache.geode.distributed.internal.tcpserver.TcpClient.getServerVersion(TcpClient.java:290) ~[geode-core-1.9.2.jar:na]
at org.apache.geode.distributed.internal.tcpserver.TcpClient.requestToServer(TcpClient.java:184) ~[geode-core-1.9.2.jar:na]
at org.apache.geode.cache.client.internal.AutoConnectionSourceImpl.queryOneLocatorUsingConnection(AutoConnectionSourceImpl.java:209) [geode-core-1.9.2.jar:na]
at org.apache.geode.cache.client.internal.AutoConnectionSourceImpl.queryOneLocator(AutoConnectionSourceImpl.java:199) [geode-core-1.9.2.jar:na]
at org.apache.geode.cache.client.internal.AutoConnectionSourceImpl.queryLocators(AutoConnectionSourceImpl.java:287) [geode-core-1.9.2.jar:na]
at org.apache.geode.cache.client.internal.AutoConnectionSourceImpl$UpdateLocatorListTask.run2(AutoConnectionSourceImpl.java:500) [geode-core-1.9.2.jar:na]
at org.apache.geode.cache.client.internal.PoolImpl$PoolTask.run(PoolImpl.java:1371) [geode-core-1.9.2.jar:na]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_232]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [na:1.8.0_232]
at org.apache.geode.internal.ScheduledThreadPoolExecutorWithKeepAlive$DelegatingScheduledFuture.run(ScheduledThreadPoolExecutorWithKeepAlive.java:276) [geode-core-1.9.2.jar:na]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_232]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_232]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_232]
After a lot of investigation I thought it was that the spring data geode client expects a spring boot geode server according to Connecting GemFire using Spring Boot and Spring Data GemFire and so I downloaded the ListRegionsOnServerFunction jar and deployed it on the GFSH server get the same result (have not yet restarted the server...) but that causes the same error condition.
If by Spring-Data-Gemfire - Unable to contact a Locator service. Operation either timed out or Locator does not exist I try and change the application.properties from
spring.data.gemfire.pool.locators=1.2.3.4[10334]
to
spring.gemfire.locators=1.2.3.4[10334]
or other variations then the app can't find the remote locator and throws:
[Timer-DEFAULT-3] o.a.g.c.c.i.AutoConnectionSourceImpl : locator localhost/127.0.0.1:10334 is not running.
Writing this question I've finally found How to connect a remote-locator in Geode and also can't PING the GFSH server from the SPRING app. However, the server bind address is setup properly for remote locator clients and various other services and UI using a locally built Geode Native Client for Geode v 1.10 can connect. I suspect PING may be disabled across this (semi-internal) network by default. I also disabled the firewall rules for ports 10334, 1099, 40404 to allow all traffic but still get the same error condition.
It turns out that from repeated INFO messages in the Spring Boot app after the connection refused:
[Timer-DEFAULT-2] o.a.g.c.c.i.AutoConnectionSourceImpl : updateLocatorInLocatorList changing locator list: loc form: LocatorAddress [socketInetAddress=UAT:10334, hostname=UAT, isIpString=false] ,loc to: UAT:10334
[Timer-DEFAULT-2] o.a.g.c.c.i.AutoConnectionSourceImpl : updateLocatorInLocatorList locator list from:[UAT:10334, /1.2.3.4:10334] to: [LocatorAddress [socketInetAddress=UAT:10334, hostname=UAT, isIpString=false], LocatorAddress [socketInetAddress=/1.2.3.4:10334, hostname=1.2.3.4, isIpString=true]]
and then running list clients on the server, the connection from the Spring Boot app to the Geode server v 1.10 is in fact established. Arrrgh!
It means the locator logic is working but this doesn't explain why after the first connection there's a java.net.ConnectException: Connection refused: connect error. Any ideas?
1 quick note about your Spring Boot application class...
#SpringBootApplication
#ClientCacheApplication(name = "CmWeb", locators = #Locator, subscriptionEnabled = true)
#EnableGemfireRepositories(basePackageClasses= {CmRequest.class})
#EnablePdx
public class CmWeb {
public static void main(String[] args) {
SpringApplication.run(CmWeb.class, args);
}
}
The following statements are true iff you are using Spring Boot for Apache Geode (or Pivotal GemFire), which is highly recommended.
When using SBDG (by declaring the correct org.springframework.geode:spring-geode-starter dependency on your application classpath), then you do not need to explicitly declare the #ClientCacheApplication, #EnableGemfireRepositories or the #EnablePdx annotations since SBDG auto-configures a ClientCache instance by default, auto-configures SD Repositories particularly when all entity classes are in the same package or sub-package as the Spring Boot app and SBDG auto-configures PDX by default, as well.
The locator = #Locator just specifies that the "DEFAULT" GemFire/Geode Pool when configured via the ClientCacheFactory should connect to the cluster via Locators, on localhost using the default Locator port, 10334. Therefore, this attribute is mostly useless and I would recommend the new #EnableClusterAware annotation from SBDG (see here).
The other attributes can be configured via Spring Boot application.properties, like so:
spring.application.name=CmWeb
spring.data.gemfire.pool.subscription-enabled=true
TIP: You can configure subscription on individually "named" Pools, even via properties, if you are using more than 1 Pool (of connections) in your application, perhaps to route different payloads based on workflows to different "grouped" servers in your cluster, etc.
You started to configure the "DEFAULT" Pool in application.properties already with...
# Configure the client's connection Pool to the servers in the cluster
spring.data.gemfire.pool.locators=1.2.3.4[10334]
Regarding...
After a lot of investigation I thought it was that the spring data geode client expects a spring boot geode server
No, SDG does not expect the cluster (of servers) to be configured or bootstrapped with Spring at all. Using Gfsh is perfectly valid. For instance. If the ListRegionsOnServerFunction is not available, SDG falls back to other means (provided by GemFire/Geode itself, which Gfsh knows and uses).
All the messages you are seeing in the Spring Boot app logs are coming from Geode itself, i.e. nothing to do with Spring. In a nutshell, and FWIW, SDG/SBDG is a facade around the Apache Geode (Pivotal GemFire) API and Java client driver. SDG/SBDG is at the mercy of this client doing the right thing, which of course, is partially dependent on proper configuration. Still... I am really just thinking out loud now since I suspect you are already well aware of (or have discovered) all of this.
I would also say the Java client and Native Client are not exactly an apple to apple comparison either. Meaning, if you developed a client using purely the Apache Geode (Pivotal GemFire) API without Spring, you'd have the exact same problem.
I have never seen a case where the first connection is establish but subsequent connections get a "Connection refused", o.O #argh
Have you tried this same configuration/arrangement with older Geode versions, e.g. 1.9?
Sorry for your troubles. I will think on this more.

java.lang.ClassNotFoundException: No ClassLoaders found for: > org.jboss.jms.client.JBossConnectionFactory

I am trying to configure jms setup between two servers. The sender is on (server A)jboss 5 and the receiver is on (server B)jboss 4(which is using remote queue from server A.
When i try to run both on jboss 5 or jboss 4, it works. But when the receiver is on jboss 4 and sender is on jboss 5, then it throws below Exception.
Caused by: java.lang.ClassNotFoundException: No ClassLoaders found for: org.jboss.jms.client.JBossConnectionFactory (no security manager: RMI class loader disabled)
Exception Trace:
javax.naming.CommunicationException [Root exception is java.lang.ClassNotFoundException: No ClassLoaders found for: org.jboss.jms.client.JBossConnectionFactory (no security manager: RMI class loader disabled)]
at org.jnp.interfaces.NamingContext.lookup(NamingContext.java:728)
at org.jnp.interfaces.NamingContext.lookup(NamingContext.java:587)
at javax.naming.InitialContext.lookup(InitialContext.java:392)
at org.jboss.ejb.plugins.jms.DLQHandler.createService(DLQHandler.java:182)
at org.jboss.system.ServiceMBeanSupport.jbossInternalCreate(ServiceMBeanSupport.java:260)
at org.jboss.system.ServiceMBeanSupport.create(ServiceMBeanSupport.java:188)
at org.jboss.ejb.plugins.jms.JMSContainerInvoker.innerStartDelivery(JMSContainerInvoker.java:510)
at org.jboss.ejb.plugins.jms.JMSContainerInvoker$ExceptionListenerImpl$ExceptionListenerRunnable.run(JMSContainerInvoker.java:1419)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.lang.ClassNotFoundException: No ClassLoaders found for: org.jboss.jms.client.JBossConnectionFactory (no security manager: RMI class loader disabled)
at sun.rmi.server.LoaderHandler.loadClass(LoaderHandler.java:375)
at sun.rmi.server.LoaderHandler.loadClass(LoaderHandler.java:165)
at java.rmi.server.RMIClassLoader$2.loadClass(RMIClassLoader.java:620)
at org.jboss.system.JBossRMIClassLoader.loadClass(JBossRMIClassLoader.java:91)
at java.rmi.server.RMIClassLoader.loadClass(RMIClassLoader.java:247)
at sun.rmi.server.MarshalInputStream.resolveClass(MarshalInputStream.java:197)
at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1575)
at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1496)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1732)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1329)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:351)
at java.rmi.MarshalledObject.get(MarshalledObject.java:142)
at org.jnp.interfaces.MarshalledValuePair.get(MarshalledValuePair.java:72)
at org.jnp.interfaces.NamingContext.lookup(NamingContext.java:652)
... 8 more
I found below link which seems similar to the issue that i am facing but not able to get exactly on from which server to which server i need to have client libraries under war file?.
https://access.redhat.com/solutions/45133
More Details:-
In my Case, jboss 4.0.4 and jboss 5.1.0 both are hosting the messages for different queues.
-- jboss 5 is hosting the message and listener is on jboss4 for QueueA.
-- jboss 4 is hosting the message and listener is on jboss 5 for QueueB.
What we want to achieve:- Jboss 4 on server A should communicate with RemoteQueues which are configured on jboss 5 on server B, which is not working and throwing above Exception. Communication is not happening between both servers.
The bottom line here is that JBoss 4.x uses JBossMQ as the JMS implementation and JBoss 5.x uses JBoss Messaging as the JMS implementation. The two are not directly compatible therefore you need to decide which server will host the messages.
If you want the JBoss 5 server to host the messages then all the clients (including those running on JBoss 4) will need to use JBoss Messaging client libraries.
If you want the JBoss 4 server to host the messages then all the clients (including those running on JBoss 5) will need to use JBossMQ client libraries.
If you're hosting messages on both JBoss 4 & JBoss 5 servers then the clients for each will need to use the corresponding client libraries for whichever broker their working with.

Wildfly10 (EAP 7) call jboss 5.0.1 EJB without legacy jars

Dears,
I'm trying to call ejb3 in jboss 5.0.1 from Wildfly 10 or EAP 7.
My code:
final Properties env = new Properties();
env.put(Context.INITIAL_CONTEXT_FACTORY, "org.jboss.naming.remote.client.InitialContextFactory");
env.put("java.naming.factory.url.pkgs", "org.jboss.ejb.client.naming");
env.put(Context.PROVIDER_URL, "remoting://localhost:1099");
env.put("org.jboss.ejb.client.scoped.context", "true");
InitialContext initialContext = new InitialContext(env);
TestBeanRemote remote = (TestBeanRemote) initialContext.lookup(
"ejb:TestEar/TestBean/TestBean!com.test.TestBeanRemote");
but it says:
Exception in thread "main" java.lang.IllegalStateException: EJBCLIENT000025: No EJB receiver available for handling [appName:BilllingFacadeCallbackEAR, moduleName:BilllingFacadeCallback, distinctName:] combination for invocation context org.jboss.ejb.client.EJBClientInvocationContext#3b088d51
at org.jboss.ejb.client.EJBClientContext.requireEJBReceiver(EJBClientContext.java:798)
at org.jboss.ejb.client.ReceiverInterceptor.handleInvocation(ReceiverInterceptor.java:128)
at org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:186)
at org.jboss.ejb.client.EJBInvocationHandler.sendRequestWithPossibleRetries(EJBInvocationHandler.java:255)
at org.jboss.ejb.client.EJBInvocationHandler.doInvoke(EJBInvocationHandler.java:200)
at org.jboss.ejb.client.EJBInvocationHandler.doInvoke(EJBInvocationHandler.java:183)
at org.jboss.ejb.client.EJBInvocationHandler.invoke(EJBInvocationHandler.java:146)
at com.sun.proxy.$Proxy2.getActions(Unknown Source)
at TestStandalone.main(TestStandalone.java:28)
Is there any solution to call legacy jboss without old jars?
There is a legacy subsystem for this but I don't know its current status.
https://github.com/jboss-set/jboss-as-legacy
The CORBA standard defines an "across the wire" standard for making remote method calls called IIOP or "Internet Inter-ORB Protocol".
You need to set up to use CORBA IIOP in order to make platform independent remote EJB calls.
Therefore, you need to:
configure JBoss 5 so that it can handle incoming IIOP calls;
configure WildFly 10/EAP 7 to make outgoing EJB invocations using IIOP.
There is some information on this in the WildFly 10 EJB3 Reference Guide although I'm not sure how up to date that is.
The issue is normally caused by a transaction reaching it's timeout value.
So it may be that the application logic is correctly handling the scenario in this case and is not attempting to retry activity
It can have several issues :
connection: Connection broken
security : user/pass invalid
EJB missing: connected, but ejb is not there
SSL
Ports
IP Address
JBoss maintains a persistent connection to the other server, so when the client sees this message it means there is no connection to a server that has the ejb you are trying to call, so a message will be logged when the connection fails to the other server.
Caused by: java.lang.IllegalStateException: EJBCLIENT000025: No EJB receiver available for handling
Can you clarify the below:
1# is your EJBs deployed on jboss 5.0.1?
2# You are invoking the EJBS from Wildfly 10 or EAP 7, means your client is deployed in Wildfly 10 or EAP 7?

Trouble saving session when mixing spring-security-gemfire and spring-security-oauth2

Background: I have a web app that utilizes AngularJS, spring-mvc, and spring-rest for delivering the UI. I have a requirement to load balance using an Elastic LB and it is not using sticky sessions; requests are round robin. I implemented session replication using spring-session with gemfire for session storage. This works well.
I need to integrate with an OAuth2 auth server (and eventually multiple OAuth2 servers) purely for authentication and the passing of userInfo. I attempted to use the spring cloud oauth2 #EnableOAuth2Sso on the web-app and hit some session serialization issues. The mere addition of the oauth2ClientContext to the session seemed to cause ClassCastException problems during session saving.
I attempted to pull down the following samples and they worked well out of the box, Particularly the UI and the Authserver.
https://github.com/spring-guides/tut-spring-security-and-angular-js
However, when I added spring session into the mix, trying to serialize to a gemfire server, I encountered the exact same issue.
Here is the stacktrace highlight:
java.lang.ClassCastException: cannot assign instance of org.springframework.beans.factory.support.StaticListableBeanFactory to field org.springframework.aop.scope.DefaultScopedObject.beanFactory of type org.springframework.beans.factory.config.ConfigurableBeanFactory in instance of org.springframework.aop.scope.DefaultScopedObject
Below is abbreviated stacktrace:
ERROR o.a.c.c.C.[.[.[/].[dispatcherServlet] : Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception
org.springframework.dao.DataAccessResourceFailureException: remote server on machine(gemfire:21800:loner):57660:9d1f3438:gemfire: : While performing a remote put; nested exception is com.gemstone.gemfire.cache.client.ServerOperationException: remote server on machine(gemfire:21800:loner):57660:9d1f3438:gemfire: : While performing a remote put
at org.springframework.data.gemfire.GemfireCacheUtils.convertGemfireAccessException(GemfireCacheUtils.java:238) ~[spring-data-gemfire-1.7.4.RELEASE.jar:1.7.4.RELEASE]
at org.springframework.data.gemfire.GemfireAccessor.convertGemFireAccessException(GemfireAccessor.java:91) ~[spring-data-gemfire-1.7.4.RELEASE.jar:1.7.4.RELEASE]
at org.springframework.data.gemfire.GemfireTemplate.put(GemfireTemplate.java:190) ~[spring-data-gemfire-1.7.4.RELEASE.jar:1.7.4.RELEASE]
at org.springframework.session.data.gemfire.GemFireOperationsSessionRepository.save(GemFireOperationsSessionRepository.java:147) ~[spring-session-1.2.1.RELEASE.jar:na]
at org.springframework.session.data.gemfire.GemFireOperationsSessionRepository.save(GemFireOperationsSessionRepository.java:35) ~[spring-session-1.2.1.RELEASE.jar:na]
at org.springframework.session.web.http.SessionRepositoryFilter$SessionRepositoryRequestWrapper.commitSession(SessionRepositoryFilter.java:244) ~[spring-session-1.2.1.RELEASE.jar:na]
at org.springframework.session.web.http.SessionRepositoryFilter$SessionRepositoryRequestWrapper.access$100(SessionRepositoryFilter.java:214) ~[spring-session-1.2.1.RELEASE.jar:na]
at org.springframework.session.web.http.SessionRepositoryFilter.doFilterInternal(SessionRepositoryFilter.java:167) ~[spring-session-1.2.1.RELEASE.jar:na]
at org.springframework.session.web.http.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:80) ~[spring-session-1.2.1.RELEASE.jar:na]
... tomcat filter chain and spring filter stuff
Caused by: com.gemstone.gemfire.cache.client.ServerOperationException: remote server on machine(gemfire:21800:loner):57660:9d1f3438:gemfire: : While performing a remote put
... gemfire internal stuff
at org.springframework.data.gemfire.GemfireTemplate.put(GemfireTemplate.java:187) ~[spring-data-gemfire-1.7.4.RELEASE.jar:1.7.4.RELEASE]
... 31 common frames omitted
Caused by: java.lang.ClassCastException: cannot assign instance of org.springframework.beans.factory.support.StaticListableBeanFactory to field org.springframework.aop.scope.DefaultScopedObject.beanFactory of type org.springframework.beans.factory.config.ConfigurableBeanFactory in instance of org.springframework.aop.scope.DefaultScopedObject
at java.io.ObjectStreamClass$FieldReflector.setObjFieldValues(ObjectStreamClass.java:2133) ~[na:1.7.0_80]
at java.io.ObjectStreamClass.setObjFieldValues(ObjectStreamClass.java:1305) ~[na:1.7.0_80]
... java.io stuff
at org.springframework.aop.framework.AdvisedSupport.readObject(AdvisedSupport.java:557) ~[spring-aop-4.3.2.RELEASE.jar:4.3.2.RELEASE]
at sun.reflect.GeneratedMethodAccessor224.invoke(Unknown Source) ~[na:na]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.7.0_80]
at java.lang.reflect.Method.invoke(Method.java:497) ~[na:1.7.0_80]
at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1058) ~[na:1.7.0_80]
... java.io stuff
at com.gemstone.gemfire.internal.InternalDataSerializer.basicReadObject(InternalDataSerializer.java:2966) ~[gemfire-8.1.0.jar:na]
at com.gemstone.gemfire.DataSerializer.readObject(DataSerializer.java:3210) ~[gemfire-8.1.0.jar:na]
at org.springframework.session.data.gemfire.AbstractGemFireOperationsSessionRepository$GemFireSessionAttributes.readObject(AbstractGemFireOperationsSessionRepository.java:800) ~[spring-session-1.2.1.RELEASE.jar:na]
at org.springframework.session.data.gemfire.AbstractGemFireOperationsSessionRepository$GemFireSessionAttributes.fromDelta(AbstractGemFireOperationsSessionRepository.java:834) ~[spring-session-1.2.1.RELEASE.jar:na]
at org.springframework.session.data.gemfire.AbstractGemFireOperationsSessionRepository$GemFireSession.fromDelta(AbstractGemFireOperationsSessionRepository.java:589) ~[spring-session-1.2.1.RELEASE.jar:na]
at com.gemstone.gemfire.internal.cache.EntryEventImpl.processDeltaBytes(EntryEventImpl.java:1345) ~[gemfire-8.1.0.jar:na]
... gemfire internal stuff
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.7.0_80]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.7.0_80]
at com.gemstone.gemfire.internal.cache.tier.sockets.AcceptorImpl$1$1.run(AcceptorImpl.java:577) ~[gemfire-8.1.0.jar:na]
... 1 common frames omitted
I found the following, https://jira.spring.io/browse/SPR-14117, which encouraged me to update some of the jars to the newest versions, hoping the spring boot versions were simply behind, however it didn't seem to help.
Version info:
spring-cloud-starter-parent: Brixton.SR4
spring-cloud-security: 1.1.2.RELEASE
spring-core: 4.3.2.RELEASE
spring-security-oauth2: 2.0.10.RELEASE
spring-session: 1.2.1.RELEASE
I've considered a few options: rewiring the OAuth2 framework to no longer use ScopedProxyMode.INTERFACES (seems daunting), use Redis vs. Gemfire, write the entire client from scratch (I've done it before... wasn't fun).
FWIW I've already added the RequestContextFilter as recommended here: OAuth2ClientContext (spring-security-oauth2) not persisted in Redis when using spring-session and spring-cloud-security
Does anyone have any guidance?
I don't know if this speaks to your problem directly but I had/have a similar problem and I think I have all the same versions as you. Seems that there are so many Spring projects and they all try to keep up with each other so sometimes there seem to be compatibility issues. I found the steps outlined here by Rob Winch fixed my issue -https://github.com/spring-projects/spring-session/issues/395