Host name resolve error for AWS Fargate task instances - wildfly

I'm trying to run a dockerized WildFly 26 EE 9.1 preview on a AWS Fargate cluster but it fails to start with
[31m09:15:52,203 ERROR [org.jboss.as.controller.management-operation] (ServerService Thread Pool -- 76) WFLYCTL0013: Operation ("add") failed - address: ([("subsystem" => "webservices")]): java.lang.RuntimeException: java.net.UnknownHostException: 7a0765d6977e48a98e3b16f7533674f7-3378193245: 7a0765d6977e48a98e3b16f7533674f7-3378193245: Name does not resolve
at org.jboss.as.webservices//org.jboss.as.webservices.dmr.WSSubsystemAdd.createServerConfig(WSSubsystemAdd.java:106)
at org.jboss.as.webservices//org.jboss.as.webservices.dmr.WSSubsystemAdd.performBoottime(WSSubsystemAdd.java:93)
at org.jboss.a...#18.0.4.Final//org.jboss.as.controller.AbstractBoottimeAddStepHandler.performBoottime(AbstractBoottimeAddStepHandler.java:144)
at org.jboss.a...#18.0.4.Final//org.jboss.as.controller.AbstractBoottimeAddStepHandler.performRuntime(AbstractBoottimeAddStepHandler.java:119)
at org.jboss.a...#18.0.4.Final//org.jboss.as.controller.AbstractAddStepHandler$1.execute(AbstractAddStepHandler.java:164)
at org.jboss.a...#18.0.4.Final//org.jboss.as.controller.AbstractOperationContext.executeStep(AbstractOperationContext.java:1045)
at org.jboss.a...#18.0.4.Final//org.jboss.as.controller.AbstractOperationContext.processStages(AbstractOperationContext.java:777)
at org.jboss.a...#18.0.4.Final//org.jboss.as.controller.AbstractOperationContext.executeOperation(AbstractOperationContext.java:466)
at org.jboss.a...#18.0.4.Final//org.jboss.as.controller.ParallelBootOperationStepHandler$ParallelBootTask.run(ParallelBootOperationStepHandler.java:384)
at org.jbos...#2.4.0.Final//org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
at org.jbos...#2.4.0.Final//org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1990)
at org.jbos...#2.4.0.Final//org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1486)
at org.jbos...#2.4.0.Final//org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1348)
at java.base/java.lang.Thread.run(Thread.java:831)
at org.jbos...#2.4.0.Final//org.jboss.threads.JBossThread.run(JBossThread.java:513)
Caused by: java.net.UnknownHostException: 7a0765d6977e48a98e3b16f7533674f7-3378193245: 7a0765d6977e48a98e3b16f7533674f7-3378193245: Name does not resolve
at java.base/java.net.InetAddress.getLocalHost(InetAddress.java:1644)
at org.jboss...#3.3.3.Final//org.jboss.ws.common.management.AbstractServerConfig.setWebServiceHost(AbstractServerConfig.java:133)
at org.jboss.as.webservices//org.jboss.as.webservices.config.ServerConfigImpl.setWebServiceHost(ServerConfigImpl.java:97)
at org.jboss.as.webservices//org.jboss.as.webservices.dmr.WSSubsystemAdd.createServerConfig(WSSubsystemAdd.java:104)
... 14 more
Caused by: java.net.UnknownHostException: 7a0765d6977e48a98e3b16f7533674f7-3378193245: Name does not resolve
at java.base/java.net.Inet4AddressImpl.lookupAllHostAddr(Native Method)
at java.base/java.net.InetAddress$PlatformNameService.lookupAllHostAddr(InetAddress.java:932)
at java.base/java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1517)
at java.base/java.net.InetAddress$NameServiceAddresses.get(InetAddress.java:851)
at java.base/java.net.InetAddress.getAllByName0(InetAddress.java:1507)
at java.base/java.net.InetAddress.getLocalHost(InetAddress.java:1639)
The cluster is deployed in a private subnet but I have a security group that allows all outgoing traffic and all traffic is routed through a NAT Gateway. I recall using the same CloudFormation scripts for WildFly 24:ish and it worked fine so it shouldn't(tm) be a networking issue.
Is the host name generally supposed to be resolvable?

I’m not sure if this issue still persists on your side, but for me it was resolved as soon as I enabled "DNS Hostnames" in VPC ("DNS resolution" was enabled already). I have no idea why this option matters, but even with disabling public IP in Service defintion of ECS the wildfly threw the error. Since the option is enabled (which only seems to enable hostnames for public ips) it is working fine.

Related

Keycloak - Connection Null issue when spun up on AWS ECS

I have checked that all relevant parameters are being parsed correctly and i am able to see them in my AWS ECS service. When the service is spun up i get the below error WARN  [org.jboss.jca.core.connectionmanager.pool.strategy.OnePool] (ServerService Thread Pool -- 60) IJ000604: Throwable while attempting to get a new connection: null: javax.resource.ResourceException: IJ031084: Unable to create connection which then follows with this error Caused by: org.postgresql.util.PSQLException: FATAL: database "keycloak" does not exist
I am unsure if the two are related. I validated relevant parameters and have tried to imitate the same issue locally.
Any help or advice would be aprpeciated
TYIA

ZuulException: Forwarding error in Spring cloud running locally

I get this very common zuul error, when running spring cloud microservices locally.
If I specify microservice address in application.yml it works fine, so I am wondering why it doesn't discovery the service address properly without explicitly specifying it, especially that this is a new behaviour.
zuul:
routes:
my-service:
path: /myservice/**
#url: http://localhost:8081
Any hints what could have caused this?
Here is some part of stacktrace.
Caused by: java.lang.RuntimeException: java.net.SocketException: Permission denied: connect
at rx.exceptions.Exceptions.propagate(Exceptions.java:57)
at rx.observables.BlockingObservable.blockForSingle(BlockingObservable.java:463)
at rx.observables.BlockingObservable.single(BlockingObservable.java:340)
at com.netflix.client.AbstractLoadBalancerAwareClient.executeWithLoadBalancer(AbstractLoadBalancerAwareClien t.java:112)
... 137 common frames omitted
Caused by: java.net.SocketException: Permission denied: connect
Thanks!

Failed to read challenge file [Caused by java.io.FileNotFoundException: /jboss/standalone/tmp/auth/local4123__.challenge (No such file or directory)

I am trying to connect two machines both running JBoss EAP 7.1.0 using a JMS bridge. Machine 1 is to act as a web server and has a WAR file deployed which is accessible, and Machine 2 is to act as the app server and has all the necessary components deployed just fine.
This is the error I am receiving:
WARN [org.apache.activemq.artemis.jms.bridge] (ServerService Thread Pool -- 72) AMQ342010: Failed to connect JMS Bridge N/A: javax.naming.CommunicationException: WFNAM00018: Failed to connect to remote host [Root exception is javax.security.sasl.SaslException: Authentication failed: all available authentication mechanisms failed:
JBOSS-LOCAL-USER: javax.security.sasl.SaslException: ELY05128: [JBOSS-LOCAL-USER] Failed to read challenge file [Caused by java.io.FileNotFoundException: /.../.../jboss/standalone/tmp/auth/local3093626581916142639.challenge (No such file or directory)]]
at org.wildfly.naming.client.remote.RemoteNamingProvider.getPeerIdentityForNaming(RemoteNamingProvider.java:110)
at org.wildfly.naming.client.remote.RemoteNamingProvider.getPeerIdentityForNaming(RemoteNamingProvider.java:53)
at org.wildfly.naming.client.NamingProvider.getPeerIdentityForNamingUsingRetry(NamingProvider.java:105)
at org.wildfly.naming.client.remote.RemoteNamingProvider.getPeerIdentityForNamingUsingRetry(RemoteNamingProvider.java:91)
at org.wildfly.naming.client.remote.RemoteContext.lambda$lookupNative$0(RemoteContext.java:189)
at org.wildfly.naming.client.NamingProvider.performExceptionAction(NamingProvider.java:222)
at org.wildfly.naming.client.remote.RemoteContext.performWithRetry(RemoteContext.java:100)
at org.wildfly.naming.client.remote.RemoteContext.lookupNative(RemoteContext.java:188)
at org.wildfly.naming.client.AbstractFederatingContext.lookup(AbstractFederatingContext.java:74)
at org.wildfly.naming.client.AbstractFederatingContext.lookup(AbstractFederatingContext.java:60)
at org.wildfly.naming.client.WildFlyRootContext.lookup(WildFlyRootContext.java:144)
at javax.naming.InitialContext.lookup(InitialContext.java:417)
at javax.naming.InitialContext.lookup(InitialContext.java:417)
at org.apache.activemq.artemis.jms.bridge.impl.JNDIFactorySupport.createObject(JNDIFactorySupport.java:46)
at org.apache.activemq.artemis.jms.bridge.impl.JNDIDestinationFactory.createDestination(JNDIDestinationFactory.java:32)
at org.apache.activemq.artemis.jms.bridge.impl.JMSBridgeImpl.setupJMSObjects(JMSBridgeImpl.java:1072)
at org.apache.activemq.artemis.jms.bridge.impl.JMSBridgeImpl.start(JMSBridgeImpl.java:398)
at org.wildfly.extension.messaging.activemq.jms.bridge.JMSBridgeService.startBridge(JMSBridgeService.java:114)
at org.wildfly.extension.messaging.activemq.jms.bridge.JMSBridgeService$1.run(JMSBridgeService.java:84)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
at org.jboss.threads.JBossThread.run(JBossThread.java:320)
The connection to the target machine (application server) is being made as the path to JBoss is that of the path on the machine, which I verified by testing with a Windows application server environment and the path was the correct Windows path to the directory where the challenge file should be, so clearly the connection is being made as the directories are being accessed. However, the .challenge file isn't present each time which understandably causes the error message.
I have scoured SO and JBoss forums for days now and nothing is resolving my issue.
I saw this post: JBOSS-LOCAL-USER: javax.security.sasl.SaslException: Failed to read server challenge
This is the same issue that I am facing, but the answer which was marked as correct doesn't help me very much. The solution, in this case, was to replace the default ApplicationRealm with a JAAS realm, but I do not know if this is what I need, and I certainly do not currently have one. I did research it, but it seemed to not be applicable to my setup, but I could be wrong.
I also tried this solution: https://access.redhat.com/solutions/3209281 (Subscription only access)
This solution was to remove default-user="$local" from here:
<security-realm name="ApplicationRealm">
<authentication>
<local default-user="$local" allowed-users="*" skip-group-loading="true"/>
I did this to both standalone-full.xml files on both machines, and it appeared to make no difference at all.
I have created application users on both machines and given them superuser privileges through the JBoss console as I figured it was probably an issue with permissions when trying to write the file but this too was to no avail. I have also verified that both user credentials are correct.
The workaround was to switch to using a core bridge instead of a JMS bridge, as per the recommendation from Justin in the comments.

How to configure PostgreSQL database for deploying alfresco on tomcat 8?

I have built alfresco(version 5.2) from source on ubuntu 16.04. I want to deploy alfresco on tomcat 8. The deployment is successful however the PostgreSQL database is not getting configured as required. I have followed the steps as given in http://docs.alfresco.com/5.1/tasks/postgresql-config.html
I observe the home page as given in image alfresco_page
Am I missing onto something here that the PostgreSQL database is not getting configured. Is there any other configuration that needs to be done that I have missed ?
UPDATE
The alfresco.log gave me this
2017-08-01 05:53:54,406 WARN [org.alfresco.web.scripts.servlet.X509ServletFilterBase] [localhost-startStop-1] clientAuth does not appear to be set for Tomcat. clientAuth must be set to 'want' for X509 Authentication
2017-08-01 05:53:54,416 WARN [org.alfresco.web.scripts.servlet.X509ServletFilterBase] [localhost-startStop-1] Attempting to set clientAuth=want through JMX...
2017-08-01 05:53:54,427 WARN [org.alfresco.web.scripts.servlet.X509ServletFilterBase] [localhost-startStop-1] Unable to set clientAuth=want through JMX.
2017-08-01 05:53:55,139 ERROR [org.apache.solr.core.CoreContainer] [coreLoadExecutor-5-thread-1] Error creating core [collection1]: Could not load conf for core collection1: Error loading solr config from solr/collection1/conf/solrconfig.xml
org.apache.solr.common.SolrException: Could not load conf for core collection1: Error loading solr config from solr/collection1/conf/solrconfig.xml
at org.apache.solr.core.ConfigSetService.getConfig(ConfigSetService.java:66)
at org.apache.solr.core.CoreContainer.create(CoreContainer.java:489)
at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:255)
at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:249)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.solr.common.SolrException: Error loading solr config from solr/collection1/conf/solrconfig.xml
at org.apache.solr.core.SolrConfig.readFromResourceLoader(SolrConfig.java:154)
at org.apache.solr.core.ConfigSetService.createSolrConfig(ConfigSetService.java:80)
at org.apache.solr.core.ConfigSetService.getConfig(ConfigSetService.java:61)
... 7 more
Caused by: java.io.IOException: Can't find resource 'solrconfig.xml' in classpath or '/root/tomcat85/output/build/webapps/solr/collection1/conf'
at org.apache.solr.core.SolrResourceLoader.openResource(SolrResourceLoader.java:362)
at org.apache.solr.core.SolrResourceLoader.openConfig(SolrResourceLoader.java:308)
at org.apache.solr.core.Config.<init>(Config.java:117)
at org.apache.solr.core.Config.<init>(Config.java:87)
at org.apache.solr.core.SolrConfig.<init>(SolrConfig.java:167)
at org.apache.solr.core.SolrConfig.readFromResourceLoader(SolrConfig.java:145)
... 9 more
2017-08-01 05:54:09,634 WARN [org.hibernate.cfg.SettingsFactory] [localhost-startStop-1] Could not obtain connection metadata
org.apache.commons.dbcp.SQLNestedException: Cannot create PoolableConnectionFactory (Connection refused. Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connections.)
at org.apache.commons.dbcp.BasicDataSource.createPoolableConnectionFactory(BasicDataSource.java:1549)
at org.apache.commons.dbcp.BasicDataSource.createDataSource(BasicDataSource.java:1388)
at org.apache.commons.dbcp.BasicDataSource.getConnection(BasicDataSource.java:1044)
at org.springframework.orm.hibernate3.LocalDataSourceConnectionProvider.getConnection(LocalDataSourceConnectionProvider.java:83)
at org.hibernate.cfg.SettingsFactory.buildSettings(SettingsFactory.java:84)
at org.hibernate.cfg.Configuration.buildSettings(Configuration.java:2079)
at org.hibernate.cfg.Configuration.buildSessionFactory(Configuration.java:1304)
at org.springframework.orm.hibernate3.LocalSessionFactoryBean.newSessionFactory(LocalSessionFactoryBean.java:863)
at org.springframework.orm.hibernate3.LocalSessionFactoryBean.buildSessionFactory(LocalSessionFactoryBean.java:782)
at org.springframework.orm.hibernate3.AbstractSessionFactoryBean.afterPropertiesSet(AbstractSessionFactoryBean.java:188)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1573)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1511)
Things to check:
Is postgres running (ps -ef|grep postgres)?
Can you use psql to connect to postgres using the db.name, db.username, and db.password that are configured in alfresco-global.properties?
Did you follow the step in the docs about editing pg_hba.conf to make sure that postgres is configured to allow password based authentication?
Also, it is exceedingly rare to need to build Alfresco from source unless you are making changes to the low-level classes themselves, which is not recommended.

NoRouteToHostException / NoSuchHostException on remote JMX call

I'm having trouble making a remote JMX call to JBoss 6 on a Centos 5.6 server. I've previously been able to do this when running the same app on a Debian server.
./twiddle.sh --server=service:jmx:rmi:///jndi/rmi://SERVER:1090/jmxconnector invoke foo:service=bar baz
Caused by: java.net.NoRouteToHostException: No route to host
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
at java.net.Socket.connect(Socket.java:529)
at java.net.Socket.connect(Socket.java:478)
at java.net.Socket.<init>(Socket.java:375)
at java.net.Socket.<init>(Socket.java:189)
at sun.rmi.transport.proxy.RMIDirectSocketFactory.createSocket(RMIDirectSocketFactory.java:22)
at sun.rmi.transport.proxy.RMIMasterSocketFactory.createSocket(RMIMasterSocketFactory.java:128)
at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:595)
The same call succeeds if I run it locally on SERVER. I've opened port 1090 with iptables, and I can connect via telnet to SERVER:1090. hostname -i returns the correct IP address.
I've also tried starting JBoss with -Djava.rmi.server.hostname=localhost. If I do this, then I get a different exception:
Caused by: java.rmi.NoSuchObjectException: no such object in table
at sun.rmi.transport.StreamRemoteCall.exceptionReceivedFromServer(StreamRemoteCall.java:255)
at sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:233)
at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:142)
at javax.management.remote.rmi.RMIServerImpl_Stub.newClient(Unknown Source)
at javax.management.remote.rmi.RMIConnector.getConnection(RMIConnector.java:2327)
at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:279)
at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:248)
at org.jboss.console.twiddle.Twiddle.createMBeanServerConnection(Twiddle.java:322)
at org.jboss.console.twiddle.Twiddle.connect(Twiddle.java:331)
at org.jboss.console.twiddle.Twiddle.access$400(Twiddle.java:60)
at org.jboss.console.twiddle.Twiddle$1.getServer(Twiddle.java:217)
It turns out that in addition to port 1090, JMX/RMI also uses a dynamically allocated port, which gets blocked by the firewall. So, if it's appropriate, disable the firewall altogether, or else this seems to be an alternative (which I've not tried yet):
http://olegz.wordpress.com/2009/03/23/jmx-connectivity-through-the-firewall/
What I did was this:
Update the file activemq.xml and specify rmiServerPort.
<managementContext createConnector="true" connectorPort="SOME_PORT" rmiServerPort="SOME_OTHER" jmxDomainName="org.apache.activemq"/>
Allow both ports by updating your iptables entry, restart activemq and it should work.