2023-01-13 11:02:36.291 INFO 1 --- [ main]
org.mongodb.driver.cluster: Cluster created with settings {hosts=mongo.apps.lab.pacosta.com:27017], mode=SINGLE, requiredClusterType=UNKNOWN, derverSelectionTimeout='30000 ms'}
2023-01-13 11:02:36.429 INFO 1 --- [costa.com:27017] org.mongodb.driver.cluster: Exception in monitor thread
while connecting to server mongo.apps.lab.pacosta.com:27017
com.mongodb.MongoSocketOpenException: Exception opening socket
at com.mongodb.internal.connection.SocketStream.open(SocketStream.java:70) ~[mongodb-driver-core-4.1.1.jar!/:na]
at com.mongodb.internal.connection.InternalStreamConnection.open(InternalStreamConnection.java:143) ~[mongodb-driver-core-4.1.1.jar!/:na]
at com.mongodb.internal.connection.DefaultServerMonitor$ServerMonitorRunnable.lookupServerDescription(DefaultServerMonitor.java:188) ~[mongodb-driver-core-4.1.1.jar!/:na]
at com.mongodb.internal.connection.DefaultServerMonitor$ServerMonitorRunnable.run(DefaultServerMonitor.java:144) ~[mongodb-driver-core-4.1.1.jar!/:na]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_212] Caused by: java.net.ConnectException: Connection refused (Connection refused)
at java.net.PlainSocketImpl.socketConnect(Native Method) ~[na:1.8.0_212]
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[na:1.8.0_212]
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[na:1.8.0_212]
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[na:1.8.0_212]
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[na:1.8.0_212]
at java.net.Socket.connect(Socket.java:589) ~[na:1.8.0_212]`
Here are screenshots of OCP routes and environment variables.
We are doing the same thing in our spring boot service using OCP version 4.11 but instead of using service ip we are taking it as a domain in mongo uri which is unable to create the connection
Recently we have upgraded ActiveMQ Artemis version from 1.5.3 to 2.21.0 and now using PostgreSQL to store the messages. Earlier we were storing the messages in file system.
ActiveMQ Artemis is used as a embedded server inside a Spring-based Java web application.
We are getting following error if connection to the PostgreSQL is temporarily interrupted:
javax.jms.IllegalStateRuntimeException: Session is closed
at org.apache.activemq.artemis.jms.client.JmsExceptionUtils.convertToRuntimeException(JmsExceptionUtils.java:59)
at org.apache.activemq.artemis.jms.client.ActiveMQJMSContext.createMapMessage(ActiveMQJMSContext.java:257)
at com..data.core.messaging.dataload.BaseDataLoaderMessageSender.getMapMessage(BaseDataLoaderMessageSender.java:220)
at com.test.data.core.messaging.dataload.BaseDataLoaderMessageSender.send(BaseDataLoaderMessageSender.java:140)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343)
at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:408)
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:834)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1415)
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: javax.jms.IllegalStateException: Session is closed
at org.apache.activemq.artemis.jms.client.ActiveMQSession.checkClosed(ActiveMQSession.java:1255)
at org.apache.activemq.artemis.jms.client.ActiveMQSession.createMapMessage(ActiveMQSession.java:169)
at org.apache.activemq.artemis.jms.client.ActiveMQJMSContext.createMapMessage(ActiveMQJMSContext.java:255)
... 132 more
The sequeunce of events are
Start the server
Shutdown DB
Perform some action on JMS queue like send a message/read messages from queue - It will fail. This is expected
Start DB
Perform some action on JMS queue like send a message/read messages from queue
In the step 5 as the connection is restored then I expect it to be successful.
ConnectionFactory is created as a Spring bean:
<bean id="queueConnectionFactory"
class="org.apache.activemq.artemis.jms.client.ActiveMQJMSConnectionFactory">
<constructor-arg value="false"/>
<constructor-arg>
<bean class="org.apache.activemq.artemis.api.core.TransportConfiguration">
<constructor-arg
value="org.apache.activemq.artemis.core.remoting.impl.invm.InVMConnectorFactory"/>
</bean>
</constructor-arg>
<property name="ConsumerWindowSize" value="0"/>
</bean>
And the JMSContext and MessageProducer are being created as
jmsContext = connectionFactory.createContext();
producer = jmsContext.createProducer();
Note: I tried to recreate ConnectionFactory programatically in case of exception but still the same issue.
Can anyone suggest what can be wrong here?
Edit 1:
Error log in case of DB connection is lost
[INFO ] 2022-06-24 10:28:53.353 [MessageBroker-1] WebSocketMessageBrokerStats - WebSocketSession[0 current WS(0)-HttpStream(0)-HttpPoll(0), 0 total, 0 closed abnormally (0 connect failure, 0 send limit, 0 transport error)], stompSubProtocol[processed CONNECT(0)-CONNECTED(0)-DISCONNECT(0)], stompBrokerRelay[null], inboundChannel[pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0], outboundChannel[pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0], sockJsScheduler[pool size = 1, active threads = 1, queued tasks = 0, completed tasks = 0]
[ERROR] 2022-06-24 10:29:11.737 [Thread-6 (ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$6#4d16a1c0)] JDBCConnectionProvider -
SQL EXCEPTIONS:
SQLState: 08001 ErrorCode: 0 Message: Connection to localhost:5432 refused. Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connections.
[WARN ] 2022-06-24 10:29:11.737 [Thread-6 (ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$6#4d16a1c0)] JDBCJournalImpl - Connection to localhost:5432 refused. Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connections.
org.postgresql.util.PSQLException: Connection to localhost:5432 refused. Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connections.
at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:280) ~[postgresql-42.2.5.jar:42.2.5]
at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:49) ~[postgresql-42.2.5.jar:42.2.5]
at org.postgresql.jdbc.PgConnection.<init>(PgConnection.java:195) ~[postgresql-42.2.5.jar:42.2.5]
at org.postgresql.Driver.makeConnection(Driver.java:454) ~[postgresql-42.2.5.jar:42.2.5]
at org.postgresql.Driver.connect(Driver.java:256) ~[postgresql-42.2.5.jar:42.2.5]
at org.apache.tomcat.jdbc.pool.PooledConnection.connectUsingDriver(PooledConnection.java:319) ~[tomcat-jdbc.jar:?]
at org.apache.tomcat.jdbc.pool.PooledConnection.connect(PooledConnection.java:212) ~[tomcat-jdbc.jar:?]
at org.apache.tomcat.jdbc.pool.ConnectionPool.createConnection(ConnectionPool.java:739) ~[tomcat-jdbc.jar:?]
at org.apache.tomcat.jdbc.pool.ConnectionPool.borrowConnection(ConnectionPool.java:671) ~[tomcat-jdbc.jar:?]
at org.apache.tomcat.jdbc.pool.ConnectionPool.getConnection(ConnectionPool.java:198) ~[tomcat-jdbc.jar:?]
at org.apache.tomcat.jdbc.pool.DataSourceProxy.getConnection(DataSourceProxy.java:132) ~[tomcat-jdbc.jar:?]
at jdk.internal.reflect.GeneratedMethodAccessor55.invoke(Unknown Source) ~[?:?]
at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344) ~[spring-aop-5.3.18.jar:5.3.18]
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:208) ~[spring-aop-5.3.18.jar:5.3.18]
at com.sun.proxy.$Proxy38.getConnection(Unknown Source) ~[?:?]
at org.springframework.jdbc.datasource.lookup.AbstractRoutingDataSource.getConnection(AbstractRoutingDataSource.java:194) ~[spring-jdbc-5.3.18.jar:5.3.18]
at org.apache.activemq.artemis.jdbc.store.drivers.JDBCConnectionProvider.getConnection(JDBCConnectionProvider.java:61) ~[artemis-jdbc-store-2.21.0.jar:2.21.0]
at org.apache.activemq.artemis.jdbc.store.journal.JDBCJournalImpl.sync(JDBCJournalImpl.java:213) ~[artemis-jdbc-store-2.21.0.jar:2.21.0]
at org.apache.activemq.artemis.jdbc.store.journal.JDBCJournalImpl$JDBCJournalSync.run(JDBCJournalImpl.java:1049) ~[artemis-jdbc-store-2.21.0.jar:2.21.0]
at org.apache.activemq.artemis.core.server.ActiveMQScheduledComponent.runForExecutor(ActiveMQScheduledComponent.java:313) ~[artemis-commons-2.21.0.jar:2.21.0]
at org.apache.activemq.artemis.core.server.ActiveMQScheduledComponent.lambda$bookedRunForScheduler$2(ActiveMQScheduledComponent.java:320) ~[artemis-commons-2.21.0.jar:2.21.0]
at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:42) ~[artemis-commons-2.21.0.jar:?]
at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:31) ~[artemis-commons-2.21.0.jar:?]
at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:65) ~[artemis-commons-2.21.0.jar:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) [artemis-commons-2.21.0.jar:?]
Caused by: java.net.ConnectException: Connection refused: connect
at java.net.PlainSocketImpl.waitForConnect(Native Method) ~[?:?]
at java.net.PlainSocketImpl.socketConnect(PlainSocketImpl.java:107) ~[?:?]
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:399) ~[?:?]
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:242) ~[?:?]
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:224) ~[?:?]
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[?:?]
at java.net.Socket.connect(Socket.java:609) ~[?:?]
at org.postgresql.core.PGStream.<init>(PGStream.java:70) ~[postgresql-42.2.5.jar:42.2.5]
at org.postgresql.core.v3.ConnectionFactoryImpl.tryConnect(ConnectionFactoryImpl.java:91) ~[postgresql-42.2.5.jar:42.2.5]
at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:192) ~[postgresql-42.2.5.jar:42.2.5]
... 28 more
[WARN ] 2022-06-24 10:29:11.833 [Thread-6 (ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$6#4d16a1c0)] server - AMQ222010: Critical IO Error, shutting down the server. file=NULL, message=Critical IO Error. Failed to process JDBC Record statements
org.postgresql.util.PSQLException: Connection to localhost:5432 refused. Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connections.
at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:280) ~[postgresql-42.2.5.jar:42.2.5]
at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:49) ~[postgresql-42.2.5.jar:42.2.5]
at org.postgresql.jdbc.PgConnection.<init>(PgConnection.java:195) ~[postgresql-42.2.5.jar:42.2.5]
at org.postgresql.Driver.makeConnection(Driver.java:454) ~[postgresql-42.2.5.jar:42.2.5]
at org.postgresql.Driver.connect(Driver.java:256) ~[postgresql-42.2.5.jar:42.2.5]
at org.apache.tomcat.jdbc.pool.PooledConnection.connectUsingDriver(PooledConnection.java:319) ~[tomcat-jdbc.jar:?]
at org.apache.tomcat.jdbc.pool.PooledConnection.connect(PooledConnection.java:212) ~[tomcat-jdbc.jar:?]
at org.apache.tomcat.jdbc.pool.ConnectionPool.createConnection(ConnectionPool.java:739) ~[tomcat-jdbc.jar:?]
at org.apache.tomcat.jdbc.pool.ConnectionPool.borrowConnection(ConnectionPool.java:671) ~[tomcat-jdbc.jar:?]
at org.apache.tomcat.jdbc.pool.ConnectionPool.getConnection(ConnectionPool.java:198) ~[tomcat-jdbc.jar:?]
at org.apache.tomcat.jdbc.pool.DataSourceProxy.getConnection(DataSourceProxy.java:132) ~[tomcat-jdbc.jar:?]
at jdk.internal.reflect.GeneratedMethodAccessor55.invoke(Unknown Source) ~[?:?]
at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344) ~[spring-aop-5.3.18.jar:5.3.18]
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:208) ~[spring-aop-5.3.18.jar:5.3.18]
at com.sun.proxy.$Proxy38.getConnection(Unknown Source) ~[?:?]
at org.springframework.jdbc.datasource.lookup.AbstractRoutingDataSource.getConnection(AbstractRoutingDataSource.java:194) ~[spring-jdbc-5.3.18.jar:5.3.18]
at org.apache.activemq.artemis.jdbc.store.drivers.JDBCConnectionProvider.getConnection(JDBCConnectionProvider.java:61) ~[artemis-jdbc-store-2.21.0.jar:2.21.0]
at org.apache.activemq.artemis.jdbc.store.journal.JDBCJournalImpl.sync(JDBCJournalImpl.java:213) ~[artemis-jdbc-store-2.21.0.jar:2.21.0]
at org.apache.activemq.artemis.jdbc.store.journal.JDBCJournalImpl$JDBCJournalSync.run(JDBCJournalImpl.java:1049) ~[artemis-jdbc-store-2.21.0.jar:2.21.0]
at org.apache.activemq.artemis.core.server.ActiveMQScheduledComponent.runForExecutor(ActiveMQScheduledComponent.java:313) ~[artemis-commons-2.21.0.jar:2.21.0]
at org.apache.activemq.artemis.core.server.ActiveMQScheduledComponent.lambda$bookedRunForScheduler$2(ActiveMQScheduledComponent.java:320) ~[artemis-commons-2.21.0.jar:2.21.0]
at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:42) ~[artemis-commons-2.21.0.jar:?]
at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:31) ~[artemis-commons-2.21.0.jar:?]
at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:65) ~[artemis-commons-2.21.0.jar:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) [artemis-commons-2.21.0.jar:?]
Caused by: java.net.ConnectException: Connection refused: connect
at java.net.PlainSocketImpl.waitForConnect(Native Method) ~[?:?]
at java.net.PlainSocketImpl.socketConnect(PlainSocketImpl.java:107) ~[?:?]
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:399) ~[?:?]
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:242) ~[?:?]
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:224) ~[?:?]
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[?:?]
at java.net.Socket.connect(Socket.java:609) ~[?:?]
at org.postgresql.core.PGStream.<init>(PGStream.java:70) ~[postgresql-42.2.5.jar:42.2.5]
at org.postgresql.core.v3.ConnectionFactoryImpl.tryConnect(ConnectionFactoryImpl.java:91) ~[postgresql-42.2.5.jar:42.2.5]
at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:192) ~[postgresql-42.2.5.jar:42.2.5]
... 28 more
[INFO ] 2022-06-24 10:29:11.845 [Thread-12] load - preDeActivate called
[WARN ] 2022-06-24 10:29:11.923 [Thread-6 (ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$6#4d16a1c0)] server - AMQ222057: It was not possible to add references due to an IO error code 6 message = JDBC Transaction failed.
[WARN ] 2022-06-24 10:29:11.923 [Thread-6 (ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$6#4d16a1c0)] server - AMQ222063: Error processing IOCallback code = 6 message = JDBC Transaction failed.
[INFO ] 2022-06-24 10:30:04.019 [Thread-12] load - deActivate called
[WARN ] 2022-06-24 10:30:04.112 [Thread-4 (ActiveMQ-client-global-threads)] client - AMQ212037: Connection failure to invm:0 has been detected: AMQ219015: The connection was disconnected because of server shutdown [code=DISCONNECTED]
[WARN ] 2022-06-24 10:30:04.112 [Thread-3 (ActiveMQ-client-global-threads)] client - AMQ212037: Connection failure to invm:0 has been detected: AMQ219015: The connection was disconnected because of server shutdown [code=DISCONNECTED]
[WARN ] 2022-06-24 10:30:04.112 [Thread-1 (ActiveMQ-client-global-threads)] client - AMQ212037: Connection failure to invm:0 has been detected: AMQ219015: The connection was disconnected because of server shutdown [code=DISCONNECTED]
[WARN ] 2022-06-24 10:30:04.112 [Thread-2 (ActiveMQ-client-global-threads)] client - AMQ212037: Connection failure to invm:0 has been detected: AMQ219015: The connection was disconnected because of server shutdown [code=DISCONNECTED]
[INFO ] 2022-06-24 10:30:04.147 [Thread-12] load - stop called
[ERROR] 2022-06-24 10:30:04.575 [DataLoad-EventLoadMessageProcessorThread] load - AMQ219017: Consumer is closed
javax.jms.IllegalStateRuntimeException: AMQ219017: Consumer is closed
at org.apache.activemq.artemis.jms.client.JmsExceptionUtils.convertToRuntimeException(JmsExceptionUtils.java:59) ~[artemis-jms-client-2.21.0.jar:2.21.0]
at org.apache.activemq.artemis.jms.client.ActiveMQJMSConsumer.receive(ActiveMQJMSConsumer.java:68) ~[artemis-jms-client-2.21.0.jar:2.21.0]
at com.test.data.core.messaging.dataload.BaseDataLoaderEventLoadMessageProcessor.processMessagesInBatch(BaseDataLoaderEventLoadMessageProcessor.java:108) [test-core-base-data-loader-22.8.0-SNAPSHOT.jar:?]
at com.test.data.core.messaging.dataload.BaseDataLoaderEventLoadMessageProcessor.run(BaseDataLoaderEventLoadMessageProcessor.java:80) [test-core-base-data-loader-22.8.0-SNAPSHOT.jar:?]
at java.lang.Thread.run(Thread.java:829) [?:?]
Caused by: javax.jms.IllegalStateException: AMQ219017: Consumer is closed
at org.apache.activemq.artemis.core.client.impl.ClientConsumerImpl.checkClosed(ClientConsumerImpl.java:951) ~[artemis-core-client-2.21.0.jar:2.21.0]
at org.apache.activemq.artemis.core.client.impl.ClientConsumerImpl.receive(ClientConsumerImpl.java:198) ~[artemis-core-client-2.21.0.jar:2.21.0]
at org.apache.activemq.artemis.core.client.impl.ClientConsumerImpl.receive(ClientConsumerImpl.java:382) ~[artemis-core-client-2.21.0.jar:2.21.0]
at org.apache.activemq.artemis.jms.client.ActiveMQMessageConsumer.getMessage(ActiveMQMessageConsumer.java:209) ~[artemis-jms-client-2.21.0.jar:2.21.0]
at org.apache.activemq.artemis.jms.client.ActiveMQMessageConsumer.receive(ActiveMQMessageConsumer.java:129) ~[artemis-jms-client-2.21.0.jar:2.21.0]
at org.apache.activemq.artemis.jms.client.ActiveMQJMSConsumer.receive(ActiveMQJMSConsumer.java:66) ~[artemis-jms-client-2.21.0.jar:2.21.0]
... 3 more
Caused by: org.apache.activemq.artemis.api.core.ActiveMQObjectClosedException: AMQ219017: Consumer is closed
at org.apache.activemq.artemis.core.client.impl.ClientConsumerImpl.checkClosed(ClientConsumerImpl.java:951) ~[artemis-core-client-2.21.0.jar:2.21.0]
at org.apache.activemq.artemis.core.client.impl.ClientConsumerImpl.receive(ClientConsumerImpl.java:198) ~[artemis-core-client-2.21.0.jar:2.21.0]
at org.apache.activemq.artemis.core.client.impl.ClientConsumerImpl.receive(ClientConsumerImpl.java:382) ~[artemis-core-client-2.21.0.jar:2.21.0]
at org.apache.activemq.artemis.jms.client.ActiveMQMessageConsumer.getMessage(ActiveMQMessageConsumer.java:209) ~[artemis-jms-client-2.21.0.jar:2.21.0]
at org.apache.activemq.artemis.jms.client.ActiveMQMessageConsumer.receive(ActiveMQMessageConsumer.java:129) ~[artemis-jms-client-2.21.0.jar:2.21.0]
at org.apache.activemq.artemis.jms.client.ActiveMQJMSConsumer.receive(ActiveMQJMSConsumer.java:66) ~[artemis-jms-client-2.21.0.jar:2.21.0]
... 3 more
[ERROR] 2022-06-24 10:30:04.575 [GS-EventLoadMessageProcessorThread] load - AMQ219017: Consumer is closed
javax.jms.IllegalStateRuntimeException: AMQ219017: Consumer is closed
at org.apache.activemq.artemis.jms.client.JmsExceptionUtils.convertToRuntimeException(JmsExceptionUtils.java:59) ~[artemis-jms-client-2.21.0.jar:2.21.0]
at org.apache.activemq.artemis.jms.client.ActiveMQJMSConsumer.receive(ActiveMQJMSConsumer.java:68) ~[artemis-jms-client-2.21.0.jar:2.21.0]
at com.test.data.core.messaging.dataload.BaseDataLoaderEventLoadMessageProcessor.processMessagesInBatch(BaseDataLoaderEventLoadMessageProcessor.java:108) [test-core-base-data-loader-22.8.0-SNAPSHOT.jar:?]
at com.test.data.core.messaging.dataload.BaseDataLoaderEventLoadMessageProcessor.run(BaseDataLoaderEventLoadMessageProcessor.java:80) [test-core-base-data-loader-22.8.0-SNAPSHOT.jar:?]
at java.lang.Thread.run(Thread.java:829) [?:?]
Caused by: javax.jms.IllegalStateException: AMQ219017: Consumer is closed
at org.apache.activemq.artemis.core.client.impl.ClientConsumerImpl.checkClosed(ClientConsumerImpl.java:951) ~[artemis-core-client-2.21.0.jar:2.21.0]
at org.apache.activemq.artemis.core.client.impl.ClientConsumerImpl.receive(ClientConsumerImpl.java:198) ~[artemis-core-client-2.21.0.jar:2.21.0]
at org.apache.activemq.artemis.core.client.impl.ClientConsumerImpl.receive(ClientConsumerImpl.java:382) ~[artemis-core-client-2.21.0.jar:2.21.0]
at org.apache.activemq.artemis.jms.client.ActiveMQMessageConsumer.getMessage(ActiveMQMessageConsumer.java:209) ~[artemis-jms-client-2.21.0.jar:2.21.0]
at org.apache.activemq.artemis.jms.client.ActiveMQMessageConsumer.receive(ActiveMQMessageConsumer.java:129) ~[artemis-jms-client-2.21.0.jar:2.21.0]
at org.apache.activemq.artemis.jms.client.ActiveMQJMSConsumer.receive(ActiveMQJMSConsumer.java:66) ~[artemis-jms-client-2.21.0.jar:2.21.0]
... 3 more
Caused by: org.apache.activemq.artemis.api.core.ActiveMQObjectClosedException: AMQ219017: Consumer is closed
at org.apache.activemq.artemis.core.client.impl.ClientConsumerImpl.checkClosed(ClientConsumerImpl.java:951) ~[artemis-core-client-2.21.0.jar:2.21.0]
at org.apache.activemq.artemis.core.client.impl.ClientConsumerImpl.receive(ClientConsumerImpl.java:198) ~[artemis-core-client-2.21.0.jar:2.21.0]
at org.apache.activemq.artemis.core.client.impl.ClientConsumerImpl.receive(ClientConsumerImpl.java:382) ~[artemis-core-client-2.21.0.jar:2.21.0]
at org.apache.activemq.artemis.jms.client.ActiveMQMessageConsumer.getMessage(ActiveMQMessageConsumer.java:209) ~[artemis-jms-client-2.21.0.jar:2.21.0]
at org.apache.activemq.artemis.jms.client.ActiveMQMessageConsumer.receive(ActiveMQMessageConsumer.java:129) ~[artemis-jms-client-2.21.0.jar:2.21.0]
at org.apache.activemq.artemis.jms.client.ActiveMQJMSConsumer.receive(ActiveMQJMSConsumer.java:66) ~[artemis-jms-client-2.21.0.jar:2.21.0]
... 3 more
Exception in thread "DataLoad-EventLoadMessageProcessorThread" Exception in thread "GS-EventLoadMessageProcessorThread" javax.jms.IllegalStateRuntimeException: AMQ219019: Session is closed
at org.apache.activemq.artemis.jms.client.JmsExceptionUtils.convertToRuntimeException(JmsExceptionUtils.java:59)
at org.apache.activemq.artemis.jms.client.ActiveMQJMSContext.recover(ActiveMQJMSContext.java:364)
at com.test.data.core.messaging.dataload.BaseDataLoaderEventLoadMessageProcessor.processMessagesInBatch(BaseDataLoaderEventLoadMessageProcessor.java:149)
at com.test.data.core.messaging.dataload.BaseDataLoaderEventLoadMessageProcessor.run(BaseDataLoaderEventLoadMessageProcessor.java:80)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: javax.jms.IllegalStateException: AMQ219019: Session is closed
at org.apache.activemq.artemis.core.client.impl.ClientSessionImpl.checkClosed(ClientSessionImpl.java:2069)
at org.apache.activemq.artemis.core.client.impl.ClientSessionImpl.rollback(ClientSessionImpl.java:1007)
at org.apache.activemq.artemis.core.client.impl.ClientSessionImpl.rollback(ClientSessionImpl.java:1000)
at org.apache.activemq.artemis.jms.client.ActiveMQSession.recover(ActiveMQSession.java:341)
at org.apache.activemq.artemis.jms.client.ActiveMQJMSContext.recover(ActiveMQJMSContext.java:362)
... 3 more
Caused by: ActiveMQObjectClosedException[errorType=OBJECT_CLOSED message=AMQ219019: Session is closed]
... 8 more
javax.jms.IllegalStateRuntimeException: AMQ219019: Session is closed
at org.apache.activemq.artemis.jms.client.JmsExceptionUtils.convertToRuntimeException(JmsExceptionUtils.java:59)
at org.apache.activemq.artemis.jms.client.ActiveMQJMSContext.recover(ActiveMQJMSContext.java:364)
at com.test.data.core.messaging.dataload.BaseDataLoaderEventLoadMessageProcessor.processMessagesInBatch(BaseDataLoaderEventLoadMessageProcessor.java:149)
at com.test.data.core.messaging.dataload.BaseDataLoaderEventLoadMessageProcessor.run(BaseDataLoaderEventLoadMessageProcessor.java:80)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: javax.jms.IllegalStateException: AMQ219019: Session is closed
at org.apache.activemq.artemis.core.client.impl.ClientSessionImpl.checkClosed(ClientSessionImpl.java:2069)
at org.apache.activemq.artemis.core.client.impl.ClientSessionImpl.rollback(ClientSessionImpl.java:1007)
at org.apache.activemq.artemis.core.client.impl.ClientSessionImpl.rollback(ClientSessionImpl.java:1000)
at org.apache.activemq.artemis.jms.client.ActiveMQSession.recover(ActiveMQSession.java:341)
at org.apache.activemq.artemis.jms.client.ActiveMQJMSContext.recover(ActiveMQJMSContext.java:362)
... 3 more
Caused by: ActiveMQObjectClosedException[errorType=OBJECT_CLOSED message=AMQ219019: Session is closed]
... 8 more
Logs is start is called without calling stop
Caused by: javax.jms.JMSException: Failed to create session factory
at org.apache.activemq.artemis.jms.client.ActiveMQConnectionFactory.createConnectionInternal(ActiveMQConnectionFactory.java:867) ~[artemis-jms-client-2.21.0.jar:2.21.0]
at org.apache.activemq.artemis.jms.client.ActiveMQConnectionFactory.createContext(ActiveMQConnectionFactory.java:306) ~[artemis-jms-client-2.21.0.jar:2.21.0]
... 139 more
Caused by: org.apache.activemq.artemis.api.core.ActiveMQNotConnectedException: AMQ219007: Cannot connect to server(s). Tried with all available servers.
at org.apache.activemq.artemis.core.client.impl.ServerLocatorImpl.createSessionFactory(ServerLocatorImpl.java:703) ~[artemis-core-client-2.21.0.jar:2.21.0]
at org.apache.activemq.artemis.jms.client.ActiveMQConnectionFactory.createConnectionInternal(ActiveMQConnectionFactory.java:865) ~[artemis-jms-client-2.21.0.jar:2.21.0]
at org.apache.activemq.artemis.jms.client.ActiveMQConnectionFactory.createContext(ActiveMQConnectionFactory.java:306) ~[artemis-jms-client-2.21.0.jar:2.21.0]
... 139 more
EDIT 2: Now sometimes threads are going into BLOCKED state while stopping the server
Thread-6
Stack Trace is:
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(java.base#11.0.15/Native Method)
at java.lang.Thread.sleep(java.base#11.0.15/Thread.java:334)
at java.util.concurrent.TimeUnit.sleep(java.base#11.0.15/TimeUnit.java:446)
at org.apache.activemq.artemis.core.server.impl.FileLockNodeManager.lock(FileLockNodeManager.java:442)
at org.apache.activemq.artemis.core.server.impl.FileLockNodeManager.writeFileLockStatus(FileLockNodeManager.java:329)
at org.apache.activemq.artemis.core.server.impl.FileLockNodeManager.setPaused(FileLockNodeManager.java:309)
at org.apache.activemq.artemis.core.server.impl.FileLockNodeManager.pauseLiveServer(FileLockNodeManager.java:269)
at org.apache.activemq.artemis.core.server.impl.LiveOnlyActivation.close(LiveOnlyActivation.java:104)
at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.stop(ActiveMQServerImpl.java:1393)
at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.stop(ActiveMQServerImpl.java:1192)
at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.stop(ActiveMQServerImpl.java:1176)
at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.stop(ActiveMQServerImpl.java:987)
at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.stop(ActiveMQServerImpl.java:981)
at org.apache.activemq.artemis.core.server.embedded.EmbeddedActiveMQ.stop(EmbeddedActiveMQ.java:168)
It's hard to say what exactly be happening without more details (e.g. logging from the broker), but in general behavior you're seeing is what I would expect. When the broker is unable to perform operations on the message store (e.g. file-system, database, etc.) this is considered a "critical" error and the broker will shut itself down. Once you restore connectivity you need to restart the broker, and there's no evidence that you're doing that.
You can register a callback with your embedded broker to be alerted when certain life-cycle events occur. Use org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl#registerActivateCallback and pass it an org.apache.activemq.artemis.core.server.ActivateCallback implementation. You could probably use this callback to restart the broker automatically in case of a failure of this kind.
Generally speaking I would recommend you use the file-based journal rather than JDBC as performance will be much better and it will be more robust.
I'm running a spring boot gradle project. Spring boot version is 2.3.2.RELEASE, MongoDB version is 4.0.5.
Per below site, retryable writes are enabled by default from MongoDB 3.6+ https://docs.mongodb.com/manual/core/retryable-writes/
I saw the following exceptions in the project, MongoReadException and MongoWriteException while reading/saving to the mongo database. I would like to know if these exceptions are automatically retried starting Spring boot 2.3.2.RELEASE.
Below are the exceptions I wanted to retry,
Caused by: org.springframework.data.mongodb.UncategorizedMongoDbException: Exception sending message; nested exception is com.mongodb.MongoSocketWriteException: Exception sending message
Caused by: org.springframework.data.mongodb.UncategorizedMongoDbException: Prematurely reached end of stream; nested exception is com.mongodb.MongoSocketReadException: Prematurely reached end of stream
To try it out, I created a spring boot demo app in my local that tries to save a document to a database whose uri is provided with invalid value in the properties file. I get MongoTimeoutException on save document which is as expected. However, I don't see any retries on the save operation.
I also tried to add the property retryWrites=true to the connection uri, it didn't work
spring.data.mongodb.uri=mongodb://XXX:XXXX#invalid-server:5555/dummyDB?&retrywrites=true
Does anyone know how to enable the retries ? How do I test it locally ?
Please help. Thanks in advance.
Below is the exception stacktrace with MongoSocketOpenException. I would like to test if this exception is auto retried, however, it's not.
nested exception is org.springframework.dao.DataAccessResourceFailureException: Timed out after 30000 ms while waiting to connect. Client view of cluster state is {type=UNKNOWN, servers=[{address=localhost:27017, type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoSocketOpenException: Exception opening socket}, caused by {java.net.ConnectException: Connection refused}}]; nested exception is com.mongodb.MongoTimeoutException: Timed out after 30000 ms while waiting to connect. Client view of cluster state is {type=UNKNOWN, servers=[{address=localhost:27017, type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoSocketOpenException: Exception opening socket}, caused by {java.net.ConnectException: Connection refused}}]] with root cause
com.mongodb.MongoTimeoutException: Timed out after 30000 ms while waiting to connect. Client view of cluster state is {type=UNKNOWN, servers=[{address=localhost:27017, type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoSocketOpenException: Exception opening socket}, caused by {java.net.ConnectException: Connection refused}}]
at com.mongodb.internal.connection.BaseCluster.getDescription(BaseCluster.java:177) ~[mongodb-driver-core-4.0.5.jar:na]
at com.mongodb.internal.connection.SingleServerCluster.getDescription(SingleServerCluster.java:41) ~[mongodb-driver-core-4.0.5.jar:na]
at com.mongodb.client.internal.MongoClientDelegate.getConnectedClusterDescription(MongoClientDelegate.java:147) ~[mongodb-driver-sync-4.0.5.jar:na]
at com.mongodb.client.internal.MongoClientDelegate.createClientSession(MongoClientDelegate.java:98) ~[mongodb-driver-sync-4.0.5.jar:na]
at com.mongodb.client.internal.MongoClientDelegate$DelegateOperationExecutor.getClientSession(MongoClientDelegate.java:278) ~[mongodb-driver-sync-4.0.5.jar:na]
at com.mongodb.client.internal.MongoClientDelegate$DelegateOperationExecutor.execute(MongoClientDelegate.java:202) ~[mongodb-driver-sync-4.0.5.jar:na]
at com.mongodb.client.internal.MongoCollectionImpl.executeSingleWriteRequest(MongoCollectionImpl.java:1008) ~[mongodb-driver-sync-4.0.5.jar:na]
at com.mongodb.client.internal.MongoCollectionImpl.executeInsertOne(MongoCollectionImpl.java:469) ~[mongodb-driver-sync-4.0.5.jar:na]
at com.mongodb.client.internal.MongoCollectionImpl.insertOne(MongoCollectionImpl.java:452) ~[mongodb-driver-sync-4.0.5.jar:na]
at com.mongodb.client.internal.MongoCollectionImpl.insertOne(MongoCollectionImpl.java:446) ~[mongodb-driver-sync-4.0.5.jar:na]
at org.springframework.data.mongodb.core.MongoTemplate.lambda$insertDocument$15(MongoTemplate.java:1442) ~[spring-data-mongodb-3.0.2.RELEASE.jar:3.0.2.RELEASE]
at org.springframework.data.mongodb.core.MongoTemplate.execute(MongoTemplate.java:566) ~[spring-data-mongodb-3.0.2.RELEASE.jar:3.0.2.RELEASE]
at org.springframework.data.mongodb.core.MongoTemplate.insertDocument(MongoTemplate.java:1436) ~[spring-data-mongodb-3.0.2.RELEASE.jar:3.0.2.RELEASE]
at org.springframework.data.mongodb.core.MongoTemplate.doInsert(MongoTemplate.java:1236) ~[spring-data-mongodb-3.0.2.RELEASE.jar:3.0.2.RELEASE]
at org.springframework.data.mongodb.core.MongoTemplate.insert(MongoTemplate.java:1168) ~[spring-data-mongodb-3.0.2.RELEASE.jar:3.0.2.RELEASE]
at org.springframework.data.mongodb.repository.support.SimpleMongoRepository.save(SimpleMongoRepository.java:85) ~[spring-data-mongodb-3.0.2.RELEASE.jar:3.0.2.RELEASE]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_101]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_101]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_101]
at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_101]
at org.springframework.data.repository.core.support.ImplementationInvocationMetadata.invoke(ImplementationInvocationMetadata.java:72) ~[spring-data-commons-2.3.2.RELEASE.jar:2.3.2.RELEASE]
at org.springframework.data.repository.core.support.RepositoryComposition$RepositoryFragments.invoke(RepositoryComposition.java:382) ~[spring-data-commons-2.3.2.RELEASE.jar:2.3.2.RELEASE]
at org.springframework.data.repository.core.support.RepositoryComposition.invoke(RepositoryComposition.java:205) ~[spring-data-commons-2.3.2.RELEASE.jar:2.3.2.RELEASE]
at org.springframework.data.repository.core.support.RepositoryFactorySupport$ImplementationMethodExecutionInterceptor.invoke(RepositoryFactorySupport.java:549) ~[spring-data-commons-2.3.2.RELEASE.jar:2.3.2.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) ~[spring-aop-5.2.8.RELEASE.jar:5.2.8.RELEASE]
at org.springframework.data.repository.core.support.QueryExecutorMethodInterceptor.doInvoke(QueryExecutorMethodInterceptor.java:155) ~[spring-data-commons-2.3.2.RELEASE.jar:2.3.2.RELEASE]
at org.springframework.data.repository.core.support.QueryExecutorMethodInterceptor.invoke(QueryExecutorMethodInterceptor.java:130) ~[spring-data-commons-2.3.2.RELEASE.jar:2.3.2.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) ~[spring-aop-5.2.8.RELEASE.jar:5.2.8.RELEASE]
at org.springframework.data.projection.DefaultMethodInvokingMethodInterceptor.invoke(DefaultMethodInvokingMethodInterceptor.java:80) ~[spring-data-commons-2.3.2.RELEASE.jar:2.3.2.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) ~[spring-aop-5.2.8.RELEASE.jar:5.2.8.RELEASE]
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:95) ~[spring-aop-5.2.8.RELEASE.jar:5.2.8.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) ~[spring-aop-5.2.8.RELEASE.jar:5.2.8.RELEASE]
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212) ~[spring-aop-5.2.8.RELEASE.jar:5.2.8.RELEASE]
at com.sun.proxy.$Proxy52.save(Unknown Source) ~[na:na]
Retries apply to operations like reads and writes.
Operations are only sent once the driver successfully connects to the server(s) you indicate in connection string/client configuration.
Your server is not accessible by the client therefore that connection isn't working therefore no operations can be performed therefore the concept of operation retries doesn't apply.
The monitoring is retried automatically, once you fix the connectivity issue between the client and the server (or start the server) the client will recognize this automatically.
I have downloaded the zip file for Apache Storm 2.2.0.
I imported the storm-starter maven project into eclipse in Windows 10. Did a Maven--> Update Project
My project did not have any errors. I did not do any updates to pom.xml.
I executed the WordCountTopology.java
I get the error NimbusLeaderNotFoundException.
I tried after stopping firewall also.
00:54:00.584 [main] INFO o.a.s.StormSubmitter - Generated ZooKeeper secret payload for MD5-digest: -xxxxxxxxxxxxxx
00:54:00.668 [main] WARN o.a.s.v.ConfigValidation - task.heartbeat.frequency.secs is a deprecated config please see class org.apache.storm.Config.TASK_HEARTBEAT_FREQUENCY_SECS for more information.
00:54:16.923 [main] WARN o.a.s.u.NimbusClient - Ignoring exception while trying to get leader nimbus info from localhost. will retry with a different seed host.
java.lang.RuntimeException: java.lang.RuntimeException: org.apache.storm.thrift.transport.TTransportException: java.net.ConnectException: Connection refused: connect
at org.apache.storm.security.auth.ThriftClient.reconnect(ThriftClient.java:108) ~[storm-client-2.2.0.jar:2.2.0]
at org.apache.storm.security.auth.ThriftClient.<init>(ThriftClient.java:69) ~[storm-client-2.2.0.jar:2.2.0]
at org.apache.storm.utils.NimbusClient.<init>(NimbusClient.java:80) ~[storm-client-2.2.0.jar:2.2.0]
at org.apache.storm.utils.NimbusClient.getConfiguredClientAs(NimbusClient.java:221) [storm-client-2.2.0.jar:2.2.0]
at org.apache.storm.utils.NimbusClient.getConfiguredClientAs(NimbusClient.java:179) [storm-client-2.2.0.jar:2.2.0]
at org.apache.storm.utils.NimbusClient.getConfiguredClient(NimbusClient.java:138) [storm-client-2.2.0.jar:2.2.0]
at org.apache.storm.blobstore.NimbusBlobStore.prepare(NimbusBlobStore.java:47) [storm-client-2.2.0.jar:2.2.0]
at org.apache.storm.utils.Utils.validateTopologyBlobStoreMap(Utils.java:1178) [storm-client-2.2.0.jar:2.2.0]
at org.apache.storm.StormSubmitter.validateConfs(StormSubmitter.java:530) [storm-client-2.2.0.jar:2.2.0]
at org.apache.storm.StormSubmitter.submitTopologyAs(StormSubmitter.java:236) [storm-client-2.2.0.jar:2.2.0]
at org.apache.storm.StormSubmitter.submitTopology(StormSubmitter.java:210) [storm-client-2.2.0.jar:2.2.0]
at org.apache.storm.StormSubmitter.submitTopology(StormSubmitter.java:173) [storm-client-2.2.0.jar:2.2.0]
at org.apache.storm.topology.ConfigurableTopology.submit(ConfigurableTopology.java:119) [storm-client-2.2.0.jar:2.2.0]
at org.apache.storm.starter.WordCountTopology.run(WordCountTopology.java:58) [classes/:?]
at org.apache.storm.topology.ConfigurableTopology.start(ConfigurableTopology.java:68) [storm-client-2.2.0.jar:2.2.0]
at org.apache.storm.starter.WordCountTopology.main(WordCountTopology.java:36) [classes/:?]
Caused by: java.lang.RuntimeException: org.apache.storm.thrift.transport.TTransportException: java.net.ConnectException: Connection refused: connect
at org.apache.storm.security.auth.TBackoffConnect.retryNext(TBackoffConnect.java:59) ~[storm-client-2.2.0.jar:2.2.0]
at org.apache.storm.security.auth.TBackoffConnect.doConnectWithRetry(TBackoffConnect.java:51) ~[storm-client-2.2.0.jar:2.2.0]
at org.apache.storm.security.auth.ThriftClient.reconnect(ThriftClient.java:98) ~[storm-client-2.2.0.jar:2.2.0]
... 15 more
Caused by: org.apache.storm.thrift.transport.TTransportException: java.net.ConnectException: Connection refused: connect
at org.apache.storm.thrift.transport.TSocket.open(TSocket.java:226) ~[storm-shaded-deps-2.2.0.jar:2.2.0]
at org.apache.storm.thrift.transport.TFramedTransport.open(TFramedTransport.java:91) ~[storm-shaded-deps-2.2.0.jar:2.2.0]
at org.apache.storm.security.auth.SimpleTransportPlugin.connect(SimpleTransportPlugin.java:101) ~[storm-client-2.2.0.jar:2.2.0]
at org.apache.storm.security.auth.TBackoffConnect.doConnectWithRetry(TBackoffConnect.java:48) ~[storm-client-2.2.0.jar:2.2.0]
at org.apache.storm.security.auth.ThriftClient.reconnect(ThriftClient.java:98) ~[storm-client-2.2.0.jar:2.2.0]
... 15 more
Caused by: java.net.ConnectException: Connection refused: connect
at java.net.DualStackPlainSocketImpl.waitForConnect(Native Method) ~[?:1.8.0_241]
at java.net.DualStackPlainSocketImpl.socketConnect(DualStackPlainSocketImpl.java:85) ~[?:1.8.0_241]
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[?:1.8.0_241]
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[?:1.8.0_241]
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[?:1.8.0_241]
at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:172) ~[?:1.8.0_241]
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[?:1.8.0_241]
at java.net.Socket.connect(Socket.java:606) ~[?:1.8.0_241]
at org.apache.storm.thrift.transport.TSocket.open(TSocket.java:221) ~[storm-shaded-deps-2.2.0.jar:2.2.0]
at org.apache.storm.thrift.transport.TFramedTransport.open(TFramedTransport.java:91) ~[storm-shaded-deps-2.2.0.jar:2.2.0]
at org.apache.storm.security.auth.SimpleTransportPlugin.connect(SimpleTransportPlugin.java:101) ~[storm-client-2.2.0.jar:2.2.0]
at org.apache.storm.security.auth.TBackoffConnect.doConnectWithRetry(TBackoffConnect.java:48) ~[storm-client-2.2.0.jar:2.2.0]
at org.apache.storm.security.auth.ThriftClient.reconnect(ThriftClient.java:98) ~[storm-client-2.2.0.jar:2.2.0]
... 15 more
org.apache.storm.utils.NimbusLeaderNotFoundException: Could not find leader nimbus from seed hosts [localhost]. Did you specify a valid list of nimbus hosts for config nimbus.seeds?
at org.apache.storm.utils.NimbusClient.getConfiguredClientAs(NimbusClient.java:250)
at org.apache.storm.utils.NimbusClient.getConfiguredClientAs(NimbusClient.java:179)
at org.apache.storm.utils.NimbusClient.getConfiguredClient(NimbusClient.java:138)
at org.apache.storm.blobstore.NimbusBlobStore.prepare(NimbusBlobStore.java:47)
at org.apache.storm.utils.Utils.validateTopologyBlobStoreMap(Utils.java:1178)
at org.apache.storm.StormSubmitter.validateConfs(StormSubmitter.java:530)
at org.apache.storm.StormSubmitter.submitTopologyAs(StormSubmitter.java:236)
at org.apache.storm.StormSubmitter.submitTopology(StormSubmitter.java:210)
at org.apache.storm.StormSubmitter.submitTopology(StormSubmitter.java:173)
at org.apache.storm.topology.ConfigurableTopology.submit(ConfigurableTopology.java:119)
at org.apache.storm.starter.WordCountTopology.run(WordCountTopology.java:58)
at org.apache.storm.topology.ConfigurableTopology.start(ConfigurableTopology.java:68)
at org.apache.storm.starter.WordCountTopology.main(WordCountTopology.java:36)
Same Eclipse in Windows 10, Under a different eclipse project, For previous storm versions' code (Version 0.0.1) Zookeeper starts, nimbus starts fine. I am able to get the output.
I did not do any cluster configuration in local/eclipse.Success Topology with version 0.0.1
I'm getting following error while trying to connect to mongodb cluster - ReplicaSet (non-dockerized) deployed on three different server via a dockerized spring boot application.
2020-04-06 19:40:06.263 INFO 1 --- [ main] org.mongodb.driver.cluster : Cluster created with settings {hosts=[196.12.7.156:27017, 196.12.7.157:27017, 196.12.7.158:27017], mode=MULTIPLE, requiredClusterType=UNKNOWN, serverSelectionTimeout='30000 ms', maxWaitQueueSize=500}
2020-04-06 19:40:06.263 INFO 1 --- [ main] org.mongodb.driver.cluster : Adding discovered server 196.12.7.156:27017 to client view of cluster
2020-04-06 19:40:06.301 INFO 1 --- [ main] org.mongodb.driver.cluster : Adding discovered server 196.12.7.157:27017 to client view of cluster
2020-04-06 19:40:06.304 INFO 1 --- [ main] org.mongodb.driver.cluster : Adding discovered server 196.12.7.158:27017 to client view of cluster
2020-04-06 19:40:06.424 INFO 1 --- [.16.8.179:27017] org.mongodb.driver.connection : Opened connection [connectionId{localValue:3, serverValue:33289}] to 196.12.7.158:27017
2020-04-06 19:40:06.427 INFO 1 --- [.16.8.179:27017] org.mongodb.driver.cluster : Monitor thread successfully connected to server with description ServerDescription{address=196.12.7.158:27017, type=REPLICA_SET_SECONDARY, state=CONNECTED, ok=true, version=ServerVersion{versionList=[3, 2, 11]}, minWireVersion=0, maxWireVersion=4, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=null, roundTripTimeNanos=1686431, setName='rs0', canonicalAddress=seabatch3:27017, hosts=[seabatch3:27017, seabatch1:27017, seabatch2:27017], passives=[], arbiters=[], primary='seabatch1:27017', tagSet=TagSet{[]}, electionId=null, setVersion=6, lastWriteDate=null, lastUpdateTimeNanos=2764696590071705}
2020-04-06 19:40:06.433 INFO 1 --- [.16.8.177:27017] org.mongodb.driver.connection : Opened connection [connectionId{localValue:1, serverValue:75063}] to 196.12.7.156:27017
2020-04-06 19:40:06.434 INFO 1 --- [.16.8.178:27017] org.mongodb.driver.connection : Opened connection [connectionId{localValue:2, serverValue:24932}] to 196.12.7.157:27017
2020-04-06 19:40:06.435 INFO 1 --- [.16.8.179:27017] org.mongodb.driver.cluster : Discovered cluster type of REPLICA_SET
2020-04-06 19:40:06.438 INFO 1 --- [.16.8.177:27017] org.mongodb.driver.cluster : Monitor thread successfully connected to server with description ServerDescription{address=196.12.7.156:27017, type=REPLICA_SET_PRIMARY, state=CONNECTED, ok=true, version=ServerVersion{versionList=[3, 2, 11]}, minWireVersion=0, maxWireVersion=4, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=null, roundTripTimeNanos=4744734, setName='rs0', canonicalAddress=seabatch1:27017, hosts=[seabatch3:27017, seabatch1:27017, seabatch2:27017], passives=[], arbiters=[], primary='seabatch1:27017', tagSet=TagSet{[]}, electionId=7fffffff0000000000000061, setVersion=6, lastWriteDate=null, lastUpdateTimeNanos=2764696601428299}
2020-04-06 19:40:06.438 INFO 1 --- [.16.8.178:27017] org.mongodb.driver.cluster : Monitor thread successfully connected to server with description ServerDescription{address=196.12.7.157:27017, type=REPLICA_SET_SECONDARY, state=CONNECTED, ok=true, version=ServerVersion{versionList=[3, 2, 11]}, minWireVersion=0, maxWireVersion=4, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=null, roundTripTimeNanos=3873250, setName='rs0', canonicalAddress=seabatch2:27017, hosts=[seabatch3:27017, seabatch1:27017, seabatch2:27017], passives=[], arbiters=[], primary='seabatch1:27017', tagSet=TagSet{[]}, electionId=null, setVersion=6, lastWriteDate=null, lastUpdateTimeNanos=2764696602199597}
2020-04-06 19:40:06.441 INFO 1 --- [.16.8.179:27017] org.mongodb.driver.cluster : Adding discovered server seabatch3:27017 to client view of cluster
2020-04-06 19:40:06.447 INFO 1 --- [.16.8.179:27017] org.mongodb.driver.cluster : Adding discovered server seabatch1:27017 to client view of cluster
2020-04-06 19:40:06.455 INFO 1 --- [.16.8.179:27017] org.mongodb.driver.cluster : Adding discovered server seabatch2:27017 to client view of cluster
2020-04-06 19:40:06.461 INFO 1 --- ['}-seabatch3:27017] org.mongodb.driver.cluster : Exception in monitor thread while connecting to server seabatch3:27017
com.mongodb.MongoSocketException: seabatch3: Name does not resolve
at com.mongodb.ServerAddress.getSocketAddresses(ServerAddress.java:211) ~[mongodb-driver-core-3.11.2.jar!/:na]
at com.mongodb.internal.connection.SocketStream.initializeSocket(SocketStream.java:75) ~[mongodb-driver-core-3.11.2.jar!/:na]
at com.mongodb.internal.connection.SocketStream.open(SocketStream.java:65) ~[mongodb-driver-core-3.11.2.jar!/:na]
at com.mongodb.internal.connection.InternalStreamConnection.open(InternalStreamConnection.java:128) ~[mongodb-driver-core-3.11.2.jar!/:na]
at com.mongodb.internal.connection.DefaultServerMonitor$ServerMonitorRunnable.run(DefaultServerMonitor.java:117) ~[mongodb-driver-core-3.11.2.jar!/:na]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_212]
Caused by: java.net.UnknownHostException: seabatch3: Name does not resolve
at java.net.Inet4AddressImpl.lookupAllHostAddr(Native Method) ~[na:1.8.0_212]
at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:929) ~[na:1.8.0_212]
at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1324) ~[na:1.8.0_212]
at java.net.InetAddress.getAllByName0(InetAddress.java:1277) ~[na:1.8.0_212]
at java.net.InetAddress.getAllByName(InetAddress.java:1193) ~[na:1.8.0_212]
at java.net.InetAddress.getAllByName(InetAddress.java:1127) ~[na:1.8.0_212]
at com.mongodb.ServerAddress.getSocketAddresses(ServerAddress.java:203) ~[mongodb-driver-core-3.11.2.jar!/:na]
... 5 common frames omitted
2020-04-06 19:40:06.466 INFO 1 --- ['}-seabatch1:27017] org.mongodb.driver.cluster : Exception in monitor thread while connecting to server seabatch1:27017
com.mongodb.MongoSocketException: seabatch1: Name does not resolve
at com.mongodb.ServerAddress.getSocketAddresses(ServerAddress.java:211) ~[mongodb-driver-core-3.11.2.jar!/:na]
at com.mongodb.internal.connection.SocketStream.initializeSocket(SocketStream.java:75) ~[mongodb-driver-core-3.11.2.jar!/:na]
at com.mongodb.internal.connection.SocketStream.open(SocketStream.java:65) ~[mongodb-driver-core-3.11.2.jar!/:na]
at com.mongodb.internal.connection.InternalStreamConnection.open(InternalStreamConnection.java:128) ~[mongodb-driver-core-3.11.2.jar!/:na]
at com.mongodb.internal.connection.DefaultServerMonitor$ServerMonitorRunnable.run(DefaultServerMonitor.java:117) ~[mongodb-driver-core-3.11.2.jar!/:na]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_212]
Caused by: java.net.UnknownHostException: seabatch1: Name does not resolve
at java.net.Inet4AddressImpl.lookupAllHostAddr(Native Method) ~[na:1.8.0_212]
at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:929) ~[na:1.8.0_212]
at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1324) ~[na:1.8.0_212]
at java.net.InetAddress.getAllByName0(InetAddress.java:1277) ~[na:1.8.0_212]
at java.net.InetAddress.getAllByName(InetAddress.java:1193) ~[na:1.8.0_212]
at java.net.InetAddress.getAllByName(InetAddress.java:1127) ~[na:1.8.0_212]
at com.mongodb.ServerAddress.getSocketAddresses(ServerAddress.java:203) ~[mongodb-driver-core-3.11.2.jar!/:na]
... 5 common frames omitted
2020-04-06 19:40:06.466 INFO 1 --- [.16.8.179:27017] org.mongodb.driver.cluster : Canonical address seabatch3:27017 does not match server address. Removing 196.12.7.158:27017 from client view of cluster
2020-04-06 19:40:06.474 INFO 1 --- [.16.8.178:27017] org.mongodb.driver.cluster : Canonical address seabatch2:27017 does not match server address. Removing 196.12.7.157:27017 from client view of cluster
2020-04-06 19:40:06.474 INFO 1 --- ['}-seabatch2:27017] org.mongodb.driver.cluster : Exception in monitor thread while connecting to server seabatch2:27017
com.mongodb.MongoSocketException: seabatch2: Name does not resolve
at com.mongodb.ServerAddress.getSocketAddresses(ServerAddress.java:211) ~[mongodb-driver-core-3.11.2.jar!/:na]
at com.mongodb.internal.connection.SocketStream.initializeSocket(SocketStream.java:75) ~[mongodb-driver-core-3.11.2.jar!/:na]
at com.mongodb.internal.connection.SocketStream.open(SocketStream.java:65) ~[mongodb-driver-core-3.11.2.jar!/:na]
at com.mongodb.internal.connection.InternalStreamConnection.open(InternalStreamConnection.java:128) ~[mongodb-driver-core-3.11.2.jar!/:na]
at com.mongodb.internal.connection.DefaultServerMonitor$ServerMonitorRunnable.run(DefaultServerMonitor.java:117) ~[mongodb-driver-core-3.11.2.jar!/:na]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_212]
Caused by: java.net.UnknownHostException: seabatch2: Name does not resolve
at java.net.Inet4AddressImpl.lookupAllHostAddr(Native Method) ~[na:1.8.0_212]
at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:929) ~[na:1.8.0_212]
at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1324) ~[na:1.8.0_212]
at java.net.InetAddress.getAllByName0(InetAddress.java:1277) ~[na:1.8.0_212]
at java.net.InetAddress.getAllByName(InetAddress.java:1193) ~[na:1.8.0_212]
at java.net.InetAddress.getAllByName(InetAddress.java:1127) ~[na:1.8.0_212]
at com.mongodb.ServerAddress.getSocketAddresses(ServerAddress.java:203) ~[mongodb-driver-core-3.11.2.jar!/:na]
... 5 common frames omitted
Initially, it connects to the mongodb cluster and get the hostnames and then application fails to start.
My guess is that since app deployed in docker container, docker is assume that mongodb is also deployed in another container and trying to find it via container names.
Any suggestion on how I can resolve this issue would help alot.
Thank you!
In the replica set configuration, each node is referred by some host name. These host names need to be resolvable and accessible from your application.
It sounds like you are using names that make sense on the host but are not resolvable in your container. From your container you have some other host name you are using in your connection string. The initial connection then succeeds but the driver then reconnects to each RS member using the names defined in the RS configuration, and this fails.
You need to either change RS configuration to use names that are resolvable both from the host and from the container, or make your existing names resolvable and accessible from the container.
"Accessible" means if the names resolve to different IP addresses in different environments, those IPs must route to the nodes in question.