getting below error while connect and close using Mysqsh with socket file for every min.
Details:
mysql version: 8.0.32
router version: 8.0.32
shell version: 8.0.32
OS version: centos 7
error in router log:
2023-02-09 16:00:01 routing INFO [7f258c18c700] [routing:bootstrap_rw] incrementing error counter for host of /opt/mysqlrouter/mysql.sock (now 0)
2023-02-09 16:00:01 routing ERROR [7f258c18c700] classic::loop() processor failed: Bad message (generic:74)
2023-02-09 16:02:01 routing INFO [7f258b169700] [routing:bootstrap_rw] /opt/mysqlrouter/mysql.sock closed connection before finishing handshake
2023-02-09 16:02:01 routing INFO [7f258b169700] [routing:bootstrap_rw] incrementing error counter for host of /opt/mysqlrouter/mysql.sock (now 0)
2023-02-09 16:02:01 routing ERROR [7f258b169700] classic::loop() processor failed: Bad message (generic:74)
2023-02-09 16:04:01 routing INFO [7f258a167700] [routing:bootstrap_rw] /opt/mysqlrouter/mysql.sock closed connection before finishing handshake
2023-02-09 16:04:01 routing INFO [7f258a167700] [routing:bootstrap_rw] incrementing error counter for host of /opt/mysqlrouter/mysql.sock (now 0)
2023-02-09 16:04:01 routing ERROR [7f258a167700] classic::loop() processor failed: Bad message (generic:74)
2023-02-09 16:06:01 routing INFO [7f2589144700] [routing:bootstrap_rw] /opt/mysqlrouter/mysql.sock closed connection before finishing handshake
2023-02-09 16:06:01 routing INFO [7f2589144700] [routing:bootstrap_rw] incrementing error counter for host of /opt/mysqlrouter/mysql.sock (now 0)
2023-02-09 16:06:01 routing ERROR [7f2589144700] classic::loop() processor failed: Bad message (generic:74)
2023-02-09 16:08:02 routing INFO [7f258c18c700] [routing:bootstrap_rw] /opt/mysqlrouter/mysql.sock closed connection before finishing handshake
2023-02-09 16:08:02 routing INFO [7f258c18c700] [routing:bootstrap_rw] incrementing error counter for host of /opt/mysqlrouter/mysql.sock (now 0)
2023-02-09 16:08:02 routing ERROR [7f258c18c700] classic::loop() processor failed: Bad message (generic:74)
Why this error getting ? is any way to fix this error ?
Related
I have setup for 3 organisations on Kubernetes cluster and it is giving me following error when I restart peer node pod and tried command -> peer channel list on bash shell,
[comm.tls] ClientHandshake -> ERRO 026 Client TLS handshake failed after 2.997205009s with error: context canceled remoteaddress=10.0.94.178:7051
[grpc] WarningDepth -> DEBU 027 [core]grpc: addrConn.createTransport failed to connect to {peer0-org1:7051 peer0-org1:7051 <nil> 0 <nil>}. Err: connection error: desc = "transport: authentication handshake failed: context canceled". Reconnecting...
Error: error getting endorser client for channel: endorser client failed to connect to peer0-org1:7051: failed to create new connection: context deadline exceeded
After some time (nearly 10-15 minutes) if I tried same command,
[comm.tls] ClientHandshake -> DEBU 024 Client TLS handshake completed in 1.48399ms remoteaddress=10.0.94.178:7051
[grpc] InfoDepth -> DEBU 025 [core]Subchannel Connectivity change to READY
It seems working and give me channel list. I am not able to find out what is reason behind this, Please help for this.
I have Artemis configuration (shared storage) with following ha-policy (for master and backup):
<ha-policy>
<shared-store>
<master>
<failover-on-shutdown>true</failover-on-shutdown>
</master>
</shared-store>
</ha-policy>
<ha-policy>
<shared-store>
<slave>
<failover-on-shutdown>true</failover-on-shutdown>
</slave>
</shared-store>
</ha-policy>
Client connection string:
(tcp://master:61616,tcp://backup:61616)?ha=true&retryInterval=1000&retryIntervalMultiplier=1.0&reconnectAttempts=10
At ~18:38 the server crashed, then at ~18:48 it recovered.
Some applications were unable to reconnect correctly without restarting with following errors...
APP 1
Master node crash:
2020-08-06 18:38:37,873 [Thread-0 (ActiveMQ-client-global-threads)] WARN org.apache.activemq.artemis.core.client - AMQ212037: Connection failure has been detected: AMQ119015: The connection was disconnected because of server shutdown [code=DISCONNECTED]
Master Node now active (the backup went into passive mode):
2020-08-06 18:47:50,949 [Thread-1 (ActiveMQ-client-global-threads)] WARN org.apache.activemq.artemis.core.client - AMQ212037: Connection failure has been detected: AMQ119015: The connection was disconnected because of server shutdown [code=DISCONNECTED]
Why? Other applications reconnected correctly
2020-08-06 18:47:59,994 [Thread-1 (ActiveMQ-client-global-threads)] WARN org.apache.activemq.artemis.core.client - AMQ212005: Tried 10 times to connect. Now giving up on reconnecting it.
2020-08-06 18:47:59,998 [Camel (camel-1) thread #4 - JmsConsumer[xxx]] WARN org.apache.camel.component.jms.DefaultJmsMessageListenerContainer - Setup of JMS message listener invoker failed for destination 'xxx' - trying to recover. Cause: Session is closed
2020-08-06 18:47:59,999 [Camel (camel-1) thread #4 - JmsConsumer[xxx]] INFO org.apache.camel.component.jms.DefaultJmsMessageListenerContainer - Successfully refreshed JMS Connection
2020-08-06 18:48:00,006 [Camel (camel-1) thread #3 - JmsConsumer[xxx]] WARN org.apache.camel.component.jms.DefaultJmsMessageListenerContainer - Setup of JMS message listener invoker failed for destination 'xxx' - trying to recover. Cause: Session is closed
This error this error was not fixed although the cluster was recovered:
2020-08-06 18:49:25,033 [Camel (camel-1) thread #5 - JmsConsumer[xxx]] INFO org.apache.camel.component.jms.DefaultJmsMessageListenerContainer - Successfully refreshed JMS Connection
2020-08-06 18:49:25,033 [Camel (camel-1) thread #7 - JmsConsumer[xxx]] WARN org.apache.camel.component.jms.DefaultJmsMessageListenerContainer - Setup of JMS message listener invoker failed for destination 'xxx' - trying to recover. Cause: AMQ119010: Connection is destroyed
APP 2
Master node crash:
2020-08-06 18:38:37.883 WARN 1 --- [Thread-1 (ActiveMQ-client-global-threads)] org.apache.activemq.artemis.core.client : AMQ212037: Connection failure to master/master:61616 has been detected: AMQ219015: The connection was disconnected because of server shutdown [code=DISCONNECTED]
2020-08-06 18:38:46.935 WARN 1 --- [Thread-1 (ActiveMQ-client-global-threads)] org.apache.activemq.artemis.core.client : AMQ212005: Tried 10 times to connect. Now giving up on reconnecting it.
2020-08-06 18:38:46.939 WARN 1 --- [DefaultMessageListenerContainer-1] o.s.j.l.DefaultMessageListenerContainer : Setup of JMS message listener invoker failed for destination 'yyyy' - trying to recover. Cause: Session is closed
2020-08-06 18:38:46.945 WARN 1 --- [DefaultMessageListenerContainer-1] o.s.j.l.DefaultMessageListenerContainer : Setup of JMS message listener invoker failed for destination 'yyyy' - trying to recover. Cause: Session is closed
2020-08-06 18:38:46.963 INFO 1 --- [Thread-7] o.s.j.c.SingleConnectionFactory : Encountered a JMSException - resetting the underlying JMS Connection
javax.jms.JMSException: ActiveMQDisconnectedException[errorType=DISCONNECTED message=AMQ219015: The connection was disconnected because of server shutdown]
at org.apache.activemq.artemis.jms.client.ActiveMQConnection$JMSFailureListener.connectionFailed(ActiveMQConnection.java:750) ~[artemis-jms-client-2.10.1.jar!/:2.10.1]
at org.apache.activemq.artemis.jms.client.ActiveMQConnection$JMSFailureListener.connectionFailed(ActiveMQConnection.java:771) ~[artemis-jms-client-2.10.1.jar!/:2.10.1]
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.callSessionFailureListeners(ClientSessionFactoryImpl.java:704) ~[artemis-core-client-2.10.1.jar!/:2.10.1]
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.failoverOrReconnect(ClientSessionFactoryImpl.java:640) ~[artemis-core-client-2.10.1.jar!/:2.10.1]
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.handleConnectionFailure(ClientSessionFactoryImpl.java:507) ~[artemis-core-client-2.10.1.jar!/:2.10.1]
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.access$600(ClientSessionFactoryImpl.java:73) ~[artemis-core-client-2.10.1.jar!/:2.10.1]
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$DelegatingFailureListener.connectionFailed(ClientSessionFactoryImpl.java:1229) ~[artemis-core-client-2.10.1.jar!/:2.10.1]
at org.apache.activemq.artemis.spi.core.protocol.AbstractRemotingConnection.callFailureListeners(AbstractRemotingConnection.java:77) ~[artemis-core-client-2.10.1.jar!/:2.10.1]
at org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.fail(RemotingConnectionImpl.java:220) ~[artemis-core-client-2.10.1.jar!/:2.10.1]
at org.apache.activemq.artemis.spi.core.protocol.AbstractRemotingConnection.fail(AbstractRemotingConnection.java:220) ~[artemis-core-client-2.10.1.jar!/:2.10.1]
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$CloseRunnable.run(ClientSessionFactoryImpl.java:1018) ~[artemis-core-client-2.10.1.jar!/:2.10.1]
at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:42) ~[artemis-commons-2.10.1.jar!/:2.10.1]
at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:31) ~[artemis-commons-2.10.1.jar!/:2.10.1]
at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:66) ~[artemis-commons-2.10.1.jar!/:2.10.1]
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[na:na]
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[na:na]
at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) ~[artemis-commons-2.10.1.jar!/:2.10.1]
Caused by: org.apache.activemq.artemis.api.core.ActiveMQDisconnectedException: AMQ219015: The connection was disconnected because of server shutdown
... 7 common frames omitted
2020-08-06 18:38:51.945 WARN 1 --- [DefaultMessageListenerContainer-2] o.s.j.l.DefaultMessageListenerContainer : Setup of JMS message listener invoker failed for destination 'xxx' - trying to recover. Cause: AMQ219010: Connection is destroyed
2020-08-06 18:39:21.965 ERROR 1 --- [DefaultMessageListenerContainer-2] o.s.j.l.DefaultMessageListenerContainer : Could not refresh JMS Connection for destination 'yyyy' - retrying using FixedBackOff{interval=5000, currentAttempts=0, maxAttempts=unlimited}. Cause: Failed to create session factory; nested exception is ActiveMQConnectionTimedOutException[errorType=CONNECTION_TIMEDOUT message=AMQ219013: Timed out waiting to receive cluster topology. Group:null]
2020-08-06 18:39:52.983 ERROR 1 --- [DefaultMessageListenerContainer-2] o.s.j.l.DefaultMessageListenerContainer : Could not refresh JMS Connection for destination 'yyy' - retrying using FixedBackOff{interval=5000, currentAttempts=0, maxAttempts=unlimited}. Cause: Failed to create session factory; nested exception is ActiveMQConnectionTimedOutException[errorType=CONNECTION_TIMEDOUT message=AMQ219013: Timed out waiting to receive cluster topology. Group:null]
2020-08-06 18:40:23.992 ERROR 1 --- [DefaultMessageListenerContainer-2] o.s.j.l.DefaultMessageListenerContainer : Could not refresh JMS Connection for destination 'yyyy' - retrying using FixedBackOff{interval=5000, currentAttempts=1, maxAttempts=unlimited}. Cause: Failed to create session factory; nested exception is ActiveMQConnectionTimedOutException[errorType=CONNECTION_TIMEDOUT message=AMQ219013: Timed out waiting to receive cluster topology. Group:null]
Master Node now active (the backup went into passive mode):
2020-08-06 18:47:50.949 WARN 1 --- [Thread-5 (ActiveMQ-client-global-threads)] org.apache.activemq.artemis.core.client : AMQ212037: Connection failure to backup/backup:61616 has been detected: AMQ219015: The connection was disconnected because of server shutdown [code=DISCONNECTED]
2020-08-06 18:47:53.145 ERROR 1 --- [DefaultMessageListenerContainer-2] o.s.j.l.DefaultMessageListenerContainer : Could not refresh JMS Connection for destination 'yyyy' - retrying using FixedBackOff{interval=5000, currentAttempts=8, maxAttempts=unlimited}. Cause: Failed to create session factory; nested exception is ActiveMQConnectionTimedOutException[errorType=CONNECTION_TIMEDOUT message=AMQ219013: Timed out waiting to receive cluster topology. Group:null]
2020-08-06 18:47:53.146 WARN 1 --- [Thread-5 (ActiveMQ-client-global-threads)] org.apache.activemq.artemis.core.client : AMQ212004: Failed to connect to server.
This error this error was not fixed although the cluster was recovered:
2020-08-06 18:47:58.147 ERROR 1 --- [DefaultMessageListenerContainer-2] o.s.j.l.DefaultMessageListenerContainer : Could not refresh JMS Connection for destination 'yyyy' - retrying using FixedBackOff{interval=5000, currentAttempts=9, maxAttempts=unlimited}. Cause: Failed to create session factory; nested exception is ActiveMQIllegalStateException[errorType=ILLEGAL_STATE message=AMQ219024: Could not select a TransportConfiguration to create SessionFactory]
2020-08-06 18:48:53.160 ERROR 1 --- [DefaultMessageListenerContainer-2] o.s.j.l.DefaultMessageListenerContainer : Could not refresh JMS Connection for destination 'yyyy' - retrying using FixedBackOff{interval=5000, currentAttempts=20, maxAttempts=unlimited}. Cause: Failed to create session factory; nested exception is ActiveMQIllegalStateException[errorType=ILLEGAL_STATE message=AMQ219024: Could not select a TransportConfiguration to create SessionFactory]
I tried to reproduce the error but could not. Maybe there are ideas for the correct setting?
Web based application intermittently throwing '500 Internal Server Error'. This application is deployed on WAS 8.0 server with IHS and WAS servers hosted on different machines. Please find below the TRACE level plugin logs at IHS end.
DEBUG: ws_common: websphereExecute: Wrote the request; reading the response (timeout 900)
DETAIL: lib_htresponse: htresponseRead: Reading the response: 64040a70 TRACE: lib_rio: Blocking for read, waiting 900
DEBUG: lib_rio.c line 896 : Read failed, rc=104
DEBUG: lib_htresponse: htresponseSetError: Setting the error to: |READ_FAILED|(1, Line: 713)
DEBUG: ws_common: websphereExecute: Failed to read from an old stream; probably Keep-Alive timeout fired
DEBUG: ws_transport: transportStreamDequeue: Checking for existing stream from the queue
DEBUG: lib_stream: destroyStream: Destroying the stream
DEBUG: lib_rio: rclose: socket 13 closed
STATS: ws_server: serverSetFailoverStatus: Server xxxxx : pendingRequests 0 failedRequests 7 affinityRequests 57 totalRequests 51.
ERROR: ws_common: websphereHandleRequest: Failed to execute the transaction to 'xxxxx' on host 'yyyyy'; will try another one
ERROR: ws_common: websphereWriteRequestReadResponse: Failed to find an app server to handle this request
ERROR: ESI: getResponse: failed to get response: rc = 10 DEBUG: ESI: esiHandleRequest: failed to get response
DEBUG: ESI: esiRequestUrlStackDestroy
DEBUG: ESI: cacheURL: '/site/index'
DETAIL: ESI: esiRequestPopUrl: '/site/index'
DEBUG: ESI: esiUrlDestroy: '/site/index'
ERROR: [://site/index] ws_common: websphereHandleRequest: Failed to handle request rc=10
DEBUG: ws_common: websphereCloseConnection
DETAIL: ws_common: websphereEndRequest: Ending the request
Please help me to get the issue resolved. Thanks in advance
rc=104 is the linux return code indicating that the socket was reset by the partner address so you need to look at the application server for errors.
I used Curator to newClient, and set retry policy, but if the connection string is wrong, retry connected will always running until out of memory, but I want the program exit once retry 3 times.
RetryPolicy retryPolicy = new ExponentialBackoffRetry(1000, 3);
CuratorFrameworkFactory.newClient(zkAddress,retryPolicy);
CuratorFrameworkFactory.builder().connectString(zkAddress)
.sessionTimeoutMs(5000).connectionTimeoutMs(12000)
.retryPolicy(retryPolicy).build();
09:03:18.810 [main-SendThread(0:0:0:0:0:0:0:1:2181)] INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server 0:0:0:0:0:0:0:1/0:0:0:0:0:0:0:1:2181. Will not attempt to authenticate using SASL (unknown error)
09:03:19.825 [main-SendThread(0:0:0:0:0:0:0:1:2181)] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused: no further information
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(Unknown Source)
at o rg.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
I am running 3 instances of ZooKeeper and the config is this:
tickTime=2000
initLimit=10
syncLimit=5
dataDir=/tmp/zookeeper1
clientPort=2181
maxClientCnxns=1000
server.1=127.0.0.1:2888:3888
server.2=127.0.0.1:2889:3889
server.3=127.0.0.1:2890:3890
I am using the leader election example code given here:
https://git-wip-us.apache.org/repos/asf?p=curator.git;a=tree;f=curator-examples/src/main/java/leader;h=73b547eadb98995c0ccbd06a5b76d0741ffef263;hb=HEAD
The code runs fine with TestingServer but when I change connection string to : "127.0.0.1:2181,127.0.0.1:2182,127.0.0.1:2183", I get the exceptions:
[main-SendThread(127.0.0.1:2183)] INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server 127.0.0.1/127.0.0.1:2183. Will not attempt to authenticate using SASL (unknown error)
[main-SendThread(127.0.0.1:2183)] INFO org.apache.zookeeper.ClientCnxn - Socket connection established, initiating session, client: /127.0.0.1:56111, server: 127.0.0.1/127.0.0.1:2183
[main-SendThread(127.0.0.1:2183)] INFO org.apache.zookeeper.ClientCnxn - Session establishment complete on server 127.0.0.1/127.0.0.1:2183, sessionid = 0x3521552283c0000, negotiated timeout = 40000
[main-EventThread] INFO org.apache.curator.framework.state.ConnectionStateManager - State change: CONNECTED
[main-SendThread(127.0.0.1:2183)] INFO org.apache.zookeeper.ClientCnxn - Unable to read additional data from server sessionid 0x3521552283c0000, likely server has closed socket, closing socket connection and attempting reconnect
[main-EventThread] INFO org.apache.curator.framework.imps.EnsembleTracker - New config event received: null
[main-EventThread] ERROR org.apache.curator.framework.imps.CuratorFrameworkImpl - Background exception was not retry-able or retry gave up
java.lang.NullPointerException
at java.io.ByteArrayInputStream.<init>(ByteArrayInputStream.java:106)
at org.apache.curator.framework.imps.EnsembleTracker.processConfigData(EnsembleTracker.java:163)
at org.apache.curator.framework.imps.EnsembleTracker.access$200(EnsembleTracker.java:48)
at org.apache.curator.framework.imps.EnsembleTracker$2.processResult(EnsembleTracker.java:134)
at org.apache.curator.framework.imps.CuratorFrameworkImpl.sendToBackgroundCallback(CuratorFrameworkImpl.java:829)
at org.apache.curator.framework.imps.CuratorFrameworkImpl.processBackgroundOperation(CuratorFrameworkImpl.java:611)
at org.apache.curator.framework.imps.WatcherRemovalFacade.processBackgroundOperation(WatcherRemovalFacade.java:151)
at org.apache.curator.framework.imps.GetConfigBuilderImpl$2.processResult(GetConfigBuilderImpl.java:210)
at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:619)
at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:528)
[main-EventThread] INFO org.apache.curator.framework.state.ConnectionStateManager - State change: SUSPENDED
What could be the issue?
I am hitting the same issue. I think it might be related to the Zookeeper 3.5.1 ClientCnxn. Even though I return back to curator 2.6.0, I still see the same stack trace. A GET_CONFIG event type is sent without the event data.
My stack trace looks like this:
org.apache.curator.framework.imps.CuratorFrameworkImpl: Background exception was not retry-able or retry gave up
! java.lang.NullPointerException: null
! at java.io.ByteArrayInputStream.(ByteArrayInputStream.java:106)
! at org.apache.curator.framework.imps.EnsembleTracker.processConfigData(EnsembleTracker.java:163)
! at org.apache.curator.framework.imps.EnsembleTracker.access$200(EnsembleTracker.java:48)
! at org.apache.curator.framework.imps.EnsembleTracker$2.processResult(EnsembleTracker.java:134)
! at org.apache.curator.framework.imps.CuratorFrameworkImpl.sendToBackgroundCallback(CuratorFrameworkImpl.java:829)
! at org.apache.curator.framework.imps.CuratorFrameworkImpl.processBackgroundOperation(CuratorFrameworkImpl.java:611)
! at org.apache.curator.framework.imps.WatcherRemovalFacade.processBackgroundOperation(WatcherRemovalFacade.java:151)
! at org.apache.curator.framework.imps.GetConfigBuilderImpl$2.processResult(GetConfigBuilderImpl.java:210)
! at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:619)
! at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:528)
If use Zookeeper 3.5.1, then curator-recipes 3.2.1+ fix this issue.