System property is set to 600 seconds.
jboss.as.management.blocking.timeout=600
But getting timeout after 300 secs while deploying WAR.
[Host Controller] 10:05:14,951 INFO [org.jboss.as.host.controller] (Host Controller Service Threads - 51) WFLYDC0082: ConcurrentServerGroupUpdateTask timed out after 305000 ms awaiting server prepared response(s) -- cancelling updates for servers
Related
I have a few streams that wake every min or so and pulling some docs from the DB and performing some actions and in the end sending messages to SNS.
The tick interval is every 1 min currently.
Every few minutes I see this error info in the log:
[INFO] [06/04/2020 07:50:32.326] [default-akka.actor.default-dispatcher-5] [default/Pool(shared->https://sns.eu-west-1.amazonaws.com:443)] Pool is now shutting down as requested.
[INFO] [06/04/2020 07:51:32.666] [default-akka.actor.default-dispatcher-15] [default/Pool(shared->https://sns.eu-west-1.amazonaws.com:443)] Pool shutting down because akka.http.host-connection-pool.idle-timeout triggered after 30 seconds.
What does it mean? Did someone have it before? 443 was worrying me.
Akka http connection pools are terminated by akka automatically if not used for a certain time (default is 30 seconds). This can be configured and set to infinite if needed.
The pools are re-created on next use but this takes some time, so the request initiating the creation will be "blocked" till the pool is re-created.
From documentation.
The time after which an idle connection pool (without pending requests) will automatically terminate itself. Set to infinite to completely disable idle timeouts.
The config parameter that controls it is
akka.http.host-connection-pool.idle-timeout
The log message points to the config parameter too
Pool shutting down because akka.http.host-connection-pool.idle-timeout
triggered after 30 seconds.
I am trying to connect to MongoDB 3.6 with Java. When I tried to interact with the database, I received the following exception
Timed out after 30000 ms while waiting for a server that matches WritableServerSelector.
Client view of cluster state is {type=REPLICA_SET, servers=[{address=localhost:27017,
type=REPLICA_SET_GHOST, roundTripTime=0.8 ms, state=CONNECTED}]. Cause: Timed out after 30000
ms while waiting for a server that matches WritableServerSelector.
Why am I getting this error? What does it mean and how to solve this ?
I'm observing Consumer instance not found error, at the time of consumer registration even if consumer instance is not timedout. Using Confluent API's.
Following are steps followed for this negative testing:
Running a script for consumer registration.
Kafka Topology : 3 ZK instance (where 2 ZK are dummy value) and 1 node cluster (single instance of rest-proxy & broker node).
When script is in progress for consumer registration, have cancelled the script & re-run it.
Have seen that for the last registered consumer, Instance not found error is returned. But, in logs after some ms, 200 OK is listed for that consumer registration request [Consumer name in shared below logs : CGStress_TEST111111111111111_6].
[2018-08-28 09:05:48,411] INFO http://localhost:8082/v1/consumer/CGStress_TEST111111111111111_5 Service: TransacationId:5 EntityId:ed_1 Authorized:true Allowed:true (io.confluent.kafkarest.resources.SecurityRestrictions)
{"X-Nssvc-serviceid":null,"Type":"API","X-Nssvc-customerid":null,"Client-IP":"127.0.0.1","Severity":"INFO","X-Cws-Transactionid":"5","message":{"request":{"content-length":81,"method":"POST","time":"2018-08-28 09:05:48.409","uri":"mr/v1/consumer/CGStress_TEST111111111111111_5","entity-id":"ed_1","user-agent":"python-requests/2.11.1"},"response":{"status_code":200,"time":"2018-08-28 09:05:48.412"}}}
[2018-08-28 09:05:48,412] INFO 127.0.0.1 - - [28/Aug/2018:09:05:48 +0000] "POST /mr/v1/consumer/CGStress_TEST111111111111111_4 HTTP/1.1" 200 205 19 (io.confluent.rest-utils.requests)
[2018-08-28 09:05:48,420] INFO http://localhost:8082/mr/v1/consumer/CGStress_TEST111111111111111_6 Service: TransacationId:6 EntityId:ed_1 Authorized:true Allowed:true (io.confluent.kafkarest.resources.SecurityRestrictions)
{"X-Nssvc-serviceid":null,"Type":"API","X-Nssvc-customerid":null,"Client-IP":"127.0.0.1","Severity":"INFO","X-Cws-Transactionid":"6","message":{"request":{"content-length":81,"method":"POST","time":"2018-08-28 09:05:48.419","uri":"mr/v1/consumer/CGStress_TEST111111111111111_6","entity-id":"ed_1","user-agent":"python-requests/2.11.1"},"response":{"status_code":404,"error_response":{"message":"Consumer instance not found.","error":40403},"time":"2018-08-28 09:05:48.421"}}}
[2018-08-28 09:05:48,423] INFO 127.0.0.1 - - [28/Aug/2018:09:05:48 +0000] "POST /mr/v1/consumer/CGStress_TEST111111111111111_5 HTTP/1.1" 200 205 15 (io.confluent.rest-utils.requests)
[2018-08-28 09:05:48,431] INFO 127.0.0.1 - - [28/Aug/2018:09:05:48 +0000] "POST /mr/v1/consumer/CGStress_TEST111111111111111_6 HTTP/1.1" 404 61 13 (io.confluent.rest-utils.requests)
[2018-08-28 09:05:49,299] WARN Client session timed out, have not heard from server in 1501ms for sessionid 0x0 (org.apache.zookeeper.ClientCnxn)
[2018-08-28 09:05:49,300] INFO Client session timed out, have not heard from server in 1501ms for sessionid 0x0, closing socket connection and attempting reconnect (org.apache.zookeeper.ClientCnxn)
[2018-08-28 09:05:49,400] INFO Opening socket connection to server localhost/0:0:0:0:0:0:0:1:32181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)
[2018-08-28 09:05:49,400] INFO Socket connection established to localhost/0:0:0:0:0:0:0:1:32181, initiating session (org.apache.zookeeper.ClientCnxn)
[2018-08-28 09:05:49,403] INFO Session establishment complete on server localhost/0:0:0:0:0:0:0:1:32181, sessionid = 0x1657f54045b00f3, negotiated timeout = 6000 (org.apache.zookeeper.ClientCnxn)
[2018-08-28 09:05:49,403] INFO zookeeper state changed (SyncConnected) (org.I0Itec.zkclient.ZkClient)
[2018-08-28 09:05:49,404] INFO [CGStress_TEST111111111111111_6_UbuntuNTP-1535447146187-7b4d0350], starting auto committer every 60000 ms (kafka.consumer.ZookeeperConsumerConnector)
{"X-Nssvc-serviceid":null,"Type":"API","X-Nssvc-customerid":null,"Client-IP":"127.0.0.1","Severity":"INFO","X-Cws-Transactionid":"6","message":{"request":{"content-length":81,"method":"POST","time":"2018-08-28 09:05:46.172","uri":"mr/v1/consumer/CGStress_TEST111111111111111_6","entity-id":"ed_1","user-agent":"python-requests/2.11.1"},"response":{"status_code":200,"time":"2018-08-28 09:05:49.405"}}}
[2018-08-28 09:05:49,409] INFO 127.0.0.1 - - [28/Aug/2018:09:05:46 +0000] "POST /mr/v1/consumer/CGStress_TEST111111111111111_6 HTTP/1.1" 200 124 3239 (io.confluent.rest-utils.requests)
root#UbuntuNTP:~/CloudServices/MsgRelay#
Is this something related to dummy zk instance added?
We are using Kafka high level consumer , and we are able to successfully consume messages but the zookeeper connections keep expiring and reestablishing.
I am wondering why are there no heartbeats to keep the connections alive:
Kafka Consumer Logs
====================
[localhost-startStop-1-SendThread(10.41.105.23:2181)] [ClientCnxn$SendThread] [line : 1096 ] - Client session timed out, have not heard from server in 2666ms for sessionid 0x153175bd3860159, closing socket connection and attempting reconnect
2016-03-08 18:00:06,750 INFO [localhost-startStop-1-SendThread(10.41.105.23:2181)] [ClientCnxn$SendThread] [line : 975 ] - Opening socket connection to server 10.41.105.23/10.41.105.23:2181. Will not attempt to authenticate using SASL (unknown error)
2016-03-08 18:00:06,823 INFO [localhost-startStop-1-SendThread(10.41.105.23:2181)] [ClientCnxn$SendThread] [line : 852 ] - Socket connection established to 10.41.105.23/10.41.105.23:2181, initiating session
2016-03-08 18:00:06,892 INFO [localhost-startStop-1-SendThread(10.41.105.23:2181)] [ClientCnxn$SendThread] [line : 1235 ] - Session establishment complete on server 10.41.105.23/10.41.105.23:2181, sessionid = 0x153175bd3860159, negotiated timeout = 4000
Zookeeper Logs
==================
[2016-03-08 17:44:37,722] INFO Accepted socket connection from /10.10.113.92:51333 (org.apache.zookeeper.server.NIOServerCnxnFactory)
[2016-03-08 17:44:37,742] INFO Client attempting to renew session 0x153175bd3860159 at /10.10.113.92:51333 (org.apache.zookeeper.server.ZooKeeperServer)
[2016-03-08 17:44:37,742] INFO Established session 0x153175bd3860159 with negotiated timeout 4000 for client /10.10.113.92:51333 (org.apache.zookeeper.server.ZooKeeperServer)
[2016-03-08 17:46:56,000] INFO Expiring session 0x153175bd3860151, timeout of 4000ms exceeded (org.apache.zookeeper.server.ZooKeeperServer)
[2016-03-08 17:46:56,001] INFO Processed session termination for sessionid: 0x153175bd3860151 (org.apache.zookeeper.server.PrepRequestProcessor)
[2016-03-08 17:46:56,011] INFO Closed socket connection for client /10.10.114.183:38324 which had sessionid 0x153175bd3860151 (org.apache.zookeeper.server.NIOServerCnxn)
Often ZooKeeper session timeouts are caused by "soft failures," which are most commonly a garbage collection pause. Turn on GC logging and see if a long GC occurs at the time the connection times out. Also, read about JVM tuning in Kafka.
[2016-03-08 17:46:56,000] INFO Expiring session 0x153175bd3860151,
timeout of 4000ms exceeded (org.apache.zookeeper.server.ZooKeeperServer)
What is Zookeeper's maxSessionTimeout?
If it's just 4000ms (4 seconds), then it's way too small.
In Cloudera distribution of Hadoop, ZK's maxSessionTimeout is by default 40s
(40000ms).
As explained in ZK configuration -
https://zookeeper.apache.org/doc/r3.4.5/zookeeperAdmin.html
it defaults 20 ticks
(and one tick by default is 2 seconds).
I'm am using JBoss EAP 6.2 as Webapplication server and Apace Modcluster for load balancing.
Whenever I try to undeploy my webapplication, I get the following warning
14:22:16,318 WARN [org.jboss.modcluster] (ServerService Thread Pool -- 136) MODCLUSTER000025: Failed to drain 2 remaining active sessions from default-host:/starrassist within 10.000000.1 seconds
14:22:16,319 INFO [org.jboss.modcluster] (ServerService Thread Pool -- 136) MODCLUSTER000021: All pending requests drained from default-host:/starrassist in 10.002000.1 seconds
and it takes forever to undeploy and the EAP server-group and node in which the application is deployed becomes unresponsive.
The only workaround is to restart the entire EAP server. My Question is, Is there an attribute that I can set in EAP or ModCluster so that the active sessions beyond a maxTimeOut would expire itself?
To control the timeout to stop a context you can use the following configuration value:
Stop Context Timeout
The amount of time, measure in units specified by
stopContextTimeoutUnit, for which to wait for clean shutdown of a
context (completion of pending requests for a distributable context;
or destruction/expiration of active sessions for a non-distributable
context).
CLI Command:
/profile=full-ha/subsystem=modcluster/mod-cluster-config=configuration/:write-attribute(name=stop-context-timeout,value=10)
Ref: Configure the mod_cluster Subsystem
Likewise if you are using JDK 8 take a look at this issue: Draining pending requests fails with Oracle JDK8