I have a Play!+Scala app using Play-2.4. Looking at the server logs, I see unrelated connection logs and exceptions. For example:
2017-11-06 21:36:45,174 ERROR [New I/O worker #1] application [?:?]
! #76020798a - Internal server error, for (CONNECT) [pl.metin2.gameforge.com:443] ->
play.api.http.HttpErrorHandlerExceptions$$anon$1: Execution exception[[NullPointerException: null]]
at play.api.http.HttpErrorHandlerExceptions$.throwableToUsefulException(HttpErrorHandler.scala:265) ~[com.typesafe.play.play_2.11-2.4.6.jar:2.4.6]
I am not connecting with gameforge at all, but there is still this message. Similarly this one:
! #75fjch9ik - Internal server error, for (OPTIONS) [sip:nm] ->
play.api.http.HttpErrorHandlerExceptions$$anon$1: Execution exception[[NullPointerException: null]]
at play.api.http.HttpErrorHandlerExceptions$.throwableToUsefulException(HttpErrorHandler.scala:265) ~[com.typesafe.play.play_2.11-2.4.6.jar:2.4.6]
at org.jboss.netty.handler.codec.http.HttpRequestDecoder.createMessage(HttpRequestDecoder.java:75) ~[io.netty.netty-3
It seems to me that this New I/O worker threadpool gets created by netty. Any kind soul has any idea, what might be causing these connections? Server is hosted on AWS EC2.
Related
I have been running PAM 7.9/JBPM 7.48 for about a year under JBOSS EAP 7.3. My JBPM's KieServer persists using SQL Server. I repeatedly deployed the KieServer yesterday but deploying today fails.
2021-12-16 15:25:53,645 ERROR [org.jboss.as.controller.management-operation] (DeploymentScanner-threads - 1) WFLYCTL0348: Timeout after [300] seconds waiting for service container stability. Operation will roll back. Step that first updated the service container was 'full-replace-deployment' at address '[]'
2021-12-16 15:26:03,649 ERROR [org.jboss.as.controller.management-operation] (DeploymentScanner-threads - 1) WFLYCTL0190: Step handler org.jboss.as.server.deployment.DeploymentHandlerUtil$4#74e289e9 for operation full-replace-deployment at address [] failed handling operation rollback -- java.util.concurrent.TimeoutException: java.util.concurrent.TimeoutException
at org.jboss.as.controller.OperationContextImpl.waitForRemovals(OperationContextImpl.java:523)
at org.jboss.as.controller.AbstractOperationContext$Step.handleResult(AbstractOperationContext.java:1518)
I have already set the property to increase the timeout for the deployment but it still complains about a 5 second timeout that must be controlled by another property
2021-12-16 13:40:47,039 ERROR [org.jboss.as.controller.management-operation] (DeploymentScanner-threads - 1) WFLYCTL0349: Timeout after [5] seconds waiting for service container stability while finalizing an operation. Process must be restarted. Step that first updated the service container was 'deploy' at address '[("deployment" => "kie-server.war")]'
I have changed the logging level to trace in order to gain all the information I can. How else can I debug / solve this issue?
There are two factors that may be contributing to this, but I don't have a good approach for addressing them.
There was a Windows Update yesterday (likely due to the recent Log4j exploit)
Some people at my company are having problems connecting to the SQL Server database. I am not seeing log messages about KieServer being unable to connect to the DB, but when it cannot reaching the DB the KieServer fails to start.
I'm working on a REST API that uses Akka, we inherited it from a previous team, and none of us have experience with Akka before this.
Akka is being used to process the data the API is returning, and acting as the HTTP server.
Recently when the API was under load, we started getting failures like so:
),HttpProtocol(HTTP/1.1)), Response: HttpResponse(500 Internal Server Error,List(),HttpEntity.Strict(text/plain; charset=UTF-8,
Error Code: 500
Type: Internal Server Error
Stack Trace:
akka.stream.StreamTcpException: The connection actor has terminated. Stopping now.
),HttpProtocol(HTTP/1.1)), Time: 6430 ms
I have no idea where the above error is happening in the code, or how to appropriately handle this error when it happens.
Can anyone give suggestions on how to trace this down further, or suggestions on how to handle and recover from these types of issues?
actually I have a TimerWS running correctly, but in a few hours I have this error
WARN [org.hornetq.core.client] (hornetq-failure-check-thread) HQ212037: Connection failure has been detected: HQ119014: Did not receive data from invm:0. It is likely the client has exited or crashed without closing its connection, or the network between the server and client has failed. You also might have configured connection-ttl and client-failure-check-period incorrectly. Please check user manual for more information. The connection will now be closed. [code=CONNECTION_TIMEDOUT]
WARN [org.hornetq.jms.server] (Thread-3 (HornetQ-client-global-threads-451033388)) Notified of connection failure in xa discovery, we will retry on the next recovery: HornetQException[errorType=NOT_CONNECTED message=HQ119006: Channel disconnected]
Caused by: com.microsoft.sqlserver.jdbc.SQLServerException: The connection is closed.
Any idea?
Something is going wrong with a client's connection to the server.
These warnings are generated from the server which determines that a client is no longer responding.When that happens, the server cleans up the server-side resources related to the client's connections.
I have seen this many times by clients which are not properly coded to close their resources when they exit (finally block).
Have a look also for network problems which break the connection.
I cannot start Distributed Transaction Coordinator service.
It stoped to work few days ago.
When I am trying to start service:
Registry properties:
RPC (For a test values was changed here to oposite and back - without any results ):
Windows logs\application logs:
53283
A MS DTC component has encountered an internal error. The process is being terminated. Error Specifics: DtcSystemShutdown (d:\w7rtm\com\complus\dtc\dtc\msdtc\src\msdtc.cpp#2539): Shutting down with an error
4111
The MS DTC service is stopping.
4102
DTC Security Configuration values (OFF = 0 and ON = 1): Network Administration of Transactions = 1,
Network Clients = 1,
Inbound Distributed Transactions using Native MSDTC Protocol = 1,
Outbound Distributed Transactions using Native MSDTC Protocol = 1,
Transaction Internet Protocol (TIP) = 0,
XA Transactions = 1,
SNA LU 6.2 Transactions = 1
Could not initialize the MS DTC Transaction Manager.
4356
Failed to initialize the MS DTC Communication Manager. Error Specifics: hr = 0x80070057, d:\w7rtm\com\complus\dtc\dtc\cm\src\ccm.cpp:2117, CmdLine: C:\Windows\System32\msdtc.exe, Pid: 5332
4358
The MS DTC Connection Manager is unable to register with RPC to use one of LRPC, TCP/IP, or UDP/IP. Please ensure that RPC is configured properly. If "ServerTcpPort" registry key is configured(DWORD value under the HKEY_LOCAL_MACHINE\Software\Microsoft\MSDTC for local DTC instance or under cluster hive for clustered DTC instance), please verify if the configured port is valid and the port is not already in use by a different component. Error Specifics:hr = 0x80070057, d:\w7rtm\com\complus\dtc\dtc\cm\src\iomgrsrv.cpp:2523, CmdLine: C:\Windows\System32\msdtc.exe, Pid: 5332
4156
String message: RPC raised an exception with a return code RPC_S_INVALIDA_ARG..
I found that we can use -resetlog command. But this doesnot resolving my problem:
Firewall is disabled.
Try to delete key HKLM\Software\Microsoft\Rpc\Internet from registry.
To get around this issue, I had to copy the log file (I accidentally deleted) to the location specified by the Local DTC Log information location.
After starting the run/debug configuration begins build and deploy application. All goes well. The only thing that provides continuously JBoss Log SocketException:
2012-07-16 12:43:30,424 DEBUG [org.jboss.naming.Naming] Error writing response to /127.0.0.1 java.net.SocketException: Software caused connection abort: socket write error at java.net.SocketOutputStream.socketWrite0(Native Method) at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92) at java.net.SocketOutputStream.write(SocketOutputStream.java:136) at java.io.ObjectOutputStream$BlockDataOutputStream.drain(ObjectOutputStream.java:1847) at java.io.ObjectOutputStream$BlockDataOutputStream.setBlockDataMode(ObjectOutputStream.java:1756) at java.io.ObjectOutputStream.writeNonProxyDesc(ObjectOutputStream.java:1257) at java.io.ObjectOutputStream.writeClassDesc(ObjectOutputStream.java:1211) at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1395) at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1158) at java.io.ObjectOutputStream.writeFatalException(ObjectOutputStream.java:1547) at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:333) at org.jnp.server.Main$BootstrapRequestHandler.run(Main.java:516) at org.jboss.util.threadpool.RunnableTaskWrapper.run(RunnableTaskWrapper.java:148) at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:756) at java.lang.Thread.run(Thread.java:662)
At issues.jboss.org/browse/JBAS-3417 offer a solution:
If you set server / etc / host with:
IP fqn name
where IP is in «Error writing response to / IP"
Ex:
... "Error writing response to / 10.1.0.1"
10.1.0.1 test.mydomain test
you resolve the problem
But I have something 127.0.0.1 localhost, it is already registered in the default c:\ Windows\system32\drivers\etc\hosts
In the Troubleshooting says:
You may notice something in the JBoss logs similar to the following:
2009-01-28 08:36:57,381 DEBUG
[org.jboss.naming.NamingService]
Error writing response to / 127.0.0.1 <[http://127.0.0.1]>
java.net.SocketException: Broken pipe
at java.net.SocketOutputStream.socketWrite0 (Native Method)
This is most likely an anomalous and benign «problem». That is, it should not affect the operation of any applications deployed to JBoss.
If this does not affect deployed application can be any way to hide it? And then the continuous output overshadows the other posts and eats memory.