I am using JBOSS 5.1. And we are using JMS(topic) for posting the messages and JMS client will take those messages, to be specific i am using durable subscription.
It works on many systems, but on one system i always see this error after two days.
2012-08-30 12:59:27,045 WARNING [sun.rmi.transport.tcp] (RMI TCP Accept-1101) RMI TCP Accept-11101: accept loop for ServerSocket[addr=/0.0.0.0,port=0,localport=11101] throws
java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:597)
at java.util.concurrent.ThreadPoolExecutor.addIfUnderMaximumPoolSize
Not sure why it is occuring only to one system, and only one JMS client is connected to Jboss for listening messages.
You should mention the details of your system, in particular the OS you are running and the Java startup parameters included in your jboss start script.
Chances are that you are running out of thread resources/file descriptors or you have set a thread stack which is not sufficient.
See this thread
Hope it helps
Related
I have deployed a simple Servlet web application on Wildfly 8.2.1 on RHEL 6.9. This application just takes post request and respond with 200 OK.
Now when the client(java client using apache-common-http client) is posting data on the web application. The web application is accepting the request but many of the requests are failing also with ERROR "Caused by java.net.ConnectException: Connection timed out (Connection timed out)" at the client side.
Here my assumption is, Wildfly has some default value for max Http connection which can be opened at any point in time. if further requests are coming which require opening a new connection, web server is rejecting them all.
could anyone here please help me with below question:
How can we check live open HTTP connections in RHEL 6.9. I mean command in RHEL to check how many connection open on port 8080?
How can we tweak the default value of the HTTP connection in wildfly?
Does HTTP connection and max thread count linked with each other. If So, Please let me know how they should be updated in wildfly configuration(standalone.xml).
How many requests can be kept in the queue by Wildfly? what will be happening to the request coming to
wildfly server if the queue is full.
NOTE: It is a kind of load testing for the webserver where traffic is high, not sure about exact value but it's high.
You're getting into some system administration topics but I'll answer what I can. First and foremost - Wildfly 8.2.1 is part of the very first release of Wildfly and I'd strongly encourage upgrading to a newer version.
To check the number of connections in a Unix-like environment you'll want to use the netstat command line. In your case, something like:
netstat -na | grep 8080 | grep EST
This will show you all the connections that are ESTABLISHED to port 8080. That would give you an snapshot of the number of connections. Pipe that to wc to get a count.
Next, finding documentation on Wildfly 8.2.1 is a bit challenging now but Wildfly 8 uses Undertow for the socket IO. That in turn uses XNIO. I found a thread that goes into some detail about configuring the I/O subsystem. Note that Wildfly 8.2.1 uses Undertow 1.1.8 which isn't documented anywhere I could find.
For your last two questions I believe they're related to the second one - the XNIO configuration includes configuration like:
<subsystem xmlns="urn:jboss:domain:io:1.0">
<worker name="default" io-threads="5" task-max-threads="50"/>
<buffer-pool name="default" buffer-size="16384" buffers-per-slice="128"/>
</subsystem>
but you'll need to dig deeper into the docs for details.
In Wildfly 19.1.0.Final the configuration looks similar to the code above other than the version is now 3.0.
I have an issue and I hope you can help me a bit.
I have to implement fast forwarding time, because I need to test something. I've wrote a python script which increment the system time with 5 seconds for every 1 real second. (5 times faster).
Then my jboss fails with some hornetq timeouts.
Do you have any ideas how I can fix this?
03/09/18 09:18:00,107 WARN
[org.hornetq.core.protocol.core.impl.RemotingConnectionImpl] (hornetq-
failure-check-thread) Connection failure has been detected: Did not
receive data from invm:0. It is likely the client has exited or crashed
without closing its connection, or the network between the server and
client has failed. You also might have configured connection-ttl and
client-failure-check-period incorrectly. Please check user manual for
more information. The connection will now be closed. [code=3]
The underlying issue is that changing the time breaks the connection-failure-detection algorithm used by the broker. The broker thinks it isn't receiving "ping" packets from clients at the proper time because you're forcing time to pass at 5x the normal rate. There is no way to fix this for remote clients aside from disabling or extending the connection TTL. However, for in-vm connections you could apply the fix from https://issues.jboss.org/browse/HORNETQ-1314 (which is not resolved in the version of HornetQ you are using) to the branch of HornetQ you're currently using and rebuild. If you don't want to rebuild you could upgrade to a version of JBoss AS (or Wildfly) which contains this fix.
im using jms in jboss Wildfly 8 for messaging. Message delivered succesfully and Receiver goes on with processing. And it takes about 15-20 Minutes, till Receiver does its job. But Server redelivers same Message after about 10 minutes. My Question is how and where can i configure Wildfly to wait for it with for example 20 Mins timelimitations. i found some helpful explenations on Wildfly documentation. But im not sure whether its right way to do this.
Jboss Documentation
Messaging Configuration
should i just add
<redelivery-delay>1200000</redelivery-delay>
<max-delivery-attempts>2</max-delivery-attempts>
in <address-setting> in standalone-full.xml
The setting you made is correct. The setting makes the server attempt to deliver messages for another 2 attempts if the first delivery is unsuccessful.
The fact that you be getting the same message several times should be related to the way you're telling the server that the message was processed.
Look at the link below and check that the acknowledge mode is correct with the operating mode of the class that receives the JMS messages.
JMS Message Delivery Reliability and Acknowledgement Patterns
Given two web apps running on the same Tomcat 6. If you do an http-call from one app to the other, will Tomcat "short circuit" this call, or will it go all the way out on the interwebz before calling home?
#thomasz answer shows the need for more detail. We're using Springs RestTemplate to do the communication. Its plugable architecture lets you provide your own ClientHttpRequestFactory.
Would it be possible to implement a ClientHttpRequest that, if the request was to localhost, it could persuade tomcat to handle it internally?
No, the request will go through all the layers, including loopback interface. Tomcat is not treating requests to the same web container differently. After all, how? You are accessing some URL via URLConnection or HttpClient or raw socket or... - Tomcat would have to somehow intercept (instrument) your application's code and dynamically replace HTTP call with some internal invocation. Possible, but very complicated.
To make matters worse, you can easily cause deadlock or starvation under high load. Imagine your Tomcat worker thread pool has 10 threads and at the same time you access the same servlet from 10 concurrent users. Every servlet now tries to connect to the same web container, but the worker thread pool is exhausted. So all these servlets are blocking, waiting for idle worker thread. But this will never happen, because they are occupying all of them!
On our Webserver, we're seeing a ton of these errors:
Application Server last connected //psoftapp.company.net_8850
bea.jolt.ServiceException: bea.jolt.JoltRemoteService(GetCertificate)call(): Timeout\nbea.jolt.SessionException: Connection recv error\nbea.jolt.JoltException: [3] NwHdlr.recv(): Timeout Error
and on our Appserver:
PSPUBDSP_dflt.27505 (0) 07/20/11 08:13:33 (JNIUTIL): Java exception thrown: java.net.SocketException: Connection reset
I'm reading some tuning documents from PeopleSoft & I found a suggestion that I've seen in a couple of places -- Reducing the tcp_wait_time_interval to 60 seconds. I think I sort of understand what this is doing - It seems that network (or socket?) connections that are no longer being used are "recycled" or made available? Can someone confirm this? Also, why are these connections unused/stale? Is it caused by people not properly logging out of the app (and just closing the browser)?
Thanks!
PSPUBDP is part of the Integration Broker application messaging framework. You could look at the Tuxedo logs or the Integration Broker Monitor too see what is going on. You may be running a high number of messages and overloading the server or possibly you have a message with errors that is somehow causing the crashes.