I'm working on an application where I want the server to send events to my JS front-end to show money transactions as they are processed.
The way I implemented works as intended; however, after a couple of hours I notice the heap increasing and consequently killing our server.
I read somewhere that it could be caused by nginx not being configured correctly, but the keep-alive is set to just a few seconds. I also saw a couple of posts regarding memory leak issues when one attempts something like the following:
val newTransaction: Enumerator[String] = {
Enumerator.generateM[String] {
Promise.timeout(getNewTransactions, checkingInterval 500)
}
}
...
Ok.stream(newTransaction &> EventSource()).as("text/event-stream")
getNewTransactions is just a method that hits Redis to check if any new transaction arrived for an account.
I tried updating my project to use the newest Scala version 2.10.2, but it seems that it doesn't solve it. In the meantime, I am resorting to JavaScript polling until I can figure it out. Does anyone have any idea how I might solve this?
Related
Here's the code:
How to ensure the javascripts/channels/chatrooms.coffee is loading and receive: (data) works? Console.log data is not loading
I posted the problem the other day. It comes from a tutorial, but somewhere after continuing work on my project, I'm not sure where the bug came up,
But the message relay job posts on the server, and my config.yml has redis with redis up and running.
There have been similar bugs, but I've worked through those solutions and it's not enough. The received: (data) console log doesn't arrive in the js console of the browser.
The App subscriptions seem to all be in order according to the tutorial, I bet it's a really simple fix.
My messaging system still works, so it's not crucial to the project, but it's a difference of having the chat system work in realtime versus with a 4 second page-refresh delay.
The last problem I had was not including jquery for my other coffeescripts, so I'm guessing the channels coffeescripts is probably a one line fix.
We use Retrofit/OkHttp3 for all network traffic from our Android application. So far everything seems to run quite smoothly.
However, we have now occasionally had our app/process run out of file handles.
Android allows for a max of 1024 file handles per process
OkHttp will create a new thread for each async call
Each thread created this way will (from our observation) be responsible for 3 new file handles (2 pipes and one socket).
We were able to debug this exactly, where each dispached async call using .enqueue() will lead to an increase of open file handles of 3.
The problem is, that the ConnectionPool in OkHttp seems to be keeping the connection threads around for much longer than they are actually needed. (This post talks of five minutes, though I haven't seen this specified anywhere.)
That means if you are quickly dispatching request, the connection pool will grow in size, and so will the number of file handles - until you reach 1024 where the app crashes.
I've seen that it is possible to limit the number of parallel calls with Dispatcher.setMaxRequests()(although it seems unclear whether this actually works, see here) - but that still doesn't quite address the issue with the open threads and file handles piling up.
How could we prevent OkHttp from creating too many file handles?
I am answering my own question here to document this issue we had. It took us a while to figure this out and I think others might encounter this too and might be glad for this answer.
Our problem was that we created one OkHttpClient per request, as we used it's builder/interceptor API to configure some per-request parameters like HTTP headers or timeouts.
By default each OkHttpClient comes with its own connection pool, which of course blows up the number of connections/threads/file handles and prevents proper reuse in the pool.
Our solution
We solved the problem by manually creating a global ConnectionPool in a singleton, and then passing that to the OkHttpClient.Builder object which builds the actual OkHttpClient.
This still allows for per-request configuration using the OkHttpClient.Builder
Makes sure all OkHttpClient instances are still using a common connection pool.
We were then able to properly size the global connection pool.
I've run into some weird issues with libSpotify. It seems that any libSpotify-based client will take ages to process requests (sometimes 20 seconds for a simple search, seconds for loading one single image, etc..) sent to Spotify servers, whereas Spotify's own desktop client for Windows works extremely well on the same system, processing requests and loading images in near realtime.
Even the demo app provided with libSpotify, called spshell, exhibits massive problems:
Did anyone experience similar problems and/or knows the cause?
Magically works again since today, no clue why.
We have a main ruleflow which calls 8 more rule flows (Rule1.rf to Rule8.rf) through an AND splitter. One of the rule flows - say Rules4.rf - is fired sometimes and not fired sometimes.
This is for an online application and we use jBoss. When the server is started, everything works fine. After many hours, for some requests, Rules4.rf is not fired at all and for others, its fired properly.
We even posted the same request again and again and the issue happens some times only. There is no difference in the logs between the success & failure requests, except for the logs from the Rules4.rf which missing in failued requests.
We are using drools 5.1 and java 6.
Please help me. This is creating a very big issue.
It is very difficult to figure out what might be going on without being able to look at the actual code and log. Could you by any chance create a JIRA and attach the process and if possible (a part of) an audit log that shows the issue?
Kris
I'm having an issue with jboss server. when i run jboss server, it stops responding( no fixed time, so cannot predict when will it stop responding after start) after that it doesn't writes anything in log file. my problem is similar to the problem described on jboss community, link given below but it doesn't have the answer. please help.
http://community.jboss.org/message/526193
--Ravi
It sounds like your jboss server is running out of threads to allocate and is waiting for a new one to become available. Try triggering a thread dump (ctrl-\) and see if you find any threads suspiciously locked and waiting in some of your code. Quite possibly you have a deadlock or memory leak somewhere in your code which is causing old threads to lock up and never be released.
Alternatively try what the guy you linked to did, i.e. increasing the amount of threads available.
edit: For some more basic advice, this post might be of use to you.