I'm working on a REST API that uses Akka, we inherited it from a previous team, and none of us have experience with Akka before this.
Akka is being used to process the data the API is returning, and acting as the HTTP server.
Recently when the API was under load, we started getting failures like so:
),HttpProtocol(HTTP/1.1)), Response: HttpResponse(500 Internal Server Error,List(),HttpEntity.Strict(text/plain; charset=UTF-8,
Error Code: 500
Type: Internal Server Error
Stack Trace:
akka.stream.StreamTcpException: The connection actor has terminated. Stopping now.
),HttpProtocol(HTTP/1.1)), Time: 6430 ms
I have no idea where the above error is happening in the code, or how to appropriately handle this error when it happens.
Can anyone give suggestions on how to trace this down further, or suggestions on how to handle and recover from these types of issues?
Related
I am having a somewhat reproducible problem on a Swift server I'm running. This is a multi-threaded server, using Kitura. The basics are: After the server has been running for a period of time, download requests start needing retries from the client (usually three retries). The attempts from the client result in the server thread not terminating. On the server, the download problem shows up like this in the log:
[INFO] REQUEST /DownloadFile: ABOUT TO END ...
And then the request never terminates.
The relevant fragment code in my server looks like this:
// <snip>
Log.info(message: "REQUEST \(request.urlURL.path): ABOUT TO END ...")
do {
try self.response.end()
Log.info(message: "REQUEST \(request.urlURL.path): STATUS CODE: \(response.statusCode)")
} catch (let error) {
Log.error(message: "Failed on `end` in failWithError: \(error.localizedDescription); HTTP status code: \(response.statusCode)")
}
Log.info(message: "REQUEST \(request.urlURL.path): COMPLETED")
// <snip>
That is, the server clearly seems to hang on the call to end (a Kitura method). See also https://github.com/crspybits/SyncServerII/blob/master/Sources/Server/Setup/RequestHandler.swift#L105
Immediately before this issue came up last time, I observed the following in my server log:
[2017-07-12T15:31:23.302Z] [ERROR] [HTTPServer.swift:194 listen(listenSocket:socketManager:)] Error accepting client connection: Error code: 5(0x5), ERROR: SSL_accept, code: 5, reason: DH lib
[2017-07-12T15:31:23.604Z] [ERROR] [HTTPServer.swift:194 listen(listenSocket:socketManager:)] Error accepting client connection: Error code: 1(0x1), ERROR: SSL_accept, code: 1, reason: Could not determine error reason.
[2017-07-12T15:31:23.995Z] [ERROR] [HTTPServer.swift:194 listen(listenSocket:socketManager:)] Error accepting client connection: Error code: 1(0x1), ERROR: SSL_accept, code: 1, reason: Could not determine error reason.
[2017-07-12T15:40:32.941Z] [ERROR] [HTTPServer.swift:194 listen(listenSocket:socketManager:)] Error accepting client connection: Error code: 1(0x1), ERROR: SSL_accept, code: 1, reason: Could not determine error reason.
[2017-07-12T15:42:43.000Z] [VERBOSE] [HTTPServerRequest.swift:215 parsingCompleted()] HTTP request from=139.162.78.135; proto=https;
[INFO] REQUEST RECEIVED: /
[2017-07-12T16:32:38.479Z] [ERROR] [HTTPServer.swift:194 listen(listenSocket:socketManager:)] Error accepting client connection: Error code: 1(0x1), ERROR: SSL_accept, code: 1, reason: Could not determine error reason.
I am not sure where this is coming from in the sense that I'm not sure if one of my client's is generating this. I do not explicitly make requests to my server with "/". (I do occasionally see requests made to my server from clients that are not mine-- it is possible this is one of these). Note that all except one of these log messages are coming from Kitura, not directly from my code. My log message is [INFO] REQUEST RECEIVED: /.
If I was a betting man, I'd say the above errors put my server into a state where afterwards, I see this download/retry behavior.
My only solution at this point is to restart the server. From which point the issue doesn't immediately happen.
Thoughts?
I'm not sure if this addresses the root problem, or if it's just a work-around, but it appears to be working. With the server as stated in the question, I had been using Kitura's built-in SSL support. I have now switched to using NGINX as a front-end, and no longer use Kitura's built-in SSL support. NGINX takes care of all the HTTPS/SSL details. Since doing this (about a month ago), and with the server running for all of that intervening time, I have not experience the not terminating issue reported in this question. See also https://github.com/crspybits/SyncServerII/issues/28
Can someone please shed some light on this cryptic exception being thrown when I try to connect and send a REST request to a local Ignite node ?
The request looks like this :
http://127.0.0.1:8080/ignite?cmd=top
There is no response from Ignite but I see the following in logs :
[GridTcpRestProtocol] Closing NIO session because of unhandled exception [cls=class o.a.i.i.util.nio.GridNioException, msg=Failed to parse incoming packet (invalid packet start) [ses=GridSelectorNioSessionImpl [selectorIdx=2, queueSize=0, writeBuf=null, readBuf=null, recovery=null, super=GridNioSessionImpl [locAddr=/127.0.0.1:8080, rmtAddr=/127.0.0.1:64501, createTime=1439344656124, closeTime=0, bytesSent=0, bytesRcvd=278, sndSchedTime=1439344656124, lastSndTime=1439344656124, lastRcvTime=1439344656124, readsPaused=false, filterChain=FilterChain[filters=[GridNioCodecFilter [parser=GridTcpRestParser [jdkMarshaller=JdkMarshaller [], routerClient=false], directMode=false]], accepted=true]], b=47]]
I am not able to figure out whats going wrong here. Searching the web for this issue I did not yeild much success. Please help. Thanks in advance.
I am writing an apache-camel RabbitMQ consumer. I would like to react somehow to connection problems (i.e. try to reconnect). Is it possible to configure apache-camel to automatically reconnect?
If not, how can I find out that a connection to the queue was interrupted? I've done the following test:
start the queue (and some producer)
start my consumer (it was getting messages as expected)
stop the queue (the messages stopped arriving, as expected, but no exception was thrown)
start the queue (no new messages were received)
I am using camel in Scala (via akka-camel), but a Java solution would be probably also OK
You can pass in the flag automaticRecoveryEnabled=true to the URI, Camel will reconnect if the connection is lost.
For automatic RabbitMQ resource recovery (Connections/Channels/Consumers/Queues/Exchanages/Bindings) when failures occur, check out Lyra (which I authored). Example usage:
Config config = new Config()
.withRecoveryPolicy(new RecoveryPolicy()
.withMaxAttempts(20)
.withInterval(Duration.seconds(1))
.withMaxDuration(Duration.minutes(5)));
ConnectionOptions options = new ConnectionOptions().withHost("localhost");
Connection connection = Connections.create(options, config);
The rest of the API is just the amqp-client API, except your resources are automatically recovered when failures occur.
I'm not sure about camel-rabbitmq specifically, but hopefully there's a way you can swap in your own resource creation via Lyra.
Current camel-rabbitmq just create a connection and the channel when the consumer or producer is started. So it don't have a chance to catch the connection exception :(.
Is it possible to let the server send messages to all connected clients without waiting for any action from them? Let me explain it :-) I've been reading the docs/examples and I have found nothing that satifies my needs: the flow is always the same; a client connects (e.g: a GET call to a Rest API), the connection is suspendend and until a new API call is received (e.g.: a POST call) the server simply waits (or at least this is what I have understood). My use case is pretty different: I want the server to send some "notifications" once new data become available. This would be my use case (pretty simplifed):
Client A connects to server
Connection is suspended since no new data is available at the moment
The server gets notified new data is available from an external
source and broadcasts it to client A
Go to step 2
What I have achieved so far is getting the connection successfully established. The next step is to solve this server issue. I must say this technology is completely new to me so it is possible I misunderstood how something works. If that's the case, let me know!
This is my stack:
Spring 3.2.0 RELEASE
Jersey 1.8
Atmosphere Jersey 1.0.13
Tomcat 7.0.40
Thank you all in advance!
UPDATE: After following this I get this warning, which I have no idea how to get rid of:
2013-06-04 09:40:36,284 WARN [org.atmosphere.cpr.AtmosphereFramework] - Failed using comet support: org.atmosphere.container.Tomcat7AsyncSupportWithWebSocket, error: Tomcat failed to detect this is a Comet application because context.xml is missing or the Http11NioProtocol Connector is not enabled.
If that's not the case, you can also remove META-INF/context.xml and WEB-INF/lib/atmosphere-compat-tomcat.jar Is the Nio or Apr Connector enabled?
2013-06-04 09:40:36,285 WARN [org.atmosphere.cpr.AtmosphereFramework] - Using org.atmosphere.container.Tomcat7BIOSupportWithWebSocket
I followed the app structure commented here, so this should not be a problem. I have noticed that by changing the transport to "websocket" instead of "long-polling" shows no errors. The server finally sends data tough :)
I followed your link and modified the code a little.
When you are in the step 3 "The server gets notified new data is available from an external source and broadcasts it to client A", you have to write a line like this:
BroadcasterFactory.getDefault().lookup("/*").broadcast(response);
At first I used the TextMessage received from my ActiveMQ Queue but I get this error, so I used a Jackson class as an object response and everything worked fine.
SEVERE: A message body writer for Java class org.apache.activemq.command.ActiveMQTextMessage, and Java type class org.apache.activemq.command.ActiveMQTextMessage, and MIME media type application/json was not found
jun 03, 2014 11:32:21 AM com.sun.jersey.spi.container.ContainerResponse write
SEVERE: The registered message body writers compatible with the MIME media type are:
application/json (JSONJAXBElementProvider, JSONArrayProvider, JSONObjectProvider, JSONRootElementProvider, JSONListElementProvider, ...)
I'm using RESTlet as Server and Client. Server is running over Tomcat and I can access it from browser.
For Client, I'm using org.restlet.resource.ClientResource. It works fine if I send a few requests to that server, but if I send a few hundred calls it breaks:
Fev 20, 2013 12:59:43 PM org.restlet.engine.connector.ClientConnectionHelper start
INFO: Starting the internal [HTTP/1.1] client
(some calls work)
Fev 20, 2013 1:00:49 PM org.restlet.util.SelectionRegistration block
WARNING: The thread blocked at the cyclic barrier has timed out
java.util.concurrent.TimeoutException
at java.util.concurrent.CyclicBarrier.dowait(CyclicBarrier.java:250)
at java.util.concurrent.CyclicBarrier.await(CyclicBarrier.java:427)
at org.restlet.util.SelectionRegistration.block(SelectionRegistration.java:191)
at org.restlet.engine.io.NbChannelInputStream.onFill(NbChannelInputStream.java:230)
at org.restlet.engine.io.Buffer.process(Buffer.java:601)
at org.restlet.engine.io.NbChannelInputStream.read(NbChannelInputStream.java:307)
at java.io.InputStream.read(InputStream.java:101)
at org.restlet.engine.io.BioUtils.copy(BioUtils.java:80)
at org.restlet.engine.io.NioUtils.copy(NioUtils.java:147)
at org.restlet.representation.ReadableRepresentation.write(ReadableRepresentation.java:104)
at org.restlet.representation.ChannelRepresentation.write(ChannelRepresentation.java:76)
at org.restlet.representation.ChannelRepresentation.write(ChannelRepresentation.java:82)
Exception in thread "main" java.io.IOException: The thread blocked at the cyclic barrier has timed out.
at org.restlet.util.SelectionRegistration.block(SelectionRegistration.java:197)
at org.restlet.engine.io.NbChannelInputStream.onFill(NbChannelInputStream.java:230)
at org.restlet.engine.io.Buffer.process(Buffer.java:601)
at org.restlet.engine.io.NbChannelInputStream.read(NbChannelInputStream.java:307)
at java.io.InputStream.read(InputStream.java:101)
at org.restlet.engine.io.BioUtils.copy(BioUtils.java:80)
at org.restlet.engine.io.NioUtils.copy(NioUtils.java:147)
at org.restlet.representation.ReadableRepresentation.write(ReadableRepresentation.java:104)
at org.restlet.representation.ChannelRepresentation.write(ChannelRepresentation.java:76)
at org.restlet.representation.ChannelRepresentation.write(ChannelRepresentation.java:82)
Caused by: java.util.concurrent.TimeoutException
at java.util.concurrent.CyclicBarrier.dowait(CyclicBarrier.java:250)
at java.util.concurrent.CyclicBarrier.await(CyclicBarrier.java:427)
at org.restlet.util.SelectionRegistration.block(SelectionRegistration.java:191)
... 11 more
I wanna use Apache HTTP Client to be able to do more calls, but I can't find documentation on how to attach it.
I had the same problem but on a live production machine which caused my server to eat up 100% of CPU. I was able to solve it by going back to using restlet 2.1.0 instead of 2.1.2. Not really an answer to the underlying problem, but definitely a solution.
Every evidence points this error to be internal in RESTlet client. Instead of using it, I used URL.getInputStream(), and with that I was able to easily do 10K sequential calls to my RESTlet server. Of course URL is way simple, it's unable to handle errors and shouldn't be used in production, but it proved RESTlet isn't worth it.
I also managed to make Benchmark comparing Axis2 and RESTlet as server, and they had same performance taking 3-5 milisecs to answer a request, I expected REST to be faster... I implemented REST server directly from Servlet and it too 0-1 milisecs to respond. Because of that I decided to drop REST and stick with SOAP Axis2, it's not slower than RESTlet and just a few milisecs slower than Servlet, but it way more flexible and powerful.