net.databinder.dispatch failing with too many keep-alive connections - scala

I use net.databinder.dispatch to call a web server with this:
import dispatch._
def request(in: String, address: java.net.URI, headers: Map[String, String]): String = {
val req = url(address.toString) << in <:< headers
val s = Http(req OK as.String)
s()
}
the client defines:
Connection: keep-alive
and the server defines:
Keep-Alive: timeout=120, max=256
If I call the webserver consecutively many times (but not in parallel), I get an error:
Exception in thread "main" java.util.concurrent.ExecutionException: dispatch.StatusCode: Unexpected response status: 500
at com.ning.http.client.providers.netty.NettyResponseFuture.abort(NettyResponseFuture.java:297)
19:32:46.474 [New I/O worker #10] DEBUG c.n.h.c.p.n.NettyAsyncHttpProvider - Unexpected response status: 500
at com.ning.http.client.providers.netty.NettyAsyncHttpProvider.abort(NettyAsyncHttpProvider.java:1326)
dispatch.StatusCode: Unexpected response status: 500
at com.ning.http.client.providers.netty.NettyAsyncHttpProvider.access$700(NettyAsyncHttpProvider.java:137)
This error does not happen with other web servers that close the connection.
What is the problem here?
The webserver?
The client?
Is net.databinder.dispatch really using the keep-alive feature (so, reusing the same socket), or creating a new one each time?
How do I modify the previous example to tell net.databinder.dispatch to use the keep-alive feature?
the version that i am using:
<dependency>
<groupId>net.databinder.dispatch</groupId>
<artifactId>dispatch-core_2.10</artifactId>
<version>0.9.5</version>
</dependency>

Related

Log url of request on InternalServerError in http4s

http4s returns InternalServerError if an unhandled error happened.
Is it possible to log url of the request when InternalServerError happened?
If yes, what is the best way to do it?
Isn't it doing this right out of the box?
Here's log from g8 template project from quick start guide https://http4s.org/v0.21/
with added route that throws runtime exception:
[ioapp-compute-1] INFO o.h.s.m.Logger - HTTP/1.1 GET /bad_joke Headers(Accept: */*, Cache-Control: no-cache, Host: localhost:8089, Connection: Keep-Alive, User-Agent: Apache-HttpClient/4.5.12 (Java/11.0.8), Accept-Encoding: gzip,deflate) body=""
[ioapp-compute-1] INFO o.h.s.m.Logger - service raised an error: class java.lang.RuntimeException
[ioapp-compute-1] ERROR o.h.s.service-errors - Error servicing request: GET /bad_joke from 127.0.0.1
java.lang.RuntimeException: haha
at pack.H4sRoutes$$anonfun$jokeRoutes$1.applyOrElse(H4sRoutes.scala:19)
at pack.H4sRoutes$$anonfun$jokeRoutes$1.applyOrElse(H4sRoutes.scala:13)
at $anonfun$combineK$1 # org.http4s.syntax.KleisliResponseOps.$anonfun$orNotFound$1(KleisliSyntax.scala:38)
at getOrElse # org.http4s.syntax.KleisliResponseOps.$anonfun$orNotFound$1(KleisliSyntax.scala:38)
at guaranteeCase$extension # org.http4s.server.middleware.RequestLogger$.$anonfun$impl$9(RequestLogger.scala:97)
at map # org.http4s.server.middleware.RequestLogger$.$anonfun$impl$9(RequestLogger.scala:101)
at flatMap # org.http4s.server.middleware.RequestLogger$.$anonfun$impl$7(RequestLogger.scala:82)
at flatMap # org.http4s.server.middleware.ResponseLogger$.$anonfun$impl$7(ResponseLogger.scala:65)
at guaranteeCase$extension # org.http4s.server.middleware.ResponseLogger$.$anonfun$impl$7(ResponseLogger.scala:88)
at main$ # pack.Main$.main(Main.scala:5)

How to hide stack trace of Kafka connect api unhandled exception

As part of registering connectors in distributed mode, when some invalid json payload is passed in API request , I am getting error in response with full stack trace, which is not desirable in my case.
Response example:
HTTP/1.1 500 Internal Server Error Connection: close Date: Fri, 26 Jul 2019 08:27:17 GMT Content-Type: application/json Content-Length: 443 Server: Jetty(9.4.11.v20180605)
{"error_code":500,"message":"Cannot construct instance of `java.util.LinkedHashMap` (although at least one Creator exists): no String-argument constructor/factory method to deserialize from String value ('')\n at [Source: (org.glassfish.jersey.message.internal.ReaderInterceptorExecutor $UnCloseableInputStream); line: 2, column: 46] (through reference chain: org.apache.k afka.connect.runtime.rest.entities.CreateConnectorRequest[\"config\"])"}
Is there any way to hide or shorten the full stack trace.
Note: I am using Hortonworks Kafka package.
The REST API returns what the REST API returns :)
So you could either (a) stop sending invalid JSON ;) (b) compile your own version of Kafka and disable the bits of output you don't want returned.

OSB - Processing Http errors from Restful service in Proxy Service

I'm invoking a REST service from an OSB Proxy Service, which is working fine if the rest service response is a valid response, i.e., there's no errors. However if the rest service replies with an http error code (e.g. 400 Bad Request), I'm not able to capture anything else except the http error code:
$fault variable in OSB:
<con:fault xmlns:con="http://www.bea.com/wli/sb/context">
<con:errorCode>BEA-382502</con:errorCode>
<con:reason>
OSB Service Callout action received an error response
</con:reason>
<con:details>
<con1:ErrorResponseDetail xmlns:con1="http://www.bea.com/wli/sb/stages/transform/config">
<con1:http-response-code>400</con1:http-response-code>
</con1:ErrorResponseDetail>
</con:details>
<con:location>
<con:node>PipelinePairNode1</con:node>
<con:pipeline>PipelinePairNode1_request</con:pipeline>
<con:stage>stage1</con:stage>
<con:path>request-pipeline</con:path>
</con:location>
</con:fault>
But the rest service, is not only replying with 400 Bad Request, but also adding a message:
<Error>
<Message>The message header contains an invalid brand code.</Message>
</Error>
Which I'm not able to process in my proxy service. Does anyone knows if it is possible to access this message details in OSB or is it a limitation?
The rest service is clearly sending the message, because invoking it directly through soap-ui I'll get the following response:
HTTP/1.1 400 Bad Request
Cache-Control: no-cache
Pragma: no-cache
Content-Length: 84
Content-Type: application/xml; charset=utf-8
Expires: -1
Server: Microsoft-IIS/8.5
X-AspNet-Version: 4.0.30319
X-Powered-By: ASP.NET
Date: Mon, 09 Feb 2015 16:15:32 GMT
<Error><Message>The message header contains an invalid brand code.</Message></Error>
Thanks
I've just found out this is an OSB bug, in case of an HTTP Error code 400 being returned, if the http-content is "Application/XML" the message is ignored.
Oracle released a Patch at the end of last month to fix this issue, which I've tested and it works. I can get the message details in the $body variable now.
Patch: 16986497
If I understand your question, you want to get in addition to the http code, the message error?
I think you should use, in your message flow, an errorHandler , so that you can catch the error and do whatever you want with.
The proxy service operation selection algorithm cannot determine the operation name from the request or returns an invalid operation (one which is not in the WSDL or null). Possible reasons include the following:
An error occurs while computing the operation.
The operation selection algorithm returns null.
The operation selection algorithm returns an operation that is not of the of the operations declared by the WSDL.
Source

SSLError (read operation timed out) instead of timeout when PUT using Requests

I'm trying to do various REST requests to a piece of equipment using the Requests package.
It is working, but on some requests, like PUTs, I'm getting a SSLError, instead of a timeout that I'm specifying.
I have the code set up to retry, up to 5 times, doubling the timeout each time (1,2,4,8, 16, 32), and if I treat the SSLError as a timeout, then it will eventually pass. Here's an example, and you can see the SSLError is occurring at the pace of the timeout interval:
2013-12-10 19:41:13.208 22294 DEBUG client [-] PUT: Request for https://192.168.200.20/api/v1/global/host-name headers {'content-type': 'application/json', 'Accept': 'application/json', 'X-auth-token': u'...omitted...'} payload {'host-name': 'TestHost'}
2013-12-10 19:41:13.209 22294 INFO requests.packages.urllib3.connectionpool [-] Starting new HTTPS connection (1): 192.168.200.20
2013-12-10 19:41:14.253 22294 ERROR client [-] EXCEPTION The read operation timed out <<< SSL Error
2013-12-10 19:41:14.255 22294 INFO requests.packages.urllib3.connectionpool [-] Starting new HTTPS connection (1): 192.168.200.20
2013-12-10 19:41:16.301 22294 ERROR client [-] EXCEPTION The read operation timed out
2013-12-10 19:41:16.302 22294 INFO requests.packages.urllib3.connectionpool [-] Starting new HTTPS connection (1): 192.168.200.20
2013-12-10 19:41:20.346 22294 ERROR client [-] EXCEPTION The read operation timed out
2013-12-10 19:41:20.348 22294 INFO requests.packages.urllib3.connectionpool [-] Starting new HTTPS connection (1): 192.168.200.20
2013-12-10 19:41:25.845 22294 DEBUG requests.packages.urllib3.connectionpool [-] "PUT /api/v1/global/host-name HTTP/1.1" 204 0 _make_request /usr/local/lib/python2.7/dist-packages/requests/packages/urllib3/connectionpool.py:289
2013-12-10 19:41:25.846 22294 DEBUG client [-] PUT: Completed [204]
As you can see the EXCEPTION log messages are the SSLError errors, but they are timeouts as they map to the timeout I specify.
In python, I'm doing this for the calls (it's spread out so here is the gist of it):
url = ('https://%(host)s/api/v1/%(resource)s' %
{'host': self.host, 'resource': resource})
...
response = self._request(method, url, try_num, timeout=timeout,
headers=headers, data=payload)
...
def _request(self, method, url, attempt, **kwargs):
...
try:
response = requests.request(method, url, verify=False, **kwargs)
except Timeout:
self.status = wexc.HTTPRequestTimeout.code
LOG.debug(_("%(method)s: Request timeout" ...)
except SSLError as se:
LOG.error("EXCEPTION %s", se)
self.status = wexc.HTTPRequestTimeout.code
If I don't handle the SSL error, then it gets caught as a ConnectionError. Any idea what I may be doing wrong?
I have this hack in place, but would like to know why I get the SSL error.
Thanks Lukasa! You're question prompted me to investigate the version. I installed requests 2.1.0 and now I see Timeout exceptions, instead of SSLError exceptions, which is what I expected to see.

Creating a WebSocket Client in Python

I am trying to learn about socket programming as well as the WebSocket protocol. I know that there are python web socket clients in existence but I am hoping to just build a toy version for my own learning. To do this I have created an extremely simple Tornado websocket server that I am running on localhost:8888. All it does is print a message when a client connects.
This is the entire server - and it works (I have tested it with a small javascript script in my browser)
import tornado.httpserver
import tornado.websocket
import tornado.ioloop
import tornado.web
class WSHandler(tornado.websocket.WebSocketHandler):
def open(self):
print('new connection')
self.write_message("Hello World")
def on_message(self, message):
print('message received %s' % message)
def on_close(self):
print('connection closed')
application = tornado.web.Application([
(r'/ws', WSHandler),
])
if __name__ == "__main__":
http_server = tornado.httpserver.HTTPServer(application)
http_server.listen(8888)
tornado.ioloop.IOLoop.instance().start()
So once I start up the server I try to run the following script
import socket
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.connect((socket.gethostbyname('localhost'), 8888))
msg = '''GET /chat HTTP/1.1
Host: server.example.com
Upgrade: websocket
Connection: Upgrade
Sec-WebSocket-Key: dGhlIHNhbXBsZSBub25jZQ==
Origin: http://example.com
Sec-WebSocket-Protocol: chat, superchat
Sec-WebSocket-Version: 13'''.encode('ascii')
print(len(msg))
sent_count = sock.send(msg)
print('sent this many bytes:', sent_count)
recv_value = sock.recv(1)
print('recvieved:', recv_value)
What I am hoping is that the server will send back the response header as specified in the RFC. Instead the sock.recv is hanging. This leads me to believe the server isn't acknowledging the websocket initial handshake. This handshake is pulled off of the RFC as well. I know that the websocket key should be random and everything, but I don't think that would cause the server to ignore the handshake (the websocket key is valid). I think I can figure the rest out once I can initiate the handshake so I am hoping that there is just some misunderstanding in either how websockets work or how to send the initial handhake.
1) When you send a message over a socket, you have no idea how many chunks it will be divided into. It may all get sent at once; or the first 3 letters may be sent, then the rest of the message; or the message may be split into 10 pieces.
2) Given 1) how is the server supposed to know when it has received all the chunks sent by the client? For instance, suppose the sever receives 1 chunk of the client's message. How does the server know whether that was the whole message or whether there are 9 more chunks coming?
3) I suggest you read this:
http://docs.python.org/2/howto/sockets.html
(Plus the links in the comments)
4) Now, why aren't you using python to create an HTTP server?
python3:
import http.server
import socketserver
PORT = 8000
handler = http.server.SimpleHTTPRequestHandler
httpd = socketserver.TCPServer(("", PORT), handler)
print("serving at port", PORT)
httpd.serve_forever()
python2:
import SimpleHTTPServer
import SocketServer
PORT = 8000
handler = SimpleHTTPServer.SimpleHTTPRequestHandler
httpd = SocketServer.TCPServer(("", PORT), handler)
print "serving at port", PORT
httpd.serve_forever()
The SimpleHTTPRequestHandler serves files out of the server program's directory and below, matching the request url to the directory structure you create. If you request '/', the server will serve up an index.html file out of the same directory the server is in. Here is an example of a client socket for python 3 (python 2 example below):
import socket
import sys
try:
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
except socket.error:
print('Failed to create socket')
sys.exit()
print('Socket Created')
#To allow you to immediately reuse the same port after
#killing your server:
s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
host = 'localhost';
port = 8000;
s.connect((host , port))
print('Socket Connected to ' + host + ' on port ', port)
#Send some data to server
message = "GET / HTTP/1.1\r\n\r\n"
try :
#Send the whole string(sendall() handles the looping for you)
s.sendall(message.encode('utf8') )
except socket.error:
print('Send failed')
sys.exit()
print('Message sent successfully')
#Now receive data
data = []
while True:
chunk = s.recv(4096) #blocks while waiting for data
if chunk: data.append(chunk.decode("utf8"))
#If the recv() returns a blank string, then the other side
#closed the socket, and no more data will be sent:
else: break
print("".join(data))
--output:--
Socket Created
Socket Connected to localhost on port 8000
Message sent successfully
HTTP/1.0 200 OK
Server: SimpleHTTP/0.6 Python/3.2.3
Date: Sat, 08 Jun 2013 09:15:18 GMT
Content-type: text/html
Content-Length: 23
Last-Modified: Sat, 08 Jun 2013 08:29:01 GMT
<div>hello world</div>
In python 3, you have to use byte strings with sockets, otherwise you will get the dreaded:
TypeError: 'str' does not support the buffer interface
Here it is in python 2.x:
import socket
import sys
try:
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
except socket.error:
print 'Failed to create socket'
sys.exit()
print('Socket Created')
#To allow you to immediately reuse the same port after
#killing your server:
s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
host = 'localhost';
port = 8000;
s.connect((host , port))
print('Socket Connected to ' + host + ' on port ', port)
#Send some data to server
message = "GET / HTTP/1.1\r\n\r\n"
try :
#Send the whole string(handles the looping for you)
s.sendall(message)
except socket.error:
print 'Send failed'
sys.exit()
print 'Message sent successfully'
#Now receive data
data = []
while True:
chunk = s.recv(4096) #blocks while waiting for data
if chunk: data.append(chunk)
#If recv() returns a blank string, then the other side
#closed the socket, and no more data will be sent:
else: break
print("".join(data))
--output:--
Message sent successfully
HTTP/1.0 200 OK
Server: SimpleHTTP/0.6 Python/2.7.3
Date: Sat, 08 Jun 2013 10:06:04 GMT
Content-type: text/html
Content-Length: 23
Last-Modified: Sat, 08 Jun 2013 08:29:01 GMT
<div>hello world</div>
Note that the header of the GET requests tells the server that HTTP 1.1 will be the protocol, i.e. the rules governing the conversation. And as the RFC for HTTP 1.1 describes, there has to be two '\r\n' sequences in the request. So the server is looking for that second '\r\n' sequence. If you delete one of the '\r\n' sequences from the request, the client will hang on the recv() because the server is still waiting for more data because the server hasn't read that second '\r\n' sequence.
Also note that you will be sending the data as bytes(in python 3), so there are not going to be any automatic '\n' conversions, and the server will be expecting the sequence '\r\n'.