I am using nghttp2 to implement a RESTful API server. I have defined two GET APIs:
/api/ping and /api/wait. While the response to the former is sent immediately, the server does some processing before responding to the latter. I allotted 4 threads to the server.
From a client (also implemented using nghttp2), I made a connection to the server and made the API calls one by one, /api/wait first followed by /api/ping. I observed using Wireshark that the two GET requests are sent over two different TCP packets. However, until the server completes processing of the /api/wait, it does not not process /api/ping, although it has other threads available.
I established two TCP connections from the client and made the two API calls on the different connections and the server processed those in parallel.
Does this mean that nghttp2 processes one TCP connection exclusively on one thread and requests from one TCP connection are processed sequentially by design? Is there any setting in nghttp2 to circumvent this? This may be a good feature for a web application (processing requests sequentially) but not an API server where APIs are independent of each other.
Related
I'm building a software agent that run on a server, this software agent act as a server manager i.e. starting/stoping Docker container, monitoring etc.
This server will host/serve many services, these services are programs running in Docker container, 1 program/service per container.
There may be so many servers and these servers aren't necessary be a high performance server, they ranges from a small VM to high performance computer. Right now, I assume that every service uses HTTP to serve request.
The function that I want to implement in this software agent is tracking the number of clients that are currently connecting (requesting) to server for every service (e.x. server A is processing 500 requests) or specific program is ok (e.x. program A is processing 100 requests, program B is processing 200 request).
I want to know this number because I want to do workload balancing across servers that host the same service.
The following is ideas that I have.
Implementing load balancer/reverse proxy inside this agent (I would use this load balancer https://github.com/nwoodthorpe/Load-Balancer-Golang). This may be the last choice because I think it will use pretty much resources for load balancing.
Letting service programs that are running on server tell agent whenever they start and finish processing request. I simply implement UDP socket server in agent to listen for a datagram that tell unique ID of request (actually, can be anything that help me distinguish specific request that being processed) and status whether is being processed or finish processing.
So, I would like to ask for a suggestion for above approaches, which one is better or how should I implement it? Is there any better approach to do this?
Maybe I haven't understood what back pressure is trying to solve, but let's say in a microservice architecture where everything is communicating via HTTP (which is backed by TCP of course). Back pressure exists so that no service gets loaded with requests more than what it can process. Now if a service is busy processing requests, won't this be propagated back to the underlying TCP connection where the HTTP requests are being read (i.e. the process won't 'read' from the underlying socket it is listening on?
I am using Camel Netty for full duplex communication over TCP socket.
My application is using the following parameters in the route.
<inOut uri="netty:tcp://{{IP-Port}}?
textline=true&sync=true&decoderMaxLineLength=1000000&autoAppendDelimiter=false&disconnect=false&producerPoolMaxActive=-1&producerPoolMinEvictableIdle=120000&keepAlive=false&noReplyLogLevel=INFO&serverExceptionCaughtLogLevel=INFO&requestTimeout=2500" />
The netty component above receives requests from a preceding wiretap in the flow.
During the day after about 8-10 hours, some of the connections show as ESTABLISHED state but will not be serving any requests. Even at the server end, these connections show as ESTABLISHED but there is no activity for hours.
When we looked at one connection closely, found that the last request attempted (not been received by server) was writing body to endpoint and got an exception org.apache.camel.processor.DefaultErrorHandler - Failed delivery for (MessageId: xxxxx on ExchangeId: ID-xxxx). On delivery attempt: 0
Since netty is being called from wiretap, after this last request, succeeding requests are not even entertained and they are blocked in wiretap itself..
I am collecting tcpdump later tonight for more details though.
Questions:
1. Why is producerPoolMinEvictable NOT kicking in to clear such stale connections?
2. How do we clear these stale connections automatically without having to
bounce the application?
3. Is there a problem using wiretap?
Appreciate suggestions to resolve this issue. Please ask for any more details needed to answer and I shall be happy to share.
Note:
camel-netty
2.11.2-
I am working with Mule ESB CE 3.5.0 and am seeing what I believe is a resource leak on the TCP connections. I am hooking up VisualVM and checking the memory. I see that it increases over time without ever decreasing.
My scenario is that I have messages being sent to Mule, Mule does its thing, and then dispatches to a remote TCP endpoint (on the same box, usually). What I did was not start up the program that would receive a message from Mule's TCP outbound endpoint. So there is nothing listening for Mule's dispatched message.
I configure my TCP connectors as following:
<tcp:connector name="TcpConnector" keepAlive="true" keepSendSocketOpen="true" sendTcpNoDelay="true" reuseAddress="true">
<reconnect-forever frequency="2000" />
<tcp:xml-protocol />
</tcp:connector>
<tcp:endpoint name="TcpEndpoint1" responseTimeout="3000" connector-ref="TcpConnector" host="${myHost}" port="${myPort}" exchange-pattern="one-way" />
My questions are:
When a flow fails to send to the TCP outbound endpoint, what happens to the message? Is the message kept in memory somewhere and once the TCP connector establishes connections to the remote endpoint, do all the accumulated messages burst through and get dispatched?
When the reconnection strategy is blocking, I assume it is a dispatcher thread that tries to establish the connection. If we have more message to dispatch, then are there more dispatcher threads that are tied up to attempting the reconnection? What happens if it is non-blocking?
Thanks!
Edit:
If I also understand the threading documentation correctly, does that mean that if I have the default threading profile set to poolExhaustedAction="RUN", and all the dispatcher threads block waiting for a connection, eventually the flow threads, and then the receiver threads will block on trying to establish the connection. When the remote application begins listening again, all the backlogged messages from the blocked threads will burst through.
So if the flow receives transient data, it should be configured to have non-blocking reconnection and since it is acceptable to throw away the messages (in my use case), then we can make do with the exception that will be thrown.
I would point you to the documentation:
Non-Blocking Reconnection
By default, a reconnection strategy will block Mule application
message processing until it is able to connect/reconnect. When you
enable non-blocking reconnection, the application does not need to
wait for all endpoints to re-connect before it restarts. Furthermore,
if a connection is lost, the reconnection takes place on a thread
separate from the application thread. Note that such behavior may or
may not be desirable, depending on your application needs.
On blocking reconnection strategies what you are going to get is that the dispatcher will get blocked, waining for an available connection. The messages are not technically kept anywhere, their flow is just stopped.
Regarding the second question it changes between transport and transport. In this very special case, given that tcp is a connection per request transport, different dispatchers will try to get a different socket form the pool of connections.
In case of non-blocking strategies you will get an exception. You can probably test it easily.
When deploying a web application running on a traditional web server, you usually restart the web server after the code updates. Due to the nature of HTTP, this is not a problem for the users. On the next request they will get the latest updates.
But what about a WebSocket server? If I restart or kill the old process all connected users will get disconnected. So my question is, what kind of strategy have you used to deploy a WebSocket server smoothly?
You're right, every connected user will be disconnected if the server restarts.
I think the less bad solution is to tell to the client to reconnect in the onClose method of the client.
WebSockets is just a transport mechanism. Libraries like socket.io exist to build on that transport -- and provide heartbeats, browser fallbacks, graceful reconnects and handle other edge-cases found in real-time applications.
In our WebSocket-enabled application, socket.io is central to ensuring our continuous deployment setup doesn't break users' active socket connections.
If clients are connected directly to sever that does all sockets networking and application logic, then yes - they will be disconnected, due to TCP layer that holds connection.
If you have gateway that clients will be connecting to, and that gateway application is running on another server, but will communicate and forward messages to logical server, then logical server will send them back and gateway will send back to client responses. With such infrastructure, you have to implement stacking of packets on gateway until it will re-establish connection with logical server. Logical server might notify gateway server before restart. That way client will have connection, it will just wont receive any responses.
Or you can implement on client side reconnection.
With HTTP, every time you navigate away, browser actually is creating socket connection to server, transmits all data and closes it (in most cases). And then all website data is local, until you navigate away.
With WebSockets it is continuous connection, and there is no reconnection on requests. Thats why you have to implement simple mechanics when WebSockets getting closing event, you will try to reconnect periodically on client side.
It is more based on your specific needs.