I'm using LWP::UserAgent to communicate with webservices on several servers; the servers are contacted one at a time. Each response might take up to 30 minutes to finish, so I set the LWP timeout to 30 minutes.
Unfortunately the same timeout also applies, if the server is not reachable at all (e.g. the webserver is down). So my application waits 30 minutes for a server, which is not running.
Is it feasable to set two seperate timeouts?
a short one, which waits for the connection to be established.
a longer one, which waits for the response, once the connection has been established.
The same timeout doesn't "also apply" if the server is not reachable. The timeout option works in a very specific way:
The request is aborted if no activity on the connection to the server is
observed for timeout seconds. This means that the time it takes for the
complete transaction and the request() method to actually return might be
longer.
As long as data is being passed, the timeout won't be triggered. You can use callback functions (see the REQUEST METHODS section of the docs) to check how long data transfer has been going on, and to exit entirely if desired.
Related
In pgjdbc we have:
loginTimeout
connectTimeout
socketTimeout
cancelSignalTimeout
But it isn't clear to me what's the difference (when are they applied) between loginTimeout, connectTimeout and socketTimeout.
As documented in the PostgreSQL JDBC documentation:
loginTimeout = int
Specify how long to wait for establishment of a database connection.
The timeout is specified in seconds.
connectTimeout = int
The timeout value used for socket connect operations. If connecting
to the server takes longer than this value, the connection is broken.
The timeout is specified in seconds and a value of zero means that it
is disabled.
socketTimeout = int
The timeout value used for socket read operations. If reading from
the server takes longer than this value, the connection is closed.
This can be used as both a brute force global query timeout and a
method of detecting network problems. The timeout is specified in
seconds and a value of zero means that it is disabled.
cancelSignalTimeout = int
Cancel command is sent out of band over its own connection, so
cancel message can itself get stuck. This property controls "connect
timeout" and "socket timeout" used for cancel commands. The timeout is
specified in seconds. Default value is 10 seconds.
The connectTimeout and socketTimeout are timeouts on low-level socket operations. The connectTimeout governs the time needed to establish a TCP socket connection. Establishing a TCP connection doesn't guarantee a login (it doesn't even guarantee that you're connecting to a PostgreSQL server, just that you connected to something that accepted your TCP connection). A socketTimeout governs the time a socket can be blocked waiting to read from a socket. This involves all reads from the server, not just during connect, but also during subsequent interaction with the server (eg executing queries).
On the other hand loginTimeout governs the PostgreSQL protocol operation of connecting and authenticating to the PostgreSQL server. This involves establishing a TCP connection followed by one or more exchanges of packets for the handshake and authentication to the PostgreSQL server (I'm not familiar with the details of the PostgreSQL protocol, so I can't be very specific).
Exchanging these packets can take additional time, or if you connected to something that isn't a PostgreSQL server the packet exchange may stall. It might be possible to solve this with careful control of both connectTimeout and socketTimeout, but there are no guarantees (eg data is being exchanged, but the login never completes). In addition, as the socketTimeout also governs all other operations on the connection, you may want to set it higher (eg for other operations that take a long time to get a response back) than you are willing to wait for the login to complete.
The cancelSignalTimeout is used as the connect and socket timeout of the separate TCP connection used for cancel commands.
After reading the source, I'd say it is like this:
connectTimeout specifies how long to wait for a TCP network connection to get established
loginTimeout specifies how long the whole process of logging into the database is allowed to take
socketTimeout specifies how long the client will wait for a response to a command from the server before throwing an error
The first two are related to establishing a connection, the third is relevant for the whole database session.
Establishing a TCP connection is part of establishing a database connection.
I'm not quite clear about the options http-request or http-response in HAProxy configuration.
Many of the parms seem to be used for modification of http request and response but what I found is it can be done using the regular option as well
What's the difference between
http-request set-src hdr(x-forwarded-for) #and
option forwardfor
Also what's the difference between:
connect timeout 5000
client timeout 5000
server timeout 5000 #And
http-request timeout 5000
I'm new to haproxy and from the documentation is written from configuration parameter perspective (like a api reference) instead of a use-case perspective (like a user-guide).
So If I asked an absurd question please do not mind and answer kindly. Thanks
What's the difference?
These first two are sort of opposites.
Configure HAProxy to use the contents of the X-Forward-For header to establish its internal concept of the source address of the request, instead of the actual IP address initiating the inbound connection:
http-request set-src hdr(x-forwarded-for)
Take the IP address of the inbound connection and add an X-Forwarded-For header for the benefit of downstream servers:
option forwardfor
Also what's the difference?
Let's take this one backwards.
First, this isn't valid in any version that I'm aware of:
http-request timeout 5000
I believe you mean this...
timeout http-request 5000
...which sets the timeout for the client to send complete, valid HTTP headers and an extra \r\n signifying end of headers. This timer doesn't usually apply to the body, if there is one -- only the request headers. If this timer fires, the transaction is aborted, a 408 Request Timeout is returned, and the client connection is forcibly closed. This timer stops once complete request headers have been received.
By default, this timeout only applies to the header part of the request,
and not to any data. As soon as the empty line is received, this timeout is
not used anymore.
http://cbonte.github.io/haproxy-dconv/1.6/configuration.html#4-timeout%20http-request
Note: http-request is something entirely different, and is used for manipulating the request during the request processing phase of the transaction lifecycle, before the request is sent to a back-end server.
Continuing, these aren't actually valid, either.
connect timeout 5000
client timeout 5000
server timeout 5000
It seems you've reversed the keywords. I believe you're thinking of these:
timeout connect 5000
That's the maximum time to wait for the back-end to accept our TCP connection by completing its share of the 3-way handshake. It has no correlation with timeout http-request, which is only timing the client sending the initial request. If this timer fires, the proxy will abort the transaction and return 503 Service Unavailable.
timeout client 5000
This one does overlap with timeout http-request, but not completely. If this timer is shorter than timeout http-request, the latter can never fire. This timer applies any time the server is expecting data from the client. The transaction aborts if this timer fires. I believe if this happens, the proxy just closes the connection.
timeout server 5000
This is time spent waiting for the server to send data. It also has no overlap with timeout http-request, since that window has already closed before this timer starts running. If we are waiting for the server to send data, and it is idle for this long, the transaction is aborted, a 504 Gateway Timeout error is by HAProxy, and the client connection is closed.
So, as you can see, the overlap here is actually pretty minimal between these three and timeout http-request.
You didn't really ask, but you'll find significant overlap between things like http-response set-header and rsp[i]rep, and http-request set-header and req[i]rep. The [req|rsp][i]rep keywords represent older functionality that is maintained for compatibility but largely obsoleted by the newer capabilities that have been introduced, and again, there's not as much overlap as there likely appears at first glance because the newer capabilities can do substantially more than the old.
I'm new to haproxy and from the documentation is written from configuration parameter perspective (like a api reference) instead of a use-case perspective (like a user-guide).
That seems like a fair point.
The situation:
Postgres 9.1 on Debian Server
Scala(Java) application using the LISTEN/NOTIFY mechanism to get notified through JDBC
As there can be very long pauses (multipla days) between notifications I ran into the problem that the underlying TCP connection silently got terminated after some time and my application stopped to receive the notifications.
When googeling for a solution I found that there is a parameter tcpKeepAlive that you can set on the connection. So I set it to true and was happy. Until the next day I saw that again my connection was dead.
As I had been suspicious there was a wireshark capture running in parallel which now turns out to be very usefull. Just about exactly two hours after the last successfull communication on the connection of interest my application sends a keepalive packet to the database server. However the server responds with RST as it seems it has already closed the connection.
The net.ipv4.tcp_keepalive_time on the server is set to 7200 which is 2 hours.
Do I need to somehow enable keepalive on the server or increase the keepalive_time?
Is this the way to go about keeping my application connected?
TL;DR: My database connection gets terminated after long inactivity. Setting tcpKeepAlive didnt fix it as server responds with RST. What to do?
As Craig suggested in the comments the problem was very likely related to some piece of network hardware in between the server and the application. The fix was to increase the frequency of the keepalive messages.
In my case the OS was Windows where you have to create a Registry key with the idle time in milliseconds after which the message should be sent. Info on that here
I have set it to 15 minutes which seems to have solved the issue.
UPDATE:
It only seemed like it solved the issue. After about two days of program run time my connection was gone again. I switched to checking the validity my connection every time I use it. This does not seem like it is the solution but it is a solution nonetheless.
So I have a Perl script on a server. (Linux...). The script takes about 3+ minutes to fully complete, (this is normal for my script). Although the server keeps disconnecting, and my browser says that the server is not responding (it timed out I guess). How can I keep the connection alive for over 3+ minutes? (The client is just waiting for a response from the server. Nothing else on the client side is happening)...
Is this even possible?
If the server is closing the connection, you need to increase the server (Apache?) script timeout, which will be a parameter to mod_cgi or mod_cgid (depending on which one you're using). If you cannot change the Apache configuration then you might experiment with sending an innocuous HTTP header (i.e. Connection: keepalive, which is the default anyway) immediately before starting your processing. This will probably be sufficient to cause Apache not to give up waiting.
I couldn't find any documentation for the -oConnectTimeout option. So I was just wondering if its just for the ssh connection to occur or for the total connection? like if I have a single line command wherein I connect to a sqlserver from command line and execute a query that takes 20 seconds, then should I keep the timeout as 5 or 10 seconds for the conn to occur, or the whole 30-35 seconds for the query to be completed also?
Its for the time it takes to connect to the server. Although I would not recommend relying on -oConnectTimeout, I'd rather close the connection when the job is done, regardless of how long it takes. You should use this value to increase the timeout more than the default TCP timeout.