According to this response you can set a time limit for a query operation via find() parameter or a collection method:
cursor = db.collection.find(max_time_ms=1)
or
cursor = db.collection.find().max_time_ms(1)
The doc says:
max_time_ms (optional): Specifies a time limit for a query operation. If the specified time is exceeded, the operation will be aborted and ExecutionTimeout is raised. Pass this as an alternative to calling max_time_ms() on the cursor.
We're currently experiencing a problem that a query runs for ~30 minutes before it eats all the RAM and the server dies. I hope this parameter gives a hard limit on the query and after the given time the server gives up.
Since our app is full of finds and cursors: is there a way how to set this parameter directly in the MongoClient constructor?
The doc says:
socketTimeoutMS: (integer or None) Controls how long (in milliseconds) the driver will wait for a response after sending an ordinary (non-monitoring) database operation before concluding that a network error has occurred. Defaults to None (no timeout).
connectTimeoutMS: (integer or None) Controls how long (in milliseconds) the driver will wait during server monitoring when connecting a new socket to a server before concluding the server is unavailable. Defaults to 20000 (20 seconds).
serverSelectionTimeoutMS: (integer) Controls how long (in milliseconds) the driver will wait to find an available, appropriate server to carry out a database operation; while it is waiting, multiple server monitoring operations may be carried out, each controlled by connectTimeoutMS. Defaults to 30000 (30 seconds).
...couldn't find another timeout and none of these seem to be the equivalent of max_time_ms. Am I missing something?
Related
I have a program within the same machine as the Postgres server. The program creates JDBC requests to fetching tuples from the underlying Postgres and then perform some processing. For a JDBC request resultSet = statement.executeQuery();, how can I measure the time spent by Postgres, i.e., Postgres receive request and get the tuple from underlying disk and return the tuple? In other words, is there any more fine-grained way than measuring the time spent around executing above code, e.g.,
// timer start
resultSet = statement.executeQuery();
// timer end
Thanks!
To measure the time spent on the server, set the PostgreSQL parameter log_min_duration_statement to a small value. Then every statement whose execution time exceeds the value will be logged, along with the time it took to process on the server.
As normal, the app can get responses quickly from MongoDB. However, if the server hasn't sent any query to MongoDB for a long time (~15min), the 1st query after that will take a longer time but subsequence queries are fine.
I guess there are a few possibilities:
MongoDB idle, 1st query will awake it.
Connection between server-MongoDB is idle or timeout, server takes time to reconnect to MongoDB.
I'm using MongoDB version 4.4 so SocketKeepAlive = true by default. Am I missing any configuration for this ?
When the Go driver detects that a context is cancelled, I think it will idle the connection and return it to the pool. Does that imply the running DB operation that it started will also be killed on the server? Or does that continue on?
You can read the official documentation about how the driver uses the context here.
Relevant section: Socket Read and Write:
When the driver retrieves a connection for an operation, it sets the socket’s read or write deadline to either the Context deadline or socket timeout, whichever is shorter.
If you cancel the Context after the execution of the Read() or Write() function but before its deadline, the behavior of the driver differs based on version.
The driver generates a separate goroutine to listen for Context cancellation when the Read() or Write() function is in progress. If the goroutine detects a cancellation, it closes the connection. The pending Read() or Write() function returns an error which the driver overwrites with the context.Canceled error.
Note that closing the connection just means putting it back to the pool as idle. There is no explict statement about cancelling the operation on the server, but obviously a signal is sent that the client abandons the operation and no longer monitors / cares for the result. The MongoDB server would be dumb not to abort the initiated operation if it can be. For example if the operation was a query, that can be aborted. If it was a write operation (such as insert or update), that may not be aborted.
I found two different places with different explanation what does socketTimeoutMS do.
The time in milliseconds to attempt a send or receive on a socket before the attempt times out. The default is never to timeout, though different drivers might vary. See the driver documentation.
From here
And following one:
The socketTimeoutMS sets the number of milliseconds a socket stays inactive after the driver has successfully connected before closing. If the value is set to 360000 milliseconds, the socket closes if there is no activity during a 30 seconds window.
From here
What does really socketTimeoutMS do?
Due to what the docs say, it will be the functionality what the driver is providing
See the driver documentation.
In your case if you're using nodejs (the link you sent) it will be as you quoted in your second quote
The socketTimeoutMS sets the number of milliseconds a socket stays inactive after the driver has successfully connected before closing. If the value is set to 360000 milliseconds, the socket closes if there is no activity during a 30 seconds window.
Just looking for an explanation of rationale for this bit of code (PoolUtiltites:293 in version 2.2.4):
dataSource.setLoginTimeout((int) TimeUnit.MILLISECONDS.toSeconds(Math.min(1000L, connectionTimeout)));
This code and the setConnectionTimeout method means that I get this behaviour:
connectionTimeout == 0, then loginTimeout = Integer.MAX_VALUE
connectionTimeout > 0 && < 100, then HikariConfig throws IllegalArgumentException
connectionTimeout >= 100 && <= 1000, then loginTimeout = connectionTimeout
connectionTeimout > 1000, then loginTimeout = 1000
That looks really weird to me!
It's almost like the Math.min should be Math.max ???
In my current project I'd like to fail connections after 30s, which is impossible in the current setup.
I'm using the 4.1 postgres jdbc driver, but I think this is not relevant to the issue above.
Many thanks - and cool pooling library!!!
Ok, there are a couple of moving parts here. First, Math.min() is a bug, it should be Math.max(). In light of that (it will be fixed) consider the following:
It is important to note that connections are created asynchronously in the pool. The setConnectionTimeout() sets the maximum time (in milliseconds) that a call to getConnection() will wait for a connection before timing out.
The DataSource loginTimeout is the maximum time that physical connection initiation to the database can take before timing out. Because HikariCP obtains connections asynchronously, if the connection attempt fails, HikariCP will continue to retry, but your calls to getConnection() will timeout appropriately. We are using the connectionTimeout in kind of a double duty for loginTimeout.
For example, lets say the pool is completely empty, and you have configured a connectionTimeout of 30 seconds. When you call getConnection() HikariCP, realizing that there are no idle connections available, starts trying to obtain a new one. There is little point in having a loginTimeout exceeding 30 seconds, in this case.
The intent of the Math.max() call is to ensure that we never set loginTimeout to 0 if the user has configured connectionTimeout to 250ms. TimeUnit.MILLESECONDS.toSeconds() would return 0 without the Math.max(). If the user has configured a connectionTimeout of 0, meaning they never want to timeout, the time conversion of Integer.MAX_VALUE results in several thousand years as a timeout (virtually never).
Having said that, and in light of how HikariCP connections to the database are obtained asynchronously, even without the Math.max() fix, you should be able to achieve application-level connection timeouts of 30s. Unless physical connections to your database exceed 1000ms you would be unaffected by the Math.min().
We are putting out a 2.2.5-rc3 release candidate in the next few hours. I will slot this fix in.