When I try to:
set wlm_query_slot_count to 10;
I get an error msg, "...query cannot run because wlm_query_slot_count is greater than the query concurrency for the queue, ... increase the query concurrency if you need more slots"
I've searched and searched, but cannot figure out where I can change the concurrency level (I'd like to raise it from its current 5 to 10).
Found the answer in AWS docs, under parameter groups
Change concurrency setting
Related
I'm facing an issue where 2 out of 10 spanner databases are showing a high CPU usage (above 40%) whereas the others are around %1 each, with almost identical or more data.
I notice one of our tables has become "unresponsive" no queries work against it. We shutdown all apps that connect to those dbs, and we also deleted all current sessions using gcloud sessions list and then gcloud session delete.
However the table is still unresponsive. A simple select like select id from mytable where name = 'test' is not responding (when tested from an app, and also from gcloud web interface), it only happens with that table, which has only a few columns with normal data and no more than 2000 records. We identified the query that could have been the source of the problem, however the table seems to be locked (only count(*) without any where clause works).
I was wondering if there is any way to "unlock" the table, kill those "transactions" that might be causing the issue, or restart those specific spanner databases, or in the worst case scenario restarting the spanner instance.
I have seen the monitoring high cpu documentation, but even if we can identify the cpu is high, we don't really know how to restart or make it back to normal before reviewing the query/ies that could have caused the issue (if that was the case).
High CPU can be caused by user initiated queries, from different types of operations. It is important to notice that your instance is where you allocate resources to be used by the underlying Cloud Spanner databases. This means, that if all of your databases are in the same instance and if some of your databases are hogging the CPU, all your other databases will struggle.
In terms of a locked table, I would be very surprised if a deadlock is the problem here, since Spanner mitigates those issues using "wound-wait" algorithm. What I suspect might be happening is that a long time is necessary to perform the query in that table and it times out. It would be nice to investigate a bit more on your problem:
How long did you wait for your query on the problematic table (before you deemed to be stuck)? It might be a problem where your query just takes long and you are timing out before getting the results. It might be a problem where there are hotspots in your table and transactions are getting aborted often, preventing you from getting results.
What error did you get when performing the query on the table? The error code can tell you more about what might be happening.
Have you tried doing a stale read on the table to see if any data is returned? If lock contention is the problem, this should succeed and return results faster for you. Thus, you can further investigate the lock usage with the statistics table (as shown below).
Inspect query statistics: you can list the queries with the highest CPU usage, find the average latency for a query and find out the queries that timeout the most. There is much more you can do, as seen here. You can also view the query statistics in cloud console. What I would verify is, by reducing the overall CPU, does your query come back without any issues? You might need more resources if so. Or you might need to reduce hotspots in your database.
Inspect Lock statistics: you can investigate lock conflicts in your rows and tables. I think that an interesting query for your problem would be Discovering which row keys and columns had long lock wait times during the selected period. You can then see if your query is reading those row keys and columns and act accordingly.
I have a free tier RDS PostgreSQL database db.t2.micro (1 core, 1 Gib). My architecture did the RDS collapse because it reaches the max number of concurrent connections.
When I query select * from pg_settings where name='max_connections', the result was 87.
I found this formula about the capacity of the max_connection attribute for each instance based on the memory:
LEAST({DBInstanceClassMemory/9531392},5000)
For my instance, the number was 104, and I modified the parameter group to this value, and still my RDS collapse.
I made a last attempt updating the max_connections to 500 believing that it'll not work because the limit is 104. But to my surprise the database worked and could handle all the concurrent connections (above 104):
Obviously, I'm missing something.
Is really there a limit of max_connections for my instance?
If I change the max_connection setting the pricing for my instance change?
Also, I'm curious about What represents the horizontal line in the graphic, because it is at the level of my initial max_connection setting, and before the change, it was not present
Thanks!
Based on the information at https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_Limits.html#RDS_Limits.MaxConnections, you should've hit the limit there.
That said, I guess one of two possible things are happening here:
AWS do not enforce the limits. I guess they could be suggesting the limits based on instance size here.
There is some kind of "burst" allowed. Similar to IOPS there could be a small "burst" balance that you can scale to temporarily without affecting service.
the intent
Basically, pagination: have a CURSOR (created using DECLARE outside any function here, but that can be changed if need be) concurrently addressed to retrieve batches of rows, implying moving the cursor position in order to fetch more than one line (FETCH count seems to be the only way to fetch more than one line).
the context
During a more global transaction (i.e. using one connection), I want to retrieve a range of rows through a cursor. To do so, I:
MOVE the cursor to the desired position (e.g. MOVE 42 FROM "mycursor")
then FETCH the amount of rows (e.g. FETCH FORWARD 10 FROM "mycursor")
However, this transaction is used by many workers (horizontally scaled), each receiving a set of "coordinates" for the cursor, like LIMIT and OFFSET: the index to MOVE to, and the amount of rows to FETCH. These workers use the DB connection through HTTP calls to a single DB API which handles the pool of connections and the transactions' liveliness.
Because of this concurrent access to the transaction/connection, I need to ensure atomic execution of the couple "MOVE then FETCH".
the setup
NodeJS workers consuming ranges of rows through a DB API
NodeJS DB API based on pg (latest)
PostgreSQL v10 (can be upgraded if required, all documentation links here are from v12 - latest)
the tries
WITH (MOVE 42 FROM "mycursor") FETCH 10 FROM "mycursor" produces a syntax error, apparently WITH doesn't handle MOVE
MOVE 42 FROM "mycursor" ; FETCH 10 FROM "mycursor" as I'm inside a transaction I suppose this could work, but anyway I'm using Node's pg which apparently doesn't handle several statements in the same call to query() (no error, but no result yielded, I didn't dig into this too much as it looks like a hack)
I'm not confident a function would guarantee atomicity, it doesn't seem to be what PARALLEL UNSAFE does, and as I'm going to have high concurrency, I'd really love some explicitly written assurances about atomicity...
the reason
I'd prefer not to rely on LIMIT/OFFSET as it would require an ORDER BY clause to ensure pagination consistency (as per the docs, ctrl-f for "unpredictable"), unless (scrollable, without hold) cursors prove to be way more resource-consuming. "Way more" because it has to be weighed with the INSENSITIVE behavior of cursors that would allow me not to acquire a lock on the underlying table during the whole process. If it's proven that pagination in this context is not feasible using cursors, I'll fall back to this solution, unless you have something better to suggest!
the human side
Hello, and thanks in advance for the help! :)
You can share connections, but you cannot to share transactions. This request is impossible for this context. Good tools doesn't share connections, when any transaction is used in this connection.
How to achieve best performance for writing huge number of records (for example 2000000) in Cassandra ?
I am using Scala, Datastax driver and phantom in my project. How can I insert these many records in database in a performant way?
2 Million isn't much. I would just use CQL copy from:
https://docs.datastax.com/en/cql/3.3/cql/cql_reference/cqlshCopy.html
The best performance could be achieved by:
using asynchronous operations;
prepared queries;
use consistency level ONE (default, don't change);
use DCAware/TokenAware load balancing policy (default, don't change);
increase the number of requests per connection from default 1024 to higher number, like, 32k;
But with asynchronous queries, the big problem is that you may push more requests that Cassandra may handle, and this could lead to BusyPoolException - to prevent this you need some kind of counting semaphore that won't allow to issue to many requests. Here is an example of such implementation.
I want to be able to query settings in another existing session. In particular the transaction_isolation. We have:
current_setting('transaction_isolation')
I want to call something like:
session_setting('transaction_isolation', backend_pid )
To find out what transaction isolation level is actually being used by an existing session/connection/back-end.
Background
We have a problem where I believe Auto-vacuum gets stuck. Running vacuum manually leaves certain tables with many (say a million) dead-tuples remaining.
That, i think, reduces performance a lot. A single row update on such tables can take over a second. Where it normally takes a millisecond.
Looking into pg_stat_activity there are quite a few apps accessing this database.
Killing off any long open read/write transaction once helped solve the problem.
(Vacuum ran, and a second later throughput leapt up by perhaps 1000 x)
In other cases that approach did not work.
It seems some of the read session may be causing the problem, even when they don't query the suspect tables. This could make sense if we had say sequential-read transaction isolation being used by these other apps' sessions.
Some of the other apps are using JDBC i think. Some ODBC. And there are a few PgAmdins joining in too.
Its hard to find out quite how the connections/session are being created directly within the bowels of some monitoring/reporting tools.
The default transaction_isolation is the normal read-committed.
We're running v9.3 postgres.
You can show all the information by running SHOW all.
If you want to show the isolation level run this:
SHOW default_transaction_isolation;
I don't think there's any way to look into one session from within another session. (I could be wrong.)
The only alternative I can think of is to expand your logging. In postgresql.conf, set
logging_collector = on
log_statement = 'all'
Restart the server.
This will log all SQL statements, including those that set the transaction isolation level. There are a lot of settings; log_connections might be a useful one.
So you'd assume that the isolation level is "read committed" unless you log a different isolation level.
As mentioned by #zubair-0:
If you want to see the isolation level for the current session, you
can use SHOW TRANSACTION ISOLATION LEVEL;