I have a Dblink query Amazon RDS (Postgres) that execute an INSERT with rows from an Amazon Redshift cluster.
The query terminates after 15/20 minutes, if not more, but I can see that all rows are being inserted after only few minutes.
I'm running these queries via JetBrains' DataGrip.
Some other similar dblink on the same connection, terminate as expected.
The only difference I see being the size of the table, which is bigger in the first case.
All these queries are simply copying the whole table. Pretty much like this:
insert into rds_table(
select *
from db_link('foreign_server',
$REDSHIFT$
select *
from redshift_table
$REDSHIFT$) as table_n(...)
);
Where "foreign server" is my connection to Redshift.
I know that the query is completed because rds_table has the same number of rows as redshift_table.
DataGrip shows the query as still running:
and won't let me run other queries until I manually stop the query.
If I do so, the inserted rows remain in the database, meaning that the transaction has already committed.
Why is this happening? Is it a problem with DataGrip or with Postgres?
How can I fix it?
Is there any other better alternative to migrate data from Redshift to RDS?
If a concurrent transaction can already see the inserted data, that means that the inserting transaction and consequently the INSERT statement must already be finished.
If DataGrip shows the statement as still running, it is lying to you.
So this must be a DataGrip bug.
Related
I am trying to understand a PostgreSQL behaviour.
In my application I am using a Postgres (AWS Aurora to be precise) database through ODBC.
I am not specifying "with hold" on the cursors used but I see in the logs that most of them are 'idle in transaction', although there is no transaction, there is just a 'select'. The statements look like:
RELEASE _EXEC_SVP_05A0C768;SAVEPOINT _EXEC_SVP_05A0C768;declare "SQL_CUR05A0E3F8" cursor with hold for select u_selskap,u_kodetyp,u_tekst from u_hrm.u_stilltyp where u_selskap=50 and u_kodetyp='T';fetch 100 in "SQL_CUR05A0E3F8"
In PgAdmin we can see how the statements are displayed.
The idling cursors seem to keep locks on the database hence subsequent queries cannot be run.
I am not able to understand why these cursors are in a transaction and which part of the chain is introducing the 'with hold' as the default creation mode is 'without hold'(as seen here: https://www.postgresql.org/docs/11/sql-declare.html).
Can it be a PostgreSQL ODBC issue?
I have a table with 500k elements. Now I want to add a new column
(type boolean, nullable = false) without a default value.
The query to do so is running like for ever.
I'm using PostgreSQL 12.1, compiled by Visual C++ build 1914, 64-bit on my Windows 2012 Server
In pgAdmin I can see the query is blocked by PID 0. But when I execute this query, I can't see the query with pid = 0
SELECT *
FROM pg_stat_activity
Can someone help me here? Why is the query blocked and how can I fix this to add a new column to my table.
UPDATE attempt:
SELECT *
FROM pg_prepared_xacts
Update
It works after rollback all prepared transaction.
ROLLBACK PREPARED 'gid goes here';
You have got stale prepared transactions. I say that as in "you have got the measles", because it is a disease for a database.
Such prepared transactions keep holding locks and block autovacuum progress, so they will bring your database to its knees if you don't take action. In addition, such transactions are persisted, so even a restart of the database won't get rid of them.
Remove them with
ROLLBACK PREPARED 'gid goes here'; /* use the transaction names shown in the view */
If you use prepared transactions, you need a distributed transaction manager. That is a piece of software that keeps track of all prepared transactions and their state and persists that information, so that no distributed transaction can become stale. Even if there is a crash, the distributed transaction manager will resolve in-doubt transactions in all involved databases.
If you don't have that, don't use prepared transactions. You now know why. Best is to set max_prepared_transactions to 0 in that case.
I'm pretty new to PostgreSQL and I'm sure I'm missing something here.
The scenario is with version 11, executing a big drop table and insert transaction on a given table with the nodejs driver, which may take 30 minutes.
While doing that, if I try to query with select on that table using the jdbc driver, the query execution waits for the transaction to finish. If I close the transaction (by finishing it or by forcing it to exit), the jdbc query becomes responsive.
I thought I can read a table with one connection while performing a transaction with another one.
What am I missing here?
Should I keep the table (without dropping it at the beginning of the transaction) ?
DROP TABLE takes an ACCESS EXCLUSIVE lock on the table, which is there precisely to prevent it from taking place concurrently with any other operation on the table. After all, DROP TABLE physically removes the table.
Since all locks are held until the end of the database transaction, all access to the dropped table is blocked until the transaction ends.
Of course the files are only removed when the transaction commits, so you might wonder why PostgreSQL doesn't let concurrent transactions read in the mean time. But that would mean that COMMIT may be blocked by a concurrent reader, or a SELECT might cause a system error in the middle of reading, both of which don't sound appealing.
I have been trying to truncate a table using SQlWorkbench. Suddenly, SqlWorkbench got freezed, while the truncate was in progress. I had to kill workbench from taskmanager. But now none of the queries are working on the table on which the truncate was aborted abruptly. For other tables queries are working fine. Need help, as I have to upload fresh data on the same table. Currently I am not even able to drop the table. What can be done to resolve this issue?
This looks like the TRUNCATE got stuck behind a lock, and then you killed the front end, while TRUNCATE kept running.
Connect to the database as superuser and examine the pg_stat_activity view; you should see some long running transactions.
Use the function pg_terminate_backend to kill these sessions by their pid.
Is it possible to drop active connections to Redshift in order to drop a database?
In my development environment I find myself recreating the schema very frequently and if there happens to be some stray process connected to the database this fails. I know it's possible to do this with Postgresql using pg_terminate_backend, but this doesn't seem to work on Redshift.
Deleting rows from the STV_SESSIONS table isn't an option, either.
Any ideas?
http://docs.aws.amazon.com/redshift/latest/dg/PG_TERMINATE_BACKEND.html
Find the PIDs of current running queries
select pid from stv_recents where status = 'Running';
and terminate all the queries with
select pg_terminate_backend(pid);