Apparently my PSQL 12 stopped working from one moment to the other.
Not accepting any insert statements anymore, while select statements for perfectly fine.
psql (12.11 (Ubuntu 12.11-0ubuntu0.20.04.1))
SSL connection (protocol: TLSv1.3, cipher: TLS_AES_256_GCM_SHA384, bits: 256, compression: off)
Type "help" for help.
cfoo=> \dt
cfoo=> select * from django_session;
cfoo=> \d django_session;
Table "public.django_session"
Column | Type | Collation | Nullable | Default
--------------+--------------------------+-----------+----------+---------
session_key | character varying(40) | | not null |
session_data | text | | not null |
expire_date | timestamp with time zone | | not null |
Indexes:
"django_session_pkey" PRIMARY KEY, btree (session_key)
"django_session_expire_date_a5c62663" btree (expire_date)
"django_session_session_key_c0390e0f_like" btree (session_key varchar_pattern_ops)
cfoo=> select * from django_session;
cfoo=> insert into django_session (session_key, session_data, expire_date) values ('123', '123', '2021-12-18 18:01:44.119+01');
And then nothing. No error, no timeout. Just staying like this.
This is the master and the slave is also not replicating right now. Not sure this is related, as I uncommented the config for that in the pg_hbl.conf already and restarted.
Any clues?
Turned out, that the streaming replication was still active and had to be removed at several places including postgresql.conf.
Postgresql was waiting for the replication to happen and therefor locked the insert.
Related
I have imported a local PostgreSQL database to a managed cluster on Digital Ocean. It will be used with a Python app that will also be hosted on Digital Ocean. I used pg_dump and pg_restore to achieve the import. Now, to make sure the import was successful, I am running some psql queries and commands via my MacOS terminal app that is set up with zsh and it connects via a shell script that prompts me for host, database name, port, user and password. I am successful in connecting to the managed cluster this way, and I can execute some queries with no problem, while others are causing errors. For example:
my_support=> \dt
List of relations
Schema | Name | Type | Owner
--------+----------------------+-------+---------
public | ages | table | doadmin
public | articles | table | doadmin
public | challenges | table | doadmin
public | cities | table | doadmin
public | comments | table | doadmin
public | messages | table | doadmin
public | relationships | table | doadmin
public | topics | table | doadmin
public | users | table | doadmin
(9 rows)
my_support=> \dt+
sh: more: command not found
my_support=>
Also:
my_support=> SELECT id,sender_id FROM messages;
id | sender_id
----+-----------
1 | 1
2 | 2
3 | 4
4 | 1
5 | 2
(5 rows)
my_support=> SELECT * FROM messages;
sh: more: command not found
my_support=>
So the terminal app seems to dislike certain characters, such as * and +, but I can't find any documentation that tells me I should escape them, or how. I tried backslash in front of them, but it did not work. And what's more confusing is that these very same queries are successful when I connected to my LOCAL copy of the database, using the very same terminal app, launched from the very same shell script.
In case it's helpful, here's what I see in the CLI when I connect:
psql (14.1, server 14.2)
SSL connection (protocol: TLSv1.3, cipher: <alphanumeric string here>, bits: 256, compression: off)
Type "help" for help.
my_support=>
Does it matter that my local PostgreSQL version is 14.1 and the server is 14.2? I'm assuming the "server" refers to the hosted environment, but it seems like something as basic as "SELECT * FROM" should not be version-dependent.
Ultimately what matters is whether my Python app (which uses psycopg library to talk to PostgreSQL) can run those queries, and I haven't test that yet. But it sure would be handy to test things on the managed cluster using my local terminal app.
BTW, I have an open ticket with Digital Ocean to answer this question, but I find SO to be faster and more helpful in most cases.
psql is trying to use a pager to display results that are longer than the number of lines in the terminal. The error message
more: command not found
indicates that the pager (more) it tries to use is not available. You can turn off the use of a pager:
\pset pager off
or set a different command to be used as the pager. See the manual for details
We recently upgraded the OS:
$ cat /etc/redhat-release
Red Hat Enterprise Linux Server release 7.6 (Maipo)
After upgrading, we are facing lot of issues with GitLab (predominantly with Postgres)..
Our GitLab is dockerized i.e. GitLab (and all its internal services including PostgreSQL) is running inside a single container. The container does not have it's own glibc, so it is using the one from the OS.
ERROR: canceling statement due to statement timeout
STATEMENT:
SELECT relnamespace::regnamespace as schemaname,
relname as relname,
pg_total_relation_size(oid) bytes FROM pg_class WHERE relkind = 'r';
The timeout messages appear continuously and this results in users facing 502 errors when accessing GitLab.
I checked the statement timeout set on the database.
gitlabhq_production=# show statement_timeout;
statement_timeout
-------------------
1min
(1 row)
I don't know what to make of this. This is probably the default setting. Is this an issue with postgres? What does this mean? Anything I can do to fix this?
EDIT:
Checked pg_stat_activity and don't see any locks as the server was rebooted earlier. The same query is running fine now but we keep seeing this issue intermittently.
Ran \d pg_class to check whether the table uses any indexes and also to check the string column.
gitlabhq_production=# \d pg_class
Table "pg_catalog.pg_class"
Column | Type | Modifiers
---------------------+-----------+-----------
relname | name | not null
relnamespace | oid | not null
reltype | oid | not null
reloftype | oid | not null
relowner | oid | not null
relam | oid | not null
relfilenode | oid | not null
reltablespace | oid | not null
relpages | integer | not null
reltuples | real | not null
relallvisible | integer | not null
reltoastrelid | oid | not null
relhasindex | boolean | not null
relisshared | boolean | not null
relpersistence | "char" | not null
relkind | "char" | not null
relnatts | smallint | not null
relchecks | smallint | not null
relhasoids | boolean | not null
relhaspkey | boolean | not null
relhasrules | boolean | not null
relhastriggers | boolean | not null
relhassubclass | boolean | not null
relrowsecurity | boolean | not null
relforcerowsecurity | boolean | not null
relispopulated | boolean | not null
relreplident | "char" | not null
relfrozenxid | xid | not null
relminmxid | xid | not null
relacl | aclitem[] |
reloptions | text[] |
Indexes:
"pg_class_oid_index" UNIQUE, btree (oid)
"pg_class_relname_nsp_index" UNIQUE, btree (relname, relnamespace)
"pg_class_tblspc_relfilenode_index" btree (reltablespace, relfilenode)
Would reindexing all tables and possibly alter tables help?
You should check whether the query us running for a minute or whether it is blocked behind a database lock. This can be seen from the pg_stat_activity row for the backend, which will show if the query is waiting for a lock or not (state=active and wait_event_type and wait_event indicate a lock).
If it is a lock, get rid of the locking transaction. It may be a prepared transaction, so check for these too.
If there is no lock at fault, it could be that your indexes have become corrupted by the operating system upgrade:
Since PostgreSQL uses operating system collations, database indexes on strings are sorted in collation order and an operating system upgrade can (and often does) lead to changed collations due to bug fixes in the C library, you should rebuild all indexes on string columns after such an upgrade.
The statement that you are showing does not use an index scan, so it should not be affected, but other statements may be.
Also, if you are using Docker, it may be that your container uses its own glibc that was not upgraded, and then you are not affected.
In a legacy PostgreSQL DB I tried to drop an existing index issuing the command:
DROP INDEX testing.idx_testing_data_model_output_data_id;
and see the error:
ERROR: index "<index name>" does not exist
But I can see the index using the \d <table name> command:
leg=# \d testing.data_model
Table "testing.data_model"
Column | Type | Modifiers
----------------+-----------------------------+---------------------------------------------------------------------------
id | bigint | not null default nextval('testing.data_model_id_seq'::regclass)
input_data | text |
output_data_id | bigint |
Indexes:
"pk_testing_data_model" PRIMARY KEY, btree (id)
"idx_testing_data_model_output_data_id" btree (output_data_id)
Ok, when I try to create the index I receive the following error:
ERROR: relation "<index name>" already exists
It seems that somehow the index creation or index dropping was not successfully complete. How can I resolve this issue?
SET search_path = <schema_name>;
Setting search_path worked for me.
I have replication slot which I want to delete but when I do delete I got an error that I can't delete from view. Any ideas?
postgres=# SELECT * FROM pg_replication_slots ;
slot_name | plugin | slot_type | datoid | database | active | xmin | catalog_xmin | restart_lsn
--------------+--------------+-----------+--------+----------+--------+------+--------------+-------------
bottledwater | bottledwater | logical | 12141 | postgres | t | | 374036 | E/FE8D9010
(1 row)
postgres=# delete from pg_replication_slots;
ERROR: cannot delete from view "pg_replication_slots"
DETAIL: Views that do not select from a single table or view are not automatically updatable.
HINT: To enable deleting from the view, provide an INSTEAD OF DELETE trigger or an unconditional ON DELETE DO INSTEAD rule.
postgres=#
Use pg_drop_replication_slot:
select pg_drop_replication_slot('bottledwater');
See the docs and this blog.
The replication slot must be inactive, i.e. no active connections. So if there's a streaming replica using the slot you must stop the streaming replica. Or you can change its recovery.conf so it doesn't use a slot anymore and restart it.
As a complement to the accepted answer, I'd like to mention that following command will not fail in case the slot does not exist (this was useful for me because I scripted that).
select pg_drop_replication_slot(slot_name) from pg_replication_slots where slot_name = 'bottledwater';
Im running Postgres 8.3 and I am having trouble running AN ALTER TABLE ADD COLUMN statement which seems to be blocked by an AccessShareLock when I run this query
SELECT t.relname,l.locktype,page,virtualtransaction,pid,mode,granted FROM pg_locks l, pg_stat_all_tables t WHERE l.relation=t.relid ORDER BY relation asc;
The table's name is dealer.
relname | locktype | page | virtualtransaction | pid | mode | granted
dealer | relation | | 2/40 | 12719 | AccessExclusiveLock | f
dealer | relation | | -1/154985751 | | AccessShareLock | t
I also ran
SELECT * FROM pg_prepared_xacts
That returned
transaction | gid | prepared | owner | database
154985751 | 131075_MS1hMzIwM2E3OmIwMjM6NTQxMGY0MzE6MWM1ZTg5OQ==_YTMyMDNhNzpiMDIzOjU0MTBmNDMxOjFjNWU4OWM= | 2014-09-19 08:01:49.650957+10 | user | database
The transaction id 154985751 looks similar to the virtualtransaction in the pg_locks table -1/154985751
I ran this command to view any processes that may be running queries on the database
ps axu | grep postgres | grep -v idle
and have confirmed there are no other processes running queries on the database.
The log file shows this after the query has been run
2014-11-14 17:25:00.794 EST (pid: 12719) LOG: statement: BEGIN;
2014-11-14 17:25:00.794 EST (pid: 12719) LOG: statement: ALTER TABLE dealer ADD bullet1 varchar;
2014-11-14 17:25:01.795 EST (pid: 12719) LOG: process 12719 still waiting for AccessExclusiveLock on relation 2321398 of database 2321293 after 1000.133 ms
2014-11-14 17:25:01.795 EST (pid: 12719) STATEMENT: ALTER TABLE dealer ADD bullet1 varchar;
What could be causing the AccessShareLock on the dealer table? Im guessing it has something to do with the transaction 154985751 is there a way to terminate a transaction with using the virtual id?
You have a prepared transaction in place. Prepared transactions - those where PREPARE TRANSACTION but not COMMIT PREPARED or ROLLBACK PREPARED has been run - hold locks, just like normal running transactions do.
Prepared transactions may be used by XA transaction managers, JTA, etc, not necessarily directly by your app. Many queuing systems use them too. If you don't know what the transaction is and you commit it or roll it back you may disrupt something that is relying on two-phase commit.
If you are certain that you know what it is you can:
COMMIT PREPARED '131075_MS1hMzIwM2E3OmIwMjM6NTQxMGY0MzE6MWM1ZTg5OQ==_YTMyMDNhNzpiMDIzOjU0MTBmNDMxOjFjNWU4OWM='
or
ROLLBACK PREPARED '131075_MS1hMzIwM2E3OmIwMjM6NTQxMGY0MzE6MWM1ZTg5OQ==_YTMyMDNhNzpiMDIzOjU0MTBmNDMxOjFjNWU4OWM='
depending on whether you wish to commit or abort the prepared xact.
You can't inspect the transaction to see what it did/does, you need to figure out what app/tool created it and why if you don't know what it is.
The identifier looks suspiciously like [number]_[base64]_[base64] so lets see what we can do with that:
postgres=> SELECT decode((string_to_array('131075_MS1hMzIwM2E3OmIwMjM6NTQxMGY0MzE6MWM1ZTg5OQ==_YTMyMDNhNzpiMDIzOjU0MTBmNDMxOjFjNWU4OWM=','_'))[2], 'base64');
decode
------------------------------------------------------------------
\x312d613332303361373a623032333a35343130663433313a31633565383939
(1 row)
postgres=> SELECT decode((string_to_array('131075_MS1hMzIwM2E3OmIwMjM6NTQxMGY0MzE6MWM1ZTg5OQ==_YTMyMDNhNzpiMDIzOjU0MTBmNDMxOjFjNWU4OWM=','_'))[3], 'base64');
decode
--------------------------------------------------------------
\x613332303361373a623032333a35343130663433313a31633565383963
(1 row)
Hm, looks like ASCII or similar, lets see:
postgres=> SELECT convert_from(decode((string_to_array('131075_MS1hMzIwM2E3OmIwMjM6NTQxMGY0MzE6MWM1ZTg5OQ==_YTMyMDNhNzpiMDIzOjU0MTBmNDMxOjFjNWU4OWM=','_'))[2], 'base64'), 'utfpostgres=> SELECT convert_from(decode((string_to_array('131075_MS1hMzIwM2E3OmIwMjM6NTQxMGY0MzE6MWM1ZTg5OQ==_YTMyMDNhNzpiMDIzOjU0MTBmNDMxOjFjNWU4OWM=','_'))[2], 'base64'), 'utf-8');
convert_from
---------------------------------
1-a3203a7:b023:5410f431:1c5e899
(1 row)
postgres=> SELECT convert_from(decode((string_to_array('131075_MS1hMzIwM2E3OmIwMjM6NTQxMGY0MzE6MWM1ZTg5OQ==_YTMyMDNhNzpiMDIzOjU0MTBmNDMxOjFjNWU4OWM=','_'))[3], 'base64'), 'utf-8');
convert_from
-------------------------------
a3203a7:b023:5410f431:1c5e89c
(1 row)
Looks vaguely GUID/UUID-ish, with odd formatting and grouping.
Maybe those identifiers will help you figure out where the xact came from.
BTW, 8.3 is exceedingly obsolete. Plan your upgrade.