PostgreSQL background process raising errors - postgresql

At about 7:45 this morning, errors started appearing in our customer's PostgreSQL logs. Nobody from my company was working on their system at that time. The error messages have nothing to do with our client's database or the software that we provided. As far as I can tell, the software we provided is operating normally.
Can anyone tell me what might be happening and how to correct it?
This is an ancient PostgreSQL installation: version 8.1 running on Windows Server 2003. We've never updated it because it is still working.
2019-08-08 07:43:23 FATAL: unrecognized configuration parameter "application_name"
2019-08-08 07:43:32 FATAL: unrecognized configuration parameter "application_name"
2019-08-08 07:43:52 FATAL: unrecognized configuration parameter "application_name"
2019-08-08 07:43:52 ERROR: syntax error at or near "has_priviledge" at character 90
2019-08-08 07:43:52 STATEMENT:
SELECT
has_table_privilege(
'pgagent.pga_job', 'INSERT, SELECT, UPDATE'
) has_priviledge
WHERE EXISTS(
SELECT has_schema_privilege('pgagent', 'USAGE')
WHERE EXISTS(
SELECT cl.oid FROM pg_class cl
LEFT JOIN pg_namespace ns ON ns.oid=relnamespace
WHERE relname='pga_job' AND nspname='pgagent'
)
)
2019-08-08 07:43:53 ERROR: column "state" does not exist
2019-08-08 07:43:53 STATEMENT: /*pga4dash*/
SELECT 'session_stats' AS chart_name, row_to_json(t) AS chart_data
FROM (SELECT
(SELECT count(*) FROM pg_stat_activity) AS "Total",
(SELECT count(*) FROM pg_stat_activity WHERE state = 'active') AS "Active",
(SELECT count(*) FROM pg_stat_activity WHERE state = 'idle') AS "Idle"
) t
UNION ALL
SELECT 'tps_stats' AS chart_name, row_to_json(t) AS chart_data
FROM (SELECT
(SELECT sum(xact_commit) + sum(xact_rollback) FROM pg_stat_database) AS "Transactions",
(SELECT sum(xact_commit) FROM pg_stat_database) AS "Commits",
(SELECT sum(xact_rollback) FROM pg_stat_database) AS "Rollbacks"
) t
UNION ALL
SELECT 'ti_stats' AS chart_name, row_to_json(t) AS chart_data
FROM (SELECT
(SELECT sum(tup_inserted) FROM pg_stat_database) AS "Inserts",
(SELECT sum(tup_updated) FROM pg_stat_database) AS "Updates",
(SELECT sum(tup_deleted) FROM pg_stat_database) AS "Deletes"
) t
UNION ALL
SELECT 'to_stats' AS chart_name, row_to_json(t) AS chart_data
FROM (SELECT
(SELECT sum(tup_fetched) FROM pg_stat_database) AS "Fetched",
(SELECT sum(tup_returned) FROM pg_stat_database) AS "Returned"
) t
UNION ALL
SELECT 'bio_stats' AS chart_name, row_to_json(t) AS chart_data
FROM (SELECT
(SELECT sum(blks_read) FROM pg_stat_database) AS "Reads",
(SELECT sum(blks_hit) FROM pg_stat_database) AS "Hits"
) t

Related

How to use pgadmin with Postgresql 15

Postgres 15 with pgAdmin was installed in Windows 11 and new database was created.
Trying to connect to database in localhost throw error
Request failed with status code 500
Postgres log contains
2022-10-25 09:37:27.159 EEST [10508] ERROR: UNION types "char" and text cannot be matched at character 745
2022-10-25 09:37:27.159 EEST [10508] STATEMENT:
SELECT cl.relkind, COALESCE(cin.nspname, cln.nspname) as nspname,
COALESCE(ci.relname, cl.relname) as relname, cl.relname as indname
FROM pg_catalog.pg_class cl
JOIN pg_catalog.pg_namespace cln ON cl.relnamespace=cln.oid
LEFT OUTER JOIN pg_catalog.pg_index ind ON ind.indexrelid=cl.oid
LEFT OUTER JOIN pg_catalog.pg_class ci ON ind.indrelid=ci.oid
LEFT OUTER JOIN pg_catalog.pg_namespace cin ON ci.relnamespace=cin.oid
WHERE cl.oid IN (SELECT objid FROM pg_catalog.pg_shdepend WHERE refobjid=16395::oid) AND cl.oid > 16383::oid
UNION ALL SELECT 'n', null, nspname, null
FROM pg_catalog.pg_namespace nsp
WHERE nsp.oid IN (SELECT objid FROM pg_catalog.pg_shdepend WHERE refobjid=16395::oid) AND nsp.oid > 16383::oid
UNION ALL SELECT CASE WHEN typtype='d' THEN 'd' ELSE 'y' END, null, typname, null
FROM pg_catalog.pg_type ty
WHERE ty.oid IN (SELECT objid FROM pg_catalog.pg_shdepend WHERE refobjid=16395::oid) AND ty.oid > 16383::oid
UNION ALL SELECT 'C', null, conname, null
FROM pg_catalog.pg_conversion co
WHERE co.oid IN (SELECT objid FROM pg_catalog.pg_shdepend WHERE refobjid=16395::oid) AND co.oid > 16383::oid
UNION ALL SELECT CASE WHEN prorettype=2279 THEN 'T' ELSE 'p' END, null, proname, null
FROM pg_catalog.pg_proc pr
WHERE pr.oid IN (SELECT objid FROM pg_catalog.pg_shdepend WHERE refobjid=16395::oid) AND pr.oid > 16383::oid
UNION ALL SELECT 'o', null, oprname || '('::text || COALESCE(tl.typname, ''::text) || CASE WHEN tl.oid IS NOT NULL
AND tr.oid IS NOT NULL THEN ','::text END || COALESCE(tr.typname, ''::text) || ')'::text, null
FROM pg_catalog.pg_operator op
LEFT JOIN pg_catalog.pg_type tl ON tl.oid=op.oprleft
LEFT JOIN pg_catalog.pg_type tr ON tr.oid=op.oprright
WHERE op.oid IN (SELECT objid FROM pg_catalog.pg_shdepend WHERE refobjid=16395::oid) AND op.oid > 16383::oid
ORDER BY 1,2,3
I installed pgadmin4-6.15-x64.exe from pgadmin.org but problem persists.
Environment:
Version
6.15
Application Mode
Desktop
Current User
pgadmin4#pgadmin.org
NW.js Version
0.62.2
Browser
Chromium 99.0.4844.84
Operating System
Windows-10-10.0.22621-SP0
PgAdmin 4 6.17 has fixed this problem.

PostGIS breaks after soft upgrade from 3.1.1 to 3.2

I've been stuck with this problem for more than a week now, and I hope someone could help me out here.
I was running Postgresql 12 with PostGIS 3.1.1 extension. More than a week ago, I accidentally performed a PostGIS upgrade by running this query
SELECT PostGIS_Extensions_Upgrade(); which returned "Upgrade completed, run SELECT postgis_full_version(); for details".
Although it seemed the upgrade went on successfully, while I check with SELECT postgis_full_version(); it returned POSTGIS="3.2.0 c3e3cc0" [EXTENSION] PGSQL="120" GEOS="3.9.1-CAPI-1.14.2" PROJ="6.3.1" GDAL="GDAL 3.0.4, released 2020/01/28 GDAL_DATA not found" LIBXML="2.9.10" LIBJSON="0.13.1" LIBPROTOBUF="1.3.3" WAGYU="0.5.0 (Internal)" TOPOLOGY RASTER. Note that GDAL_DATA is no longer found.
This causes functions like ST_Transform breaks with error as following: SQL Error [XX000]: ERROR: could not form projection (LWPROJ) from 'srid=4326' to 'srid=4269'.
So far I have tried reinstalled postgresql-12, postgresql-client-12, postgis, postgresql-12-postgis-3 and postgresql-12-postgis-3-scripts on my Ubuntu 20.04 system, to no avail.
I tried downgrading to PostGIS to 3.1.4 (somehow apt repos only have 3.1.4 not 3.1.1), which returned POSTGIS="3.1.4 ded6c34" [EXTENSION] PGSQL="120" GEOS="3.9.1-CAPI-1.14.2" PROJ="7.2.1" GDAL="GDAL 3.3.2, released 2021/09/01" LIBXML="2.9.10" LIBJSON="0.13.1" LIBPROTOBUF="1.3.3" WAGYU="0.5.0 (Internal)" (core procs from "3.2.0 c3e3cc0" need upgrade) TOPOLOGY (topology procs from "3.2.0 c3e3cc0" need upgrade) RASTER (raster procs from "3.2.0 c3e3cc0" need upgrade) when querying SELECT postgis_full_version();. There didn't seem any error but the ST_Transform query still gave the same error above.
I am also tried to use the pg_upgrade function to upgrade the database to Postgresql 14. While running with the --check option, the results says "Clusters are compatible". But when running the actual upgrade, it breaks at the "Creating dump of database schemas" of the postgres database. Dumps of other databases went through just fine. The "pg_upgrade_dump_13427.log" log shows the following.
pg_dump: error: query was: SELECT classid, objid, refobjid FROM pg_depend WHERE refclassid = 'pg_extension'::regclass AND deptype = 'e' ORDER BY 3
command: "/usr/lib/postgresql/14/bin/pg_dump" --host /var/lib/postgresql --port 50432 --username postgres --schema-only --quote-all-identifiers --binary-upgrade --format=custom --file="pg_upgrade_dump_13427.custom" 'dbname=postgres' >> "pg_upgrade_dump_13427.log" 2>&1
pg_dump: WARNING: terminating connection because of crash of another server process
DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
HINT: In a moment you should be able to reconnect to the database and repeat your command.
pg_dump: error: query failed: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
pg_dump: error: query was: SELECT c.tableoid, c.oid, c.relname, (SELECT pg_catalog.array_agg(acl ORDER BY row_n) FROM (SELECT acl, row_n FROM pg_catalog.unnest(coalesce(c.relacl,pg_catalog.acldefault(CASE WHEN c.relkind = 'S' THEN 's' ELSE 'r' END::"char",c.relowner))) WITH ORDINALITY AS perm(acl,row_n) WHERE NOT EXISTS ( SELECT 1 FROM pg_catalog.unnest(coalesce(pip.initprivs,pg_catalog.acldefault(CASE WHEN c.relkind = 'S' THEN 's' ELSE 'r' END::"char",c.relowner))) AS init(init_acl) WHERE acl = init_acl)) as foo) AS relacl, (SELECT pg_catalog.array_agg(acl ORDER BY row_n) FROM (SELECT acl, row_n FROM pg_catalog.unnest(coalesce(pip.initprivs,pg_catalog.acldefault(CASE WHEN c.relkind = 'S' THEN 's' ELSE 'r' END::"char",c.relowner))) WITH ORDINALITY AS initp(acl,row_n) WHERE NOT EXISTS ( SELECT 1 FROM pg_catalog.unnest(coalesce(c.relacl,pg_catalog.acldefault(CASE WHEN c.relkind = 'S' THEN 's' ELSE 'r' END::"char",c.relowner))) AS permp(orig_acl) WHERE acl = orig_acl)) as foo) as rrelacl, CASE WHEN privtype = 'e' THEN (SELECT pg_catalog.array_agg(acl ORDER BY row_n) FROM (SELECT acl, row_n FROM pg_catalog.unnest(pip.initprivs) WITH ORDINALITY AS initp(acl,row_n) WHERE NOT EXISTS ( SELECT 1 FROM pg_catalog.unnest(pg_catalog.acldefault(CASE WHEN c.relkind = 'S' THEN 's' ELSE 'r' END::"char",c.relowner)) AS privm(orig_acl) WHERE acl = orig_acl)) as foo) END AS initrelacl, CASE WHEN privtype = 'e' THEN (SELECT pg_catalog.array_agg(acl) FROM (SELECT acl, row_n FROM pg_catalog.unnest(pg_catalog.acldefault(CASE WHEN c.relkind = 'S' THEN 's' ELSE 'r' END::"char",c.relowner)) WITH ORDINALITY AS privp(acl,row_n) WHERE NOT EXISTS ( SELECT 1 FROM pg_catalog.unnest(pip.initprivs) AS initp(init_acl) WHERE acl = init_acl)) as foo) END as initrrelacl, c.relkind, c.relnamespace, (SELECT rolname FROM pg_catalog.pg_roles WHERE oid = c.relowner) AS rolname, c.relchecks, c.relhastriggers, c.relhasindex, c.relhasrules, 'f'::bool AS relhasoids, c.relrowsecurity, c.relforcerowsecurity, c.relfrozenxid, c.relminmxid, tc.oid AS toid, tc.relfrozenxid AS tfrozenxid, tc.relminmxid AS tminmxid, c.relpersistence, c.relispopulated, c.relreplident, c.relpages, am.amname, CASE WHEN c.relkind = 'f' THEN (SELECT ftserver FROM pg_catalog.pg_foreign_table WHERE ftrelid = c.oid) ELSE 0 END AS foreignserver, CASE WHEN c.reloftype <> 0 THEN c.reloftype::pg_catalog.regtype ELSE NULL END AS reloftype, d.refobjid AS owning_tab, d.refobjsubid AS owning_col, (SELECT spcname FROM pg_tablespace t WHERE t.oid = c.reltablespace) AS reltablespace, array_remove(array_remove(c.reloptions,'check_option=local'),'check_option=cascaded') AS reloptions, CASE WHEN 'check_option=local' = ANY (c.reloptions) THEN 'LOCAL'::text WHEN 'check_option=cascaded' = ANY (c.reloptions) THEN 'CASCADED'::text ELSE NULL END AS checkoption, tc.reloptions AS toast_reloptions, c.relkind = 'S' AND EXISTS (SELECT 1 FROM pg_depend WHERE classid = 'pg_class'::regclass AND objid = c.oid AND objsubid = 0 AND refclassid = 'pg_class'::regclass AND deptype = 'i') AS is_identity_sequence, EXISTS (SELECT 1 FROM pg_attribute at LEFT JOIN pg_init_privs pip ON (c.oid = pip.objoid AND pip.classoid = 'pg_class'::regclass AND pip.objsubid = at.attnum)WHERE at.attrelid = c.oid AND ((SELECT pg_catalog.array_agg(acl ORDER BY row_n) FROM (SELECT acl, row_n FROM pg_catalog.unnest(coalesce(at.attacl,pg_catalog.acldefault('c',c.relowner))) WITH ORDINALITY AS perm(acl,row_n) WHERE NOT EXISTS ( SELECT 1 FROM pg_catalog.unnest(coalesce(pip.initprivs,pg_catalog.acldefault('c',c.relowner))) AS init(init_acl) WHERE acl = init_acl)) as foo) IS NOT NULL OR (SELECT pg_catalog.array_agg(acl ORDER BY row_n) FROM (SELECT acl, row_n FROM pg_catalog.unnest(coalesce(pip.initprivs,pg_catalog.acldefault('c',c.relowner))) WITH ORDINALITY AS initp(acl,row_n) WHERE NOT EXISTS ( SELECT 1 FROM pg_catalog.unnest(coalesce(at.attacl,pg_catalog.acldefault('c',c.relowner))) AS permp(orig_acl) WHERE acl = orig_acl)) as foo) IS NOT NULL OR CASE WHEN privtype = 'e' THEN (SELECT pg_catalog.array_agg(acl ORDER BY row_n) FROM (SELECT acl, row_n FROM pg_catalog.unnest(pip.initprivs) WITH ORDINALITY AS initp(acl,row_n) WHERE NOT EXISTS ( SELECT 1 FROM pg_catalog.unnest(pg_catalog.acldefault('c',c.relowner)) AS privm(orig_acl) WHERE acl = orig_acl)) as foo) END IS NOT NULL OR CASE WHEN privtype = 'e' THEN (SELECT pg_catalog.array_agg(acl) FROM (SELECT acl, row_n FROM pg_catalog.unnest(pg_catalog.acldefault('c',c.relowner)) WITH ORDINALITY AS privp(acl,row_n) WHERE NOT EXISTS ( SELECT 1 FROM pg_catalog.unnest(pip.initprivs) AS initp(init_acl) WHERE acl = init_acl)) as foo) END IS NOT NULL))AS changed_acl, pg_get_partkeydef(c.oid) AS partkeydef, c.relispartition AS ispartition, pg_get_expr(c.relpartbound, c.oid) AS partbound FROM pg_class c LEFT JOIN pg_depend d ON (c.relkind = 'S' AND d.classid = c.tableoid AND d.objid = c.oid AND d.objsubid = 0 AND d.refclassid = c.tableoid AND d.deptype IN ('a', 'i')) LEFT JOIN pg_class tc ON (c.reltoastrelid = tc.oid AND c.relkind <> 'p') LEFT JOIN pg_am am ON (c.relam = am.oid) LEFT JOIN pg_init_privs pip ON (c.oid = pip.objoid AND pip.classoid = 'pg_class'::regclass AND pip.objsubid = 0) WHERE c.relkind in ('r', 'S', 'v', 'c', 'm', 'f', 'p') ORDER BY c.oid
command: "/usr/lib/postgresql/13/bin/pg_dump" --host /var/lib/postgresql --port 50432 --username postgres --schema-only --quote-all-identifiers --binary-upgrade --format=custom --file="pg_upgrade_dump_13427.custom" 'dbname=postgres' >> "pg_upgrade_dump_13427.log" 2>&1
pg_dump: WARNING: terminating connection because of crash of another server process
DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
HINT: In a moment you should be able to reconnect to the database and repeat your command.
pg_dump: error: query failed: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
pg_dump: error: query was: SELECT classid, objid, refobjid FROM pg_depend WHERE refclassid = 'pg_extension'::regclass AND deptype = 'e' ORDER BY 3
It seems the dependencies of postgres extensions are broken. How do I fix that? Can someone please help?

Backup of postgres database giving ssl error

I have been using using DBeaver and I got a new laptop. With a fresh install adding the postgres remote database exactly as it is in the other computer.
I can see all the tables and data in the database. I just cannot take a backup.
When I right click tools->backup and I click start (same process works on old laptop) I am getting the error below. If I do this with my local dev database it works fine as expected. I can do a backup on my laptop both directly before and directly after trying on the new laptop.
C:\Program Files\PostgreSQL\10\bin\pg_dump.exe --format=c -n public --verbose --host=compute-1.amazonaws.com --port=5432 --username=elkjlkjkjyb d5kjhkjhjkht5
Backup started at Sun Sep 02 22:10:20 EDT 2018
pg_dump: last built-in OID is 16383
pg_dump: reading extensions
pg_dump: identifying extension members
pg_dump: reading schemas
pg_dump: reading user-defined tables
pg_dump: [archiver (db)] query failed: SSL error: sslv3 alert bad record mac
pg_dump: [archiver (db)] query was: SELECT c.tableoid, c.oid, c.relname, (SELECT pg_catalog.array_agg(acl ORDER BY row_n) FROM (SELECT acl, row_n FROM pg_catalog.unnest(coalesce(c.relacl,pg_catalog.acldefault(CASE WHEN c.relkind = 'S' THEN 's' ELSE 'r' END::"char",c.relowner))) WITH ORDINALITY AS perm(acl,row_n) WHERE NOT EXISTS ( SELECT 1 FROM pg_catalog.unnest(coalesce(pip.initprivs,pg_catalog.acldefault(CASE WHEN c.relkind = 'S' THEN 's' ELSE 'r' END::"char",c.relowner))) AS init(init_acl) WHERE acl = init_acl)) as foo) AS relacl, (SELECT pg_catalog.array_agg(acl ORDER BY row_n) FROM (SELECT acl, row_n FROM pg_catalog.unnest(coalesce(pip.initprivs,pg_catalog.acldefault(CASE WHEN c.relkind = 'S' THEN 's' ELSE 'r' END::"char",c.relowner))) WITH ORDINALITY AS initp(acl,row_n) WHERE NOT EXISTS ( SELECT 1 FROM pg_catalog.unnest(coalesce(c.relacl,pg_catalog.acldefault(CASE WHEN c.relkind = 'S' THEN 's' ELSE 'r' END::"char",c.relowner))) AS permp(orig_acl) WHERE acl = orig_acl)) as foo) as rrelacl, NULL AS initrelacl, NULL as initrrelacl, c.relkind, c.relnamespace, (SELECT rolname FROM pg_catalog.pg_roles WHERE oid = c.relowner) AS rolname, c.relchecks, c.relhastriggers, c.relhasindex, c.relhasrules, c.relhasoids, c.relrowsecurity, c.relforcerowsecurity, c.relfrozenxid, c.relminmxid, tc.oid AS toid, tc.relfrozenxid AS tfrozenxid, tc.relminmxid AS tminmxid, c.relpersistence, c.relispopulated, c.relreplident, c.relpages, CASE WHEN c.reloftype <> 0 THEN c.reloftype::pg_catalog.regtype ELSE NULL END AS reloftype, d.refobjid AS owning_tab, d.refobjsubid AS owning_col, (SELECT spcname FROM pg_tablespace t WHERE t.oid = c.reltablespace) AS reltablespace, array_remove(array_remove(c.reloptions,'check_option=local'),'check_option=cascaded') AS reloptions, CASE WHEN 'check_option=local' = ANY (c.reloptions) THEN 'LOCAL'::text WHEN 'check_option=cascaded' = ANY (c.reloptions) THEN 'CASCADED'::text ELSE NULL END AS checkoption, tc.reloptions AS toast_reloptions, c.relkind = 'S' AND EXISTS (SELECT 1 FROM pg_depend WHERE classid = 'pg_class'::regclass AND objid = c.oid AND objsubid = 0 AND refclassid = 'pg_class'::regclass AND deptype = 'i') AS is_identity_sequence, EXISTS (SELECT 1 FROM pg_attribute at LEFT JOIN pg_init_privs pip ON (c.oid = pip.objoid AND pip.classoid = 'pg_class'::regclass AND pip.objsubid = at.attnum)WHERE at.attrelid = c.oid AND ((SELECT pg_catalog.array_agg(acl ORDER BY row_n) FROM (SELECT acl, row_n FROM pg_catalog.unnest(coalesce(at.attacl,pg_catalog.acldefault('c',c.relowner))) WITH ORDINALITY AS perm(acl,row_n) WHERE NOT EXISTS ( SELECT 1 FROM pg_catalog.unnest(coalesce(pip.initprivs,pg_catalog.acldefault('c',c.relowner))) AS init(init_acl) WHERE acl = init_acl)) as foo) IS NOT NULL OR (SELECT pg_catalog.array_agg(acl ORDER BY row_n) FROM (SELECT acl, row_n FROM pg_catalog.unnest(coalesce(pip.initprivs,pg_catalog.acldefault('c',c.relowner))) WITH ORDINALITY AS initp(acl,row_n) WHERE NOT EXISTS ( SELECT 1 FROM pg_catalog.unnest(coalesce(at.attacl,pg_catalog.acldefault('c',c.relowner))) AS permp(orig_acl) WHERE acl = orig_acl)) as foo) IS NOT NULL OR NULL IS NOT NULL OR NULL IS NOT NULL))AS changed_acl, pg_get_partkeydef(c.oid) AS partkeydef, c.relispartition AS ispartition, pg_get_expr(c.relpartbound, c.oid) AS partbound FROM pg_class c LEFT JOIN pg_depend d ON (c.relkind = 'S' AND d.classid = c.tableoid AND d.objid = c.oid AND d.objsubid = 0 AND d.refclassid = c.tableoid AND d.deptype IN ('a', 'i')) LEFT JOIN pg_class tc ON (c.reltoastrelid = tc.oid) LEFT JOIN pg_init_privs pip ON (c.oid = pip.objoid AND pip.classoid = 'pg_class'::regclass AND pip.objsubid = 0) WHERE c.relkind in ('r', 'S', 'v', 'c', 'm', 'f', 'p') ORDER BY c.oid
Backup finished at Sun Sep 02 22:10:21 EDT 2018
Process exit code: 1
UPDATE
I believe this has to do with a large query problem instead of a ssl problem. I did an insert with 50 rows and no problem. As soon as i go over a certain number of characters it seems to break connecting to the server. Also I believe the pg_dump is a problem because the query it tries to do is a very large query. This is a local problem because it doesnt happen on my other laptop.
Probably some max setting in Postgres 10.5-2 or JDBC. My old laptop has the same JDBC version as my new one. Old laptop has Postgres 10.4-1 though.
So this turns out not to be related directly with Postgres.
I have a MSI GS65. It has a hardware network driver called killer network.
The driver has settings you can modify. One of those setting is to optimize network speeds. It seems that this setting will drop larger connections. When I am doing a backup or a large query through dbeaver, it drops the connection and stops the backup/query.
Took me 2 years to find this issue. I hope this helps any devs with a MSI computer with network issues.

PostgreSQL - get effective permissions for specified roles on each object type

I am trying to write a query that would (given a list of roles, and list of databases), list effective permissions for object of type database, schema, and table (to start with)
I have been trying to use has_XXX_privilege() functions but the output feels awkward...
Given 3 roles, for example, (app_rwc, app_rw, app_r) and a single db test_db I'd like to get output like this
role, obj_type, obj_name, has_permissions, missing_premissions
app_rwc, DATABASE, test_db, CREATE+CONNECT+TEMPORARY", NULL
app_rw, DATABASE, test_db, CONNECT+TEMPORARY, CREATE
app_r, DATABASE, test_db, CONNECT+TEMPORARY, CREATE
app_rwc, SCHEMA, audit, CREATE+USAGE, NULL
app_rwc, SCHEMA, shared, CREATE+USAGE, NULL
app_rw, SCHEMA, audit, USAGE, CREATE
app_rw, SCHEMA, shared, USAGE, CREATE
app_r, SCHEMA, audit, USAGE, CREATE
app_r, SCHEMA, audit, USAGE, CREATE
app_rwc, TABLE, audit.trail, SELECT+INSERT+UPDATE+DELETE+REFERENCES+TRIGGERS, TRUNCATE
etc
etc
So far this is what I got and it kind of works except it is verbose...
If anyone has a better approach please advise - thanks.
WITH
databases AS (
SELECT * FROM (VALUES ('app_prod')) AS t(database_name)
),
roles AS (
SELECT * FROM (VALUES ('app_rwc'), ('app_rw'), ('app_r')) AS t(role_name)
),
db_permissions AS (
SELECT * FROM (VALUES ('CREATE'), ('CONNECT'), ('TEMPORARY')) AS t(permission_name)
),
schemas AS (
SELECT
schema_name
FROM
information_schema.schemata
WHERE
catalog_name IN (SELECT database_name FROM databases)
AND schema_owner IN (SELECT role_name FROM roles)
),
schema_permissions AS (
SELECT * FROM (VALUES ('CREATE'), ('USAGE')) AS t(permission_name)
),
tables AS (
SELECT table_schema, table_name
FROM information_schema.tables
WHERE
table_catalog IN (SELECT database_name FROM databases)
AND table_schema IN (SELECT schema_name FROM schemas)
AND table_type IN ('BASE TABLE') -- , 'VIEW'
),
table_permissions AS (
SELECT * FROM (VALUES ('SELECT'), ('INSERT'), ('UPDATE'), ('DELETE'), ('TRUNCATE'), ('REFERENCES'), ('TRIGGER')) AS t(permission_name)
)
-- ----------------------------------------------------------------------------
SELECT
'DATABASE' AS obj_type
, databases.database_name AS obj_name
, roles.role_name
, db_permissions.permission_name
, has_database_privilege(roles.role_name, databases.database_name, db_permissions.permission_name) AS has_permission
FROM
databases
CROSS JOIN roles
CROSS JOIN db_permissions
-- ----------------------------------------------------------------------------
UNION ALL
-- ----------------------------------------------------------------------------
SELECT
'SCHEMA' AS obj_type
, schemas.schema_name AS obj_name
, roles.role_name
, schema_permissions.permission_name
, has_schema_privilege(roles.role_name, schemas.schema_name, schema_permissions.permission_name) AS has_permission
FROM
schemas
CROSS JOIN roles
CROSS JOIN schema_permissions
-- ----------------------------------------------------------------------------
UNION ALL
-- ----------------------------------------------------------------------------
SELECT
'TABLE' AS obj_type
, tables.table_schema || '.' || tables.table_name AS obj_name
, roles.role_name
, table_permissions.permission_name
, has_table_privilege(roles.role_name, (tables.table_schema || '.' || tables.table_name),table_permissions.permission_name) AS has_permission
FROM
tables
CROSS JOIN roles
CROSS JOIN table_permissions
UPDATE #1 - Here is expanded query (does types, sequences, and functions) with aggregation (Thanks to #filiprem for the tip!) Still rather large, but it does what I want it to do.
WITH
databases AS (
SELECT unnest('{app_prod}'::text[]) AS dbname
),
roles AS (
SELECT unnest('{app_rwc,app_rw,app_r}'::text[]) AS rname
),
permissions AS (
SELECT 'DATABASE' AS ptype, unnest('{CREATE,CONNECT,TEMPORARY}'::text[]) AS pname
UNION ALL
SELECT 'SCHEMA' AS ptype, unnest('{CREATE,USAGE}'::text[]) AS pname
UNION ALL
SELECT 'TABLE' AS ptype, unnest('{SELECT,INSERT,UPDATE,DELETE,TRUNCATE,REFERENCES,TRIGGER}'::text[]) AS pname
UNION ALL
SELECT 'SEQUENCE' AS ptype, unnest('{USAGE,SELECT,UPDATE}'::text[]) AS pname
UNION ALL
SELECT 'TYPE' AS ptype, unnest('{USAGE}'::text[]) AS pname
UNION ALL
SELECT 'FUNCTION' AS ptype, unnest('{EXECUTE}'::text[]) AS pname
),
schemas AS (
SELECT schema_name AS sname
FROM information_schema.schemata
WHERE catalog_name IN (SELECT dbname FROM databases) -- show schemas that exist in specified DB
AND schema_owner IN (SELECT rname FROM roles) -- show schemas that are owned by specified roles
OR schema_name IN ('public') -- always include these
--OR schema_name IN ('public', 'information_schema', 'pg_catalog')
),
tables AS (
SELECT table_schema AS tschema, table_name AS tname
FROM information_schema.tables
WHERE table_catalog IN (SELECT dbname FROM databases)
AND table_schema IN (SELECT sname FROM schemas)
AND table_type IN ('BASE TABLE') -- , 'VIEW'
),
sequences AS (
SELECT schemaname AS seqschema, sequencename AS seqname
FROM pg_sequences
WHERE schemaname IN (SELECT sname FROM schemas)
),
types AS (
SELECT nspname AS typeschema, typname AS typename, CASE typtype WHEN 'c' THEN 'composite' WHEN 'd' THEN 'domain' WHEN 'e' THEN 'enum' WHEN 'r' THEN 'range' ELSE 'other' END AS typekind
FROM pg_type INNER JOIN pg_namespace ON pg_type.typnamespace = pg_namespace.oid
WHERE nspname IN (SELECT sname FROM schemas)
AND typtype NOT IN ('b','p') -- exclude base and pseudo types
AND typname NOT IN (SELECT seqname FROM sequences) -- exclude sequences
),
functions AS (
SELECT nspname AS fnschema, proname AS fnname, pg_proc.oid AS fnoid, pg_get_function_arguments(pg_proc.oid) AS fnargs
FROM pg_proc INNER JOIN pg_namespace ON pg_proc.pronamespace = pg_namespace.oid
WHERE nspname IN (SELECT sname FROM schemas)
),
final AS (
SELECT
permissions.ptype
, databases.dbname AS obj_name
, roles.rname
, permissions.pname
, has_database_privilege(roles.rname, databases.dbname, permissions.pname) AS has_permission
FROM
databases
CROSS JOIN roles
CROSS JOIN permissions
WHERE
permissions.ptype = 'DATABASE'
UNION ALL -- ----------------------------------------------------------------------------------------------------------
SELECT
permissions.ptype
, schemas.sname AS obj_name
, roles.rname
, permissions.pname
, has_schema_privilege(roles.rname, schemas.sname, permissions.pname) AS has_permission
FROM
schemas
CROSS JOIN roles
CROSS JOIN permissions
WHERE
permissions.ptype = 'SCHEMA'
UNION ALL -- ----------------------------------------------------------------------------------------------------------
SELECT
permissions.ptype
, tables.tschema || '.' || tables.tname AS obj_name
, roles.rname
, permissions.pname
, has_table_privilege(roles.rname, (tables.tschema || '.' || tables.tname), permissions.pname) AS has_permission
FROM
tables
CROSS JOIN roles
CROSS JOIN permissions
WHERE
permissions.ptype = 'TABLE'
UNION ALL -- ----------------------------------------------------------------------------------------------------------
SELECT
permissions.ptype
, sequences.seqschema || '.' || sequences.seqname AS obj_name
, roles.rname
, permissions.pname
, has_sequence_privilege(roles.rname, (sequences.seqschema || '.' || sequences.seqname), permissions.pname) AS has_permission
FROM
sequences
CROSS JOIN roles
CROSS JOIN permissions
WHERE
permissions.ptype = 'SEQUENCE'
UNION ALL -- ----------------------------------------------------------------------------------------------------------
SELECT
permissions.ptype || ' - ' || types.typekind
, types.typeschema || '.' || types.typename AS obj_name
, roles.rname
, permissions.pname
, has_type_privilege(roles.rname, (types.typeschema || '.' || types.typename), permissions.pname) AS has_permission
FROM
types
CROSS JOIN roles
CROSS JOIN permissions
WHERE
permissions.ptype = 'TYPE'
UNION ALL -- ----------------------------------------------------------------------------------------------------------
SELECT
permissions.ptype
, functions.fnschema || '.' || functions.fnname || '(' || fnargs || ')' AS obj_name
, roles.rname
, permissions.pname
, has_function_privilege(roles.rname, functions.fnoid, permissions.pname) AS has_permission
FROM
functions
CROSS JOIN roles
CROSS JOIN permissions
WHERE
permissions.ptype = 'FUNCTION'
)
-- ====================================================================================================================
SELECT
rname AS role_name
, ptype AS object_type
, obj_name AS object_name
, string_agg(DISTINCT CASE WHEN has_permission THEN pname END, ',') AS granted_permissions
, string_agg(DISTINCT CASE WHEN NOT has_permission THEN pname END, ',') AS missing_premissions
FROM
final
GROUP BY 1, 2, 3
ORDER BY 1, 2, 3
Your query is good, you just need to add some aggregation. This is a start:
select obj_type, obj_name, role_name,
array_agg(distinct case when has_permission then permission_name end),
array_agg(distinct case when not has_permission then permission_name end)
from ( /* your query */ ) AS q1
group by 1,2,3
order by 1,2,3

Wrong row count using Catalog Queries

I got this query on the following page:
http://docs.aws.amazon.com/redshift/latest/dg/c_join_PG_examples.html
It lists the table names along with row count. But the row count seems to be wrong (10 times than actual)
select datname, nspname, relname, sum(rows) as rows
from pg_class, pg_namespace, pg_database, stv_tbl_perm
where pg_namespace.oid = relnamespace
and pg_class.oid = stv_tbl_perm.id
and pg_database.oid = stv_tbl_perm.db_id
and datname ='tickit'
group by datname, nspname, relname
order by datname, nspname, relname;
Is the sum(rows) correct?
Here, try mine. It includes the data size as well. :-D
/*
Ordered view of table sizes.
*/
CREATE VIEW my_schema.vw_table_summary
AS
SELECT db_name
,schema_name
,table_name
,TO_CHAR(rows_total,'999,999,999,999') rows_total
,TO_CHAR(rows_sorted,'999,999,999,999') rows_sorted
,CASE WHEN rows_total > 0
THEN ROUND((CONVERT(NUMERIC,rows_sorted)/CONVERT(NUMERIC,rows_total))*100,2)
ELSE 0.00 END percent_sorted
,COALESCE(size_in_gb,0) size_in_gb
FROM (SELECT id table_id
,datname db_name
,nspname schema_name
,relname table_name
,SUM(rows) rows_total
,SUM(sorted_rows) rows_sorted
FROM stv_tbl_perm
JOIN pg_class
ON pg_class.oid = stv_tbl_perm.id
JOIN pg_namespace
ON pg_namespace.oid = relnamespace
JOIN pg_database
ON pg_database.oid = stv_tbl_perm.db_id
WHERE name NOT LIKE 'pg_%'
AND name NOT LIKE 'stl_%'
AND name NOT LIKE 'stp_%'
AND name NOT LIKE 'padb_%'
AND nspname <> 'pg_catalog'
GROUP BY id, datname, nspname, relname
ORDER BY id, datname, nspname, relname) tbl_det
LEFT
JOIN (SELECT tbl table_id
,ROUND(CONVERT(REAL,COUNT(*))/1024,2) size_in_gb
FROM stv_blocklist bloc
GROUP BY tbl) tbl_size
ON tbl_size.table_id = tbl_det.table_id
ORDER BY db_name
,schema_name
,table_name
;