Postgres: log_statement = 'none' is ignored when query comes from C++ - postgresql

I am blocked by the following problem.
The log_statement of postgres on a server is set to ddl (or all). The database is created by my application, written in C++. Queries are sent to DB by using libpq - PQexec.
Every query is logged twice, i don't know why (I am not a C++ programmer nor expert in postgres):
Apr 3 02:26:44 xxx postgres[12345]: [8-1] [2020-04-03 02:26:44.487 CDT] [s:xxx.694d] [u:user] [a:[unknown]] [db:postgres] [p:12345] [clnt:[local]] LOG: statement: CREATE USER "gingillo" WITH PASSWORD 'giggio';
Apr 3 02:26:44 xxx postgres[12345]: [9-1] [2020-04-03 02:26:44.487 CDT] [s:xxx.694d] [u:user] [a:[unknown]] [db:postgres] [p:12345] [clnt:[local]] LOG: AUDIT: SESSION,1,1,ROLE,CREATE ROLE,,,"CREATE USER ""gingillo"" WITH PASSWORD <REDACTED>",<not logged>
As you already imaged, I don't want to log passwords, so I changed the current query (1):
CREATE USER "gingillo" WITH PASSWORD 'giggio'
to be following (2):
BEGIN;SET LOCAL log_statement = 'none';CREATE USER "gingillo" WITH PASSWORD 'giggio';COMMIT;
If I run manually the query (2), I get the result I want, just one log is logged containing REDACTED instead of the password:
Apr 3 02:26:44 xxx postgres[12345]: [9-1] [2020-04-03 02:26:44.487 CDT] [s:xxx.694d] [u:user] [a:[unknown]] [db:postgres] [p:12345] [clnt:[local]] LOG: AUDIT: SESSION,1,1,ROLE,CREATE ROLE,,,"CREATE USER ""gingillo"" WITH PASSWORD <REDACTED>",<not logged>
But when the same query is run by C++, I have double log, showing even the log_statement stuff:
Apr 8 06:44:24 xxx postgres[27171]: [8-1] [2020-04-08 06:44:24.489 CDT] [s:xxx.6a23] [u:user] [a:[unknown]] [db:postgres] [p:27171] [clnt:[local]] LOG: statement: BEGIN;SET LOCAL log_statement = 'none';CREATE USER "gingillo" WITH PASSWORD 'giggio';COMMIT;
Apr 8 06:44:24 xxx postgres[27171]: [9-1] [2020-04-08 06:44:24.490 CDT] [s:xxx.6a23] [u:user] [a:[unknown]] [db:postgres] [p:27171] [clnt:[local]] LOG: AUDIT: SESSION,1,1,ROLE,CREATE ROLE,,,"BEGIN;SET LOCAL log_statement = 'none';CREATE USER ""gingillo"" WITH PASSWORD <REDACTED>",<not logged>
Does anybody have any idea how to disable the first log? What am I doing wrong?

SET LOCAL scope is only for the current transaction.
Try:
Either to run SET log_statement=none that persists during the database session unless ROLLBACK is issued.
or to run SET LOCAL log_statement=none for each transaction.
It looks C++ code is setting log_statement parameter: the best solution should be to remove it since it already uses another logging.

You are sending PostgreSQL a single "statement" which contains 4 actual statements. By the time the "SET LOCAL log_statement = 'none'" is processed, the damage has already been done as the entire multi-statement statement has already been logged. You need to send each statement separately if you want to have part of it effect the settings for the other part of it.
Alternatively, just avoid setting log_statement to 'all' in the first place, as it seems to be unneeded.

Related

Disable logging of logical replication statements in Postgres 13.1

I have a simple process that is reading logical replication messages from postgres. This process runs every second and generates a lot of messages in the postgres logs like:
2021-02-15 20:35:11.032 UTC [35] STATEMENT: SELECT * FROM pg_logical_slot_get_changes('lazy_cloud', NULL, NULL);
2021-02-15 20:35:11.032 UTC [35] LOG: logical decoding found consistent point at 0/167C618
2021-02-15 20:35:11.032 UTC [35] DETAIL: There are no running transactions.
I've configured logging with the following settings:
log_min_messages=ERROR
log_statement=none
log_replication_commands=0
But, the logical replication logs are still produced.
Is there a setting to disable these messages? I can use sed or something like that, but would prefer a built in solution.
There is no way to disable that message short of setting
log_min_messages = fatal
in postgresql.conf, but that is not a smart idea, because then you'd miss out on all error messages in the log file and essentially disable logging.

Origin of a postgres timeout impossible to determine

I have a mobile application (C#) which call an API using phoenix and ecto.
This API makes several calls to the postgres database. Each call lasts approximately 60ms and we make approximately 25 calls to the database.
But at some point, I get a timeout from the database.
Here is the postgres error:
2020-06-04 09:40:03.503 CEST [24455] postgres#view_models ERROR: canceling statement due to user request
2020-06-04 09:40:03.503 CEST [24455] postgres#view_models STATEMENT: SELECT DISTINCT ON (i0."intervention_id") i0."intervention_id" FROM "interventions" AS i0 LEFT OUTER JOIN "appointments" AS a1 ON a1."intervention_id" = i0."intervention_id" WHERE ((i0."account_id" = $1) AND ((i0."updated_at" > $2) OR (a1."updated_at" > $3))) LIMIT 1
2020-06-04 09:40:03.504 CEST [24455] postgres#view_models LOG: could not send data to client: Broken pipe
2020-06-04 09:40:03.504 CEST [24455] postgres#view_models FATAL: connection to client lost`
Here is the ecto error:
DBConnection.ConnectionError: tcp recv: closed (the connection was closed by the pool, possibly due to a timeout or because the pool has been terminated)
The postgres statement_tiemout parameter is 0
Here is the ecto config:
config :query_backend, QueryBackend.V1.Repo,
username: System.get_env("POSTGRES_USERNAME"),
password: System.get_env("POSTGRES_PASSWORD"),
database: System.get_env("VIEW_POSTGRES_DB_NAME"),
hostname: System.get_env("POSTGRES_HOST"),
pool_size: 10,
queue_target: 3_000,
queue_interval: 15_000,
connect_timeout: 20_000,
timeout: 30_000
On the mobile, the HTTP client timeout is 15s.
This problem is reproduced only by our customers and our technical team cannot reproduce it.
Can you tell me if my configuration is valid? What is the origin of timeout? the mobile HTTP client, Ecto, Postgres ?
Thank you in advance for your help
Best regards.
Julien.

Hsqldb Server, alias does not exist

According to the log the server was startet with alias "myrepos" as expected.
But if I try to connect to this alias, I get an error, also visible in the log (last line).
What could be the cause?
[Server#28fc19eb]: Initiating startup sequence...
[Server#28fc19eb]: Server socket opened successfully in 6 ms.
[Server#28fc19eb]: Database [index=0, id=0, db=file:/Users/t..../myrepos, alias=myrepos ] opened sucessfully in 1238 ms.
[Server#28fc19eb]: Startup sequence completed in 1247 ms.
[Server#28fc19eb]: 2016-04-08 10:32:33.871 HSQLDB server 2.3.3 is online on port 9001
[Server#28fc19eb]: To close normally, connect and execute SHUTDOWN SQL
[Server#28fc19eb]: From command line, use [Ctrl]+[C] to abort abruptly
[Server#28fc19eb]: [Thread[HSQLDB Connection #2304d78b,5,HSQLDB Connections #28fc19eb]]: database alias=myrepos does not exist
Solved. I had spaces behind the database name in my configuration, which had been incorporated to the alias: 'alias=myrepos ]'
So, the alias actually wasn't "myrepos" but "myrepos "

Postgresql - LDAP Authentication against Active Directory (AD) - trouble from linux server while ok from windows server

EDIT
I put the same pg_hba rule on the postgresql installed on my WINDOWS laptop, and it works... so I changed titled : how can I make my linux server work to authenticate users with AD, like the windows Pg server does ?
/ EDIT
I need to perform authentication of postgresql db users with our Active Directory servers. I've tested lots of configurations but so far, I couldn't find why postgresql users can't be authenticated with this authentication methode.
LDAP : Active Directory / Postgresql : Postgresql 9.4
Here is the pg_hba rule I use :
host myDB myUser localhost ldap ldapserver="192.168.10.1" ldapbasedn="DC=companygroup,DC=priv" ldapbinddn="cn=LDAP - Lecture,ou=Users,ou=Specials Objects,dc=companygroup,dc=priv" ldapbindpasswd="ldapPassWord" ldapsearchattribute="sAMAccountName"
When logging with 'myUser' with the correct password for this user, I have the following logs in postgresql log file :
2015-11-18 10:01:50 CET [25991-1] [unknown]#[unknown] LOG: 00000: connection received: host=127.0.0.1 port=39074
2015-11-18 10:01:50 CET [25991-2] [unknown]#[unknown] LOCATION: BackendInitialize, postmaster.c:4003
2015-11-18 10:01:50 CET [25991-3] myUser#myDB LOG: 00000: could not search LDAP for filter "(sAMAccountName=myUser)" on server "192.168.10.1": Operations error
2015-11-18 10:01:50 CET [25991-4] myUser#myDB LOCATION: CheckLDAPAuth, auth.c:2030
2015-11-18 10:01:50 CET [25991-5] myUser#myDB FATAL: 28000: LDAP authentication failed for user "myUser"
2015-11-18 10:01:50 CET [25991-6] myUser#myDB DETAIL: Connection matched pg_hba.conf line 104: "host myDB myUser localhost ldap ldapserver="192.168.10.1" ldapbasedn="DC=companygroup,DC=priv" ldapbinddn="cn=LDAP - Lecture,ou=Users,ou=Specials Objects,dc=companygroup,dc=priv" ldapbindpasswd="ldapPassWord" ldapsearchattribute="sAMAccountName"
I saw that if I change somehow ldapbinddn or ldapbindpasswd, I have another error like 'couldn't perform initial LDAP bind for ldapbinddn "...". so these parameters should be ok.
"Operations error" was no very detailed, so I tcpdump the authentication process and here is what I found. It seems that Postgres perform two queries :
First to search the user via the search attribute. This action seems OK because in the response of Active Directory, I saw information tied to my user.
Then another query is performed. On this one, the real message from the LDAP Active directory server is :
LdapErr: DSID-0C0906E8, comment: In order to perform this operation a successful bind must be completed on the connection., data 0 , v1db1
On this second query, I see that PG seem to change slightly the base search with
"DC=ForestDnsZones,DC=companygroup,dc=priv" instead of "DC=companygroup,DC=priv"
(I saw it in the tcp trace :
LDAPMessage searchRequest(3) "DC=ForestDnsZones,DC=companygroup,dc=priv" wholeSubtree ...
)
When I tried a research thanks to windows soft "ldapbrowser", I was abled to find my account with a simple filter (sAMAccountName=myUser), with the searchDN DC=companygroup,DC=priv
Is my understanding correct ? is it possible that the search is not successful just because of the basedn that is maybe changed ? or do I miss something else ?
Based on your log, your configuration for ldapsearchattribute="sAMAccountName" doesn't work.
You can use a LDAP tool such as LDAPAdmin or OpenLDAP to test your filter. Make sure that the above return result when your filter the attribute below
(sAMAccountName=myUser)

Postgres crashes for long query

My postgres crashes for long query. It's on Debian 7 64bit, and postgresql-9.3.2. I uses all default configuration. Could anyone suggest what problem it could be? thanks.
--part1:
SELECT r1.f2 as b, r1.e as l
FROM r r8,r r7,r r6,r r5,r r4,r r3,r r2,r r1
WHERE
r1.f2=r2.f1 AND
r1.f2=r3.f1 AND
r1.f2=r4.f1 AND
r1.f1=r5.f2 AND
r1.f1=r8.f1 AND
r2.f1=r3.f1 AND
r2.f1=r4.f1 AND
r2.f2=r6.f2 AND
r2.f2=r7.f1 AND
r3.f1=r4.f1 AND
r3.f2=r7.f2 AND
r3.f2=r8.f2 AND
r4.f2=r5.f1 AND
r4.f2=r6.f1 AND
r5.f1=r6.f1 AND
r5.f2=r8.f1 AND
r6.f2=r7.f1 AND
r7.f2=r8.f2 AND
r1.d=1 AND
r2.d=1 AND
r3.d=2 AND
r4.d=2 AND
r5.d=2 AND
r6.d=2 AND
r7.d=2 AND
r8.d=2
-- part2
group by r1.f2,r1.e
having
calc_empty_a() AND
calc_empty_b();
In the query, calc_empty_a() are just empty boolean functions (return true), so they should have no problem.
If I run the query in client, the server crashes. There is nothing useful information in the log (please refer to the error info at end of the post).
If I run the part 1 query, the query works well.
If I first run the part 1 query, then I run the whole query, it works well.
If I change the query, reduce the r numbers, for example, there are only r1 to r6 FROM tables, delete the predicates with r8, r7, but keep the part 2's GROUP BY and HAVING clause. The query still works well.
If the query have one empty function in HVING clause, the query also works well, but will crash if there are two functions.
The following query works well
SELECT r1.f2 as b, r1.f1 as a , r1.e as e
FROM r r8,r r7,r r6,r r5,r r4,r r3,r r2,r r1
WHERE
r1.f2=r2.f1 AND
r1.f2=r3.f1 AND
r1.f2=r4.f1 AND
r1.f1=r5.f2 AND
r1.f1=r8.f1 AND
r2.f1=r3.f1 AND
r2.f1=r4.f1 AND
r2.f2=r6.f2 AND
r2.f2=r7.f1 AND
r3.f1=r4.f1 AND
r3.f2=r7.f2 AND
r3.f2=r8.f2 AND
r4.f2=r5.f1 AND
r4.f2=r6.f1 AND
r5.f1=r6.f1 AND
r5.f2=r8.f1 AND
r6.f2=r7.f1 AND
r7.f2=r8.f2
group by r1.f2,r1.f1, r1.e
having
calc_empty_a() AND
calc_empty_a();
I have set the os to use strict overcommit mode:
sysctl -w vm.overcommit_memory=2
Error info:
at client
The connection to the server was lost. Attempting reset: Succeeded.
at server
LOG: terminating any other active server processes
WARNING: terminating connection because of crash of another server process
DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
HINT: In a moment you should be able to reconnect to the database and repeat your command.
WARNING: terminating connection because of crash of another server process
DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
HINT: In a moment you should be able to reconnect to the database and repeat your command.
LOG: all server processes terminated; reinitializing
LOG: database system was interrupted; last known up at 2014-11-07 16:47:03 GMT
LOG: database system was not properly shut down; automatic recovery in progress
LOG: redo starts at 0/2126C98
LOG: record with zero length at 0/21A9D98
LOG: redo done at 0/21A9D68
LOG: last completed transaction was at log time 2014-11-07 16:47:26.844406+00
LOG: autovacuum launcher started
LOG: database system is ready to accept connections