I am trying to get my postgresql 9.3 server to log all sql that runs longer than 1 second. I have set:
log_min_duration=1s
log_statement='mod'
log_duration=off
for most queries, the logging is working correclty, but some statements, such as "CREATE TABLE AS" or "INSERT" are not logging the statement. The log the duration, but not the complete sql.
Has anyone else seen this type of issue before, or know why postgres may not be logging the sql? (possibly a transaction block?) Any help is greatly appreciated, thanks.
The correct parameter is log_min_duration_statement and not log_min_duration.
And this note, from docs,
Note: When using this option together with log_statement, the text of
statements that are logged because of log_statement will not be
repeated in the duration log message. If you are not using syslog, it
is recommended that you log the PID or session ID using
log_line_prefix so that you can link the statement message to the
later duration message using the process ID or session ID.
consider is a good explanation for problem.
Related
I need to log all activity for some specific user on database. I have set up the logging with ALTER ROLE username SET log_statement TO 'all'; and the logging works fine, all queries from user are logged. The problem is that for this user queries to Postgres internal schemas (pg_catalog) from clients like psql and pgAdmin are also logged. I have a bunch of lines with SELECT pg_catalog.quote_ident(n.nspname) || '.' || pg_catalog.quote_ident(c.relname).... in the log that are of no use to me. Even worse this queries are more then one line in the log so it's not easy to filter them out.
Is it possible to somehow restrict the logging only to one specific database or schema and not to include queries to other schemas like pg_catalog?
I don't know if the standard logging utility in postgres has that option (my guess is no). But maybe it's worth a look to the pgaudit external library for postgres.
The module pgadmin is designed to generate audit logs, but it uses the standard postgres logging tool. You can tweak several parameters to customize the logs, and it has a specific parameter that I think is perfect for your use case. From the documentation:
pgaudit.log_catalog
Specifies that session logging should be enabled in the case where all
relations in a statement are in pg_catalog. Disabling this setting
will reduce noise in the log from tools like psql and PgAdmin that
query the catalog heavily.
The default is on.
I hope it helps!
Change your logging format from text to csv (log_destination=csvlog) — you can then import the data to the database and then filter out the queries you are not interested in:
Using CSV-Format Log Output
I have configured my PostgreSQL 9.4 install to log statements that take over 500 ms using log_min_duration_statement
Now, I have a query that is expected to take in the order of 2 seconds. So it gets recorded every time is run.
I would like to whitelist this query so it doesn't show up in the logs or, even better, to configure log_min_duration_statement to only log that query is it takes over 5000 ms without affecting the rest.
There is no direct way to do that, but there are some workarounds. You can set log_min_duration_statement from your client code before you run that one query. Or you could run that query under a different user and have that user configured differently.
I want to log the actual sql statements executed against a POSTGRES instance. I am aware that I can enable logging of the sql statements. Unfortunately, this doesn't log the actual sql, but rather a parsed version, with certain parameters stripped out and listed separately.
Is there a tool for reliably reconstituting this output into executable sql statements?
Or is there a way of intercepting the sql that is send to the postgres instance, such that that sql can be logged?
We want to be able to replay these sql statements against another database.
Thanks for your help!
Actually, PostgreSQL does log exactly the SQL that got executed. It doesn't strip parameters out. Rather, it doesn't interpolate them in, it logs what the application sent, with bind parameters separate. If your app sends insert into x(a,b) values ($1, $2) with bind params 42 and 18, that's what gets logged.
There's no logging option to interpolate bind parameters into the query string.
Your last line is the key part. You don't want logging at all. You're trying to do statement based replication via the logs. This won't work well, if at all, due to volatile functions, the search_path, per-user settings, sequence allocation order/gap issues, and more. If you want replication don't try to do it by log parsing.
If you want to attempt statement-based replication look into PgPool-II. It has a limited ability to do so, with caveats aplenty.
Via setting log_statement to all on postgresql.conf. See the documentation for runtime-config-logging
I have the following setting in my postgresql.conf
log_statement='all'
log_duration=on
When I run psql, and run any query like select/create user/ etc, it logs the statement as well as the duration. If I give the query to connect to a different database
\c <database_name>
it logs neither the duration nor the statement. I thought it might not log duration and statement for meta-commands that starts with a backslash. But it does log the statement and duration for some of them, like \dt \l.
I think there might be a list of queries/commands, for which log will be generated or not. is there any such list?
Having said that, the original issue, that led me to dig this is --
My application (golang-react application) interacts with postgresql and logs statement and its duration. But after that it logs two more durations, whose statement is not printed. (I have commented log_min_duration_statement), so I'm not sure what queries, this duration is for.
Also if I do SELECT query,xact_start,query_start FROM pg_stat_activity; at that time to see the running queries,it does not show any other query than the one which I see in the logs.
How can I know which queries is this duration for?
But after that it logs two more durations, whose statement is not printed.
These are probably the parse, bind and execute durations for the three phases of statement execution, if you have log_duration = on. It's only done in three phases via client applications - psql runs in a single go.
If you just want the total, use log_min_duration_statement = 0 instead.
Yes, that's confusing. Maybe it's worth writing it up and submitting it as a usability problem report to pgsql-general.
I am currently trying to debug "idle in transaction" scenarios in my application.I can find out the process id and transaction start time for a query with state 'idle in transaction' by looking at pg_state_activity.
select pid,query from pg_stat_activity where state='idle in transaction' OR state='idle'
Is there any way to identify list of all queries executed within a transaction corresponding to the query with 'idle in transaction'
Are you attempting to get a list of all previous statements run by the transaction that is now showing up as idle in transaction?
If so, there is no easy and safe way to do so at the SQL level. You should use CSV logging mode to analyze your query history and group queries up into transactions. Handily, you can do this with SQL, by COPYing the CSV into a PostgreSQL table for easier analysis.
Alternately, use ordinary text logs, and set a log_line_prefix that includes the transaction ID and process ID.
(I could've sworn I saw an extension for debugging that collected a query trace, but cannot find it now, and I'm not sure it's that useful as you must run a command on the problem connection to extract the data it's collected).