Using pg badger to generate daily reports of the database but the report is empty ? ANy ideas why?
thanks
log_min_duration_statement = -1
log_duration = on
log_line_prefix = '%t [%p]: user=%u,db=%d,app=%a,client=%h'
log_checkpoints = 'on'
log_lock_waits = 'on'
log_temp_files = 0
log_autovacuum_min_duration = 0
log_error_verbosity = 'default'
lc_messages='en_US.UTF-8'
-bash-4.2$ pgbadger -f stderr -d dbdev /var/lib/pgsql/13/data/log/postgresql-*.log -x html -o
/var/lib/pgsql/13/data/log/pgbadger_dbdevv_date +\%F.html
[========================>] Parsed 23526878 bytes of 23526878
(100.00%), queries: 0, events: 0 LOG: Ok, generating html report...
The pgBadger documentation states:
You must first enable SQL query logging to have something to parse:
log_min_duration_statement = 0
I have a task to import some data from Postgres database to Oracle via dblink
The connection between Postgres and Oracle works good, but unfortunately, when I try read data from created view (in Oracle database), I spotted a problem with data encoding - special national characters (Polish).
Source Postgres database have a UTF8 encoding, but Oracle have a AL32UTF8
Postgres:
select server_encoding
-
UTF8
Oracle:
select * from v$nls_parameters where parameter like '%CHARACTERSET';
-
PARAMETER VALUE
NLS_CHARACTERSET AL32UTF8
NLS_NCHAR_CHARACTERSET AL16UTF16
When I use command "isql -v" (on destination machine with Oracle database) and later "select * from table;", everything works good, but when I use this same select from Oracle database using dblink my data encoding is broken
For example:
from odbc:
isql -v
select * from table;
[ID][Name]
0,Warszawa
1,Kraków
2,Gdańsk
from oracle using dblink:
select * from table#dblink;
[ID][Name]
0,Warszawa
1,KrakĂłw
2,Gdańsk
/etd/odbc.ini:
[ODBC Data Sources]
[Postgres_DB]
Description = Postgres_DB
Driver = /usr/lib64/psqlodbcw.so
DSN = Postgres_DB
Trace = Yes
TraceFile = /tmp/odbc_sql_postgresdb.log
Database = database
Servername = server
UserName = user
Password = secret
Port = 5432
Protocol = 8.4
ReadOnly = Yes
RowVersioning = No
ShowSystemTables = No
ShowOidColumn = No
FakeOidIndex = No
SSLmode = require
Charset = UTF8
$ORACLE_HOME/hs/admin/initPostgres_DB.ora:
HS_FDS_CONNECT_INFO = Postgres_DB
HS_FDS_TRACE_LEVEL=DEBUG
HS_FDS_SHAREABLE_NAME = /usr/lib64/libodbc.so
HS_FDS_SUPPORT_STATISTICS = FALSE
HS_LANGUAGE=AL32UTF8
set ODBCINI=/etc/odbc.ini
I have installed these packages:
postgresql-libs.x8664 - 8.4.20-8.el69
postgresql-odbc.x8664 - 08.04.0200-1.el6
unixODBC.x8664 - 2.2.14-14.el6
unixODBC-devel.x86_64 - 2.2.14-14.el6
Please help me.. I need to have the correct data in Oracle..
Thank you very much
There is the question. Sphinx, version 2.1.6. I used to rt(real time) index, but when indexing display message in koncole:
using config file 'sphinx.conf'...
skipping non-plain index 'rt'...
But at a connection to sphinxbase and write query mysql> desc rt - displays:
+------------+--------+
| Field | Type |
+------------+--------+
| id | bigint |
| id | field |
| first_name | field |
| last_name | field |
+------------+--------+
This is default data?? They do not meet my request. How to work with index rt?
Sphinx.conf.
source database
{
type = mysql
sql_host = 127.0.0.1
sql_user = test
sql_pass = test
sql_db = community
sql_port = 3306
mysql_connect_flags = 32 # enable compression
sql_query_pre = SET NAMES utf8
sql_query_pre = SET SESSION query_cache_type=OFF
}
source rt : database
{
sql_query_range = SELECT MIN(id),MAX(id) FROM mbt_accounts
sql_query = SELECT id AS 'accountId', first_name AS 'fname', last_name AS 'lname' FROM mbt_accounts WHERE id >= 0 AND id<= 1000
sql_range_step = 1000
sql_ranged_throttle = 1000 # milliseconds
}
index rt
{
source = rt
type = rt
path = /etc/sphinxsearch/rtindex
rt_mem_limit = 700M
rt_field = accountId
rt_field = fname
rt_field = lname
rt_attr_string = fname
rt_attr_string = lname
charset_type = utf-8
charset_table = 0..9, A..Z->a..z, _, -, a..z, U+410..U+42F->U+430..U+44F, U+430..U+44F, U+401->U+451, U+451
}
searchd
{
listen = localhost:9312 # port for API
listen = localhost:9306:mysql41 #port for a SphinxQL
log = /var/log/sphinxsearch/searchd.log
binlog_path = /var/log/sphinxsearch/
query_log = /var/log/sphinxsearch/query.log
query_log_format = sphinxql
pid_file = /var/run/sphinxsearch/searchd.pid
workers = threads
max_matches = 1000
read_timeout = 5
client_timeout = 300
max_children = 30
max_packet_size = 8M
binlog_flush = 2
binlog_max_log_size = 90M
thread_stack = 8M
expansion_limit = 500
rt_flush_period = 1800
collation_server = utf8_general_ci
compat_sphinxql_magics = 0
prefork_rotation_throttle = 100
}
Thanks.
indexer only works with indexes that have a 'source' - ie plain disk indexesd. ie indexer does the stuff in the source to get the data to create the index.
RT (Real Time) indexes work very differently. indexer is not involved with RT indexes at all. They are handled totally by searchd.
To add data to a RT index, you need to run a bunch of SphinxQL commands (INSERT, UPDATE etc) that actually add the data to the index.
(DESCRIBE works, because searchd knows the 'structure' of the index (you told it via the rt_field etc) - even if never inserted any data)
Ah, I think you are asking why the structure is different. That's probably because the index was probably created before, you modified sphinx.conf. If you change the definiton of a RT index, you need to 'destroy' the index, to allow it be recreated again.
The simplest way is to shutdown searchd, delete the index files, delete the binlog (it no longer relevent) and then restart searchd.
searchd --stopwait
rm /etc/sphinxsearch/rtindex*
rm /path/to/binlog* #(you dont define a path, so it must be the default, which varies)
searchd #(starts searchd again)
Every time I invoke a parameterized query I get too much output in the log file. For example, when inserting 3 users into a table I get the following log output:
2013-10-29 06:01:43 EDT LOG: duration: 0.000 ms parse <unnamed>: INSERT INTO users (login,role,password) VALUES
($1,$2,$3)
,($4,$5,$6)
,($7,$8,$9)
2013-10-29 06:01:43 EDT LOG: duration: 0.000 ms bind <unnamed>: INSERT INTO users (login,role,password) VALUES
($1,$2,$3)
,($4,$5,$6)
,($7,$8,$9)
2013-10-29 06:01:43 EDT DETAIL: parameters: $1 = 'guest', $2 = 'user', $3 = '123', $4 = 'admin', $5 = 'admin', $6 = '123', $7 = 'mark', $8 = 'power user', $9 = '123'
2013-10-29 06:01:43 EDT LOG: execute <unnamed>: INSERT INTO users (login,role,password) VALUES
($1,$2,$3)
,($4,$5,$6)
,($7,$8,$9)
2013-10-29 06:01:43 EDT DETAIL: parameters: $1 = 'guest', $2 = 'user', $3 = '123', $4 = 'admin', $5 = 'admin', $6 = '123', $7 = 'mark', $8 = 'power user', $9 = '123'
2013-10-29 06:01:43 EDT LOG: duration: 4.000 ms
Notice, that the whole query appears three times - for parse, for bind and for execute. And the complete set of parameters appears twice - for bind and for execute.
Note, that this extra verbosity is only present when running parameterized queries.
Here is my config:
C:\Program Files\PostgreSQL\9.2\data>findstr log_ postgresql.conf
# "postgres -c log_connections=on". Some parameters can be changed at run time
log_destination = 'stderr' # Valid values are combinations of
log_directory = 'pg_log' # directory where log files are written,
log_filename = 'postgresql-%Y-%m-%d_%H%M%S.log' # log file name pattern,
log_file_mode = 0600 # creation mode for log files,
#log_truncate_on_rotation = off # If on, an existing log file with the
#log_rotation_age = 1d # Automatic rotation of logfiles will
#log_rotation_size = 10MB # Automatic rotation of logfiles will
#syslog_facility = 'LOCAL0'
#syslog_ident = 'postgres'
log_min_messages = notice # values in order of decreasing detail:
log_min_error_statement = error # values in order of decreasing detail:
log_min_duration_statement = 0 # -1 is disabled, 0 logs all statements
#log_checkpoints = off
#log_connections = off
#log_disconnections = off
#log_duration = off
#log_error_verbosity = default # terse, default, or verbose messages
#log_hostname = off
log_line_prefix = '%t ' # special values:
#log_lock_waits = off # log lock waits >= deadlock_timeout
log_statement = 'all' # none, ddl, mod, all
#log_temp_files = -1 # log temporary files equal or larger
log_timezone = 'US/Eastern'
#log_parser_stats = off
#log_planner_stats = off
#log_executor_stats = off
#log_statement_stats = off
#log_autovacuum_min_duration = -1 # -1 disables, 0 logs all actions and
C:\Program Files\PostgreSQL\9.2\data>
So, my question is how can I reduce the verbosity of the log for parameterized queries without affecting the other queries? Ideally, I would like to have the query SQL and its parameters logged just once.
I don't think that's possible out of the box. You could write a logging hook and filter log entries.
The same issue I was facing before Above config which you have mentioned just change log_min_duration_statement=-1 ( disable) the query will be once only logged
BUT you have enabled the duration it will just log duration 3 times but not the query
Like
2013-10-29 06:01:43 EDT LOG: duration: 0.000 ms
2013-10-29 06:01:43 EDT LOG: duration: 0.000 ms
2013-10-29 06:01:43 EDT LOG: execute <unnamed>: INSERT INTO users (login,role,password) VALUES
($1,$2,$3)
,($4,$5,$6)
,($7,$8,$9)
2013-10-29 06:01:43 EDT DETAIL: parameters: $1 = 'guest', $2 = 'user', $3 = '123', $4 = 'admin', $5 = 'admin', $6 = '123', $7 = 'mark', $8 = 'power user', $9 = '123'
2013-10-29 06:01:43 EDT LOG: duration: 4.000 ms
I'm running a log (log_min_duration_statement = 200) to analyse some slow queries in PostgreSQL 9.0 but the statements for worst queries aren't being logged. Is there any way I can find out what the queries actually are?
(some values replaced with *** for brevity and privacy.)
2012-06-29 02:10:39 UTC LOG: duration: 266.658 ms statement: SELECT *** FROM "oauth_accesstoken" WHERE "oauth_accesstoken"."token" = E'***'
2012-06-29 02:10:40 UTC LOG: duration: 1797.400 ms statement:
2012-06-29 02:10:49 UTC LOG: duration: 1670.132 ms statement:
2012-06-29 02:10:50 UTC LOG: duration: 354.336 ms statement: SELECT *** FROM ***
...
There are some log file destination options in postgresql.conf, as shown below. I suggest to use csvlog.
log_destination = 'csvlog'
logging_collector = on
log_directory = '/var/applog/pg_log/1922/'
log_rotation_age = 1d
log_rotation_size = 10MB
log_statement = 'ddl' # none, ddl, mod, all
log_min_duration_statement = 200
After making any changes, you need to reload the postgresql.conf file.
It turns out because I was keeping an eye on the logs with tail -f path | grep 'duration .+ ms' any statement starting with a newline was not visible. I was mainly doing this to highlight the duration string.