Not able to increase TPS in Postgres BDR set up using pgbench utility - postgresql-9.4

I am getting constant TPS as 500 while using pgbench with different TPS as input .How to increase the TPS in Postgres BDR set up, please suggest .
bdr_version used : 1.0.3-2017-11-21-15283ba
Sample Output and pgbench command input
Scenario 1:
Shared_Buffer = 1024 MB and throttling TPS as 2000
command used:
pgbench -h <hostname> -p 5432 -U postgres -d <dbname> -c 50 -j 50 -r -R 2000
O/P :
transaction type: TPC-B (sort of)
scaling factor: 1
query mode: simple
number of clients: 50
number of threads: 50
duration: 2400 s
number of transactions actually processed: 1200964
latency average: 894071.370 ms
latency stddev: -nan ms
rate limit schedule lag: avg 893971.489 (max 1819008.190) ms
tps = 500.382936 (including connections establishing)
tps = 500.385107 (excluding connections establishing)
statement latencies in milliseconds:
0.109225 \set nbranches 1 * :scale
0.100045 \set ntellers 10 * :scale
0.085917 \set naccounts 100000 * :scale
0.064758 \setrandom aid 1 :naccounts
0.057230 \setrandom bid 1 :nbranches
0.052089 \setrandom tid 1 :ntellers
0.049827 \setrandom delta -5000 5000
0.471368 BEGIN;
0.678317 UPDATE pgbench_accounts SET abalance = abalance + :delta WHERE aid = :aid;
0.599331 SELECT abalance FROM pgbench_accounts WHERE aid = :aid;
80.328257 UPDATE pgbench_tellers SET tbalance = tbalance + :delta WHERE tid = :tid;
15.537372 UPDATE pgbench_branches SET bbalance = bbalance + :delta WHERE bid = :bid;
0.698180 INSERT INTO pgbench_history (tid, bid, aid, delta, mtime) VALUES (:tid, :bid, :aid, :delta, CURRENT_TIMESTAMP);
0.995450 END;
Thu Aug 16 09:52:30 UTC 2018
Scenario 2:
Shared_Buffer = 1024 MB and throttling TPS as 6000
Command Used:
pgbench -h <hostname> -p 5432 -U postgres -d <dbname> -c 50 -j 50 -r -R 6000
O/P:
transaction type: TPC-B (sort of)
scaling factor: 1
query mode: simple
number of clients: 50
number of threads: 50
duration: 2400 s
number of transactions actually processed: 1184746
latency average: 1096729.554 ms
latency stddev: -nan ms
rate limit schedule lag: avg 1096628.305 (max 2208483.098) ms
tps = 493.625936 (including connections establishing)
tps = 493.629106 (excluding connections establishing)
statement latencies in milliseconds:
0.108491 \set nbranches 1 * :scale
0.098740 \set ntellers 10 * :scale
0.084497 \set naccounts 100000 * :scale
0.064168 \setrandom aid 1 :naccounts
0.056658 \setrandom bid 1 :nbranches
0.051678 \setrandom tid 1 :ntellers
0.049427 \setrandom delta -5000 5000
0.480755 BEGIN;
0.696514 UPDATE pgbench_accounts SET abalance = abalance + :delta WHERE aid = :aid;
0.607114 SELECT abalance FROM pgbench_accounts WHERE aid = :aid;
81.404448 UPDATE pgbench_tellers SET tbalance = tbalance + :delta WHERE tid = :tid;
15.775295 UPDATE pgbench_branches SET bbalance = bbalance + :delta WHERE bid = :bid;
0.708403 INSERT INTO pgbench_history (tid, bid, aid, delta, mtime) VALUES (:tid, :bid, :aid, :delta, CURRENT_TIMESTAMP);
1.009272 END;
Thu Aug 16 11:33:17 UTC 2018

Related

High CPU usage caused during select query

On running a select query with a QPS of 300+, query latency increases up to 20k ms with a CPU usage of ~100%. I couldn't identify what the issue could be over here!?
System details:
vCPUs: 16
RAM: 64 GiB
Create hypertable:
CREATE TABLE test_table (time TIMESTAMPTZ NOT NULL, val1 TEXT NOT NULL, val2 DOUBLE PRECISION NOT NULL, val3 DOUBLE PRECISION NOT NULL);
SELECT create_hypertable('test_table', 'time');
SELECT add_retention_policy('test_table', INTERVAL '2 day');
Query:
SELECT * from test_table where val1='abc'
Timescale version:
2.5.1
No. of rows in test-table:
6.9M
Latencies:
p50=8493.722203,
p75=8566.074792,
p95=21459.204448,
p98=21459.204448,
p99=21459.204448,
p999=21459.204448
CPU usage of postgres SELECT query as per the htop command.

How to get P99 latency result using pgbench?

I am trying to use pgbench to perform a test on PolarDB for postgreSQL.
This is the command I used to perform the test.
pgbench -M prepared -r -c 16 -j 4 -T 30 -p 10001 -d pgbench -l
And this is the result
... ...
client 2 sending P0_10
client 2 receiving
client 2 receiving
client 14 receiving
transaction type: <builtin: TPC-B (sort of)>
scaling factor: 32
query mode: prepared
number of clients: 16
number of threads: 4
duration: 30 s
number of transactions actually processed: 49126
latency average = 9.772 ms
tps = 1637.313156 (including connections establishing)
tps = 1637.438330 (excluding connections establishing)
statement latencies in milliseconds:
1.128 \set aid random(1, 100000 * :scale)
0.068 \set bid random(1, 1 * :scale)
0.040 \set tid random(1, 10 * :scale)
0.041 \set delta random(-5000, 5000)
0.104 BEGIN;
3.815 UPDATE pgbench_accounts SET abalance = abalance + :delta WHERE aid = :aid;
0.590 SELECT abalance FROM pgbench_accounts WHERE aid = :aid;
1.188 UPDATE pgbench_tellers SET tbalance = tbalance + :delta WHERE tid = :tid;
1.440 UPDATE pgbench_branches SET bbalance = bbalance + :delta WHERE bid = :bid;
0.327 INSERT INTO pgbench_history (tid, bid, aid, delta, mtime) VALUES (:tid, :bid, :aid, :delta, CURRENT_TIMESTAMP);
0.481 END;
I wonder if there is a way to calculate P99 from the result, or there is some extra parameter I need provide to pgbench?
The -l caused it to write log files. You need to look in those log files for the latencies. For me that looks something like this:
cat pgbench_log.107915*|wc
36635 219810 1033548
sort pgbench_log.107915* -k3rn|head -n 366|tail -n 1
13 990 65184 0 195589 166574
so about 65.184 ms was the 99% latency. I would question if that actually means anything, though. After all, the very last transaction had to wait nearly 30 seconds before it even got is turn on the server, so why wasn't its 'latency' 30 seconds? What about the transactions that never got their turn at all?

insert blocks with ExclusiveLock on index in Postgres

Database - Google Cloud SQL, Postgresql 11.
We have a partitioned table, with one of the partitions, which is the most active, having 4 bln rows (3Tb).
The table has several columns being a primary key and one of the columns is json with a gin index.
CREATE TABLE messages (
key_col1,
key_col2,
message_type text NOT NULL,
message jsonb NOT NULL,
CONSTRAINT messages_pkey PRIMARY KEY (key_col1, key_col2, message_type)
)
PARTITION BY LIST (message_type);
CREATE INDEX gin_index ON messages USING gin (message);
CREATE TABLE messages_type_1 PARTITION OF messages FOR VALUES IN ('MessageType1')
The table is frequently inserted to, with ON CONFLICT DO NOTHING.
Lately, some of the inserts started blocking. It goes on for around 25 minutes. then it resolves, all the inserts that have heaped up get done, and some other insert gets blocked.
This query
SELECT a.datname,
l.relation::regclass,
l.transactionid,
l.mode,
l.GRANTED,
a.usename,
a.query,
a.query_start,
age(now(), a.query_start) AS "age",
a.pid
FROM pg_stat_activity a
JOIN pg_locks l ON l.pid = a.pid
WHERE MODE <> 'RowExclusiveLock'
ORDER BY a.query_start;
shows these locks
| relation | transactions | mode | granted | query | query_start
+---------------------------------------------------------------------------------------------
[NULL] [NULL] ExclusiveLock true INSERT INTO ... 2020-07-28 19:20:37
[NULL] [NULL] ExclusiveLock true INSERT INTO ... 2020-07-28 19:20:37
[NULL] 1480644902 ExclusiveLock true INSERT INTO ... 2020-07-28 19:20:37
gin_index [NULL] ExclusiveLock true INSERT INTO ... 2020-07-28 19:20:37
These are autovacuum settings
autovacuum_analyze_scale_factor 0.1
autovacuum_analyze_threshold 50
autovacuum_freeze_max_age 200000000
autovacuum_max_workers 3
autovacuum_multixact_freeze_max_age 400000000
autovacuum_naptime 1min
autovacuum_vacuum_cost_delay 20ms
autovacuum_vacuum_cost_limit -1
autovacuum_vacuum_scale_factor 0.2
autovacuum_vacuum_threshold 50
autovacuum_work_mem -1
vacuum_cleanup_index_scale_factor 0.1
vacuum_cost_delay 0
vacuum_cost_limit 1000
vacuum_cost_page_dirty 20
vacuum_cost_page_hit 1
vacuum_cost_page_miss 10
vacuum_defer_cleanup_age 0
vacuum_freeze_min_age 50000000
vacuum_freeze_table_age 150000000
vacuum_multixact_freeze_min_age 5000000
vacuum_multixact_freeze_table_age 150000000
Are these exclusive locks normal? How could I debug this further?

Geoserver WFS + PostgreSQL with large table impossibly slow

I have a PostgreSQL table of point locations that contains 9.5 million rows. I am attempting to run this query:
/geoserver/workspace/ows?service=WFS&version=1.0.0&request=GetFeature&typeName=workspace:layer&maxFeatures=50&outputFormat=application/json but it takes multiple minutes to respond.
When I look at the geoserver logs I see this:
Request: getFeature
service = WFS
version = 1.0.0
baseUrl = http://my_geoserver_url/geoserver/
query[0]:
typeName[0] = {workspace}layer
maxFeatures = 50
outputFormat = application/json
resultType = results
19 Nov 15:26:04 INFO [wfs.json] - about to encode JSON
At that point it stalls for many minutes.
When I look at the currently active queries of my PostgreSQL server I see this:
SELECT count(*) FROM "public"."layer"
This query by itself takes 335 seconds to return a response. First of all, wtf? Even if it has to count row by row, 9.5 million isn't THAT many rows. Is there any way to speed up this operation?
Second, why is it trying to do SELECT count(*) FROM "public"."layer" and is there any way to prevent it? I specified maxFeatures = 50, so why count them?
Version: PostgreSQL 11.2 (Debian 11.2-1.pgdg90+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 6.3.0-18+deb9u1) 6.3.0 20170516, 64-bi
Machine: n1-standard-2 (2 vCPUs, 7.5 GB memory) (from Google Cloud)
Variables:
allow_system_table_mods off
application_name Navicat
archive_command (disabled)
archive_mode off
archive_timeout 0
array_nulls on
authentication_timeout 60
autovacuum on
autovacuum_analyze_scale_factor 0.1
autovacuum_analyze_threshold 50
autovacuum_freeze_max_age 200000000
autovacuum_max_workers 3
autovacuum_multixact_freeze_max_age 400000000
autovacuum_naptime 60
autovacuum_vacuum_cost_delay 20
autovacuum_vacuum_cost_limit -1
autovacuum_vacuum_scale_factor 0.2
autovacuum_vacuum_threshold 50
autovacuum_work_mem -1
backend_flush_after 0
backslash_quote safe_encoding
bgwriter_delay 200
bgwriter_flush_after 64
bgwriter_lru_maxpages 100
bgwriter_lru_multiplier 2
block_size 8192
bonjour off
bonjour_name
bytea_output hex
check_function_bodies on
checkpoint_completion_target 0.5
checkpoint_flush_after 32
checkpoint_timeout 300
checkpoint_warning 30
client_encoding UNICODE
client_min_messages notice
cluster_name
commit_delay 0
commit_siblings 5
config_file /var/lib/postgresql/data/postgresql.conf
constraint_exclusion partition
cpu_index_tuple_cost 0.005
cpu_operator_cost 0.0025
cpu_tuple_cost 0.01
cursor_tuple_fraction 0.1
data_checksums off
data_directory /var/lib/postgresql/data
data_directory_mode 0700
data_sync_retry off
DateStyle ISO, MDY
db_user_namespace off
deadlock_timeout 1000
debug_assertions off
debug_pretty_print on
debug_print_parse off
debug_print_plan off
debug_print_rewritten off
default_statistics_target 100
default_tablespace
default_text_search_config pg_catalog.english
default_transaction_deferrable off
default_transaction_isolation read committed
default_transaction_read_only off
default_with_oids off
dynamic_library_path $libdir
dynamic_shared_memory_type posix
effective_cache_size 524288
effective_io_concurrency 1
enable_bitmapscan on
enable_gathermerge on
enable_hashagg on
enable_hashjoin on
enable_indexonlyscan on
enable_indexscan on
enable_material on
enable_mergejoin on
enable_nestloop on
enable_parallel_append on
enable_parallel_hash on
enable_partition_pruning on
enable_partitionwise_aggregate off
enable_partitionwise_join off
enable_seqscan on
enable_sort on
enable_tidscan on
escape_string_warning on
event_source PostgreSQL
exit_on_error off
external_pid_file
extra_float_digits 0
force_parallel_mode off
from_collapse_limit 8
fsync on
full_page_writes on
geqo on
geqo_effort 5
geqo_generations 0
geqo_pool_size 0
geqo_seed 0
geqo_selection_bias 2
geqo_threshold 12
gin_fuzzy_search_limit 0
gin_pending_list_limit 4096
hba_file /var/lib/postgresql/data/pg_hba.conf
hot_standby on
hot_standby_feedback off
huge_pages try
ident_file /var/lib/postgresql/data/pg_ident.conf
idle_in_transaction_session_timeout 0
ignore_checksum_failure off
ignore_system_indexes off
integer_datetimes on
IntervalStyle postgres
jit off
jit_above_cost 100000
jit_debugging_support off
jit_dump_bitcode off
jit_expressions on
jit_inline_above_cost 500000
jit_optimize_above_cost 500000
jit_profiling_support off
jit_provider llvmjit
jit_tuple_deforming on
join_collapse_limit 8
krb_caseins_users off
krb_server_keyfile FILE:/etc/postgresql-common/krb5.keytab
lc_collate en_US.utf8
lc_ctype en_US.utf8
lc_messages en_US.utf8
lc_monetary en_US.utf8
lc_numeric en_US.utf8
lc_time en_US.utf8
listen_addresses *
lo_compat_privileges off
local_preload_libraries
lock_timeout 0
log_autovacuum_min_duration -1
log_checkpoints off
log_connections off
log_destination stderr
log_directory log
log_disconnections off
log_duration off
log_error_verbosity default
log_executor_stats off
log_file_mode 0600
log_filename postgresql-%Y-%m-%d_%H%M%S.log
log_hostname off
log_line_prefix %m [%p]
log_lock_waits off
log_min_duration_statement -1
log_min_error_statement error
log_min_messages warning
log_parser_stats off
log_planner_stats off
log_replication_commands off
log_rotation_age 1440
log_rotation_size 10240
log_statement none
log_statement_stats off
log_temp_files -1
log_timezone UTC
log_truncate_on_rotation off
logging_collector off
maintenance_work_mem 65536
max_connections 100
max_files_per_process 1000
max_function_args 100
max_identifier_length 63
max_index_keys 32
max_locks_per_transaction 64
max_logical_replication_workers 4
max_parallel_maintenance_workers 2
max_parallel_workers 8
max_parallel_workers_per_gather 2
max_pred_locks_per_page 2
max_pred_locks_per_relation -2
max_pred_locks_per_transaction 64
max_prepared_transactions 0
max_replication_slots 10
max_stack_depth 2048
max_standby_archive_delay 30000
max_standby_streaming_delay 30000
max_sync_workers_per_subscription 2
max_wal_senders 10
max_wal_size 1024
max_worker_processes 8
min_parallel_index_scan_size 64
min_parallel_table_scan_size 1024
min_wal_size 80
old_snapshot_threshold -1
operator_precedence_warning off
parallel_leader_participation on
parallel_setup_cost 1000
parallel_tuple_cost 0.1
password_encryption md5
port 5432
post_auth_delay 0
postgis.backend geos
pre_auth_delay 0
quote_all_identifiers off
random_page_cost 4
restart_after_crash on
row_security on
search_path public, "$user", public
segment_size 131072
seq_page_cost 1
server_encoding UTF8
server_version 11.2 (Debian 11.2-1.pgdg90+1)
server_version_num 110002
session_preload_libraries
session_replication_role origin
shared_buffers 16384
shared_preload_libraries
ssl off
ssl_ca_file
ssl_cert_file server.crt
ssl_ciphers HIGH:MEDIUM:+3DES:!aNULL
ssl_crl_file
ssl_dh_params_file
ssl_ecdh_curve prime256v1
ssl_key_file server.key
ssl_passphrase_command
ssl_passphrase_command_supports_reload off
ssl_prefer_server_ciphers on
standard_conforming_strings on
statement_timeout 0
stats_temp_directory pg_stat_tmp
superuser_reserved_connections 3
synchronize_seqscans on
synchronous_commit on
synchronous_standby_names
syslog_facility local0
syslog_ident postgres
syslog_sequence_numbers on
syslog_split_messages on
tcp_keepalives_count 9
tcp_keepalives_idle 7200
tcp_keepalives_interval 75
temp_buffers 1024
temp_file_limit -1
temp_tablespaces
TimeZone UTC
timezone_abbreviations Default
trace_notify off
trace_recovery_messages log
trace_sort off
track_activities on
track_activity_query_size 1024
track_commit_timestamp off
track_counts on
track_functions none
track_io_timing off
transaction_deferrable off
transaction_isolation read committed
transaction_read_only off
transform_null_equals off
unix_socket_directories /var/run/postgresql
unix_socket_group
unix_socket_permissions 0777
update_process_title on
vacuum_cleanup_index_scale_factor 0.1
vacuum_cost_delay 0
vacuum_cost_limit 200
vacuum_cost_page_dirty 20
vacuum_cost_page_hit 1
vacuum_cost_page_miss 10
vacuum_defer_cleanup_age 0
vacuum_freeze_min_age 50000000
vacuum_freeze_table_age 150000000
vacuum_multixact_freeze_min_age 5000000
vacuum_multixact_freeze_table_age 150000000
wal_block_size 8192
wal_buffers 512
wal_compression off
wal_consistency_checking
wal_keep_segments 0
wal_level replica
wal_log_hints off
wal_receiver_status_interval 10
wal_receiver_timeout 60000
wal_retrieve_retry_interval 5000
wal_segment_size 16777216
wal_sender_timeout 60000
wal_sync_method fdatasync
wal_writer_delay 200
wal_writer_flush_after 128
work_mem 4096
xmlbinary base64
xmloption content
zero_damaged_pages off
explain (analyze, buffers, format text):
Buffers: shared hit=2940 read=1601607
-> Gather (cost=1655905.64..1655905.85 rows=2 width=8) (actual time=191335.534..191349.259 rows=3 loops=1)
Workers Planned: 2
Workers Launched: 2
Buffers: shared hit=2940 read=1601607
-> Partial Aggregate (cost=1654905.64..1654905.65 rows=1 width=8) (actual time=191277.227..191277.227 rows=1 loops=3)
Buffers: shared hit=2940 read=1601607
-> Parallel Seq Scan on forecast (cost=0.00..1644833.91 rows=4028691 width=0) (actual time=8.037..190681.354 rows=3222220 loops=3)
Buffers: shared hit=2940 read=1601607
Planning Time: 0.351 ms
Execution Time: 191349.424 ms
Indexes:
CREATE INDEX forecast_init_utc ON public.forecast USING btree (forecast_init_utc)
CREATE UNIQUE INDEX forecast_pkey ON public.forecast USING btree (ogc_fid)
CREATE INDEX forecast_wkb_geometry_geom_idx ON public.forecast USING gist (wkb_geometry)
CREATE INDEX forecast_init_local ON public.forecast USING btree (forecast_init_local)
CREATE INDEX country_code ON public.forecast USING btree (country_code)
CREATE INDEX store_num ON public.forecast USING btree (store_num)
Edit: It appears that VACUUM ANALYZE did the trick. It took 728 seconds to run, but now the query returns quickly. How often do I need to run that?
Here is the new explain (analyze, buffers, format text):
Finalize Aggregate (cost=204220.77..204220.78 rows=1 width=8) (actual time=7116.754..7116.755 rows=1 loops=1)
Buffers: shared hit=525705 read=25977
-> Gather (cost=204220.55..204220.77 rows=2 width=8) (actual time=7115.613..7123.979 rows=3 loops=1)
Workers Planned: 2
Workers Launched: 2
Buffers: shared hit=525705 read=25977
-> Partial Aggregate (cost=203220.55..203220.57 rows=1 width=8) (actual time=7051.177..7051.178 rows=1 loops=3)
Buffers: shared hit=525705 read=25977
-> Parallel Index Only Scan using store_num on forecast (cost=0.43..193125.85 rows=4037882 width=0) (actual time=18.684..6564.663 rows=3229577 loops=3)
Heap Fetches: 0
Buffers: shared hit=525705 read=25977
Planning Time: 0.896 ms
Execution Time: 7124.113 ms
#a_horse_with_no_name's suggestion to run vaccum analyze on the table made a huge difference but there is a setting to turn of those select count(*) queries entirely. , It is located in Layers -> <Layer Name> -> Publishing -> Skip the counting of the numberMatched attribute in the admin panel.

Average after fixed interval and Group By in SQL

Is it possible to Average after every fixed interval and group by one column in MSSQL ?
Suppose I have a table A as under:
NAME Interval Data1 Data2
1 0.01 1 4
1 0.05 4 2
1 0.09 7 6
1 0.11 1 2
1 0.15 7 6
1 0.18 3 1
1 0.19 2 5
2 0.209 9 0
I want the Output to group by Name and run average every 10 counts.
So for expamle
Name - 1
Interval Start - 0
Interval End - 10
Data 1 Avg - 4 [(1 + 4 + 7) / 3]
Data 3 Avg - 4 [(4 + 2 + 6) / 3]
AND
Name - 1
Interval Start - 10
Interval End - 20
Data 1 Avg - 3.25 [(1 + 7 + 3 + 2) / 4]
Data 3 Avg - 3.50 [(2 + 6 + 1 + 5) / 4]
So I want the Ouput as below. The interval per "Name" column is different.
Name Interval-Start Interval-End DataAvg1 DataAvg2
1 0 10 4 4
1 10 20 3.25 3.50
2 0 10 0 0
2 10 20 0 0
2 20 30 9 0
I used the below query, but cant figure out logic per interval.
SELECT Name, Interval, AVG(Data1) AS Data1Avg, AVG(Data2) AS Data2Avg
FROM TableA
GROUP BY Name;
Can someone please help me with it.
using cursor and temp table
--drop table dbo.#result
--drop table dbo.#steps
CREATE TABLE dbo.#result
(
[Name] varchar(50),
[Interval-Start] float,
[Interval-End] float,
[DataAvg1] float,
[DataAvg2] float
)
CREATE TABLE dbo.#steps
(
[IntervalStart] float,
[IntervalEnd] float
)
declare #min int, #max int, #step float
DECLARE #Name varchar(50), #IntervalStart float, #IntervalEnd float;
set #min = 0
set #max = 1
set #step = 0.1
insert into #steps
select #min + Number * #step IntervalStart, #min + Number * #step + #step IntervalEnd
from master..spt_values
where type = 'P' and number between 0 and (#max - #min) / #step
DECLARE _cursor CURSOR FOR
SELECT [Name], [IntervalStart], [IntervalEnd] FROM
(select [Name] from [TableA] Group by [Name]) t
INNER JOIN #steps on 1=1
OPEN _cursor;
FETCH NEXT FROM _cursor
INTO #Name, #IntervalStart, #IntervalEnd;
WHILE ##FETCH_STATUS = 0
BEGIN
insert into dbo.#result
select #Name, #IntervalStart, #IntervalEnd, AVG(CAST(Data1 as FLOAT)), AVG(CAST(Data2 as FLOAT))
FROM [TableA]
where [NAME] = #Name and Interval between #IntervalStart and #IntervalEnd
FETCH NEXT FROM _cursor
INTO #Name, #IntervalStart, #IntervalEnd;
END
CLOSE _cursor;
DEALLOCATE _cursor;
select * from dbo.#result