I am doing POC on document data store so for that I have selected MongoDB and CouchBase for evaluation.
Environment Detail is as below
Machine : Centos 6.7, 2 core cpu, CPU MHz: 2494.078, RAM : 7 GB (6 GB Free)
MongoDB db version v3.2.0 with default configuration
CouchBase Version: 4.1.0-5005 Enterprise Edition (Cluster RAM: 3GB, Bucket RAM: 1GB)
Document Size : 326 B
Following is the result of POC
+--------------+---------------------------------------------+--------------+--------------+--------------+--------------+--------------+-------------+-------------+
| Operation | insert (in 10 batch each bacth is of 100K ) | select query | select query | select query | select query | select query | range query | range query |
+--------------+---------------------------------------------+--------------+--------------+--------------+--------------+--------------+-------------+-------------+
| Record Count | 1000K | 0 | 100 | 33k | 140k | 334k | 114k | 460k |
| Mongo | 99 sec | 568ms | 792ms | 1500ms | 3800ms | 7800ms | - | 15387ms |
| CouchBase | 370 sec | 8ms | 250ms | 6700ms | 28000ms | 69000ms | 28644ms | - |
+--------------+---------------------------------------------+--------------+--------------+--------------+--------------+--------------+-------------+-------------+
Client: I have used JAVA sdk and spring data.
There is big diffrence in performance of couchbase and mongodb on single node. Is there there any configuration parameter to increase performnace of couchbase?
It appears the current version of Spring Data MongoDB uses WriteConcern.UNACKNOWLEGED - it's fire and forget. You should enable WriteResultChecking.EXCEPTION or use WriteConcern.ACKNOWLEDGED.
http://docs.spring.io/spring-data/mongodb/docs/current/reference/html/#mongo.mongo-3.write-concern
What durability options are you using? Are you running out of bucket memory? 2 cpus is on the the low side for couchbase minimum requirements, if compaction is running at the same time as your test I would expect that to make a difference. This can be disabled in the settings.
Couchbase on a single node is not something I would ever run in production, minium 3 nodes, so if you have the time increasing your node count might give you some more meaningful figures
If you have 6GB of memory available, you might want to increase the amount of memory allocated to your bucket. In MongoDB 3.2, WiredTiger will use 60% of memory minus 1GB. For 7GB, that's 2.6 to 3.2GB (I'm not sure if it is 60% of available or 60% of total). Perhaps configure your bucket to match that.
Related
I am trying to create PostgreSQL - Etcd - Patroni(PEP) cluster. There are lots of examples on the internet and I have created one and it runs perfect. Yet, this architecture should comply with my company' s backup solution which is NetApp. We are putting the database into backup mode with "SELECT pg_start_backup('test_backup', true);" and then copy all the data files to backup directory.
PEP cluster has a small problem with this solution. Taking backup is running fine, but restoration point is not that much good. In order to restore the leader of the PEP cluster I need stop the database and then move the backup files to the data directory and finally start the restoration. At this point Patroni says the restoration node is a new cluster. Here is the error:
raise PatroniFatalException('Failed to bootstrap cluster')
patroni.exceptions.PatroniFatalException: 'Failed to bootstrap cluster'
2022-04-11 12:49:29,930 INFO: No PostgreSQL configuration items changed, nothing to reload.
2022-04-11 12:49:29,942 INFO: Lock owner: None; I am pgsql_node1
2022-04-11 12:49:29,962 INFO: trying to bootstrap a new cluster
The files belonging to this database system will be owned by user "postgres".
Also, when I check the patroni cluster status I saw this:
root#4cddca032454:/data/backup# patronictl -c /etc/patroni/config.yml list
+ Cluster: pgsql (7085327534197401486) --------+----+-----------+
| Member | Host | Role | State | TL | Lag in MB |
+-------------+------------+---------+---------+----+-----------+
| pgsql_node1 | 172.17.0.6 | Replica | stopped | | unknown |
| pgsql_node2 | 172.17.0.7 | Replica | running | 11 | 0 |
| pgsql_node3 | 172.17.0.8 | Replica | running | 11 | 0 |
+-------------+------------+---------+---------+----+-----------+
At this point I have a PEP cluster without a leader. So, how can I solve this issue?
(Note: The restoration node attempted to join right cluster because, before starting the restoration I check cluster status and got this result:
root#4cddca032454:/data/backup# patronictl -c /etc/patroni/config.yml list
+ Cluster: pgsql (7085327534197401486) --------+----+-----------+
| Member | Host | Role | State | TL | Lag in MB |
+-------------+------------+---------+---------+----+-----------+
| pgsql_node2 | 172.17.0.7 | Replica | running | 11 | 0 |
| pgsql_node3 | 172.17.0.8 | Replica | running | 11 | 0 |
+-------------+------------+---------+---------+----+-----------+
pgsql_node1 is not there.
)
As explained here, "https://patroni.readthedocs.io/en/latest/existing_data.html#existing-data" I can create a new cluster after restoration but my priority saving the cluster. Or do I think wrong, all this steps are same with the converting a standalone PostgreSQL database to PEP cluster?
Please let me know if you need any data or something is not clear.
Here is my leader node patroni config file:
scope: "cluster"
namespace: "/cluster/"
name: 8d454a228d251
restapi:
listen: 172.17.0.2:8008
connect_address: 172.17.0.2:8008
etcd:
host: 172.17.0.2:2379
dcs:
ttl: 30
loop_wait: 10
retry_timeout: 10
maximum_lag_on_failover: 1048576
check_timeline: true
postgresql:
use_pg_rewind: true
remove_data_directory_on_rewind_failure: true
remove_data_directory_on_diverged_timelines: true
use_slots: true
postgresql:
listen: 0.0.0.0:5433
connect_address: 172.17.0.2:5432
use_unix_socket: true
data_dir: /data/postgresql/
bin_dir: /usr/lib/postgresql/14/bin
config_dir: /etc/postgresql/14/main
authentication:
replication:
username: "patroni_replication"
password: "123123"
superuser:
username: "patroni_superuser"
password: "123123"
parameters:
unix_socket_directories: '/var/run/postgresql/'
logging_collector: 'on'
log_directory: '/var/log/postgresql'
log_filename: 'postgresql-14-8d454a228d25.log'
restore_command: 'cp /data/backup/%f %p'
recovery_target_timeline: 'latest'
promote_trigger_file: '/tmp/promote'
Thanks!
If I understand well you want to restore the primary server (leader) restoring the data directory with a new set of backup files.
After doing the restore in data directory of leader you need to recreate the patroni cluster (remove the keys in DCS) with patronictl remove option.
Example:
stop pgsql_node2
stop pgsql_node3
stop pgsql_node1
on pgsql_node1:
patronictl -c /etc/patroni/config.yml remove <clustername>
start pgsql_node1
start pgsql_node2
start pgsql_node3
I'm testing failover using RDS Aurora PostgreSQL.
First, create RDS Aurora PostgreSQL and access the writer cluster to create users table.
$ CREATE TABLE users (
id SERIAL PRIMARY KEY NOT NULL,
name varchar(10) NOT NULL,
createAt TIMESTAMP DEFAULT Now() );
And I added one row and checked the table.
$ INSERT INTO users(name) VALUES ('test');
$ SELECT * FROM users;
+----+--------+----------------------------+
| id | name | createdAt |
+----+--------+----------------------------+
| 1 | test | 2022-02-02 23:09:57.047981 |
+----+--------+----------------------------+
After failover of RDS Aurora Cluster, I added another row and checked the table.
$ INSERT INTO users(name) VALUES ('temp');
$ SELECT * FROM users;
+-----+--------+----------------------------+
| id | name | createdAt |
+-----+--------+----------------------------+
| 1 | test | 2022-02-01 11:09:57.047981 |
| 32 | temp | 2022-02-01 11:25:57.047981 |
+-----+--------+----------------------------+
After failover, the id value that should be 2 became 32.
Why is this happening?
Is there any way to solve this problem?
That is to be expected. Index modifications are not WAL logged whenever nextval is called, because that could become a performance bottleneck. Rather, a WAL record is written every 32 calls. That means that the sequence can skip some values after a crash or failover to the standby.
You may want to read my ruminations about gaps in sequences.
Sorry, newbie in SQL. I have such table in Google Cloud SQL (MySql).
How can I get time difference between surrounding rows like 164 and 165?
I want to get time period (downtime) when no one sencor worked with the condition that downtime more 20 minutes.
autoID | Datetime | Number_of_sensor
163 | 2020-04-06 13:46:42 | C3
164 | 2020-04-06 13:46:45 | C4
165 | 2020-04-06 15:10:48 | C3
166 | 2020-04-06 15:46:48 | C4
I tried something but cann't get result.
You would have to use something called Window Functions, which are only available on MySQL after version 8.0
Google Cloud SQL for MySQL is only up version 5.7 for now.
However if you use Postgres that will be able to run as a Window Function SQL Query.
You can use the native TIMESTAMPDIFF() function, using the values from 2 tables offset by 1 as arguments, and join those 2 tables. This will work with MySQL 5.7 which is used on Google Cloud SQL.
SELECT a.autoID, a.Datetime , TIMESTAMPDIFF(MINUTE,a.Datetime,COALESCE(b.Datetime,0)) as downtime
FROM test a
LEFT JOIN test b on a.autoID=b.autoID-1
HAVING downtime > 20;
is it possible to check (as not rooted user/SQL question) to check, if my connection from client to server uses SSL (my destination server cas uses both - secured and not secured connection)?
as of 9.5:
https://www.postgresql.org/docs/current/static/monitoring-stats.html#PG-STAT-SSL-VIEW
The pg_stat_ssl view will contain one row per backend or WAL sender
process, showing statistics about SSL usage on this connection. It can
be joined to pg_stat_activity or pg_stat_replication on the pid column
to get more details about the connection.
t=# set role notsu ;
SET
Time: 9.289 ms
t=> select * from pg_stat_ssl where pid = pg_backend_pid();
pid | ssl | version | cipher | bits | compression | clientdn
-------+-----+---------+--------+------+-------------+----------
43767 | f | | | | |
(1 row)
Time: 10.846 ms
t=> \du+ notsu
List of roles
Role name | Attributes | Member of | Description
-----------+------------+-----------+-------------
notsu | | {} |
the above shows my connection is not SSL
Our Postgres BDR database system stopped replicating data between the nodes.
When I did a check using the pg_xlog_location_diff I noticed that there is a growing buffer in the replication slot.
SELECT slot_name, database, active, pg_xlog_location_diff(pg_current_xlog_insert_location(), restart_lsn) AS retained_bytes
FROM pg_replication_slots
WHERE plugin = 'bdr';
slot_name | database | active | retained_bytes
-----------------------------------------+--------------+--------+----------------
bdr_26702_6275336279642079463_1_20305__ | ourdatabase | f | 32253352
I also noticed that the slot is marked as active=false.
SELECT * FROM pg_replication_slots;
-[ RECORD 1 ]+----------------------------------------
slot_name | bdr_26702_6275336279642079463_1_20305__
plugin | bdr
slot_type | logical
datoid | 26702
database | ourdatabase
active | f
xmin |
catalog_xmin | 8041
restart_lsn | 0/5F0C6C8
I increased the Postgres logging level, but then only messages I see in the log are:
LOCATION: LogicalIncreaseRestartDecodingForSlot, logical.c:886
DEBUG: 00000: updated xmin: 1 restart: 0
LOCATION: LogicalConfirmReceivedLocation, logical.c:958
DEBUG: 00000: failed to increase restart lsn: proposed 0/7DCE6F8, after 0/7DCE6F8, current candidate 0/7DCE6F8, current after 0/7DCE6F8, flushed up to 0/7DCE6F8
Please let me know if you have an idea how I can re-activate the replication slot and allow the replication to resume.
Except if you have really huuuuuge amount of data, I cannot see any reason for not recreating the replication from scratch. Stop the slave, delete the slot on master, delete data directory on slave, create new slot (with the same name to avoid further changes on slave), do pg_basebackup.
You can find a good tutorial here.