I'm trying to backup a MySQL database running inside a k8s cluster (k3s). This cluster is running locally on a Raspberry Pi. I've built a custom image based on Alpine Linux that contains a script to create the backup using mysqldump:
kubectl exec <pod_name> -n <namespace_name> -- /usr/bin/mysqldump -u <db_user> --password=<db_password> --verbose <db_name> > <file_name>
When I run the mysqldump command from inside the database pod, it completes successfully after 10-15 seconds. But when this command gets executed from inside the Alpine pod it somehow takes a lot longer (2m40s). At that point it stops/aborts the kubectl exec command (due to a timeout?) and the script uploads a corrupt sql file because the mysqldump command wasn't finished with the backup.
Expected verbose output:
-- Connecting to localhost...
-- Retrieving table structure for table _prisma_migrations...
-- Sending SELECT query...
-- Retrieving rows...
-- Retrieving table structure for table database...
-- Sending SELECT query...
-- Retrieving rows...
-- Retrieving table structure for table recycleBin...
-- Sending SELECT query...
-- Retrieving rows...
-- Retrieving table structure for table user...
-- Sending SELECT query...
-- Retrieving rows...
-- Disconnecting from localhost...
Received verbose output:
-- Connecting to localhost...
-- Retrieving table structure for table _prisma_migrations...
-- Sending SELECT query...
-- Retrieving rows...
-- Retrieving table structure for table database...
-- Sending SELECT query...
-- Retrieving rows...
-- Retrieving table structure for table recycleBin...
-- Sending SELECT query...
-- Retrieving rows...
I have 2 questions:
Why does the mysqldump command take so much longer in the Alpine pod compared to the database pod?
Why isn't the kubectl exec command waiting until the mysqldump command has finished taking the backup? Why does it suddenly decide it's time to disconnect and move on?
It is possible that you're getting disconnected because kubectl thinks the connection is dead due to no data from the other side.
Instead of:
kubectl exec <pod_name> -n <namespace_name> -- /usr/bin/mysqldump -u <db_user> --password=<db_password> --verbose <db_name> > <file_name>
Try:
kubectl exec <pod_name> -n <namespace_name> -- /usr/bin/mysqldump -u <db_user> --password=<db_password> --verbose <db_name> | tee <file_name>
This will instead output to stdout as well as sending to the file. This will guarantee that kubectl will see data coming back and will not disconnect you if this is a no-data problem. The downside to this is that you are getting the entire SQL data being pumped back to you on stdout
The other alternative (and one I would personally recommend), is installing something like screen or tmux or another terminal multiplexer and doing the dump in that so you can disconnect from the pod and not worry about kubectl disconnecting you.
EDIT: Just to clarify, I meant installing the multiplexer inside the pod (e.g. inside the docker image you use to create the pod)
Related
I'm still a relative newbie to Postgresql, so pardon if this is simple ignorance.
I've setup a active/read-only pacemaker cluster of Postgres v9.4 per the cluster labs documentation.
I'm trying to verify that both databases are indeed in sync. I'm doing the dump on both hosts and checking the diff between the output. The command I'm using is:
pg_sql -U myuser mydb >dump-node-1.sql
Pacemaker shows the database status as 'sync' and querying Postgres directly also seems to indicate the sync is good... (Host .59 is my read-only standby node)
psql -c "select client_addr,sync_state from pg_stat_replication;"
+---------------+------------+
| client_addr | sync_state |
+---------------+------------+
| 192.16.111.59 | sync |
+---------------+------------+
(1 row)
However, when I do a dump on the read-only host I end up with all my tables having 'public.' added to the front of the names. So table foo on the master node dumps as 'foo' whereas on the read-only node it dumps as 'public.foo'. I don't understand why this is happening... I had done a 9.2 Postgresql cluster in a similar setup and didn't see this issue. I don't have tables in the public schema on the master node...
Hope someone can help me understand what is going on.
Much appreciated!
Per a_horse_with_no_name, the security updates in 9.4.18 changed the way the dump is written compared to 9.4.15. I didn't catch that one node was still running an older version. The command that identified the problem was his suggestion to run:
psql -c "select version();"
I have installed, configured my Postgres XC 1.0.2 on Ubuntu. I was able to setup the GTM, Coordinator, nodes 1 & 2 with no challenges. But when attempting to distribute the data amongst the two nodes, I'm getting the below errors which are actually contradicting each other.
postgres=# CREATE TABLE DIST_REP(T INT) DISTRIBUTE BY REPLICATION TO NODE datanode1,datanode2;
ERROR: relation "dist_rep" already exists
postgres=# INSERT INTO DIST_REP VALUES(GENERATE_SERIES(1,100));
ERROR: relation "dist_rep" does not exist
Answering my own question here, this seemed to be recurring most of the times when I try to create a table and insert data in Postgres-XC. I tried restarting the Coordinator, seemed to work sometimes, not promising though. Just a workaround and it was also impossible to drop the table here. Here's the command for stopping and restarting the Coordinator.
To stop:
$ /usr/local/pgsql/bin/pg_ctl stop -D /data_local/pgxc/data_coord1/ -Z coordinator -l /tmp/logfile_cord -mf
To start:
/usr/local/pgsql/bin/pg_ctl start -D /data_local/pgxc/data_coord1 -Z coordinator -l /tmp/logfile_cord
Note: /usr/local/pgsql/bin was my PostgreSQL location and /data_local/pgxc was my Postgres-XC location.
Just a quick question .
I have one heavy function in PostgreSQL 9.3 , how I can check if the function is still running after several hours and how to run a function in background in psql ( my connection is unstable from time to time)
Thanks
For long running functions, it can be useful to have them RAISE LOG or RAISE NOTICE from time to time, indicating progress. If they're looping over millions of records, you might emit a log message every few thousand records.
Some people also (ab)use a SEQUENCE, where they get the nextval of the sequence in their function, and then directly read the sequence value to check progress. This is crude but effective. I prefer logging whenever possible.
To deal with disconnects, run psql on the remote side over ssh rather than connecting to the server directly over the PostgreSQL protocol. As Christian suggests, use screen so the remote psql doesn't get killed when the ssh session dies.
Alternately, you can use the traditional unix command nohup, which is available everywhere:
nohup psql -f the_script.sql </dev/null &
which will run psql in the background, writing all output and errors to a file named nohup.out.
You may also find that if you enable TCP keepalives, you don't lose remote connections anyway.
pg_stat_activity is a good hint to check if your function is still running. Also use screen or tmux on the server to ensure that it will survive a reconnect.
1 - Login to the psql console.
$ psql -U user -d database
2- Issue a \x command to format the results.
3- SELECT * from pg_stat_activity;
4- Scroll until you see your function name on the list. It should have an active status.
5- Check if there are any blocks on the table that your function relies on using:
SELECT k_locks.pid AS pid_blocking, k_activity.usename AS user_blocking, k_activity.query AS query_blocking, locks.pid AS pid_blocked, activity.usename AS user_blocked, activity.query AS query_blocked, to_char(age(now(), activity.query_start), 'HH24h:MIm:SSs') AS blocking_age FROM pg_catalog.pg_locks locks JOIN pg_catalog.pg_stat_activity activity ON locks.pid = activity.pid JOIN pg_catalog.pg_locks k_locks ON locks.locktype = k_locks.locktype and locks.database is not distinct from k_locks.database and locks.relation is not distinct from k_locks.relation and locks.page is not distinct from k_locks.page and locks.tuple is not distinct from k_locks.tuple and locks.virtualxid is not distinct from k_locks.virtualxid and locks.transactionid is not distinct from k_locks.transactionid and locks.classid is not distinct from k_locks.classid and locks.objid is not distinct from k_locks.objid and locks.objsubid is not distinct from k_locks.objsubid and locks.pid <> k_locks.pid JOIN pg_catalog.pg_stat_activity k_activity ON k_locks.pid = k_activity.pid WHERE k_locks.granted and not locks.granted ORDER BY activity.query_start;
I am running the YCSB tool on mongodb for benchmarking db and I notice that once I load a workload ( workloada for example) and run a transaction ( target 1500 for example) I am not able to run another transaction without dropping the entire database and loading the database again. the reason being that if I run another transaction without dropping and loading the database I get the error the "duplicate key error".
It looks like the first transaction entered some keys which the second transaction also tries to insert. Is there a workaround for this? Or is there something wrong with what I am doing.
this is the command I use for loading :
./bin/ycsb load mongodb -P workloads/workloada
-p mongodb.url=<ip_address>:27020
-p mongodb.maxconnections=150 -s
-p mongodb.writeConcern=normal
-target 3500 -threads 200 > <output-file>
Here is command I use for the transaction phase
./bin/ycsb load mongodb
-P workloads/workloada
-p mongodb.url=<IP_address>:27020
-p mongodb.maxconnections=100 -s
-p mongodb.writeConcern=normal
-target 1500 -threads 100 > <output_file>
When you load once you can run YCSB as many times as you want.
But loading again will give you the error as records are already loaded. Hence, you will have to delete the directory on which you are loading MongoDB.
The question is old but adding a response anyways.
That's how it is meant to behave: the load phase ideally runs once inserting data into MongoDB, followed by whatever workloads you intend to run.
Look here in YCSB wiki for an example of a sequence in which workloads can be run. This page in the wiki runs through the list of everything needed to run a test.
If load is what you intend to benchmark then you should drop the collection and the database between and before your 'load' operations.
I am trying to restore a database using mysqldump, but nothing seems to happen. I get some output on the screen, but the program stops before it imports anything and does not report any errors.
I am trying to restore a dump using the syntax
mysqldump --log-error=/root/dumplog --verbose --user=myuser mydatabasename < /root/dump.sql
I get no entries in the MySQL log, and in dumplog, all I get is this:
-- Connecting to localhost...
-- Disconnecting from localhost...
The dump file is like 15mb
You don't use mysqldump to restore. To restore you would do:
mysql -uUser -p dbname < /path/to/file.sql