I'm backing up my database that's in a docker container, and since the total filesize is too large to fit onto the remaining disk space I execute it via SSH and dump it onto my local pc with this command (I'm using Ubuntu default bash):
docker-machine ssh my-machine-image "docker exec container-id /bin/sh -c 'mongodump --archive -u=admin --authenticationDatabase=admin -p=mongo-pwd --db=my-db --collection=my-collection --gzip'" > myfile.dump
This works pretty great, however I'm having trouble getting it to work with the --query command. Mongodump requires it to be in strict JSON and I'm having trouble with getting the nested quotes in bash to work. My most successful attempt (aka it actually successfuly executed the command instead of returning a syntax/JSON error) was with a string literal like this, however that seems to parse the JSON wrong, since it always returns 0 documents, no matter the query:
docker-machine ssh my-machine-image "docker exec container-id /bin/sh -c $'mongodump --archive -u=admin --authenticationDatabase=admin -p=mongo-pwd --db=my-db --collection=my-collection --query=\'{ \"_id\": { \"$oid\": \"some_random_object_id\" } }\' --gzip'" > myfile.dump
What is the correct way to pass strict JSON to the --query parameter with this amount of nested quotes?
Since you have multiple layers of quoting it would be easiest to assign each layer to a variable. Then use bash's printf %q to automatically quote any string for use in a shell.
#! /usr/bin/env bash
json='{"_id": { "'"$oid"'": "some_random_object_id" } }'
cmd="mongodump --archive -u=admin --authenticationDatabase=admin -p=mongo-pwd --db=my-db --collection=my-collection --query=$(printf %q "$json") --gzip"
sshCmd="docker exec container-id /bin/sh -c $(printf %q "$cmd")"
docker-machine ssh my-machine-image "$sshCmd"
Related
I am trying to use the following approach in order to backup PostgreSQL database that is in a Docker container:
cmd /c 'docker exec -t postgres-dev pg_dump dvdrental -U postgres -c -v >
C:\\postgresql\\backup\\dvdrental_%date:~-4,4%.%date:~-7,2%.%date:~-10,2%_%time:~0,2%.%time:~3,2%.sql'
However, it gives the following file that has the size ok 0 KB and with the extension of %tme:
dvdrental_2021.05.24_%tme
I tried some different combinations, but it still the same and I am not able to get backup with the following format:
dvdrental_2021.05.24_15.30
So, how can I get the expected result as mentioned above? I use Windows, but I think it is not important as the command is executed on Docker container.
I have the following command that I'm running (renamed some variables):
docker exec docker_name sh -c 'psql dbname -U joeadmin -c "update table set field='really_longstringwithabunf3493829#########=';"'
When I run this, it will throw the following error:
ERROR: syntax error at or near ";"
LINE 1: ...longstringwithabunf3493829#########=;
^
How can I escape or retain the single quotes so this will work. Also, if this is just a horrible way of approaching this, I'm open to other suggestions. The use case is making changes to a db on a docker container before exporting data out of it.
Working with strings can be a bit ugly at times with Docker. In your case, I would suggest using double-quotes first, then escape inner double-quotes with \:
docker exec docker_name sh -c "psql dbname -U joeadmin -c \"update table set field='really_longstringwithabunf3493829#########=';\""
A cleaner option would be to create an environment variable:
STR="really_longstringwithabunf3493829#########="
docker exec docker_name sh -c "psql dbname -U joeadmin -c \"update table set field='${STR}';\""
Bear in mind that string interpolation happens in the host OS, not in the container (unless you escape the dollar-sign with \${STR}
Still cleaner would just be to create a file and then copy it into the container:
echo "update table set field='really_longstringwithabunf3493829#########=';" > ~/myfile.txt
docker cp ~/myfile.txt docker_name:/tmp
docker exec docker_name sh -c "psql dbname -U joeadmin -f /tmp/myfile.txt"
When using the psql command line utility on my local machine, I have the option to use the -q or --quiet switch to tell Postgres to do it's work quietly - i.e. it won't print every single INSERT statement to the console if you're doing a large import.
Here's an example of how I'm using it:
psql -q -d <SOME_DATABASE> -f <SOME_SQL_FILE>
However, when using the pg:psql command line utility in Heroku, that option doesn't seem to be available. So I'm currently having to use it like so:
heroku pg:psql DATABASE -a <SOME_HEROKU_APP> < <SOME_SQL_FILE>
which produces a lot of output to my console (hundreds of thousands of lines), because of the large size of the SQL file I'm importing. Whenever I try to use the -q or --quiet option, something like this:
heroku pg:psql DATABASE -q -a <SOME_HEROKU_APP> < <SOME_SQL_FILE>
it'll throw an error saying that -q is not a valid option.
Is there some way to enable quiet mode when running Postgres commands in Heroku?
heroku pg:psql is just a wrapper onto your local psql binary (https://github.com/heroku/heroku/blob/master/lib/heroku/command/pg.rb#L151)
So, given this - you are able to do:
psql `heroku config:get DATABASE_URL -a <yourappname>`
to get a psql connection and consequently pass -q other options accordingly.
I have tried following command to split backup file but it is always showing error illegal option split:
pg_dump.exe" -h localhost -p 5432 -U
postgres --inserts | split -b 2m – backup.sql -f "D:\post\filename.sql"
db_name
you use pipe (|) and unix split command as argument to pg_dump.exe. It won't work. Consider trying 7zip volumes for that. Or any other command line splitter
I am migrating an application into Docker. One of the issues that I am bumping into is what is the correct way to load the initial data into PostgreSQL running in Docker? My typical method of restoring a database backup file are not working. I have tried the following ways:
gunzip -c mydbbackup.sql.gz | psql -h <docker_host> -p <docker_port> -U <dbuser> -d <db> -W
That does not work, because PostgreSQL is prompting for a password, and I cannot enter a password because it is reading data from STDOUT. I cannot use the $PGPASSWORD environment variable, because the any environment variable I set in my host is not set in my container.
I also tried a similar command above, except using the -f flag, and specify the path to a sql backup file. This does not work because my file is not on my container. I could copy the file to my container with the ADD statement in my Dockerfile, but this does not seem right.
So, I ask the community. What is the preferred method on loading PostgreSQL database backups into Docker containers?
I cannot use the $PGPASSWORD environment variable, because the any
environment variable I set in my host is not set in my container.
I don't use docker, but your container looks like a remote host in the command shown, with psql running locally. So PGPASSWORD never has to to be set on the remote host, only locally.
If the problems boils down to adding a password to this command:
gunzip -c mydbbackup.sql.gz |
psql -h <docker_host> -p <docker_port> -U <dbuser> -d <db> -W
you may submit it using several methods (in all cases, don't use the -W option to psql)
hardcoded in the invocation:
gunzip -c mydbbackup.sql.gz |
PGPASSWORD=something psql -h <docker_host> -p <docker_port> -U <dbuser> -d <db>
typed on the keyboard
echo -n "Enter password:"
read -s PGPASSWORD
export PGPASSWORD
gunzip -c mydbbackup.sql.gz |
psql -h <docker_host> -p <docker_port> -U <dbuser> -d <db>
Note about the -W or --password option to psql.
The point of this option is to ask for a password to be typed first thing, even if the context makes it unnecessary.
It's frequently misunderstood as the equivalent of the -poption of mysql. This is a mistake: while -p is required on password-protected connections, -W is never required and actually goes in the way when scripting.
-W, --password
Force psql to prompt for a password before connecting to a
database.
This option is never essential, since psql will automatically
prompt for a password if the server demands password
authentication. However, psql will waste a connection attempt
finding out that the server wants a password. In some cases it is
worth typing -W to avoid the extra connection attempt.