I am migrating an application into Docker. One of the issues that I am bumping into is what is the correct way to load the initial data into PostgreSQL running in Docker? My typical method of restoring a database backup file are not working. I have tried the following ways:
gunzip -c mydbbackup.sql.gz | psql -h <docker_host> -p <docker_port> -U <dbuser> -d <db> -W
That does not work, because PostgreSQL is prompting for a password, and I cannot enter a password because it is reading data from STDOUT. I cannot use the $PGPASSWORD environment variable, because the any environment variable I set in my host is not set in my container.
I also tried a similar command above, except using the -f flag, and specify the path to a sql backup file. This does not work because my file is not on my container. I could copy the file to my container with the ADD statement in my Dockerfile, but this does not seem right.
So, I ask the community. What is the preferred method on loading PostgreSQL database backups into Docker containers?
I cannot use the $PGPASSWORD environment variable, because the any
environment variable I set in my host is not set in my container.
I don't use docker, but your container looks like a remote host in the command shown, with psql running locally. So PGPASSWORD never has to to be set on the remote host, only locally.
If the problems boils down to adding a password to this command:
gunzip -c mydbbackup.sql.gz |
psql -h <docker_host> -p <docker_port> -U <dbuser> -d <db> -W
you may submit it using several methods (in all cases, don't use the -W option to psql)
hardcoded in the invocation:
gunzip -c mydbbackup.sql.gz |
PGPASSWORD=something psql -h <docker_host> -p <docker_port> -U <dbuser> -d <db>
typed on the keyboard
echo -n "Enter password:"
read -s PGPASSWORD
export PGPASSWORD
gunzip -c mydbbackup.sql.gz |
psql -h <docker_host> -p <docker_port> -U <dbuser> -d <db>
Note about the -W or --password option to psql.
The point of this option is to ask for a password to be typed first thing, even if the context makes it unnecessary.
It's frequently misunderstood as the equivalent of the -poption of mysql. This is a mistake: while -p is required on password-protected connections, -W is never required and actually goes in the way when scripting.
-W, --password
Force psql to prompt for a password before connecting to a
database.
This option is never essential, since psql will automatically
prompt for a password if the server demands password
authentication. However, psql will waste a connection attempt
finding out that the server wants a password. In some cases it is
worth typing -W to avoid the extra connection attempt.
Related
I have some .sql files with thousands of INSERT statements in them and need to run these inserts on my PostgreSQL database in order to add them to a table. The files are that large that it is impossible to open them and copy the INSERT statements into an editor window and run them there. I found on the Internet that you can use the following by navigating to the bin folder of your PostgreSQL install:
psql -d myDataBase -a -f myInsertFile
In my case:
psql -d HIGHWAYS -a -f CLUSTER_1000M.sql
I am then asked for a password for my user, but I cannot enter anything and when I hit enter I get this error:
psql: FATAL: password authentication failed for user "myUsername"
Why won't it let me enter a password. Is there a way round this as it is critical that I can run these scripts?
I got around this issue by adding a new entry in my pg_hba.conf file with the following structure:
# IPv6 local connections:
host myDbName myUserName ::1/128 trust
The pg_hba.conf file can usually be found in the 'data' folder of your PostgreSQL install.
Of course, you will get a fatal error for authenticating, because you do not include a user name...
Try this one, it is OK for me :)
psql -U username -d myDataBase -a -f myInsertFile
If the database is remote, use the same command with host
psql -h host -U username -d myDataBase -a -f myInsertFile
You should do it like this:
\i path_to_sql_file
See:
You have four choices to supply a password:
Set the PGPASSWORD environment variable. For details see the manual: http://www.postgresql.org/docs/current/static/libpq-envars.html
Use a .pgpass file to store the password. For details see the manual: http://www.postgresql.org/docs/current/static/libpq-pgpass.html
Use "trust authentication" for that specific user: http://www.postgresql.org/docs/current/static/auth-methods.html#AUTH-TRUST
Since PostgreSQL 9.1 you can also use a connection string: https://www.postgresql.org/docs/current/static/libpq-connect.html#LIBPQ-CONNSTRING
Use this to execute *.sql files when the PostgreSQL server is located in a difference place:
psql -h localhost -d userstoreis -U admin -p 5432 -a -q -f /home/jobs/Desktop/resources/postgresql.sql
-h PostgreSQL server IP address
-d database name
-U user name
-p port which PostgreSQL server is listening on
-f path to SQL script
-a all echo
-q quiet
Then you are prompted to enter the password of the user.
EDIT: updated based on the comment provided by #zwacky
If you are logged in into psql on the Linux shell the command is:
\i fileName.sql
for an absolute path and
\ir filename.sql
for the relative path from where you have called psql.
export PGPASSWORD=<password>
psql -h <host> -d <database> -U <user_name> -p <port> -a -w -f <file>.sql
Via the terminal log on to your database and try this:
database-# >#pathof_mysqlfile.sql
or
database-#>-i pathof_mysqlfile.sql
or
database-#>-c pathof_mysqlfile.sql
You can give both user name and PASSSWORD on the command line itself with the "-d" parameter
psql -d "dbname='urDbName' user='yourUserName' password='yourPasswd' host='yourHost'" -f yourFileName.sql
you could even do it in this way:
sudo -u postgres psql -d myDataBase -a -f myInsertFile
If you have sudo access on machine and it's not recommended for production scripts just for test on your own machine it's the easiest way.
2021 Solution
if your PostgreSQL database is on your system locally.
psql dbname < sqldump.sql username
If its hosted online
psql -h hostname dbname < sqldump.sql username
If you have any doubts or questions, please ask them in the comments.
Walk through on how to run an SQL on the command line for PostgreSQL in Linux:
Open a terminal and make sure you can run the psql command:
psql --version
which psql
Mine is version 9.1.6 located in /bin/psql.
Create a plain textfile called mysqlfile.sql
Edit that file, put a single line in there:
select * from mytable;
Run this command on commandline (substituting your username and the name of your database for pgadmin and kurz_prod):
psql -U pgadmin -d kurz_prod -a -f mysqlfile.sql
The following is the result I get on the terminal (I am not prompted for a password):
select * from mytable;
test1
--------
hi
me too
(2 rows)
psql -h localhost -d userstoreis -U admin -p 5432 -a -q -f /home/jobs/Desktop/resources/postgresql.sql
Parameter explanations:
-h PostgreSQL server IP address
-d database name
-U user name
-p port which PostgreSQL server is listening on
-f path to SQL script
-a all echo
-q quiet
You can open a command prompt and run as administrator. Then type
../bin>psql -f c:/...-h localhost -p 5432 -d databasename -U "postgres"
Password for user postgres: will show up.
Type your password and enter. I couldn't see the password what I was typing, but this time when I press enter it worked. Actually I was loading data into the database.
I achived that wrote (located in the directory where my script is)
::someguy#host::$sudo -u user psql -d my_database -a -f file.sql
where -u user is the role who owns the database where I want to execute the script then the psql connects to the psql console after that -d my_database loads me in mydatabase finally -a -f file.sql where -a echo all input from the script and -f execute commands from file.sql into mydatabase, then exit.
I'm using:
psql (PostgreSQL) 10.12
on (Ubuntu 10.12-0ubuntu0.18.04.1)
A small improvement in #wingman__7 's 2021 answer: if your username contains certain characters (an underscore in my case), you need to pass it with the -U flag.
This worked for me:
$ psql -h db.host -d db_name -U my_user < query.sql
Try using the following command in the command line console:
psql -h localhost -U postgres -f restore.sql
My older Postgresql servers accepts remote connections.
I'm trying to transfer the database from the old server to a new one. On the new server when logged in as root, I execute the following command
pg_dump -h oldserverip -U mydb -C mydb | psql -h localhost -d mydb -U mydb
I then get prompted with the following
"Password for user mydb: Password: "
Why is it asking for password twice?
and after I input the password, it waits there and nothing happens.
Any idea what the problem is?
Because you are using the -h flag twice, you are being asked for password twice. The second time you are getting asked for the password, there is no prompt because the pipe | is masking it. If you just “trust” that the password prompt is there, and you type in the password, things will proceed.
But that’s probably not what you want to do. If the passwords for both servers are the same, you can do export PGPASSWORD=<password> before issuing your command. If you fee uncomfortable with this, you could use a .pgpass file
Another option is to set PGPASSWORD for both the pg_dump and psql commands, like so:
PGPASSWORD=<password> pg_dump -h oldserverip -U mydb -C mydb | PGPASSWORD=<otherpassword> psql -h localhost -d mydb -U mydb
Finally, if your pg_hba.conf has a local all all trust entry, you can simply omit the -h localhost arg on the psql side, and you’ll only be prompted for a password once.
Take your pick!
Disclosure: I work for EnterpriseDB (EDB)
I am trying to automate database creation process with a shell script and one thing I've hit a road block with passing a password to psql.
Here is a bit of code from the shell script:
psql -U $DB_USER -h localhost -c"$DB_RECREATE_SQL"
How do I pass a password to psql in a non-interactive way?
Set the PGPASSWORD environment variable inside the script before calling psql
PGPASSWORD=pass1234 psql -U MyUsername myDatabaseName
For reference, see http://www.postgresql.org/docs/current/static/libpq-envars.html
Edit
Since Postgres 9.2 there is also the option to specify a connection string or URI that can contain the username and password. Syntax is:
$ psql postgresql://[user[:password]#][host][:port][,...][/dbname][?param1=value1&...]
Using that is a security risk because the password is visible in plain text when looking at the command line of a running process e.g. using ps (Linux), ProcessExplorer (Windows) or similar tools, by other users.
See also this question on Database Administrators
From the official documentation:
It is also convenient to have a ~/.pgpass file to avoid regularly having to type in passwords. See Section 30.13 for more information.
...
This file should contain lines of the following format:
hostname:port:database:username:password
The password field from the first line that matches the current connection parameters will be used.
in one line:
export PGPASSWORD='password'; psql -h 'server name' -U 'user name' -d 'base name' -c 'command'
with command a sql command such as "select * from schema.table"
or more readable:
export PGPASSWORD='password'
psql -h 'server name' -U 'user name' -d 'base name' \
-c 'command' (eg. "select * from schema.table")
I tend to prefer passing a URL to psql:
psql "postgresql://$DB_USER:$DB_PWD#$DB_SERVER/$DB_NAME"
This gives me the freedom to name my environment variables as I wish and avoids creating unnecessary files.
This requires libpq. The documentation can be found here.
On Windows:
Assign value to PGPASSWORD: C:\>set PGPASSWORD=pass
Run command: C:\>psql -d database -U user
Ready
Or in one line,
set PGPASSWORD=pass&& psql -d database -U user
Note the lack of space before the && !
This can be done by creating a .pgpass file in the home directory of the (Linux) User.
.pgpass file format:
<databaseip>:<port>:<databasename>:<dbusername>:<password>
You can also use wild card * in place of details.
Say I wanted to run tmp.sql without prompting for a password.
With the following code you can in *.sh file
echo "192.168.1.1:*:*:postgres:postgrespwd" > $HOME/.pgpass
echo "` chmod 0600 $HOME/.pgpass `"
echo " ` psql -h 192.168.1.1 -p 5432 -U postgres postgres -f tmp.sql `
An alternative to using the PGPASSWORD environment variable is to use the conninfo string according to the documentation:
An alternative way to specify connection parameters is in a conninfo
string or a URI, which is used instead of a database name. This
mechanism give you very wide control over the connection.
$ psql "host=<server> port=5432 dbname=<db> user=<user> password=<password>"
postgres=>
If its not too late to add most of the options in one answer:
There are a couple of options:
set it in the pgpass file. link
set an environment variable and get it from there:
export PGPASSWORD='password'
and then run your psql to login or even run the command from
there:
psql -h clustername -U username -d testdb
On windows you will have to use "set" :
set PGPASSWORD=pass and then login to the psql bash.
Pass it via URL & env variable:
psql "postgresql://$USER_NAME:$PASSWORD#$HOST_NAME/$DB_NAME"
Just to add more clarity.
You can assign the password to the PGPASSWORD variable.
So instead of the below which will require you to type the password:
psql --host=aurora-postgres.cluster-fgshdjdf.eu-west-1.rds.amazonaws.com --port=5432 --user=my_master_user --password --dbname=postgres
We will replace the --password flag with PGPASSWORD=QghyumjB3ZtCQkdf. So it will be:
PGPASSWORD=QghyumjB3ZtCQkdf psql --host=aurora-postgres.cluster-fgshdjdf.eu-west-1.rds.amazonaws.com --port=5432 --user=my_master_user --dbname=postgres
This way you will not be required to type the password.
Added content of pg_env.sh to my .bashrc:
cat /opt/PostgreSQL/10/pg_env.sh
#!/bin/sh
# The script sets environment variables helpful for PostgreSQL
export PATH=/opt/PostgreSQL/10/bin:$PATH
export PGDATA=/opt/PostgreSQL/10/data
export PGDATABASE=postgres
export PGUSER=postgres
export PGPORT=5433
export PGLOCALEDIR=/opt/PostgreSQL/10/share/locale
export MANPATH=$MANPATH:/opt/PostgreSQL/10/share/man
with addition of (as per user4653174 suggestion)
export PGPASSWORD='password'
psql postgresql://myawsumuser:myPassword#127.0.0.1:5432/myawsumdb
I want to export my database as a .sql file.
Can someone help me? The solutions I have found don't work.
A detailed description please.
On Windows 7.
pg_dump defaults to plain SQL export. both data and structure.
open command prompt and
run pg_dump -U username -h localhost databasename >> sqlfile.sql
Above command is preferable as most of the times there will be an error which will be something like - ...FATAL: Peer authentication failed for user ...
In windows, first, make sure the path is added in environment variables PATH
C:\Program Files\PostgreSQL\12\bin
After a successful path adding restart cmd and type command
pg_dump -U username -p portnumber -d dbname -W -f location
this command will export both schema and data
for only schema use -s in place of -W
and for only data use -a.
replace each variable like username, portnumber, dbname and location according to your situation
everything is case sensitive, make sure you insert everything correctly,
and to import
psql -h hostname -p port_number -U username -f your_file.sql databasename
make sure your db is created or creation query is present in .sql file
Documentation: https://www.postgresql.org/docs/current/app-pgdump.html
Go to your command line and run
pg_dump -U userName -h localhost -d databaseName > ~/Desktop/cmsdump.sql
I have some .sql files with thousands of INSERT statements in them and need to run these inserts on my PostgreSQL database in order to add them to a table. The files are that large that it is impossible to open them and copy the INSERT statements into an editor window and run them there. I found on the Internet that you can use the following by navigating to the bin folder of your PostgreSQL install:
psql -d myDataBase -a -f myInsertFile
In my case:
psql -d HIGHWAYS -a -f CLUSTER_1000M.sql
I am then asked for a password for my user, but I cannot enter anything and when I hit enter I get this error:
psql: FATAL: password authentication failed for user "myUsername"
Why won't it let me enter a password. Is there a way round this as it is critical that I can run these scripts?
I got around this issue by adding a new entry in my pg_hba.conf file with the following structure:
# IPv6 local connections:
host myDbName myUserName ::1/128 trust
The pg_hba.conf file can usually be found in the 'data' folder of your PostgreSQL install.
Of course, you will get a fatal error for authenticating, because you do not include a user name...
Try this one, it is OK for me :)
psql -U username -d myDataBase -a -f myInsertFile
If the database is remote, use the same command with host
psql -h host -U username -d myDataBase -a -f myInsertFile
You should do it like this:
\i path_to_sql_file
See:
You have four choices to supply a password:
Set the PGPASSWORD environment variable. For details see the manual: http://www.postgresql.org/docs/current/static/libpq-envars.html
Use a .pgpass file to store the password. For details see the manual: http://www.postgresql.org/docs/current/static/libpq-pgpass.html
Use "trust authentication" for that specific user: http://www.postgresql.org/docs/current/static/auth-methods.html#AUTH-TRUST
Since PostgreSQL 9.1 you can also use a connection string: https://www.postgresql.org/docs/current/static/libpq-connect.html#LIBPQ-CONNSTRING
Use this to execute *.sql files when the PostgreSQL server is located in a difference place:
psql -h localhost -d userstoreis -U admin -p 5432 -a -q -f /home/jobs/Desktop/resources/postgresql.sql
-h PostgreSQL server IP address
-d database name
-U user name
-p port which PostgreSQL server is listening on
-f path to SQL script
-a all echo
-q quiet
Then you are prompted to enter the password of the user.
EDIT: updated based on the comment provided by #zwacky
If you are logged in into psql on the Linux shell the command is:
\i fileName.sql
for an absolute path and
\ir filename.sql
for the relative path from where you have called psql.
export PGPASSWORD=<password>
psql -h <host> -d <database> -U <user_name> -p <port> -a -w -f <file>.sql
Via the terminal log on to your database and try this:
database-# >#pathof_mysqlfile.sql
or
database-#>-i pathof_mysqlfile.sql
or
database-#>-c pathof_mysqlfile.sql
You can give both user name and PASSSWORD on the command line itself with the "-d" parameter
psql -d "dbname='urDbName' user='yourUserName' password='yourPasswd' host='yourHost'" -f yourFileName.sql
you could even do it in this way:
sudo -u postgres psql -d myDataBase -a -f myInsertFile
If you have sudo access on machine and it's not recommended for production scripts just for test on your own machine it's the easiest way.
2021 Solution
if your PostgreSQL database is on your system locally.
psql dbname < sqldump.sql username
If its hosted online
psql -h hostname dbname < sqldump.sql username
If you have any doubts or questions, please ask them in the comments.
Walk through on how to run an SQL on the command line for PostgreSQL in Linux:
Open a terminal and make sure you can run the psql command:
psql --version
which psql
Mine is version 9.1.6 located in /bin/psql.
Create a plain textfile called mysqlfile.sql
Edit that file, put a single line in there:
select * from mytable;
Run this command on commandline (substituting your username and the name of your database for pgadmin and kurz_prod):
psql -U pgadmin -d kurz_prod -a -f mysqlfile.sql
The following is the result I get on the terminal (I am not prompted for a password):
select * from mytable;
test1
--------
hi
me too
(2 rows)
psql -h localhost -d userstoreis -U admin -p 5432 -a -q -f /home/jobs/Desktop/resources/postgresql.sql
Parameter explanations:
-h PostgreSQL server IP address
-d database name
-U user name
-p port which PostgreSQL server is listening on
-f path to SQL script
-a all echo
-q quiet
You can open a command prompt and run as administrator. Then type
../bin>psql -f c:/...-h localhost -p 5432 -d databasename -U "postgres"
Password for user postgres: will show up.
Type your password and enter. I couldn't see the password what I was typing, but this time when I press enter it worked. Actually I was loading data into the database.
I achived that wrote (located in the directory where my script is)
::someguy#host::$sudo -u user psql -d my_database -a -f file.sql
where -u user is the role who owns the database where I want to execute the script then the psql connects to the psql console after that -d my_database loads me in mydatabase finally -a -f file.sql where -a echo all input from the script and -f execute commands from file.sql into mydatabase, then exit.
I'm using:
psql (PostgreSQL) 10.12
on (Ubuntu 10.12-0ubuntu0.18.04.1)
A small improvement in #wingman__7 's 2021 answer: if your username contains certain characters (an underscore in my case), you need to pass it with the -U flag.
This worked for me:
$ psql -h db.host -d db_name -U my_user < query.sql
Try using the following command in the command line console:
psql -h localhost -U postgres -f restore.sql