I was messing around with postgres in docker and wanted to connect to it via url URI.
If the password contains a substring that could be transformed as a character thanks to decodeURIComponent, it will be treated as well, leading to a connection fail.
Is it possible to specify some character in the uri that should not be considered as special characters to decode?
Here is an example:
%23 is automatically encoded as #
$> docker run --name test -p 5433:5432 -e POSTGRES_USER=user -e POSTGRES_DB=db -e POSTGRES_PASSWORD=secret%23 --rm -d postgres
$> psql postgres://user:secret%23#localhost:5433/db
psql: error: could not connect to server: FATAL: password authentication failed for user "user"
$> PGPASSWORD=secret%23 psql --user=user --host=localhost --port=5433 --dbname=db
psql (12.3, server 13.2 (Debian 13.2-1.pgdg100+1))
WARNING: psql major version 12, server major version 13.
Some psql features might not work.
Type "help" for help.
db=#
Working example (without a percent followed by two digits):
$> docker run --name test2 -p 5434:5432 -e POSTGRES_USER=user -e POSTGRES_DB=db -e POSTGRES_PASSWORD=secret# --rm -d postgres
$> psql postgres://user:secret%23#localhost:5434/db
psql (12.3, server 13.2 (Debian 13.2-1.pgdg100+1))
WARNING: psql major version 12, server major version 13.
Some psql features might not work.
Type "help" for help.
db=#
$> PGPASSWORD=secret# psql --user=user --host=localhost --port=5434 --dbname=db
psql (12.3, server 13.2 (Debian 13.2-1.pgdg100+1))
WARNING: psql major version 12, server major version 13.
Some psql features might not work.
Type "help" for help.
db=#
psql postgres://user:secret%23#localhost:5433/db does not work with a password set to secret%23 but does with one set to secret#.
Is it possible to use database uri with a password like secret%23?
The doc says the connection URI must be encoded, so your value of %23 must be encoded to %2523.
Related
I have installed PostgreSQL server in Google Colab as follows :
# Install postgresql server
!apt update > /dev/null
!apt install postgresql > /dev/null
!service postgresql start
and have then configured the 'postgres' userid and database as follows :
# Setup a password `pass` for username `postgres`
!sudo -u postgres psql -U postgres -c "ALTER USER postgres PASSWORD 'pass';"
#
# Setup a database with name `praxis` to be used
!sudo -u postgres psql -U postgres -c 'DROP DATABASE IF EXISTS praxisdb;'
!sudo -u postgres psql -U postgres -c 'CREATE DATABASE praxisdb;'
Subsequently, I have created a table, inserted data and run the select command
!psql -h localhost -p 5432 -Upostgres -W -dpraxisdb -c 'create table dept ... ;'
!psql -h localhost -p 5432 -Upostgres -W -dpraxisdb -c "INSERT INTO Dept ... ;"
!psql -h localhost -p 5432 -Upostgres -W -dpraxisdb -c "select * from dept;"
All this works perfectly but each time, I am prompted to enter the password. I wish to avoid having to enter the password each time. Based on what I read in the documentation on using password files, I created a file ~/.pgpass as follows :
!echo "localhost:5432:praxisdb:postgres:pass" > ~/.pgpass
!chmod 0600 ~/.pgpass
Now, when I execute any command, eg.
!psql -c "select * from dept;"
I get the error
psql: error: FATAL: role "root" does not exist
Where does this role root come from? I looked at the file ~/.pgpass and noted that its owner and group is root and I changed that to postgres using chown, chgrp, but that does not solve the problem. What else should I do in this case to solve the problem.
The version of Postgres and the OS is as follows :
!sudo -u postgres psql -V
psql (PostgreSQL) 12.13 (Ubuntu 12.13-0ubuntu0.20.04.1)
You misunderstand how the password file works. You still have to specify host, port, database and user in your connection request. The client library then searches the matching entry in the password file and reads the password.
I have some .sql files with thousands of INSERT statements in them and need to run these inserts on my PostgreSQL database in order to add them to a table. The files are that large that it is impossible to open them and copy the INSERT statements into an editor window and run them there. I found on the Internet that you can use the following by navigating to the bin folder of your PostgreSQL install:
psql -d myDataBase -a -f myInsertFile
In my case:
psql -d HIGHWAYS -a -f CLUSTER_1000M.sql
I am then asked for a password for my user, but I cannot enter anything and when I hit enter I get this error:
psql: FATAL: password authentication failed for user "myUsername"
Why won't it let me enter a password. Is there a way round this as it is critical that I can run these scripts?
I got around this issue by adding a new entry in my pg_hba.conf file with the following structure:
# IPv6 local connections:
host myDbName myUserName ::1/128 trust
The pg_hba.conf file can usually be found in the 'data' folder of your PostgreSQL install.
Of course, you will get a fatal error for authenticating, because you do not include a user name...
Try this one, it is OK for me :)
psql -U username -d myDataBase -a -f myInsertFile
If the database is remote, use the same command with host
psql -h host -U username -d myDataBase -a -f myInsertFile
You should do it like this:
\i path_to_sql_file
See:
You have four choices to supply a password:
Set the PGPASSWORD environment variable. For details see the manual: http://www.postgresql.org/docs/current/static/libpq-envars.html
Use a .pgpass file to store the password. For details see the manual: http://www.postgresql.org/docs/current/static/libpq-pgpass.html
Use "trust authentication" for that specific user: http://www.postgresql.org/docs/current/static/auth-methods.html#AUTH-TRUST
Since PostgreSQL 9.1 you can also use a connection string: https://www.postgresql.org/docs/current/static/libpq-connect.html#LIBPQ-CONNSTRING
Use this to execute *.sql files when the PostgreSQL server is located in a difference place:
psql -h localhost -d userstoreis -U admin -p 5432 -a -q -f /home/jobs/Desktop/resources/postgresql.sql
-h PostgreSQL server IP address
-d database name
-U user name
-p port which PostgreSQL server is listening on
-f path to SQL script
-a all echo
-q quiet
Then you are prompted to enter the password of the user.
EDIT: updated based on the comment provided by #zwacky
If you are logged in into psql on the Linux shell the command is:
\i fileName.sql
for an absolute path and
\ir filename.sql
for the relative path from where you have called psql.
export PGPASSWORD=<password>
psql -h <host> -d <database> -U <user_name> -p <port> -a -w -f <file>.sql
Via the terminal log on to your database and try this:
database-# >#pathof_mysqlfile.sql
or
database-#>-i pathof_mysqlfile.sql
or
database-#>-c pathof_mysqlfile.sql
You can give both user name and PASSSWORD on the command line itself with the "-d" parameter
psql -d "dbname='urDbName' user='yourUserName' password='yourPasswd' host='yourHost'" -f yourFileName.sql
you could even do it in this way:
sudo -u postgres psql -d myDataBase -a -f myInsertFile
If you have sudo access on machine and it's not recommended for production scripts just for test on your own machine it's the easiest way.
2021 Solution
if your PostgreSQL database is on your system locally.
psql dbname < sqldump.sql username
If its hosted online
psql -h hostname dbname < sqldump.sql username
If you have any doubts or questions, please ask them in the comments.
Walk through on how to run an SQL on the command line for PostgreSQL in Linux:
Open a terminal and make sure you can run the psql command:
psql --version
which psql
Mine is version 9.1.6 located in /bin/psql.
Create a plain textfile called mysqlfile.sql
Edit that file, put a single line in there:
select * from mytable;
Run this command on commandline (substituting your username and the name of your database for pgadmin and kurz_prod):
psql -U pgadmin -d kurz_prod -a -f mysqlfile.sql
The following is the result I get on the terminal (I am not prompted for a password):
select * from mytable;
test1
--------
hi
me too
(2 rows)
psql -h localhost -d userstoreis -U admin -p 5432 -a -q -f /home/jobs/Desktop/resources/postgresql.sql
Parameter explanations:
-h PostgreSQL server IP address
-d database name
-U user name
-p port which PostgreSQL server is listening on
-f path to SQL script
-a all echo
-q quiet
You can open a command prompt and run as administrator. Then type
../bin>psql -f c:/...-h localhost -p 5432 -d databasename -U "postgres"
Password for user postgres: will show up.
Type your password and enter. I couldn't see the password what I was typing, but this time when I press enter it worked. Actually I was loading data into the database.
I achived that wrote (located in the directory where my script is)
::someguy#host::$sudo -u user psql -d my_database -a -f file.sql
where -u user is the role who owns the database where I want to execute the script then the psql connects to the psql console after that -d my_database loads me in mydatabase finally -a -f file.sql where -a echo all input from the script and -f execute commands from file.sql into mydatabase, then exit.
I'm using:
psql (PostgreSQL) 10.12
on (Ubuntu 10.12-0ubuntu0.18.04.1)
A small improvement in #wingman__7 's 2021 answer: if your username contains certain characters (an underscore in my case), you need to pass it with the -U flag.
This worked for me:
$ psql -h db.host -d db_name -U my_user < query.sql
Try using the following command in the command line console:
psql -h localhost -U postgres -f restore.sql
I'm having an issue getting a database restore using pg_restore to work. I've got Postgres 12.2 installed on my Mac running High Sierra (10.13.6). It was installed with Brew.
I created a simple database "foo" with table "foo" to test this out:
Nates-MacBook-Pro:k8s natereed$ psql -d foo
psql (12.2, server 12.1)
Type "help" for help.
foo=# SELECT table_schema,table_name
FROM information_schema.tables
WHERE table_schema='public' ORDER BY table_schema,table_name;
table_schema | table_name
--------------+------------
public | foo
(1 row)
Generate dump:
pg_dump -Fc foo -f foo.backup
When I try to restore, nothing happens. pg_restore just hangs:
pg_restore -h localhost -p 5432 -U postgres -v -Fc -f foo.backup
I've tried various permutations of this command, but every time it just hangs. In Activity Monitor, I see the process but it is not using any CPU.
Online help says:
pg_restore restores a PostgreSQL database from an archive created by pg_dump.
Usage:
pg_restore [OPTION]... [FILE]
General options:
-d, --dbname=NAME connect to database name
-f, --file=FILENAME output file name
-F, --format=c|d|t backup file format (should be automatic)
-l, --list print summarized TOC of the archive
-v, --verbose verbose mode
-V, --version output version information, then exit
-?, --help show this help, then exit
For pg_restore -f is parameter for the output file: it is not the input file. Remove -f in your example:
pg_restore -h localhost -p 5432 -U postgres -v -Fc foo.backup
I downloaded the xbrldb_SEC_pg_2014-11-02.pg.gzip postgres pg_dump file from arelle.org. I then ran the schema ddl file in pgAdminIII and it recreated all of the databases, functions, etc.
When I try to restore the databases using the following:
desktop:~/Downloads$ sudo postgres zcat xbrldb_SEC_pg_2014-11-02.pg.gzip | psql -U postgres public
I get:
sudo: postgres: command not found psql: FATAL: Peer authentication failed for user "postgres"
I can zcat the file into a file to expand it. Looks like it is a pg_dump file.
postgres=> pg_restore -a /home/jeremy/Downloads/xbrldb_SEC_pg_2014-11-02.txt
postgres-> ;
ERROR: syntax error at or near "pg_restore"
LINE 1: pg_restore -a /home/jeremy/Downloads/xbrldb_SEC_pg_2014-11-0...
^
postgres=> pg_restore -a postgres /home/jeremy/Downloads/xbrldb_SEC_pg_2014-11-02.txt;
ERROR: syntax error at or near "pg_restore"
LINE 1: pg_restore -a postgres /home/jeremy/Downloads/xbrldb_SEC_pg_...
So then I tried to use PG Admin III, and my output:
/usr/bin/pg_restore --host localhost --port 5432 --username "postgres" --dbname "public" --role "postgres" --no-password --section data --data-only --exit-on-error --table accession --schema public --verbose "/home/jeremy/Downloads/xbrldb_SEC_pg_2014-11-02.backup"
pg_restore: [archiver] input file appears to be a text format dump. Please use psql.
Process returned exit code 1.
May I please ask what I need to do to get the databases restored?
Does anyone know what I need to do to get the database updated from 2014-11-02 to the current date?
You should run psql as postgres user, not zcat, so try to use following:
zcat xbrldb_SEC_pg_2014-11-02.pg.gzip | sudo -u postgres psql public
PS pg_restore is an utility, not a PostgreSQL command, that means you should run it from command line, not from psql.
I have some .sql files with thousands of INSERT statements in them and need to run these inserts on my PostgreSQL database in order to add them to a table. The files are that large that it is impossible to open them and copy the INSERT statements into an editor window and run them there. I found on the Internet that you can use the following by navigating to the bin folder of your PostgreSQL install:
psql -d myDataBase -a -f myInsertFile
In my case:
psql -d HIGHWAYS -a -f CLUSTER_1000M.sql
I am then asked for a password for my user, but I cannot enter anything and when I hit enter I get this error:
psql: FATAL: password authentication failed for user "myUsername"
Why won't it let me enter a password. Is there a way round this as it is critical that I can run these scripts?
I got around this issue by adding a new entry in my pg_hba.conf file with the following structure:
# IPv6 local connections:
host myDbName myUserName ::1/128 trust
The pg_hba.conf file can usually be found in the 'data' folder of your PostgreSQL install.
Of course, you will get a fatal error for authenticating, because you do not include a user name...
Try this one, it is OK for me :)
psql -U username -d myDataBase -a -f myInsertFile
If the database is remote, use the same command with host
psql -h host -U username -d myDataBase -a -f myInsertFile
You should do it like this:
\i path_to_sql_file
See:
You have four choices to supply a password:
Set the PGPASSWORD environment variable. For details see the manual: http://www.postgresql.org/docs/current/static/libpq-envars.html
Use a .pgpass file to store the password. For details see the manual: http://www.postgresql.org/docs/current/static/libpq-pgpass.html
Use "trust authentication" for that specific user: http://www.postgresql.org/docs/current/static/auth-methods.html#AUTH-TRUST
Since PostgreSQL 9.1 you can also use a connection string: https://www.postgresql.org/docs/current/static/libpq-connect.html#LIBPQ-CONNSTRING
Use this to execute *.sql files when the PostgreSQL server is located in a difference place:
psql -h localhost -d userstoreis -U admin -p 5432 -a -q -f /home/jobs/Desktop/resources/postgresql.sql
-h PostgreSQL server IP address
-d database name
-U user name
-p port which PostgreSQL server is listening on
-f path to SQL script
-a all echo
-q quiet
Then you are prompted to enter the password of the user.
EDIT: updated based on the comment provided by #zwacky
If you are logged in into psql on the Linux shell the command is:
\i fileName.sql
for an absolute path and
\ir filename.sql
for the relative path from where you have called psql.
export PGPASSWORD=<password>
psql -h <host> -d <database> -U <user_name> -p <port> -a -w -f <file>.sql
Via the terminal log on to your database and try this:
database-# >#pathof_mysqlfile.sql
or
database-#>-i pathof_mysqlfile.sql
or
database-#>-c pathof_mysqlfile.sql
You can give both user name and PASSSWORD on the command line itself with the "-d" parameter
psql -d "dbname='urDbName' user='yourUserName' password='yourPasswd' host='yourHost'" -f yourFileName.sql
you could even do it in this way:
sudo -u postgres psql -d myDataBase -a -f myInsertFile
If you have sudo access on machine and it's not recommended for production scripts just for test on your own machine it's the easiest way.
2021 Solution
if your PostgreSQL database is on your system locally.
psql dbname < sqldump.sql username
If its hosted online
psql -h hostname dbname < sqldump.sql username
If you have any doubts or questions, please ask them in the comments.
Walk through on how to run an SQL on the command line for PostgreSQL in Linux:
Open a terminal and make sure you can run the psql command:
psql --version
which psql
Mine is version 9.1.6 located in /bin/psql.
Create a plain textfile called mysqlfile.sql
Edit that file, put a single line in there:
select * from mytable;
Run this command on commandline (substituting your username and the name of your database for pgadmin and kurz_prod):
psql -U pgadmin -d kurz_prod -a -f mysqlfile.sql
The following is the result I get on the terminal (I am not prompted for a password):
select * from mytable;
test1
--------
hi
me too
(2 rows)
psql -h localhost -d userstoreis -U admin -p 5432 -a -q -f /home/jobs/Desktop/resources/postgresql.sql
Parameter explanations:
-h PostgreSQL server IP address
-d database name
-U user name
-p port which PostgreSQL server is listening on
-f path to SQL script
-a all echo
-q quiet
You can open a command prompt and run as administrator. Then type
../bin>psql -f c:/...-h localhost -p 5432 -d databasename -U "postgres"
Password for user postgres: will show up.
Type your password and enter. I couldn't see the password what I was typing, but this time when I press enter it worked. Actually I was loading data into the database.
I achived that wrote (located in the directory where my script is)
::someguy#host::$sudo -u user psql -d my_database -a -f file.sql
where -u user is the role who owns the database where I want to execute the script then the psql connects to the psql console after that -d my_database loads me in mydatabase finally -a -f file.sql where -a echo all input from the script and -f execute commands from file.sql into mydatabase, then exit.
I'm using:
psql (PostgreSQL) 10.12
on (Ubuntu 10.12-0ubuntu0.18.04.1)
A small improvement in #wingman__7 's 2021 answer: if your username contains certain characters (an underscore in my case), you need to pass it with the -U flag.
This worked for me:
$ psql -h db.host -d db_name -U my_user < query.sql
Try using the following command in the command line console:
psql -h localhost -U postgres -f restore.sql