I have developed a shell script whose job is to take the dump of postgres DB. Below is the snippet:
#!/bin/sh
today=$(date +"%Y-%m-%d")
yes "password" | sudo -S sudo su - postgres <<EOF
/usr/pgsql-11/bin/pg_dump -U postgres -d db_name > /home/db_backup/db_name_$today.sql
EOF
exit
However, this script is NOT running because of the below reason:
[sudo] password for user: Sorry, Try again
However, when I use sudo su - postgres and then provide password, it is working as expected. And interestingly, if now I run the above shell script after the login, it runs absolutely fine.
What I am missing here.
It is dangerous to store passwords in scripts, so please do not do it.
Modify your /etc/sudoers file by running sudo visudo and adding a line like this at the bottom:
%sudo ALL=(postgres) NOPASSWD: /usr/bin/psql
This allows anyone with sudo permission to run /usr/bin/psql to postgres on any host (ALL) with no password.
Now your script should work this way:
#!/bin/sh
today=$(date +"%Y-%m-%d")
sudo -b -n -H -u postgres /usr/pgsql-11/bin/pg_dump -U postgres -d db_name > /home/db_backup/db_name_$today.sql
Make sure postgres can write to the directory /home/db_backup/.
I've really been struggling over the past few days trying to setup some docker containers and shell scripts to create an environment for my application to run in.
The tall and short is that I have a web server which requires a database to operate. My aim is to have end users unzip the content onto their docker machine, run a build script (which just builds the relevant docker images), then run a OneTime.sh script (which creates the volumes and databases necessary), during this script, they are prompted for what user name and password they would like for the super user of the database.
The problem I'm having is getting those values to the docker image. Here is my script:
# Create the volumes for the data backend database.
docker volume create --name psql-data-etc
docker volume create --name psql-data-log
docker volume create --name psql-data-lib
# Create data store database
echo -e "\n${TITLE}[Data Store Database]${NC}"
docker run -v psql-data-etc:/etc/postgresql -v psql-data-log:/var/log/postgresql -v psql-data-lib:/var/lib/postgresql -p 9001:5432 -P --name psql-data-onetime postgres-setup
# Close containers
docker stop psql-data-onetime
docker rm psql-data-onetime
docker stop psql-transactions-onetime
docker rm psql-transactions-onetime
And here is the docker file:
FROM ubuntu
#Required environment variables: USERNAME, PASSWORD, DBNAME
# Add the PostgreSQL PGP key to verify their Debian packages.
# It should be the same key as https://www.postgresql.org/media/keys/ACCC4CF8.asc
RUN apt-key adv --keyserver hkp://p80.pool.sks-keyservers.net:80 --recv-keys B97B0AFCAA1A47F044F244A07FCC7D46ACCC4CF8
# Add PostgreSQL's repository. It contains the most recent stable release
# of PostgreSQL, ``9.3``.
RUN echo "deb http://apt.postgresql.org/pub/repos/apt/ precise-pgdg main" > /etc/apt/sources.list.d/pgdg.list
# Install ``python-software-properties``, ``software-properties-common`` and PostgreSQL 9.3
# There are some warnings (in red) that show up during the build. You can hide
# them by prefixing each apt-get statement with DEBIAN_FRONTEND=noninteractive
RUN apt-get update && apt-get install -y python-software-properties software-properties-common postgresql-9.3 postgresql-client-9.3 postgresql-contrib-9.3
# Note: The official Debian and Ubuntu images automatically ``apt-get clean``
# after each ``apt-get``
# Run the rest of the commands as the ``postgres`` user created by the ``postgres-9.3`` package when it was ``apt-get installed``
USER postgres
# Complete configuration
USER root
RUN echo "host all all 0.0.0.0/0 md5" >> /etc/postgresql/9.3/main/pg_hba.conf
RUN echo "listen_addresses='*'" >> /etc/postgresql/9.3/main/postgresql.conf
# Expose the PostgreSQL port
EXPOSE 5432
# Add VOLUMEs to allow backup of config, logs and databases
RUN mkdir -p /var/run/postgresql && chown -R postgres /var/run/postgresql
VOLUME ["/etc/postgresql", "/var/log/postgresql", "/var/lib/postgresql"]
# Run setup script
ADD Setup.sh /
CMD ["sh", "Setup.sh"]
The script 'Setup.sh' is the following:
echo -n " User name: "
read user
echo -n " Password: "
read password
echo -n " Database Name: "
read dbname
/etc/init.d/postgresql start
/usr/lib/postgresql/9.3/bin/psql --command "CREATE USER $user WITH SUPERUSER PASSWORD '$password';"
/usr/lib/postgresql/9.3/bin/createdb -O $user $dbname
exit
Why doesn't this work? (I don't get prompted to enter the text, and it throws an error that the parameters are bad). What is the proper way to do something like this? It feels like it's probably a pretty common problem to solve, but I cannot for the life of me find any non convoluted examples of this behaviour.
The main purpose of this is to make life easier for the end user, so if I could just prompt them for the user name, password, and dbname, (plus calling the correct scripts), that would be ideal.
EDIT:
After running the log file looks like this:
User name:
Password:
Database Name:
Usage: /etc/init.d/postgresql {start|stop|restart|reload|force-reload|status} [version ..]
EDIT 2:
After updating to CMD ["sh", "-x", "Setup.sh"]
I get:
echo -n User name:
+read user
:bad variable nameuser
echo -n Password:
+read password
:bad variable namepassword
echo -n Database Name:
+read dbname
:bad variable dbname
I've created an image for docker which hosts a postgresql server. In the dockerfile, the environment variable 'USER', and I pass a constant password into the a run of psql:
USER postgres
RUN /etc/init.d/postgresql start && psql --command "CREATE USER docker WITH SUPERUSER PASSWORD 'docker';" && createdb -O docker docker
Ideally either before or after calling 'docker run' on this image, I'd like the caller to have to input these details into the command line, so that I don't have to store them anywhere.
I'm not really sure how to go about this. Does docker have any support for reading stdin into an environment variable? Or perhaps there's a better way of handling this all together?
At build time
You can use build arguments in your Dockerfile:
ARG password=defaultPassword
USER postgres
RUN /etc/init.d/postgresql start && psql --command "CREATE USER docker WITH SUPERUSER PASSWORD '$password';" && createdb -O docker docker
Then build with:
$ docker build --build-arg password=superSecretPassword .
At run time
For setting the password at runtime, you can use an environment variable (ENV) that you can evaluate in an entrypoint script (ENTRYPOINT):
ENV PASSWORD=defaultPassword
ADD entrypoint.sh /docker-entrypoint.sh
USER postgres
ENTRYPOINT /docker-entrypoint.sh
CMD ["postgres"]
Within the entrypoint script, you can then create a new user with the given password as soon as the container starts:
pg_ctl -D /var/lib/postgresql/data \
-o "-c listen_addresses='localhost'" \
-w start
psql --command "CREATE USER docker WITH SUPERUSER PASSWORD '$password';"
postgres pg_ctl -D /var/lib/postgresql/data -m fast -w stop
exec $#
You can also have a look at the Dockerfile and entrypoint script of the official postgres image, from which I've borrowed most of the code in this answer.
A note on security
Storing secrets like passwords in environment variables (both build and run time) is not incredibly secure (unfortunately, to my knowledge, Docker does not really offer any better solution for this, right now). An interesting discussion on this topic can be found in this question.
You could use environment variable in your Dockerfile and override the default value when you call docker run using -e or --env argument.
Also you will need to amend the init script to run psql command on startup referenced by the CMD instruction.
I have troubles restarting a dockerized postgres database (I use Core OS). The database is started in a bash script using the command
# boot.sh
sudo -i -u postgres /usr/lib/postgresql/9.3/bin/postgres -D /var/lib/postgresql/9.3/main -c config_file=/etc/postgresql/9.3/main/postgresql.conf
which works. I have another script called by confd which is run when some etcd keys change (this part is ok, the file is correctly called) and must restart postgres (not reload, because some config changes require a restart). Here are the main options I tried, which failed...
# restart.sh
sudo -u postgres /usr/lib/postgresql/9.3/bin/pg_ctl --pgdata=/var/lib/postgresql/9.3/main restart
systematically raises an error:
%FATAL: lock file "postmaster.pid" already exists
%HINT: Is another postmaster (PID 273) running in data directory "/var/lib/postgresql/9.3/main"?
Furthermore,
# restart.sh
rm /var/lib/postgresql/9.3/main/postmaster.pid
sudo -i -u postgres /usr/lib/postgresql/9.3/bin/postgres -D /var/lib/postgresql/9.3/main -c config_file=/etc/postgresql/9.3/main/postgresql.conf
,
rm /var/lib/postgresql/9.3/main/postmaster.pid
/etc/init.d/postgresql start
,
/etc/init.d/postgresql restart
and
exec su postgres -c "/usr/lib/postgresql/9.3/bin/postgres -D /var/lib/postgresql/9.3/main -c config_file=/etc/postgresql/9.3/main/postgresql.conf"
fail with
ERROR exit status 1
Any thought? Thank you in advance!
For me, changing the config and doing
$ docker restart <postgres_container>
on the host works just fine.
I have some .sql files with thousands of INSERT statements in them and need to run these inserts on my PostgreSQL database in order to add them to a table. The files are that large that it is impossible to open them and copy the INSERT statements into an editor window and run them there. I found on the Internet that you can use the following by navigating to the bin folder of your PostgreSQL install:
psql -d myDataBase -a -f myInsertFile
In my case:
psql -d HIGHWAYS -a -f CLUSTER_1000M.sql
I am then asked for a password for my user, but I cannot enter anything and when I hit enter I get this error:
psql: FATAL: password authentication failed for user "myUsername"
Why won't it let me enter a password. Is there a way round this as it is critical that I can run these scripts?
I got around this issue by adding a new entry in my pg_hba.conf file with the following structure:
# IPv6 local connections:
host myDbName myUserName ::1/128 trust
The pg_hba.conf file can usually be found in the 'data' folder of your PostgreSQL install.
Of course, you will get a fatal error for authenticating, because you do not include a user name...
Try this one, it is OK for me :)
psql -U username -d myDataBase -a -f myInsertFile
If the database is remote, use the same command with host
psql -h host -U username -d myDataBase -a -f myInsertFile
You should do it like this:
\i path_to_sql_file
See:
You have four choices to supply a password:
Set the PGPASSWORD environment variable. For details see the manual: http://www.postgresql.org/docs/current/static/libpq-envars.html
Use a .pgpass file to store the password. For details see the manual: http://www.postgresql.org/docs/current/static/libpq-pgpass.html
Use "trust authentication" for that specific user: http://www.postgresql.org/docs/current/static/auth-methods.html#AUTH-TRUST
Since PostgreSQL 9.1 you can also use a connection string: https://www.postgresql.org/docs/current/static/libpq-connect.html#LIBPQ-CONNSTRING
Use this to execute *.sql files when the PostgreSQL server is located in a difference place:
psql -h localhost -d userstoreis -U admin -p 5432 -a -q -f /home/jobs/Desktop/resources/postgresql.sql
-h PostgreSQL server IP address
-d database name
-U user name
-p port which PostgreSQL server is listening on
-f path to SQL script
-a all echo
-q quiet
Then you are prompted to enter the password of the user.
EDIT: updated based on the comment provided by #zwacky
If you are logged in into psql on the Linux shell the command is:
\i fileName.sql
for an absolute path and
\ir filename.sql
for the relative path from where you have called psql.
export PGPASSWORD=<password>
psql -h <host> -d <database> -U <user_name> -p <port> -a -w -f <file>.sql
Via the terminal log on to your database and try this:
database-# >#pathof_mysqlfile.sql
or
database-#>-i pathof_mysqlfile.sql
or
database-#>-c pathof_mysqlfile.sql
You can give both user name and PASSSWORD on the command line itself with the "-d" parameter
psql -d "dbname='urDbName' user='yourUserName' password='yourPasswd' host='yourHost'" -f yourFileName.sql
you could even do it in this way:
sudo -u postgres psql -d myDataBase -a -f myInsertFile
If you have sudo access on machine and it's not recommended for production scripts just for test on your own machine it's the easiest way.
2021 Solution
if your PostgreSQL database is on your system locally.
psql dbname < sqldump.sql username
If its hosted online
psql -h hostname dbname < sqldump.sql username
If you have any doubts or questions, please ask them in the comments.
Walk through on how to run an SQL on the command line for PostgreSQL in Linux:
Open a terminal and make sure you can run the psql command:
psql --version
which psql
Mine is version 9.1.6 located in /bin/psql.
Create a plain textfile called mysqlfile.sql
Edit that file, put a single line in there:
select * from mytable;
Run this command on commandline (substituting your username and the name of your database for pgadmin and kurz_prod):
psql -U pgadmin -d kurz_prod -a -f mysqlfile.sql
The following is the result I get on the terminal (I am not prompted for a password):
select * from mytable;
test1
--------
hi
me too
(2 rows)
psql -h localhost -d userstoreis -U admin -p 5432 -a -q -f /home/jobs/Desktop/resources/postgresql.sql
Parameter explanations:
-h PostgreSQL server IP address
-d database name
-U user name
-p port which PostgreSQL server is listening on
-f path to SQL script
-a all echo
-q quiet
You can open a command prompt and run as administrator. Then type
../bin>psql -f c:/...-h localhost -p 5432 -d databasename -U "postgres"
Password for user postgres: will show up.
Type your password and enter. I couldn't see the password what I was typing, but this time when I press enter it worked. Actually I was loading data into the database.
I achived that wrote (located in the directory where my script is)
::someguy#host::$sudo -u user psql -d my_database -a -f file.sql
where -u user is the role who owns the database where I want to execute the script then the psql connects to the psql console after that -d my_database loads me in mydatabase finally -a -f file.sql where -a echo all input from the script and -f execute commands from file.sql into mydatabase, then exit.
I'm using:
psql (PostgreSQL) 10.12
on (Ubuntu 10.12-0ubuntu0.18.04.1)
A small improvement in #wingman__7 's 2021 answer: if your username contains certain characters (an underscore in my case), you need to pass it with the -U flag.
This worked for me:
$ psql -h db.host -d db_name -U my_user < query.sql
Try using the following command in the command line console:
psql -h localhost -U postgres -f restore.sql