System is Debian 11.2 with PostgreSQL 11.5.
I created a database and table as below:
CREATE DATABASE dbname OWNER=postgres
ENCODING= 'UTF8'
\c dbname
CREATE TABLE test(
id serial primary key,
site varchar(100) NOT NULL,
username char(30) NOT NULL,
password char(300) NOT NULL,
note varchar(200) DEFAULT NULL
);
Create bash file as below:
#!/bin/bash
res_user='me'
db_user='postgres'
db_name='dbname'
table_name='test'
sym_key='key'
#insert 4 columns
su $db_user <<EOFU
psql -d "$db_name" -U "$db_user" << EOF
INSERT INTO $table_name (site,username,password,note) VALUES ('v4','u3',pgp_sym_encrypt('password','key','cipher-algo=aes128,compress-algo=0,convert-crlf=1,sess-key=0,s2k-mode=3'),'note3');
EOF
EOFU
#column note has no output
password_arr=($(su $db_user <<EOFU
psql -tAq --field-separator= -d "$db_name" -U "$db_user" << EOF
SELECT "username",pgp_sym_decrypt(password::bytea,'key'),"note" FROM "$table_name" WHERE "site" LIKE '%v4%';
EOF
EOFU
))
echo "${password_arr[1]}" #output is passwordnote3
echo "${password_arr[2]}" #no ouput?
The expect output is:
${password_arr[1]} is `password`
${password_arr[2]} is `note3`
Run above bash script, but output "${password_arr[2]}" has no value,"${password_arr[1]}" is passwordnote3. Where is the problem?
I found the issue. The problem is you specified --field-separator to "nothing" instead of a space. It should be --field-separator=" ". This allowed the output of pgp_sym_decrypt() to concatenate with note. The username field however always had spaces probably since it has a fixed width of 30.
I also suggest that you reduce the number of row outputs to 1, and also enable "noglob" option when you're relying on word splitting. This can be done with set -f. You can also use read to get the needed fields. See How to split a string into an array in Bash?.
I'm trying to write a bash script that will create a Postgres database, as well as the user and the user privileges to access that database. I'm using Postgres 9.6. I have the below ...
create_db_command="SELECT 'CREATE DATABASE $DB_NAME' WHERE NOT EXISTS (SELECT FROM pg_database WHERE datname = '$DB_NAME')\gexec"
drop_owned_by_command="DROP OWNED BY $DB_USER;"
drop_role_command="DROP ROLE IF EXISTS $DB_USER;"
create_user_command="create user $DB_USER with encrypted password '$DB_PASS';"
grant_privs_command="grant all privileges on database $DB_NAME to $DB_USER;"
PGPASSWORD=$ROOT_PASSWORD
# This command creates the db if it doesn't already exist
echo "SELECT 'CREATE DATABASE $DB_NAME' WHERE NOT EXISTS (SELECT FROM pg_database WHERE datname = '$DB_NAME')\gexec" | psql -U$PG_USER
psql -U$PG_USER $DB_NAME -c "$drop_owned_by_command"
psql -U$PG_USER -c "$drop_role_command"
psql -U$PG_USER -c "$create_user_command"
psql -U$PG_USER -c "$grant_privs_command"
The problem is when the script is run the very first time, the command
DROP OWNED BY $DB_USER;
fails because the user does not yet exist. Is there a way to write the above command so that it will only run if the user exists? Similar to DROP USER IF EXISTS ..., but DROP OWNED has no IF EXISTS clause.
You can use a similar technique like you already have for CREATE DATABASE.
In the shell:
drop_owned_by_command="SELECT 'DROP OWNED BY $DB_USER' FROM pg_roles WHERE rolname = '$DB_USER'\gexec"
echo $drop_owned_by_command | psql -U$PG_USER $DB_NAME
The SELECT only returns a row (containing the DDL command) if the role a actually exists. This is in turn executed by the psql command \gexec.
So we have a combination of SQL and psql commands and cannot use psql -c since, quoting the manual on --command:
command must be either a command string that is completely parsable by the server (i.e., it contains no psql-specific features),
or a single backslash command. Thus you cannot mix SQL and psql
meta-commands within a -c option.
Instead, pipe the echo to psql like demonstrated - and like suggested in the manual and in my related answer below and like you already do for CREATE DATABASE.
Related:
Simulate CREATE DATABASE IF NOT EXISTS for PostgreSQL?
Shell script to execute pgsql commands in files
When I do this command :
SELECT language, title FROM cms_title WHERE language != 'en' AND title != 'Blog';
I see the pages I want to copy from a DB on the development server to the production server.
So, my aim is to dump those specific pages and then insert them in the production DB.
My question is :
Is it possible ? Is it the best practice / way of doing it ?
Thank you so much in advance for the time spent on my request.
I do it the following way:
1.create dump:
psql -h dbhost1 -d dbname -U dbuser -c "copy(SELECT language, title FROM cms_title WHERE language != 'en' AND title != 'Blog') to stdout" > dump.tsv`
2.import dump:
psql -h dbhost2 -d dbname -U dbuser -c 'copy cms_title from stdin' < dump.tsv
This will append imported data to the table.
I can see 4 approaches:
1. export that columns as .csv and import from it on seccond DB
COPY (
SELECT language, title FROM cms_title WHERE language != 'en' AND title != 'Blog'
) TO '/path/to/csv/cms_title_dump.csv' WITH CSV HEADER DELIMITER ';';
put that .csv in path accessible for destination server on that server perform importing:
COPY cms_title FROM '/path/to/csv/cms_title_dump.csv' DELIMITER ';' CSV;
or \copy to use the command in psql + db
2. dump it using pg_dump and restore (pg_restore) on seccond DB
dump:
pg_dump opencare -F c -h db_address -U db_user -p sb_port -t cms_title -f /path/to/dump/cms_title.dump;
restore:
pg_restore -h dest_db_address -U dest_db_user -p dest_dp_port -d db_name /path/to/dump/cms_title.dump;
In this case you must remember that pg_dump performs dump of WHOLE table data. You can't choose subset of rows from table to dump. So this option is appropiate when doing table mirroring from ona DB to another.
3. use dblink module for PostgreSQL to perform coss-database queries
4. generate INSERT query and execute it on seccond server:
`SELECT
'INSERT INTO cms_title(language, title) VALUES '
|| string_agg( '(' || language || ', '|| title || ')', ',' )
|| ';'
FROM cms_title WHERE language != 'en' AND title != 'Blog';`
I usually doing like this For Import specific column dump
Create cms_title_dump.csv file.
touch /home/www/html/cms_title_dump.csv
give the write permission
chmoc 777 /home/www/html/cms_title_dump.csv
execute this query
COPY (SELECT language, title, FROM cms_title WHERE language != 'en' AND title != 'Blog' ) to '/home/www/html/cms_title_dump.csv' DELIMITER ',' CSV HEADER
change the file permission. as Read only
I have a bash script that opens a file and executes a bunch of psql queries.
I want these queries to be echoed/print as and when they run.
How do I do the same ?
I have tried using \echo for inserts & inside stored procedures too, but it doesn't seem to work. How do I do it ?
Use psql --echo-all.
$ psql --echo-all -c "SELECT 1;"
SELECT 1;
?column?
----------
1
(1 row)
The only way that I know, that you can echo anything during the execution of a PostgreSQL function (named stored procedure), is with raise. This command is used to trown exceptions, but you can throw a NOTICE level exception, that will not interfere on the function execution.
Maybe it is not exactly what you want, but is a good workaround. The way that PostgreSQL execute their procedures, don't allow runtime echos (like Sybase ou Ms SQL Server). See this examples (It will only work inside functions):
raise notice 'Some message';
It will output:
NOTICE: Some message
Or passing vars to the debug:
raise notice 'Inserting '%' in '%'.',var_value,var_table;
When var_table = 'customers' and var_value = 'Joe Doe', it will output:
NOTICE: Inserting 'Joe Doe' in 'customers'
--echo-queries (for shell script)
PGPASS='passwd'
su -c "PGPASSWORD=${PGPASS} psql -d postgres --echo-queries -qc "\pset border 2;" -c "show data_directory;"" postgres
From the Postgres documentation page ( please note that the syntax of psql has remained largely unchanged over versions ), it is clearer with an example of a DDL.
There are several ways to echo. -e to echo just the queries only.
$ psql -ec "create table t1 ( c1 int ) " ;
create table t1 ( c1 int )
CREATE TABLE
If you do not want the "CREATE TABLE" message add a "-q" flag as well
$ psql -eqc "create table t1 ( c1 int ) " ;
create table t1 ( c1 int )
You can add header and footer to your script file:
\set origin_ECHO :ECHO
\set ECHO all
--****** YOUR SCRIPT TEXT *****
--.........
--*****************************
\set ECHO :origin_ECHO
What's the correct way to copy entire database (its structure and data) to a new one in pgAdmin?
Postgres allows the use of any existing database on the server as a template when creating a new database. I'm not sure whether pgAdmin gives you the option on the create database dialog but you should be able to execute the following in a query window if it doesn't:
CREATE DATABASE newdb WITH TEMPLATE originaldb OWNER dbuser;
Still, you may get:
ERROR: source database "originaldb" is being accessed by other users
To disconnect all other users from the database, you can use this query:
SELECT pg_terminate_backend(pg_stat_activity.pid) FROM pg_stat_activity
WHERE pg_stat_activity.datname = 'originaldb' AND pid <> pg_backend_pid();
A command-line version of Bell's answer:
createdb -O ownername -T originaldb newdb
This should be run under the privileges of the database master, usually postgres.
To clone an existing database with postgres you can do that
/* KILL ALL EXISTING CONNECTION FROM ORIGINAL DB (sourcedb)*/
SELECT pg_terminate_backend(pg_stat_activity.pid) FROM pg_stat_activity
WHERE pg_stat_activity.datname = 'SOURCE_DB' AND pid <> pg_backend_pid();
/* CLONE DATABASE TO NEW ONE(TARGET_DB) */
CREATE DATABASE TARGET_DB WITH TEMPLATE SOURCE_DB OWNER USER_DB;
IT will kill all the connection to the source db avoiding the error
ERROR: source database "SOURCE_DB" is being accessed by other users
In production environment, where the original database is under traffic, I'm simply using:
pg_dump production-db | psql test-db
Don't know about pgAdmin, but pgdump gives you a dump of the database in SQL. You only need to create a database by the same name and do
psql mydatabase < my dump
to restore all of the tables and their data and all access privileges.
First, sudo as the database user:
sudo su postgres
Go to PostgreSQL command line:
psql
Create the new database, give the rights and exit:
CREATE DATABASE new_database_name;
GRANT ALL PRIVILEGES ON DATABASE new_database_name TO my_user;
\d
Copy structure and data from the old database to the new one:
pg_dump old_database_name | psql new_database_name
In pgAdmin you can make a backup from your original database, and then just create a new database and restore from the backup just created:
Right click the source database, Backup... and dump to a file.
Right click, New Object, New Database... and name the destination.
Right click the new database, Restore... and select your file.
Copying an "under load" db
I pieced this approach together with the examples from above. I'm working on an "under load" server and got the error when I attempted the approach from #zbyszek. I also was after a "command line only" solution.
createdb: database creation failed: ERROR: source database "exampledb" is being accessed by other users.
Here's what worked for me (Commands prepended with nohup to move output into a file and protect from a server disconnect):
nohup pg_dump exampledb > example-01.sql
createdb -O postgres exampledbclone_01
my user is "postgres"
nohup psql exampledbclone_01 < example-01.sql
What's the correct way to copy entire database (its structure and data) to a new one in pgAdmin?
Answer:
CREATE DATABASE newdb WITH TEMPLATE originaldb;
Tried and tested.
Here's the whole process of creating a copying over a database using only pgadmin4 GUI (via backup and restore)
Postgres comes with Pgadmin4. If you use macOS you can press CMD+SPACE and type pgadmin4 to run it. This will open up a browser tab in chrome.
Steps for copying
1. Create the backup
Do this by rightclicking the database -> "backup"
2. Give the file a name.
Like test12345. Click backup. This creates a binary file dump, it's not in a .sql format
3. See where it downloaded
There should be a popup at the bottomright of your screen. Click the "more details" page to see where your backup downloaded to
4. Find the location of downloaded file
In this case, it's /users/vincenttang
5. Restore the backup from pgadmin
Assuming you did steps 1 to 4 correctly, you'll have a restore binary file. There might come a time your coworker wants to use your restore file on their local machine. Have said person go to pgadmin and restore
Do this by rightclicking the database -> "restore"
6. Select file finder
Make sure to select the file location manually, DO NOT drag and drop a file onto the uploader fields in pgadmin. Because you will run into error permissions. Instead, find the file you just created:
7. Find said file
You might have to change the filter at bottomright to "All files". Find the file thereafter, from step 4. Now hit the bottomright "Select" button to confirm
8. Restore said file
You'll see this page again, with the location of the file selected. Go ahead and restore it
9. Success
If all is good, the bottom right should popup an indicator showing a successful restore. You can navigate over to your tables to see if the data has been restored propery on each table.
10. If it wasn't successful:
Should step 9 fail, try deleting your old public schema on your database. Go to "Query Tool"
Execute this code block:
DROP SCHEMA public CASCADE; CREATE SCHEMA public;
Now try steps 5 to 9 again, it should work out
EDIT - Some additional notes. Update PGADMIN4 if you are getting an error during upload with something along the lines of "archiver header 1.14 unsupported version" during restore
From the documentation, using createdb or CREATE DATABASE with templates is not encouraged:
Although it is possible to copy a database other than template1 by
specifying its name as the template, this is not (yet) intended as a
general-purpose “COPY DATABASE” facility. The principal limitation is
that no other sessions can be connected to the template database while
it is being copied. CREATE DATABASE will fail if any other connection
exists when it starts; otherwise, new connections to the template
database are locked out until CREATE DATABASE completes.
pg_dump or pg_dumpall is a good way to go for copying database AND ALL THE DATA. If you are using a GUI like pgAdmin, these commands are called behind the scenes when you execute a backup command. Copying to a new database is done in two phases: Backup and Restore
pg_dumpall saves all of the databases on the PostgreSQL cluster. The disadvantage to this approach is that you end up with a potentially very large text file full of SQL required to create the database and populate the data. The advantage of this approach is that you get all of the roles (permissions) for the cluster for free. To dump all databases do this from the superuser account
pg_dumpall > db.out
and to restore
psql -f db.out postgres
pg_dump has some compression options that give you much smaller files. I have a production database I backup twice a day with a cron job using
pg_dump --create --format=custom --compress=5 --file=db.dump mydatabase
where compress is the compression level (0 to 9) and create tells pg_dump to add commands to create the database. Restore (or move to new cluster) by using
pg_restore -d newdb db.dump
where newdb is the name of the database you want to use.
Other things to think about
PostgreSQL uses ROLES for managing permissions. These are not copied by pg_dump. Also, we have not dealt with the settings in postgresql.conf and pg_hba.conf (if you're moving the database to another server). You'll have to figure out the conf settings on your own. But there is a trick I just discovered for backing up roles. Roles are managed at the cluster level and you can ask pg_dumpall to backup just the roles with the --roles-only command line switch.
For those still interested, I have come up with a bash script that does (more or less) what the author wanted. I had to make a daily business database copy on a production system, this script seems to do the trick. Remember to change the database name/user/pw values.
#!/bin/bash
if [ 1 -ne $# ]
then
echo "Usage `basename $0` {tar.gz database file}"
exit 65;
fi
if [ -f "$1" ]
then
EXTRACTED=`tar -xzvf $1`
echo "using database archive: $EXTRACTED";
else
echo "file $1 does not exist"
exit 1
fi
PGUSER=dbuser
PGPASSWORD=dbpw
export PGUSER PGPASSWORD
datestr=`date +%Y%m%d`
dbname="dbcpy_$datestr"
createdbcmd="CREATE DATABASE $dbname WITH OWNER = postgres ENCODING = 'UTF8' TABLESPACE = pg_default LC_COLLATE = 'en_US.UTF-8' LC_CTYPE = 'en_US.UTF-8' CONNECTION LIMIT = -1;"
dropdbcmp="DROP DATABASE $dbname"
echo "creating database $dbname"
psql -c "$createdbcmd"
rc=$?
if [[ $rc != 0 ]] ; then
rm -rf "$EXTRACTED"
echo "error occured while creating database $dbname ($rc)"
exit $rc
fi
echo "loading data into database"
psql $dbname < $EXTRACTED > /dev/null
rc=$?
rm -rf "$EXTRACTED"
if [[ $rc != 0 ]] ; then
psql -c "$dropdbcmd"
echo "error occured while loading data to database $dbname ($rc)"
exit $rc
fi
echo "finished OK"
PostgreSQL 9.1.2:
$ CREATEDB new_db_name -T orig_db_name -O db_user;
To create database dump
cd /var/lib/pgsql/
pg_dump database_name> database_name.out
To resote database dump
psql -d template1
CREATE DATABASE database_name WITH ENCODING 'UTF8' LC_CTYPE 'en_US.UTF-8' LC_COLLATE 'en_US.UTF-8' TEMPLATE template0;
CREATE USER role_name WITH PASSWORD 'password';
ALTER DATABASE database_name OWNER TO role_name;
ALTER USER role_name CREATEDB;
GRANT ALL PRIVILEGES ON DATABASE database_name to role_name;
CTR+D(logout from pgsql console)
cd /var/lib/pgsql/
psql -d database_name -f database_name.out
If the database has open connections, this script may help. I use this to create a test database from a backup of the live-production database every night. This assumes that you have an .SQL backup file from the production db (I do this within webmin).
#!/bin/sh
dbname="desired_db_name_of_test_enviroment"
username="user_name"
fname="/path to /ExistingBackupFileOfLive.sql"
dropdbcmp="DROP DATABASE $dbname"
createdbcmd="CREATE DATABASE $dbname WITH OWNER = $username "
export PGPASSWORD=MyPassword
echo "**********"
echo "** Dropping $dbname"
psql -d postgres -h localhost -U "$username" -c "$dropdbcmp"
echo "**********"
echo "** Creating database $dbname"
psql -d postgres -h localhost -U "$username" -c "$createdbcmd"
echo "**********"
echo "** Loading data into database"
psql -d postgres -h localhost -U "$username" -d "$dbname" -a -f "$fname"
Using pgAdmin, disconnect the database that you want to use as a template. Then you select it as the template to create the new database, this avoids getting the already in use error.
pgAdmin4:
1.Select DB you want to copy and disconnect it
Rightclick
"Disconnect DB"
2.Create a new db next to the old one:
Give it a name.
In the "definition" tab select
the first table as an Template (dropdown menu)
Hit create and just left click on the new db to reconnect.
If you want to copy whole schema you can make a pg_dump with following command:
pg_dump -h database.host.com -d database_name -n schema_name -U database_user --password
And when you want to import that dump, you can use:
psql "host=database.host.com user=database_user password=database_password dbname=database_name options=--search_path=schema_name" -f sql_dump_to_import.sql
More info about connection strings: https://www.postgresql.org/docs/current/libpq-connect.html#LIBPQ-CONNSTRING
Or then just combining it in one liner:
pg_dump -h database.host.com -d postgres -n schema_name -U database_user --password | psql "host=database.host.com user=database_user password=database_password dbname=database_name options=--search_path=schema_name”
Open the Main Window in pgAdmin and then open another Query Tools Window
In the main windows in pgAdmin,
Disconnect the "templated" database that you want to use as a template.
Goto the Query Tools Window
Run 2 queries as below
SELECT pg_terminate_backend(pg_stat_activity.pid)
FROM pg_stat_activity
WHERE pg_stat_activity.datname = 'TemplateDB' AND pid <> pg_backend_pid();
(The above SQL statement will terminate all active sessions with TemplateDB and then you can now select it as the template to create the new TargetDB database, this avoids getting the already in use error.)
CREATE DATABASE 'TargetDB'
WITH TEMPLATE='TemplateDB'
CONNECTION LIMIT=-1;
New versions of pgAdmin (definitely 4.30) support creating new databases from template. All you need to populate are new database name and existing template database.
CREATE DATABASE newdb WITH TEMPLATE originaldb OWNER dbuser;
If you have using Ubuntu.
1 way
createdb -O Owner -T old_db_name new_db_name
2 way
createdb test_copy
pg_dump old_db_name | psql test_copy
Try this:
CREATE DATABASE newdb WITH ENCODING='UTF8' OWNER=owner TEMPLATE=templatedb LC_COLLATE='en_US.UTF-8' LC_CTYPE='en_US.UTF-8' CONNECTION LIMIT=-1;
gl XD