I am trying to write a script to import a database schema from a remote machine that only accepts ssh connections to a local one.
I managed to do anything except keep the same encoding has the remote database.
I found out that the solution was using pg_dump with -C (create) and that way I would be able to create the database with the same encoding but I faced a problem... there is a table space in the remote database and I dont want to import it.
I know that recent versions of psql already have the no-tablespace argument... but unlucky me, I'm not allowed to upgrade the postgres version.
Could someone tell me a way to remove all the tablespace ocurrences on a sql dump? like with sed or something.
Thanks a lot!
I used to switch tablespaces between installations by piping pg_dump through sed where I altered the TABLESPACE clause.
You can also just remove it and additionally remove CREATE TABLESPACE ... from the dump file with any editor and you are good to load it to another DB cluster.
I long since moved on to newer versions where I can use the --no-tablespaces option. Depending on your setup, a shell command could look something like this in Linux - from the top of my head, only tested cursory:
pgdump -h 123.456.7.89 -p 5432 mydb \
| sed \
-e' /^CREATE TABLESPACE / d' \
-e 's/ *TABLESPACE .*;/;/' \
-e "s/SET default_tablespace = .*;/SET default_tablespace = '';/"
| psql -p5432 mylocaldb
-e' /^CREATE TABLESPACE / d' ... delete lines beginning with "CREATE TABLESPACE ".
-e 's/ *TABLESPACE .*;/;/' ... trim the tablespace clause (always at the end of the line in pg_dump output) from CREATE TABLE or CREATE INDEX statements.
-e "s/SET default_tablespace = .*;/SET default_tablespace = '';" .. do away with any other default tablespace than the empty string - which signifies the default tablespace of the current db. Note the use of double quote ", so I can easily enter single quotes '.
If you know the name of the tablespace involved you can narrow this down. There is a theoretical possibility that a data line could start like one of the search terms. I have never encountered problems myself, though.
Check out a page like this for more info on sed.
Related
I try to restore a large table
pg_restore.exe -U postgres -d db_name --clean --if-exists --single-transaction F:\Backups\PostgreSQL\data.dump.gz
So I have a read lock for a few minutes. How to restore data with zero downtime for reading? I need only reading.
You would need to not do the --clean and instead do --data-only, but then do a DELETE from tablename inside the same transaction, before the COPY. I don't think there is a way to make pg_restore do this for you, but you could dump the output of pg_restore to a file and edit it, or use something like sed or perl to inject the DELETE.
This should work for table names which don't need to be quoted, and assuming none of the data being copied has first column which starts with 'COPY ':
pg_restore --data-only --single-transaction dmp.dmp -f -| perl -pe 's/^COPY ([\w.]+)/delete from $1; copy $1/' | psql -U postgres -d db_name
However, your schema changing method doesn't seem so dirty to me. It still requires a momentary access exclusive lock, so it isn't really zero downtime, but it might be unnoticable downtime if it can acquire said lock quickly enough.
I'm using a windows batch file to connect to postgres using psql. I'm issuing commands like this....
SET PGPASSWORD=postgres
psql -U postgres -d postgres -c "DROP USER IF EXISTS foo;"
This works fine for running one, short SQL command against the database. But I'm having trouble with two related issues
How to continue a single long SQL command over multiple lines, and
How to run multiple commands.
Example 1.....
psql -U postgres -d postgres -c "CREATE DATABASE foo
WITH OWNER = bar
ENCODING = 'UTF8'
TABLESPACE = mytabspace;"
Example 2.....
psql -U postgres -d postgres -c "
ALTER TABLE one ALTER COLUMN X TYPE INTEGER;
ALTER TABLE two ALTER COLUMN Y TYPE INTEGER;"
Neither of these will work as shown, I've done some googling and found some suggestions for doing this with linux, and have experimented with various carats, backslashes and underscores, but just don't seem to be able to split the commands across lines.
I'm aware of the -f option to run a file, but I'm trying to avoid that.
Any suggestions?
The line continuation character in batch is the ^. See this Q&A
So end the line with space+caret ^ and make sure the following line begins with a space.
You will also have to escape double quoted areas that span several lines with a caret for this to work.
Since the line is unquoted then for the batch parser you will also have to escape any special chararacters like <>|& also with a caret.
psql -U postgres -d postgres -c ^"CREATE DATABASE foo ^
WITH OWNER = bar ^
ENCODING = 'UTF8' ^
TABLESPACE = mytabspace;"
psql -U postgres -d postgres -c ^" ^
ALTER TABLE one ALTER COLUMN X TYPE INTEGER; ^
ALTER TABLE two ALTER COLUMN Y TYPE INTEGER;"
I run scripts against my database like this...
psql -d myDataBase -a -f myInsertFile.sql
The only problem is I want to be able to specify in this command what schema to run the script against. I could call set search_path='my_schema_01' but the files are supposed to be portable. How can I do this?
You can create one file that contains the set schema ... statement and then include the actual file you want to run:
Create a file run_insert.sql:
set schema 'my_schema_01';
\i myInsertFile.sql
Then call this using:
psql -d myDataBase -a -f run_insert.sql
More universal way is to set search_path (should work in PostgreSQL 7.x and above):
SET search_path TO myschema;
Note that set schema myschema is an alias to above command that is not available in 8.x.
See also: http://www.postgresql.org/docs/9.3/static/ddl-schemas.html
Main Example
The example below will run myfile.sql on database mydatabase using schema myschema.
psql "dbname=mydatabase options=--search_path=myschema" -a -f myfile.sql
The way this works is the first argument to the psql command is the dbname argument. The docs mention a connection string can be provided.
If this parameter contains an = sign or starts with a valid URI prefix
(postgresql:// or postgres://), it is treated as a conninfo string
The dbname keyword specifies the database to connect to and the options keyword lets you specify command-line options to send to the server at connection startup. Those options are detailed in the server configuration chapter. The option we are using to select the schema is search_path.
Another Example
The example below will connect to host myhost on database mydatabase using schema myschema. The = special character must be url escaped with the escape sequence %3D.
psql postgres://myuser#myhost?options=--search_path%3Dmyschema
The PGOPTIONS environment variable may be used to achieve this in a flexible way.
In an Unix shell:
PGOPTIONS="--search_path=my_schema_01" psql -d myDataBase -a -f myInsertFile.sql
If there are several invocations in the script or sub-shells that need the same options, it's simpler to set PGOPTIONS only once and export it.
PGOPTIONS="--search_path=my_schema_01"
export PGOPTIONS
psql -d somebase
psql -d someotherbase
...
or invoke the top-level shell script with PGOPTIONS set from the outside
PGOPTIONS="--search_path=my_schema_01" ./my-upgrade-script.sh
In Windows CMD environment, set PGOPTIONS=value should work the same.
I'm using something like this and works very well:* :-)
(echo "set schema 'acme';" ; \
cat ~/git/soluvas-framework/schedule/src/main/resources/org/soluvas/schedule/tables_postgres.sql) \
| psql -Upostgres -hlocalhost quikdo_app_dev
Note: Linux/Mac/Bash only, though probably there's a way to do that in Windows/PowerShell too.
This works for me:
psql postgresql://myuser:password#myhost/my_db -f myInsertFile.sql
In my case, I wanted to add schema to a file dynamically so that whatever schema name user will provide from the cli, I will run sql file with that provided schema name.
For this, I replaced some text in the sql file. First I added {{schema}} in the file like this
CREATE OR REPLACE FUNCTION {{schema}}.usp_dailygaintablereportdata(
then replace {{schema}} dynamically with user provided schema name with the help of sed command
sed -i "s/{{schema}}/$pgSchemaName/" $filename
result=$(psql -U $user -h $host -p $port -d $dbName -f "$filename" 2>&1)
sed -i "s/$pgSchemaName/{{schema}}/" $filename
First replace is done, then target file is run and then again our replace is reverted back
I was facing similar problems trying to do some dat import on an intermediate schema (that later we move on to the final one). As we rely on things like extensions (for example PostGIS), the "run_insert" sql file did not fully solved the problem.
After a while, we've found that at least with Postgres 9.3 the solution is far easier... just create your SQL script always specifying the schema when refering to the table:
CREATE TABLE "my_schema"."my_table" (...);
COPY "my_schema"."my_table" (...) FROM stdin;
This way using psql -f xxxxx works perfectly, and you don't need to change search_paths nor use intermediate files (and won't hit extension schema problems).
We're working on a website, and when we develop locally (one of us from Windows), we use sqlite3, but on the server (linux) we use postgres. We'd like to be able to import the production database into our development process, so I'm wondering if there is a way to convert from a postgres database dump to something sqlite3 can understand (just feeding it the postgres's dumped SQL gave many, many errors). Or would it be easier just to install postgres on windows? Thanks.
I found this blog entry which guides you to do these steps:
Create a dump of the PostgreSQL database.
ssh -C username#hostname.com pg_dump --data-only --inserts YOUR_DB_NAME > dump.sql
Remove/modify the dump.
Remove the lines starting with SET
Remove the lines starting with SELECT pg_catalog.setval
Replace true for ‘t’
Replace false for ‘f’
Add BEGIN; as first line and END; as last line
Recreate an empty development database. bundle exec rake db:migrate
Import the dump.
sqlite3 db/development.sqlite3
sqlite> delete from schema_migrations;
sqlite> .read dump.sql
Of course connecting via ssh and creating a new db using rake are optional
STEP1: make a dump of your database structure and data
pg_dump --create --inserts -f myPgDump.sql \
-d myDatabaseName -U myUserName -W myPassword
STEP2: delete everything except CREATE TABLES and INSERT statements out of myPgDump.sql (using text editor)
STEP3: initialize your SQLite database passing structure and data of your Postgres dump
sqlite3 myNewSQLiteDB.db -init myPgDump.sql
STEP4: use your database ;)
Taken from https://stackoverflow.com/a/31521432/1680728 (upvote there):
The sequel gem makes this a very relaxing procedure:
First install Ruby, then install the gem by running gem install sequel.
In case of sqlite, it would be like this: sequel -C postgres://user#localhost/db sqlite://db/production.sqlite3
Credits to #lulalala .
You can use pg2sqlite for converting pg_dump output to sqlite.
# Making dump
pg_dump -h host -U user -f database.dump database
# Making sqlite database
pg2sqlite -d database.dump -o sqlite.db
Schemas is not supported by pg2sqlite, and if you dump contains schema then you need to remove it. You can use this script:
# sed 's/<schema name>\.//' -i database.dump
sed 's/public\.//' -i database.dump
pg2sqlite -d database.dump -o sqlite.db
Even though there are many very good helpful answers here, I just want to mark this as answered. We ended up going with the advice of the comments:
I'd just switch your development environment to PostgreSQL, developing on top of one database (especially one as loose and forgiving as SQLite) but deploying on another (especially one as strict as PostgreSQL) is generally a recipe for aggravation and swearing. –
#mu is too short
To echo mu's response, DON'T DO THIS..DON'T DO THIS..DON'T DO THIS. Develop and deploy on the same thing. It's bad engineering practice to do otherwise. – #Kuberchaun
So we just installed postgres on our dev machines. It was easy to get going and worked very smoothly.
In case one needs a more automatized solution, here's a head start:
#!/bin/bash
$table_name=TABLENAMEHERE
PGPASSWORD="PASSWORD" /usr/bin/pg_dump --file "results_dump.sql" --host "yourhost.com" --username "username" --no-password --verbose --format=p --create --clean --disable-dollar-quoting --inserts --column-inserts --table "public.${table_name}" "memseq"
# Some clean ups
perl -0777 -i.original -pe "s/.+?(INSERT)/\1/is" results_dump.sql
perl -0777 -i.original -pe "s/--.+//is" results_dump.sql
# Remove public. prefix from table name
sed -i "s/public.${table_name}/${table_name}/g" results_dump.sql
# fix binary blobs
sed -i "s/'\\\\x/x'/g" results_dump.sql
# use transactions to make it faster
echo 'BEGIN;' | cat - results_dump.sql > temp && mv temp results_dump.sql
echo 'END;' >> results_dump.sql
# clean the current table
sqlite3 results.sqlite "DELETE FROM ${table_name};"
# finally apply changes
sqlite3 results.sqlite3 < results_dump.sql && \
rm results_dump.sql && \
rm results_dump.sql.original
when I faced with same issue I did not find any useful advices on Internet. My source PostgreSQL db had very complicated schema.
You just need to remove from your db-file manually everything besides table creating
More details - here
It was VERY easy for me to do using the taps gem as described here:
http://railscasts.com/episodes/342-migrating-to-postgresql
And I've started using the Postgres.app on my Mac (no install needed, drop the app in your Applications directory, although might have to add one line to your PATH envirnment variable as described in the documentation), with Induction.app as a GUI tool to view/query the database.
I have taken a dump of a database named temp1, by using the follwing command
$ pg_dump -i -h localhost -U postgres -F c -b -v -f pub.backup temp1
Now I want to restore the dump in a different database called "db_temp" , but in that I just want that all the tables should be created in a "temp_schema" ( not the default schema which is in the fms temp1 database ) which is in the "db_temp" database.
Is there any way to do this using pg_restore command?
Any other method also be appreciated!
A quick and dirty way:
1) rename default schema:
alter schema public rename to public_save;
2) create new schema as default schema:
create schema public;
3) restore data
pg_restore -f pub.backup db_temp [and whatever other options]
4) rename schemas according to need:
alter schema public rename to temp_schema;
alter schema public_save rename to public;
There is a simple solution:
Create your backup dump in plain SQL format (format "p" using the parameter --format=p or -F p)
Edit your pub.backup.sql dump with your favorite editor and add the following two lines at the top of your file:
create schema myschema;
SET search_path TO myschema;
Now you can restore your backup dump with the command
psql -f pub.backup.sql
The set search_path to <schema> command will set myschema as the default, so that new tables and other objects are created in this schema, independently of the "default" schema where they lived before.
There's no way in pg_restore itself. What you can do is use pg_restore to generate SQL output, and then send this through for example a sed script to change it. You need to be careful about how you write that sed script though, so it doesn't match and change things inside your data.
Probably the easiest method would be to simply rename the schema after restore, ie with the following SQL:
ALTER SCHEMA my_schema RENAME TO temp_schema
I believe that because you're using the compressed archive format for the output of pg_dump you can't alter it before restoring. The option would be to use the default output and do a search and replace on the schema name, but that would be risky and could perhaps cause data to be corrupted if you were not careful.
If you only have a few tables then you can restore one table at a time, pg_restore accepts -d database when you specify -t tablename. Of course, you'll have to set up the schema before restoring the tables and then sort out the indexes and constraints when you're done restoring the tables.
Alternatively, set up another server on a different port, restore using the new PostgreSQL server, rename the schema, dump it, and restore into your original database. This is a bit of a kludge of course but it will get the job done.
If you're adventurous you might be able to change the database name in the dump file using a hex editor. I think it is only mentioned in one place in the dump and as long as the new and old database names are the same it should work. YMMV, don't do anything like this in a production environment, don't blame me if this blows up and levels your home town, and all the rest of the usual disclaimers.
Rename the schema in a temporary database.
Export the schema:
pg_dump --schema-only --schema=prod > prod.sql
Create a new database. Restore the export:
psql -f prod.sql
ALTER SCHEMA prod RENAME TO somethingelse;
pg_dump --schema-only --schema=somethingelse > somethingelse.sql
(delete the database)
For the data you can just modify the set search_path at the top.
As noted, there's no direct support in pg_dump, psql or pg_restore to change the schema name during a dump/restore process. But it's fairly straightforward to export using "plain" format then modify the .sql file. This Bash script does the basics:
rename_schema () {
# Change search path so by default everything will go into the specified schema
perl -pi -e "s/SET search_path = $2, pg_catalog/SET search_path = $3, pg_catalog, $2;/" "$1"
# Change 'ALTER FUNCTION foo.' to 'ALTER FUNCTION bar.'
perl -pi -e 's/^([A-Z]+ [A-Z]+) '$2'\./$1 '$3'./' "$1"
# Change the final GRANT ALL ON SCHEMA foo TO PUBLIC
perl -pi -e 's/SCHEMA '$2'/SCHEMA '$3'/' "$1"
}
Usage:
pg_dump --format plain --schema=foo --file dump.sql MYDB
rename_schema dump.sql foo bar
psql -d MYDB -c 'CREATE SCHEMA bar;'
psql -d MYDB -f dumpsql
The question is pretty old, but maybe can help some one.
Streaming the output of pg_restore to sed and replace the schema name in order to import the dump to a different schema.
Something like:
pg_restore ${dumpfile} | \
sed -e "s/OWNER TO ${source_owner}/OWNER TO ${target_owner}/" \
-e "s/${source_schema}/${target_schema}/" | \
psql -h ${pgserver} -d ${dbname} -U ${pguser}