I'm trying to dump a huge database and compress the dump in order to not have to wait hours till it's done.
I dump the database the following way:
pg_dump -Fc -U -v | gzip > db$(date +%d-%m-%y_%H-%M).tar.gz
This leaves me with a compressed tar file.
I know want to unzip it in order to have a .tar file only:
tar -xvf xxx.tar.gz
This leaves me with an error message saying This does not look like a tar archive file
My goal is to then import it via psql.
I do not see what I am doing wrong – according to the Postgres documentation on dumps, I can use -Fc to dump in any wanted format? Thank you
It is not a tar archive, even if you named it like that, so you cannot use tar to unpack it.
To restore, you could use
zcat xxx.tar.gz | pg_restore -d target_database -U dbuser
By the way, it is pretty pointless to compress a custom format dump, since it is already compressed. You could choose better compression during pg_dump with the -Z 9 option.
I have downloaded the dvd rental.tar file on my desktop. followed the below instructions
pwd
/Users/O2/tools/postgres
mkdir dvdrentaldb
initdb dvdrentaldb
I get the following message
initdb directory "dvdrentaldb' exists but is not empty If you want to create a new database system either remove or empty the directory or run initdb with an argument other than "dvdrentaldb"
I have downloaded the dvd rental.tar file on my desktop. followed the below instructions
pwd
/Users/O2/tools/postgres
mkdir dvdrentaldb
initdb dvdrentaldb
post this i am using the following code
pg_restore -U O2 -d dvdrentaldb dvdrental.tar
i get syntax error at or near "pg_restore"
I get the following message
initdb directory "dvdrentaldb' exists but is not empty If you want to create a new database system either remove or empty the directory or run initdb with an argument other than "dvdrentaldb"
Got this from reddit
https://www.reddit.com/r/PostgreSQL/comments/93uuhg/can_anyone_get_this_sample_database_tutorial_to/
createuser -s dvdrental
createdb dvdrental -O dvdrental
pg_restore -U dvdrental -d dvdrental ./dvdrental.tar
first off im relatively new to pgadmin and postgresql
so i have a backup done with pgadmin in plain text which dumps the data i need from one table into a file called lodging. it came from a plans_to_lodge table from our live server. now to test a new web application front end i want to pg_restore it but this is what i get
> pg_restore -h localhost -d TestingDatabase2 -t lodging_of_plans -n plan_authentication "C:\lodging"
pg_restore: [archiver] could not open input file "C:\lodging": No such file or directory
the documentation doesnt give an example for a table restore
and yes the file is at that location in the C drive
The issue was it being the root folder, so when i put the file named lodging into another folder like so
C:\folder\lodging
it worked now onto more errors
During preparing database according to How to import OpenStreetMap data into PostgreSQL i always get error message C:/Program: No such file or directory after writing psql -U postgres -d gis -f PATH_TO_POSTGRES/share/contrib/postgis-1.5/postgis.sql In my case path looks like : C:\Program Files\PostgreSQL\9.3\share\contrib\postgis-2.1\postgis.sql
Thank you for help or advices
psql -U postgres -d gis
-f "C:\Program Files\PostgreSQL\9.3\share\contrib\postgis-2.1\postgis.sql"
Here Program Files is separated with space so you need to add double quote around the path.
We're working on a website, and when we develop locally (one of us from Windows), we use sqlite3, but on the server (linux) we use postgres. We'd like to be able to import the production database into our development process, so I'm wondering if there is a way to convert from a postgres database dump to something sqlite3 can understand (just feeding it the postgres's dumped SQL gave many, many errors). Or would it be easier just to install postgres on windows? Thanks.
I found this blog entry which guides you to do these steps:
Create a dump of the PostgreSQL database.
ssh -C username#hostname.com pg_dump --data-only --inserts YOUR_DB_NAME > dump.sql
Remove/modify the dump.
Remove the lines starting with SET
Remove the lines starting with SELECT pg_catalog.setval
Replace true for ‘t’
Replace false for ‘f’
Add BEGIN; as first line and END; as last line
Recreate an empty development database. bundle exec rake db:migrate
Import the dump.
sqlite3 db/development.sqlite3
sqlite> delete from schema_migrations;
sqlite> .read dump.sql
Of course connecting via ssh and creating a new db using rake are optional
STEP1: make a dump of your database structure and data
pg_dump --create --inserts -f myPgDump.sql \
-d myDatabaseName -U myUserName -W myPassword
STEP2: delete everything except CREATE TABLES and INSERT statements out of myPgDump.sql (using text editor)
STEP3: initialize your SQLite database passing structure and data of your Postgres dump
sqlite3 myNewSQLiteDB.db -init myPgDump.sql
STEP4: use your database ;)
Taken from https://stackoverflow.com/a/31521432/1680728 (upvote there):
The sequel gem makes this a very relaxing procedure:
First install Ruby, then install the gem by running gem install sequel.
In case of sqlite, it would be like this: sequel -C postgres://user#localhost/db sqlite://db/production.sqlite3
Credits to #lulalala .
You can use pg2sqlite for converting pg_dump output to sqlite.
# Making dump
pg_dump -h host -U user -f database.dump database
# Making sqlite database
pg2sqlite -d database.dump -o sqlite.db
Schemas is not supported by pg2sqlite, and if you dump contains schema then you need to remove it. You can use this script:
# sed 's/<schema name>\.//' -i database.dump
sed 's/public\.//' -i database.dump
pg2sqlite -d database.dump -o sqlite.db
Even though there are many very good helpful answers here, I just want to mark this as answered. We ended up going with the advice of the comments:
I'd just switch your development environment to PostgreSQL, developing on top of one database (especially one as loose and forgiving as SQLite) but deploying on another (especially one as strict as PostgreSQL) is generally a recipe for aggravation and swearing. –
#mu is too short
To echo mu's response, DON'T DO THIS..DON'T DO THIS..DON'T DO THIS. Develop and deploy on the same thing. It's bad engineering practice to do otherwise. – #Kuberchaun
So we just installed postgres on our dev machines. It was easy to get going and worked very smoothly.
In case one needs a more automatized solution, here's a head start:
#!/bin/bash
$table_name=TABLENAMEHERE
PGPASSWORD="PASSWORD" /usr/bin/pg_dump --file "results_dump.sql" --host "yourhost.com" --username "username" --no-password --verbose --format=p --create --clean --disable-dollar-quoting --inserts --column-inserts --table "public.${table_name}" "memseq"
# Some clean ups
perl -0777 -i.original -pe "s/.+?(INSERT)/\1/is" results_dump.sql
perl -0777 -i.original -pe "s/--.+//is" results_dump.sql
# Remove public. prefix from table name
sed -i "s/public.${table_name}/${table_name}/g" results_dump.sql
# fix binary blobs
sed -i "s/'\\\\x/x'/g" results_dump.sql
# use transactions to make it faster
echo 'BEGIN;' | cat - results_dump.sql > temp && mv temp results_dump.sql
echo 'END;' >> results_dump.sql
# clean the current table
sqlite3 results.sqlite "DELETE FROM ${table_name};"
# finally apply changes
sqlite3 results.sqlite3 < results_dump.sql && \
rm results_dump.sql && \
rm results_dump.sql.original
when I faced with same issue I did not find any useful advices on Internet. My source PostgreSQL db had very complicated schema.
You just need to remove from your db-file manually everything besides table creating
More details - here
It was VERY easy for me to do using the taps gem as described here:
http://railscasts.com/episodes/342-migrating-to-postgresql
And I've started using the Postgres.app on my Mac (no install needed, drop the app in your Applications directory, although might have to add one line to your PATH envirnment variable as described in the documentation), with Induction.app as a GUI tool to view/query the database.