Running 9.5 on Ubuntu 16.04
Unsure what database was backed up, I think 8.4
When I run pg_restore I get
pg_restore: implied data-only restore
--
-- PostgreSQL database dump
--
-- Started on 30608-10-13 11:53:01 MDT
SET statement_timeout = 0;
SET lock_timeout = 0;
SET client_encoding = 'SQL_ASCII';
SET standard_conforming_strings = off;
SET check_function_bodies = false;
SET client_min_messages = warning;
SET escape_string_warning = off;
SET row_security = off;
-- Completed on 2018-09-06 11:12:06 MDT
--
-- PostgreSQL database dump complete
--
When I run pg_restore -l
;
; Archive created at 30608-10-13 11:53:01 MDT
; dbname:
; TOC Entries: -1835365408
; Compression: -1
; Dump Version: 1.11-0
; Format: CUSTOM
; Integer: 4 bytes
; Offset: 8 bytes
;
;
; Selected TOC Entries:
;
Obviously there is a timestamp issue, and clearly the TOC entries and compression is off.
Not sure where to go from here. The file size indicates it should be a complete backup. I have multiple backups from same time frame, and they all report similar when I try to restore them.
Is there any way to uncompress the data portion of the file. I can hexedit and see the schema, but the data is in postgres's binary compressed (-Fc) format. I just need to find and verify a few entries, so if there's a manual way to inspect and search, that would work.
Any help is appreciated.
Yes, you could extract it just using -f parameter:
pg_restore -f /path/to/target/file /path/from/source/dump
More info here.
I couldn't see the timestamp issue in the output you pasted or I'm missing something.
Though, generally, if you would like to restore only few records, you may want to try -L of pg_restore in order to to reorder them and restore.
For example,
pg_restore -l dbdump > dblist
Edit dblist file (By comment them out or leave only those items you want to restore)
pg_restore -L dblist dbdump
pg_restore -d newdb dbdump
Related
I wish to pg_dump a specific schema from one database and pg_restore it into an already existing database without dropping it.
The command I have been using for the pg_dump is as follows:
pg_dump -n mySchema -Z 9 -b -f mySchema.sql.gz -F c -U ${db_user} -h ${db_host} ${db_name}
-F c generates a custom format file suitable for pg_restore.
I'm aware that if I included the --clean flag it would 'clean (drop) database objects prior to outputting the commands for creating them.' according to the documentation here.
I did not include this flag.
However, when I run head on the generated file, I can see the following within it: DROP DATABASE myDatabase. Why is it here? I'm afraid that if I do a pg_restore with this file that it will drop my existing database.
Here is what head returns:
PGDMP
ymyDatabase11.8"11.10 (Ubuntu 11.10-1.pgdg18.04+1)�0ENCODINENCODINGSET client_encoding = 'UTF8';
false�00
STDSTRINGS
STDSTRINGS(SET standard_conforming_strings = 'on';
false�00
SEARCHPATH
SEARCHPATH8SELECT pg_catalog.set_config('search_path', '', false);
false�126221356smarDATABASEwCREATE DATABASE myDatabase WITH TEMPLATE = template0 ENCODING = 'UTF8' LC_COLLATE = 'en_US.UTF-8' LC_CTYPE = 'en_US.UTF-8';
DROP DATABASE myDatabase;
myUserfalse�00DATABASE smartACL/GRANT CONNECT ON DATABASE myDatabase TO readaccess;
A custom format dump includes the DDL to drop the listed objects.
That is because with a custom format dump, you can specify the --clean option with pg_restore, that is, you don't need if you need the DROP statements until the dump is restored.
However, if you don't use --clean with pg_restore, the DROP statements won't be executed, so you don't have to worry.
I've try to load a dump to a new database and all seems to work :
user#vpsXXXX:~$ pg_dump -U user -d database < mydump.sql
--
-- PostgreSQL database dump
--
-- Dumped from database version 10.6 (Ubuntu 10.6-0ubuntu0.18.04.1)
-- Dumped by pg_dump version 10.6 (Ubuntu 10.6-0ubuntu0.18.04.1)
SET statement_timeout = 0;
SET lock_timeout = 0;
SET idle_in_transaction_session_timeout = 0;
SET client_encoding = 'UTF8';
SET standard_conforming_strings = on;
SELECT pg_catalog.set_config('search_path', '', false);
SET check_function_bodies = false;
SET client_min_messages = warning;
SET row_security = off;
--
-- Name: plpgsql; Type: EXTENSION; Schema: -; Owner:
--
CREATE EXTENSION IF NOT EXISTS plpgsql WITH SCHEMA pg_catalog;
--
-- Name: EXTENSION plpgsql; Type: COMMENT; Schema: -; Owner:
--
COMMENT ON EXTENSION plpgsql IS 'PL/pgSQL procedural language';
--
-- PostgreSQL database dump complete
--
When I look the tables on a software like Postico, there is no tables except the Postgres ones. My dump is complete when I look the SQL file.
Do you know a tip to know what happens ?
Thanks !
Dump and restore operations are best performed as postgres user. The easiest way to achieve this is to become the postgres UNIX user.
The initial command had the mistake of confusing pg_dump with psql.
I'd like to export the entire schema using pg_dump as well as the data within a single table. I know I can do something like this:
pg_dump -s mydatabase > db.sql; pg_dump -a -t some_table mydatabase >> db.sql
However, it's somewhat inelegant, and the output of the second half repeats a lot of settings and comments that the first half already provides such as:
-- Dumped from database version 9.6.8
-- Dumped by pg_dump version 9.6.8
SET statement_timeout = 0;
SET lock_timeout = 0;
...
Is there a better way to do this?
using both is restricted
t=# \! pg_dump -d t -h localhost -p 5400 -s -a -t so12
pg_dump: options -s/--schema-only and -a/--data-only cannot be used together
so obviously - no. you have to do them sequentially...
SET SESSION statements just set the session "parameters" - they won't duplicated data or such - why you want to avoid them?..
I can't imagine any faster commands executed, e./g. compared to select now():
t=# \timing on
Timing is on.
t=# SET statement_timeout = 0;
SET
Time: 0.312 ms
t=# select now();
now
------------------------------
2018-04-01 17:40:37.52727+01
(1 row)
Time: 17.182 ms
I'm trying to copy a table from one database to another database (NOT schema). The code I used in terminal is as below:
pg_dump -U postgres -t OldSchema.TableToCopy OldDatabase | psql -U postgres -d NewDatabase
When I press Enter it requests postgres password I enter my pass and then It requests psql password. I enter it and press Enter. I receive lots of:
invalid command \N
ERROR: relation "TableToCopy" does not exist
Both tables have UTF8 encoding. Am I doing something wrong?
OS: windows XP
Error output:
psql:TblToCopy.sql:39236: invalid command \N
psql:TblToCopy.sql:39237: invalid command \N
psql:TblToCopy.sql:39238: invalid command \N
.
.
.
After Hundreds of above errors, the terminal echoes:
psql:TblToCopy.sql:39245: ERROR: syntax error at or near "509"
LINE 1: 509 some gibberish words and letters here
And Finally:
sql:TblToCopy.sql:39245: ERROR: relation "TableToCopy" does not exist
EDIT
I read this response to the same problem \N error with psql , it says to use INSERT instead of COPY, but in the file pg_dump created COPY. How to say to pg_dump to use INSERT instead of COPY?
I converted the file with iconv to utf-8. Now that error has gone but I have a new error. In this particular case when I use psql to import data to database something new happens. Table gets created but without data. It says:
SET
SET
SET
SET
SET
SET
SET
SET
CREATE TABLE
ALTER TABLE
psql:tblNew.sql:39610: ERROR: value too long for type character(3)
CONTEXT: COPY words, line 1, column first_two_letters: "سر"
ALTER TABLE
ALTER TABLE
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE TRIGGER
I've tried to create a database with Encoding: UTF8 with a table and insert the two UTF-8 encoded characters the COPY command is trying to insert and it works when using INSERT.
CREATE DATABASE test
WITH OWNER = postgres
ENCODING = 'UTF8'
TABLESPACE = pg_default
LC_COLLATE = 'English_United States.1252'
LC_CTYPE = 'English_United States.1252'
CONNECTION LIMIT = -1;
CREATE TABLE x
(
first_two_letters character(3)
)
WITH (
OIDS=FALSE
);
ALTER TABLE x
OWNER TO postgres;
INSERT INTO x(
first_two_letters)
VALUES ('سر');
According to http://rishida.net/tools/conversion/ for the failing COPY the Unicode code points are:
U+0633 U+0631
which are two characters, which means you should be able to store them in a column defined as character(3), which stores strings up to 3 characters (not bytes) in length.
and if we try to INSERT, it succeeds:
INSERT INTO x(
first_two_letters)
VALUES (U&'\0633\0631');
From the pgdump documentation you can INSERT instead of COPY by using the --inserts option
--inserts
Dump data as INSERT commands (rather than COPY). This will make restoration very slow; it is mainly useful for making dumps that can
be loaded into non-PostgreSQL databases. However, since this option
generates a separate command for each row, an error in reloading a row
causes only that row to be lost rather than the entire table contents.
Note that the restore might fail altogether if you have rearranged
column order. The --column-inserts option is safe against column order
changes, though even slower.
Try to use this instead for Step 1:
pg_dump -U postgres -t OldSchema."TableToCopy" --inserts OldDatabase > Table.sql
I've also tried to COPY from a table to a file and use COPY to import and for me it works.
Are you sure your client and server database encoding is UTF8 ?
Firstly, export the table named "x" from schema "public" on database "test" to a plain text SQL file:
pg_dump -U postgres -t public."x" test > x.sql
which creates the x.sql file that contains:
--
-- PostgreSQL database dump
--
SET statement_timeout = 0;
SET lock_timeout = 0;
SET client_encoding = 'UTF8';
SET standard_conforming_strings = on;
SET check_function_bodies = false;
SET client_min_messages = warning;
SET search_path = public, pg_catalog;
SET default_tablespace = '';
SET default_with_oids = false;
--
-- Name: x; Type: TABLE; Schema: public; Owner: postgres; Tablespace:
--
CREATE TABLE x (
first_two_letters character(3)
);
ALTER TABLE public.x OWNER TO postgres;
--
-- Data for Name: x; Type: TABLE DATA; Schema: public; Owner: postgres
--
COPY x (first_two_letters) FROM stdin;
سر
\.
--
-- PostgreSQL database dump complete
--
Secondly, import with:
psql -U postgres -d test -f x.sql
The table name should be quoted , as the following
pg_dump -U postgres -t OldSchema."TableToCopy" OldDatabase | psql -U postgres -d NewDatabase
And I suggest you do the job in two steps
Step 1
pg_dump -U postgres -t OldSchema."TableToCopy" OldDatabase > Table.sql
If step 1 goes ok then do the step2.
Step 2
psql -U postgres -d NewDatabase -f Table.sql
I have a database that was set up with the default character set SQL_ASCII. I want to switch it to UNICODE. Is there an easy way to do that?
First off, Daniel's answer is the correct, safe option.
For the specific case of changing from SQL_ASCII to something else, you can cheat and simply poke the pg_database catalogue to reassign the database encoding. This assumes you've already stored any non-ASCII characters in the expected encoding (or that you simply haven't used any non-ASCII characters).
Then you can do:
update pg_database set encoding = pg_char_to_encoding('UTF8') where datname = 'thedb'
This will not change the collation of the database, just how the encoded bytes are converted into characters (so now length('£123') will return 4 instead of 5). If the database uses 'C' collation, there should be no change to ordering for ASCII strings. You'll likely need to rebuild any indices containing non-ASCII characters though.
Caveat emptor. Dumping and reloading provides a way to check your database content is actually in the encoding you expect, and this doesn't. And if it turns out you did have some wrongly-encoded data in the database, rescuing is going to be difficult. So if you possibly can, dump and reinitialise.
To change the encoding of your database:
Dump your database
Drop your database,
Create new database with the different encoding
Reload your data.
Make sure the client encoding is set correctly during all this.
Source: http://archives.postgresql.org/pgsql-novice/2006-03/msg00210.php
Dumping a database with a specific encoding and try to restore it on another database with a different encoding could result in data corruption.
Data encoding must be set BEFORE any data is inserted into the database.
Check this :
When copying any other database, the encoding and locale settings cannot be changed from those of the source database, because that might result in corrupt data.
And this :
Some locale categories must have their values fixed when the database is created. You can use different settings for different databases, but once a database is created, you cannot change them for that database anymore. LC_COLLATE and LC_CTYPE are these categories. They affect the sort order of indexes, so they must be kept fixed, or indexes on text columns would become corrupt. (But you can alleviate this restriction using collations, as discussed in Section 22.2.) The default values for these categories are determined when initdb is run, and those values are used when new databases are created, unless specified otherwise in the CREATE DATABASE command.
I would rather rebuild everything from the begining properly with a correct local encoding on your debian OS as explained here :
su root
Reconfigure your local settings :
dpkg-reconfigure locales
Choose your locale (like for instance for french in Switzerland : fr_CH.UTF8)
Uninstall and clean properly postgresql :
apt-get --purge remove postgresql\*
rm -r /etc/postgresql/
rm -r /etc/postgresql-common/
rm -r /var/lib/postgresql/
userdel -r postgres
groupdel postgres
Re-install postgresql :
aptitude install postgresql-9.1 postgresql-contrib-9.1 postgresql-doc-9.1
Now any new database will be automatically be created with correct encoding, LC_TYPE (character classification), and LC_COLLATE (string sort order).
Daniel Kutik's answer is correct, but it can be even more safe, with database renaming.
So, the truly safe way is:
Create new database with the different encoding and name
Dump your database
Restore dump to the new DB
Test that your application runs correctly with the new DB
Rename old DB to something meaningful
Rename new DB
Test application again
Drop the old database
In case of emergency, just rename DBs back
# dump into file
pg_dump myDB > /tmp/myDB.sql
# create an empty db with the right encoding (on older versions the escaped single quotes are needed!)
psql -c 'CREATE DATABASE "tempDB" WITH OWNER = "myself" LC_COLLATE = '\''de_DE.utf8'\'' TEMPLATE template0;'
# import in the new DB
psql -d tempDB -1 -f /tmp/myDB.sql
# rename databases
psql -c 'ALTER DATABASE "myDB" RENAME TO "myDB_wrong_encoding";'
psql -c 'ALTER DATABASE "tempDB" RENAME TO "myDB";'
# see the result
psql myDB -c "SHOW LC_COLLATE"
I had the same issue in postgres 11 and I did change the database encoding using the below steps,
to update all the list of encoding
SET client_encoding = 'UTF8';
UPDATE pg_database SET datcollate='en_US.UTF-8', datctype='en_US.UTF-8' WHERE datname='postgres';
update pg_database set encoding = pg_char_to_encoding('UTF8') where datname = 'dbname' ;
make sure to apply the update statment in template0 and template1 and postgres Database
postgres=# UPDATE pg_database SET datcollate='en_US.UTF-8', datctype='en_US.UTF-8' WHERE datname='postgres';
UPDATE 1
postgres=# UPDATE pg_database SET datcollate='en_US.UTF-8', datctype='en_US.UTF-8' WHERE datname='template0';
UPDATE 1
postgres=# UPDATE pg_database SET datcollate='en_US.UTF-8', datctype='en_US.UTF-8' WHERE datname='template1';
UPDATE 1