I use
pg_dump -d xx_sjcbhs --table=wd555.ft_bjgzflcbmxb --column-inserts > e:\temp\ontable2.sql
backup a table. The command is Ok, but the backup file have no insert statement, and no data.
--
-- PostgreSQL database dump
--
-- Dumped from database version 14.5
-- Dumped by pg_dump version 14.5
SET statement_timeout = 0;
SET lock_timeout = 0;
SET idle_in_transaction_session_timeout = 0;
SET client_encoding = 'UTF8';
SET standard_conforming_strings = on;
SELECT pg_catalog.set_config('search_path', '', false);
SET check_function_bodies = false;
SET xmloption = content;
SET client_min_messages = warning;
SET row_security = off;
SET default_tablespace = '';
--
-- Name: ft_bjgzflcbmxb; Type: TABLE; Schema: wd555; Owner: postgres
--
CREATE TABLE wd555.ft_bjgzflcbmxb (
xxbsm text NOT NULL,
xxmc text NOT NULL,
tyshxydm text NOT NULL,
id text NOT NULL,
yyyymm text NOT NULL,
bjdm text NOT NULL,
cbflbm text NOT NULL,
cbfl text NOT NULL,
je numeric(18,2) NOT NULL,
zgh text NOT NULL,
rysxdm text NOT NULL,
ftbz text,
ftsm text,
dftje numeric(18,2) NOT NULL,
zrs numeric(18,2) NOT NULL,
bjrs numeric(18,2) NOT NULL
)
PARTITION BY LIST (yyyymm);
--
-- PostgreSQL database dump complete
You got what you requested, because a partitioned table contains no data. You have to dump the partitions as well if you want to dump the data. To makes that easier, note that the --table option accepts a pattern as argument, so that you don't have to enumerate all partitions.
Related
I wanted to backup my db with no data, but my PK is autoincrementing in no order. On some tables it goes 1,3,5,6,7.., some columns it is 1,3,4.. and some columns even start from 2.
Here is my sql code example:
SET statement_timeout = 0;
SET lock_timeout = 0;
SET idle_in_transaction_session_timeout = 0;
SET client_encoding = 'UTF8';
SET standard_conforming_strings = on;
SELECT pg_catalog.set_config('search_path', '', false);
SET check_function_bodies = false;
SET xmloption = content;
SET client_min_messages = warning;
SET row_security = off;
CREATE SCHEMA "Codes";
ALTER SCHEMA "Codes" OWNER TO postgres;
CREATE TABLE "Codes"."VehicleColor" (
"ID" integer NOT NULL,
"Name" character varying(255) NOT NULL,
"Code" character varying(30),
"Note" character varying(2000),
"Active" boolean NOT NULL
);
ALTER TABLE "Codes"."VehicleColor" OWNER TO postgres;
ALTER TABLE "Codes"."VehicleColor" ALTER COLUMN "ID" ADD GENERATED ALWAYS AS IDENTITY (
SEQUENCE NAME "Codes"."VehicleColor_ID_seq"
START WITH 1
INCREMENT BY 1
NO MINVALUE
NO MAXVALUE
CACHE 1
);
ALTER TABLE ONLY "Codes"."VehicleColor"
ADD CONSTRAINT "VehicleColor_pkey" PRIMARY KEY ("ID");
SELECT pg_catalog.setval('"Codes"."VehicleColor_ID_seq"', 1, true);
With this last line Select pg_catalog.setval... table VehicleColor PK order was 1,3,5,6,7.. and without this last line, PK order is 1,3,4..
Does anyone has idea why this happens?
I created a postgres database dump with pg_dump. I would like to now why the backup file is so verbose.
a) What is the purpose of this config:
SET statement_timeout = 0;
SET lock_timeout = 0;
SET idle_in_transaction_session_timeout = 0;
SET client_encoding = 'UTF8';
SET standard_conforming_strings = on;
SELECT pg_catalog.set_config('search_path', '', false);
SET check_function_bodies = false;
SET xmloption = content;
SET client_min_messages = warning;
SET row_security = off;
SET default_tablespace = '';
SET default_table_access_method = heap;
b) Why is a creation of a table so verbose:
CREATE TABLE public.tag (
id integer NOT NULL,
name character varying(100) NOT NULL,
description text
);
ALTER TABLE public.tag OWNER TO postgres;
CREATE SEQUENCE public.tag_id_seq
AS integer
START WITH 1
INCREMENT BY 1
NO MINVALUE
NO MAXVALUE
CACHE 1;
ALTER TABLE public.tag_id_seq OWNER TO postgres;
ALTER SEQUENCE public.tag_id_seq OWNED BY public.tag.id;
ALTER TABLE ONLY public.tag ALTER COLUMN id SET DEFAULT nextval('public.tag_id_seq'::regclass);
Why not just like I created it before:
CREATE TABLE tag (
id SERIAL PRIMARY KEY,
name VARCHAR(100) NOT NULL,
description TEXT
);
c) Why is COPY ... FROM stdin; used rather than INSERT INTO ... VALUES ?
COPY public.tag (id, name, description) FROM stdin;
1 vegetarian No animals.
2 vegan No animal products.
3 vegetable
\.
SELECT pg_catalog.setval('public.tag_id_seq', 3, true);
ALTER TABLE ONLY public.tag
ADD CONSTRAINT tag_pkey PRIMARY KEY (id);
I would suggest spending some time here pg_dump. In mean time:
a) That sets up the database environment to be compatible with the database you dumped from. Also a security feature SELECT pg_catalog.set_config('search_path', '', false); See here CVE-2018-1058 for more on that.
b) This is because a dump has three parts 'pre-data, data, post-data`. See link above for complete explanation. By default it will dump all three, but keeps the parts separate. It also allows for things like triggers that work best if restored separately and later so they are not running when inputting the data.
c) COPY is orders of magnitude faster then INSERTS. You can specify using INSERTS, but I would recommend not unless you are moving data to another SQL database that does not understand COPY.
After defining the table, the script generated by PGAdmin via rightclick->SCRIPT->CREATE SCRIPT, looks like this:
-- DROP TABLE public.arr;
CREATE TABLE public.arr
(
rcdno integer NOT NULL DEFAULT nextval('arr_rcdno_seq'::regclass),
rcdstate character varying(10) COLLATE pg_catalog."default" DEFAULT 'ACTIVE'::character varying,
rcdserial integer NOT NULL DEFAULT 0,
msrno integer NOT NULL DEFAULT nextval('arr_msrno_seq'::regclass),
mobileid character varying(50) COLLATE pg_catalog."default" NOT NULL,
edittimestamp timestamp without time zone DEFAULT now(),
editor character varying(20) COLLATE pg_catalog."default" NOT NULL,
CONSTRAINT pk_arr PRIMARY KEY (mobileid, msrno, rcdserial)
)
After using >pg_dump -s -U webmaster -W -F p Test > c:\temp\Test.sql, the table definition in the script OMITS the defaults for the 2 original "serial" columns. This means that the pg_dump script doesn't create the table correctly when it is run!
--
-- Name: arr; Type: TABLE; Schema: public; Owner: webmaster
--
CREATE TABLE public.arr (
rcdno integer NOT NULL, -- default missing!
rcdstate character varying(10) DEFAULT 'ACTIVE'::character varying,
rcdserial integer DEFAULT 0 NOT NULL,
msrno integer NOT NULL, -- default missing!
mobileid character varying(50) NOT NULL,
edittimestamp timestamp without time zone DEFAULT now(),
editor character varying(20) NOT NULL
);
ALTER TABLE public.arr OWNER TO webmaster;
--
-- Name: arr_msrno_seq; Type: SEQUENCE; Schema: public; Owner: webmaster
--
CREATE SEQUENCE public.arr_msrno_seq
AS integer
START WITH 1
INCREMENT BY 1
NO MINVALUE
NO MAXVALUE
CACHE 1;
ALTER TABLE public.arr_msrno_seq OWNER TO webmaster;
--
-- Name: arr_msrno_seq; Type: SEQUENCE OWNED BY; Schema: public; Owner: webmaster
--
ALTER SEQUENCE public.arr_msrno_seq OWNED BY public.arr.msrno;
--
-- Name: arr_rcdno_seq; Type: SEQUENCE; Schema: public; Owner: webmaster
--
CREATE SEQUENCE public.arr_rcdno_seq
AS integer
START WITH 1
INCREMENT BY 1
NO MINVALUE
NO MAXVALUE
CACHE 1;
ALTER TABLE public.arr_rcdno_seq OWNER TO webmaster;
--
-- Name: arr_rcdno_seq; Type: SEQUENCE OWNED BY; Schema: public; Owner: webmaster
--
ALTER SEQUENCE public.arr_rcdno_seq OWNED BY public.arr.rcdno;
EDIT: After the generated script has been run, the statement
into public.arr(mobileid, editor)
values
('12345', 'nolaspeaker')
generates
ERROR: null value in column "rcdno" violates not-null constraint DETAIL: Failing row contains (null, ACTIVE, 0, null, 12345, 2020-08-18 08:54:41.34052, nolaspeaker). SQL state: 23502
Which indicates that I am correct to assert that the script omits the default values for the rcdno column!
The ALTER TABLE statements that set the default values are towards the end of the dump, close to where the ALTER SEQUENCE ... OWNED BY are.
If they don't get restored, that might be because there was a problem creating the sequences.
Look at the error messages from the restore, particularly the first ones (later ones are often consequences of the earlier ones).
The statements that assign the default values to the applicable columns come much later in the script
--
-- Name: arr rcdno; Type: DEFAULT; Schema: public; Owner: webmaster
--
ALTER TABLE ONLY public.arr ALTER COLUMN rcdno SET DEFAULT nextval('public.arr_rcdno_seq'::regclass);
--
-- Name: arr msrno; Type: DEFAULT; Schema: public; Owner: webmaster
--
ALTER TABLE ONLY public.arr ALTER COLUMN msrno SET DEFAULT nextval('public.arr_msrno_seq'::regclass);
So you have to be careful if you cut-and-paste a table definition out of the script!
I have a dump, where the data and the structure is in the public schema. I want to restore it into a schema with a custom name - how can I do that?
EDIT V 2:
My dump file is from heroku, and looks like this at the beginning:
PGDMP
!
pd6rq1i7f3kcath9.1.59.1.6<Y
0ENCODINENCODINGSET client_encoding = 'UTF8';
falseZ
00
STDSTRINGS
STDSTRINGS)SET standard_conforming_strings = 'off';
false[
126216385d6rq1i7f3kcatDATABASE?CREATE DATABASE d6rq1i7f3kcath WITH TEMPLATE = template0 ENCODING = 'UTF8' LC_COLLATE = 'en_US.UTF-8' LC_CTYPE = 'en_US.UTF-8';
DROP DATABASE d6rq1i7f3kcath;
uc0lt9t3fj0da4false26152200publicSCHEMACREATE SCHEMA public;
DROP SCHEMA public;
postgresfalse\
SCHEMA publicCOMMENT6COMMENT ON SCHEMA public IS 'standard public schema';
postgresfalse5?307916392plpgsql EXTENSION?CREATE EXTENSION IF NOT EXISTS plpgsql WITH SCHEMA pg_catalog;
DROP EXTENSION plpgsql;
false]
00EXTENSION plpgsqlCOMMENT#COMMENT ON EXTENSION plpgsql IS 'PL/pgSQL procedural language';
false212?125516397_final_mode(anyarrayFUNCTION?CREATE FUNCTION _final_mode(anyarray) RETURNS anyelement
LANGUAGE sql IMMUTABLE
AS $_$
SELECT a
FROM unnest($1) a
GROUP BY 1
ORDER BY COUNT(1) DESC, 1
LIMIT 1;
$_$;
,DROP FUNCTION public._final_mode(anyarray);
publicuc0lt9t3fj0da4false5?125516398mode(anyelement) AGGREGATE?CREATE AGGREGATE mode(anyelement) (
SFUNC = array_append,
STYPE = anyarray,
INITCOND = '{}',
FINALFUNC = _final_mode
);
(DROP AGGREGATE public.mode(anyelement);
publicuc0lt9t3fj0da4false5224?125916399 advert_candidate_collector_failsTABLECREATE TABLE advert_candidate_collector_fails (
id integer NOT NULL,
advert_candidate_collector_status_id integer,
exception_message text,
stack_trace text,
url text,
created_at timestamp without time zone,
updated_at timestamp without time zone
);
4DROP TABLE public.advert_candidate_collector_fails;
publicuc0lt9t3fj0da4false5?125916405'advert_candidate_collector_fails_id_seSEQUENCE?CREATE SEQUENCE advert_candidate_collector_fails_id_seq
START WITH 1
INCREMENT BY 1
NO MINVALUE
NO MAXVALUE
CACHE 1;
>DROP SEQUENCE public.advert_candidate_collector_fails_id_seq;
publicuc0lt9t3fj0da4false1615^
00'advert_candidate_collector_fails_id_seqSEQUENCE OWNED BYeALTER SEQUENCE advert_candidate_collector_fails_id_seq OWNED BY advert_candidate_collector_fails.id;
publicuc0lt9t3fj0da4false162_
00'advert_candidate_collector_fails_id_seq
SEQUENCE SETRSELECT pg_catalog.setval('advert_candidate_collector_fails_id_seq', 13641, true);
publicuc0lt9t3fj0da4false162?125916407#advert_candidate_collector_statusesTABLE?CREATE TABLE advert_candidate_collector_statuses (
id integer NOT NULL,
data_source_id character varying(120),
state character varying(15) DEFAULT 'Queued'::character varying,
source_name character varying(30),
collector_type character varying(30),
started_at timestamp without time zone,
ended_at timestamp without time zone,
times_failed integer DEFAULT 0,
created_at timestamp without time zone,
updated_at timestamp without time zone
);
7DROP TABLE public.advert_candidate_collector_statuses;
publicuc0lt9t3fj0da4false240424055?125916412*advert_candidate_collector_statuses_id_seSEQUENCE?CREATE SEQUENCE advert_candidate_collector_statuses_id_seq
START WITH 1
INCREMENT BY 1
NO MINVALUE
NO MAXVALUE
CACHE 1;
ADROP SEQUENCE public.advert_candidate_collector_statuses_id_seq;
publicuc0lt9t3fj0da4false1635`
00*advert_candidate_collector_statuses_id_seqSEQUENCE OWNED BYkALTER SEQUENCE advert_candidate_collector_statuses_id_seq OWNED BY advert_candidate_collector_statuses.id;
publicuc0lt9t3fj0da4false164a
00*advert_candidate_collector_statuses_id_seq
SEQUENCE SETVSELECT pg_catalog.setval('advert_candidate_collector_statuses_id_seq', 133212, true);
publicuc0lt9t3fj0da4false164?125916414advertsTABLE"CREATE TABLE adverts (
id integer NOT NULL,
car_id integer NOT NULL,
source_name character varying(20),
url text,
first_extraction timestamp without time zone,
last_observed_at timestamp without time zone,
created_at timestamp without time zone,
updated_at timestamp without time zone,
source_id character varying(255),
deactivated_at timestamp without time zone,
seller_id integer NOT NULL,
data_source_id character varying(100),
price integer,
availability_state character varying(15)
);
ROP TABLE public.adverts;
publicuc0lt9t3fj0da4false5?125916420adverts_id_seSEQUENCEpCREATE SEQUENCE adverts_id_seq
START WITH 1
INCREMENT BY 1
#Tometzky's solution isn't quite right, at least with 9.2's pg_dump. It'll create the table in the new schema, but pg_dump schema-qualifies the ALTER TABLE ... OWNER TO statements, so those will fail:
postgres=# CREATE DATABASE demo;
\cCREATE DATABASE
postgres=# \c demo
You are now connected to database "demo" as user "postgres".
demo=# CREATE TABLE public.test ( dummy text );
CREATE TABLE
demo=# \d
List of relations
Schema | Name | Type | Owner
--------+------+-------+----------
public | test | table | postgres
(1 row)
demo=# \q
$
$ pg_dump -U postgres -f demo.sql demo
$ sed -i 's/^SET search_path = public, pg_catalog;$/SET search_path = testschema, pg_catalog;/' demo.sql
$ grep testschema demo.sql
SET search_path = testschema, pg_catalog;
$ dropdb -U postgres demo
$ createdb -U postgres demo
$ psql -U postgres -c 'CREATE SCHEMA testschema;' demo
CREATE SCHEMA
$ psql -U postgres -f demo.sql -v ON_ERROR_STOP=1 -v QUIET=1 demo
psql:demo.sql:40: ERROR: relation "public.test" does not exist
$ psql demo
demo=> \d testschema.test
Table "testschema.test"
Column | Type | Modifiers
--------+------+-----------
dummy | text |
You will also need to edit the dump to remove the schema-qualification on public.test or change it to the new schema name. sed is a useful tool for this.
I could've sworn the correct way to do this was with pg_dump -Fc -n public -f dump.dbbackup then pg_restore into a new schema, but I can't seem to find out exactly how right now.
Update: Nope, it looks like sed is your best bet. See I want to restore the database with a different schema
Near the beginning of a dump file (created with pg_dump databasename) is a line:
SET search_path = public, pg_catalog;
Just change it to:
SET search_path = your_schema_name, pg_catalog;
Also you'll need to search for
ALTER TABLE public.
and replace with:
ALTER TABLE your_schema_name.
I keep getting:
SQL error: ERROR: could not create
unique index
"service_import_checksum_key" DETAIL:
Key (checksum)=() is duplicated.
In statement:
ALTER TABLE "public"."service_import" ADD CONSTRAINT "service_import_checksum_key" UNIQUE ("checksum")
But this constraint ISN'T a duplicate. There is no other constraint like this anywhere in the entire database and I have no idea why on earth it keeps insisting it's a duplicate. I'm assuming this is some weird nuance of postgres that I'm missing here.
What am I doing wrong?
Table dump:
--
-- PostgreSQL database dump
--
SET statement_timeout = 0;
SET client_encoding = 'UTF8';
SET standard_conforming_strings = off;
SET check_function_bodies = false;
SET client_min_messages = warning;
SET escape_string_warning = off;
SET search_path = public, pg_catalog;
SET default_tablespace = '';
SET default_with_oids = false;
--
-- Name: service_import; Type: TABLE; Schema: public; Owner: cvs_tar; Tablespace:
--
CREATE TABLE service_import (
id integer NOT NULL,
name character varying(32) NOT NULL,
importfile character varying(64) NOT NULL,
reportfile character varying(64) NOT NULL,
percent smallint NOT NULL,
message text NOT NULL,
stamp timestamp without time zone DEFAULT now() NOT NULL,
complete smallint DEFAULT 0 NOT NULL,
checksum character varying(40) NOT NULL
);
ALTER TABLE public.service_import OWNER TO cvs_tar;
--
-- Name: service_imports_id_seq; Type: SEQUENCE; Schema: public; Owner: cvs_tar
--
CREATE SEQUENCE service_imports_id_seq
START WITH 1
INCREMENT BY 1
NO MINVALUE
NO MAXVALUE
CACHE 1;
ALTER TABLE public.service_imports_id_seq OWNER TO cvs_tar;
--
-- Name: service_imports_id_seq; Type: SEQUENCE OWNED BY; Schema: public; Owner: cvs_tar
--
ALTER SEQUENCE service_imports_id_seq OWNED BY service_import.id;
--
-- Name: id; Type: DEFAULT; Schema: public; Owner: cvs_tar
--
ALTER TABLE service_import ALTER COLUMN id SET DEFAULT nextval('service_imports_id_seq'::regclass);
--
-- Name: service_import_name_key; Type: CONSTRAINT; Schema: public; Owner: cvs_tar; Tablespace:
--
ALTER TABLE ONLY service_import
ADD CONSTRAINT service_import_name_key UNIQUE (name);
--
-- Name: service_import_pkey; Type: CONSTRAINT; Schema: public; Owner: cvs_tar; Tablespace:
--
ALTER TABLE ONLY service_import
ADD CONSTRAINT service_import_pkey PRIMARY KEY (id);
--
-- Name: service_import_complete_idx; Type: INDEX; Schema: public; Owner: cvs_tar; Tablespace:
--
CREATE INDEX service_import_complete_idx ON service_import USING btree (complete);
--
-- Name: service_import_stamp_idx; Type: INDEX; Schema: public; Owner: cvs_tar; Tablespace:
--
CREATE INDEX service_import_stamp_idx ON service_import USING btree (stamp);
--
-- PostgreSQL database dump complete
--
Read the error message again:
SQL error: ERROR: could not create unique index "service_import_checksum_key"
DETAIL: Key (checksum)=() is duplicated.
Looks like it is telling you that there are duplicate values in the checksum column and you're trying to enforce uniqueness on that column with your constraint. The constraint isn't duplicated, the data has duplicates.
Furthermore, the "()" part indicates that you have multiple empty strings in the checksum column. Unique constraints allow multiple NULL values (since NULL = NULL is NULL which is not true) but empty strings are not NULL.
An example to clarify what's going on:
=> CREATE TABLE x (s VARCHAR NULL);
CREATE TABLE
=> INSERT INTO x (s) VALUES (''), ('a'), ('');
INSERT 0 3
=> ALTER TABLE x ADD CONSTRAINT ux UNIQUE(s);
NOTICE: ALTER TABLE / ADD UNIQUE will create implicit index "ux" for table "x"
ERROR: could not create unique index "ux"
DETAIL: Key (s)=() is duplicated.
=> delete from x where s='';
DELETE 2
=> INSERT INTO x (s) VALUES ('a');
INSERT 0 1
=> ALTER TABLE x ADD CONSTRAINT ux UNIQUE(s);
NOTICE: ALTER TABLE / ADD UNIQUE will create implicit index "ux" for table "x"
ERROR: could not create unique index "ux"
DETAIL: Key (s)=(a) is duplicated.
In particular, note what the ERROR and DETAIL are saying and compare that to the INSERTs.