pg_dump setting of sequences - postgresql

I've recently started developing apps with PostgreSQL as backend DB (imposed on me) with no previous experience of Postgres. So far it hasn't been too bad, but now I run into a problem to which I cannot find answer for.
I created a batch scripts that runs a pg_dump command for a particular database on the server. This batch file is executed on schedule by the pgAgent.
The pg_dump itself seems to work ok. All the database structure and data are dumped to a file. However the sequences are all set to 1. For example for table tbl_departments the sequence dump looks like this:
CREATE SEQUENCE "tbl_departments_iID_seq"
START WITH 1
INCREMENT BY 1
NO MINVALUE
NO MAXVALUE
CACHE 1;
ALTER TABLE "tbl_departments_iID_seq" OWNER TO postgres;
ALTER SEQUENCE "tbl_departments_iID_seq" OWNED BY tbl_departments."iID";
In this particular example the sequence should be set to start with 8, since the last inserted record has iID = 7.
How do I make the pg_dump set the sequence starting number the next one available for each table?
The command for dump is:
%PGBIN%pg_dump -h 192.168.0.112 -U postgres -F p -b -v --inserts -f "\\192.168.0.58\PostgresDB\backup\internals_db.sql" Internals
EDIT:
I think I have found the issue, although I still don't know how to resolve this:
If I open pgAdmin and generate CREATE script for tbl_departments, it look like this:
CREATE TABLE tbl_departments
(
"iID" serial NOT NULL, -- id, autoincrement
"c150Name" character varying(150) NOT NULL, -- human readable name for department
"bRetired" boolean NOT NULL DEFAULT false, -- if TRUE that it is no longer active
"iParentDept" integer NOT NULL DEFAULT 0, -- ID of the parent department
CONSTRAINT tbl_departments_pkey PRIMARY KEY ("iID")
)
The pg_dump statement is:
CREATE TABLE tbl_departments (
"iID" integer NOT NULL,
"c150Name" character varying(150) NOT NULL,
"bRetired" boolean DEFAULT false NOT NULL,
"iParentDept" integer DEFAULT 0 NOT NULL
);
ALTER TABLE tbl_departments OWNER TO postgres;
COMMENT ON TABLE tbl_departments IS 'list of departments';
COMMENT ON COLUMN tbl_departments."iID" IS 'id, autoincrement';
COMMENT ON COLUMN tbl_departments."c150Name" IS 'human readable name for department';
COMMENT ON COLUMN tbl_departments."bRetired" IS 'if TRUE that it is no longer active';
COMMENT ON COLUMN tbl_departments."iParentDept" IS 'ID of the parent department';
CREATE SEQUENCE "tbl_departments_iID_seq"
START WITH 1
INCREMENT BY 1
NO MINVALUE
NO MAXVALUE
CACHE 1;
ALTER TABLE "tbl_departments_iID_seq" OWNER TO postgres;
ALTER SEQUENCE "tbl_departments_iID_seq" OWNED BY tbl_departments."iID";
INSERT INTO tbl_departments VALUES (1, 'Information Technologies', false, 0);
INSERT INTO tbl_departments VALUES (2, 'Quality Control', false, 0);
INSERT INTO tbl_departments VALUES (3, 'Engineering', false, 0);
INSERT INTO tbl_departments VALUES (5, 'Quality Assurance', false, 0);
INSERT INTO tbl_departments VALUES (6, 'Production', false, 2);
ALTER TABLE ONLY tbl_departments
ADD CONSTRAINT tbl_departments_pkey PRIMARY KEY ("iID");
SELECT pg_catalog.setval('"tbl_departments_iID_seq"', 1, false);
the pg_dump sets the iID column to integer rather than serial, which disabled the auto-incrementation. The setval is also set to 1 rather than 7 as one would expect.
When I open the front-end application and go to add new department it fails because all I am providing is: name of new department, active/disabled (true/false), ID of parent dept. (0 if no parent).
I am expecting for the new record primary key iID to be created automatically by the DB, which as far as I know is an expected basic feature of any RDBMS.
because the pg_dump converts the serials to integers the auto-incrementation stops.

There is no reason for concern.
The generated SQL file will restore current values of sequences.
Open the file with an editor and look for setval.
There should be lines like this:
SELECT pg_catalog.setval('test_id_seq', 1234, true);
If you cannot find them it means that INSERT commands set the proper value of a sequence.
As Craig noticed, the current value of the sequence had to be equal to 1 at the time of dump of the original database. You have probably inserted iID values directly, not using default. In that case the sequence is not used.
Therefore I suggest start from the beginning, but in two databases:
make an sql dump like in the question,
create a new database,
run the sql script in the new database,
check whether corresponding serial columns have the same declaration in both databases,
compare current values of corresponding sequences in both databases.

the pg_dump sets the iID column to integer rather than serial, which disabled the auto-incrementation.
That's normal. See the manual.
SERIAL is basically just shorthand for CREATE SEQUENCE and then an integer column that makes that sequence its default for nextval('seq_name').
The setval is also set to 1 rather than 7 as one would expect.
I can only explain that one by assuming that the sequence start point is 1 in the DB. Perhaps due to a prior attempt at running DDL that altered it, such as a setval or alter sequence?
setval it to the start point you expect. Then, so long as you don't run other setval commands, alter sequence commands, etc, you'll be fine.
Or maybe the app inserted values directly, without using the sequence?

SELECT setval(pg_get_serial_sequence('public.table', 'id'), max(id)+1) FROM public.table;

Related

How does pg_restore handle GENERATED ALWAYS AS IDENTITY columns?

If I pg_restore --data-only into a table like this:
CREATE TABLE foo (
id int4 NOT NULL GENERATED ALWAYS AS IDENTITY,
bar varchar
);
what happens to the id column? Is the GENERATED ALWAYS ignored and the ids from the pg_dump file inserted, or are new ids generated?
The pg_restore docs say, under the --data-only flag:
Table data, large objects, and sequence values are restored, if present in the archive.
Does "sequence values" here include GENERATED ALWAYS AS IDENTITY?
I'm using Postgres 11 but it would be interesting to know if this behaviour has been the same since all generally supported versions (>=9.5).
With INSERT, you need to use OVERRIDING SYSTEM VALUE if you want to overrule the default value from the identity column, but with COPY there are no such restrictions.
That makes it simple for pg_dump. Here is an example of what a dump of a table with identity column looks like:
/* section = pre-data */
CREATE TABLE laurenz.identity (
id integer NOT NULL,
value text NOT NULL
);
ALTER TABLE laurenz.identity ALTER COLUMN id ADD GENERATED ALWAYS AS IDENTITY (
SEQUENCE NAME laurenz.identity_id_seq
START WITH 1
INCREMENT BY 1
NO MINVALUE
NO MAXVALUE
CACHE 1
);
/* section = data */
COPY laurenz.identity (id, value) FROM stdin;
1 one
2 two
3 three
-10 weird
-5 also weird
\.
/* section = post-data */
SELECT pg_catalog.setval('laurenz.identity_id_seq', 3, true);

SET*5 set_config ------------ (1 row) SET*4 CREATE TABLE ERROR: syntax error at or near "AS" LINE 2: AS integer ^ Import error: exit status 3 [duplicate]

Postgresql lost the autoincrement feature after a restore. My database was created on Windows 10 (v 10.1) and I restored it to Postgresql on Ubuntu (v 9.6). Now that I posted the question I saw that the versions are different. I didn't use any obscure feature, only tables, functions, and columns with serials. Also, the restore process didn't complain about anything. I checked the dump options but I couldn't find anything that caused the problem.
With Pgadmin right-clicking the table > scripts > create a script on my original table gives this:
CREATE TABLE public.produto
(
produto_id integer NOT NULL DEFAULT nextval('produto_produto_id_seq'::regclass),
...
);
In my server, the restored database. It seems it lost the feature.
CREATE TABLE public.produto
(
produto_id integer NOT NULL,
...
);
You didn't check for errors during restore of the database; there should have been a few.
A dump of a table like yours will look like this in PostgreSQL v10 (this is 10.3 and it looks slightly different in 10.1, but that's irrelevant to this case):
CREATE TABLE public.produto (
produto_id integer NOT NULL
);
ALTER TABLE public.produto OWNER TO laurenz;
CREATE SEQUENCE public.produto_produto_id_seq
AS integer
START WITH 1
INCREMENT BY 1
NO MINVALUE
NO MAXVALUE
CACHE 1;
ALTER TABLE public.produto_produto_id_seq OWNER TO laurenz;
ALTER SEQUENCE public.produto_produto_id_seq
OWNED BY public.produto.produto_id;
ALTER TABLE ONLY public.produto
ALTER COLUMN produto_id
SET DEFAULT nextval('public.produto_produto_id_seq'::regclass);
Now the problem is that AS integer was introduced to CREATE SEQUENCE in PostgreSQL v10, so that statement will fail with a syntax error in 9.6.
What is the consequence?
The table is created like in the first statement.
The third statement creating the sequence fails.
All the following statements that require the sequence will also fail.
Note: It is not supported to downgrade PostgeSQL with dump and restore.
The solution is to manually edit the dump until it works, in particular you'll have to remove the AS integer or AS bigint clause in CREATE SEQUENCE.

Postgres: difference between DEFAULT in CREATE TABLE and ALTER TABLE in database dump

In database dump created with pg_dump, some tables have DEFAULTs in the CREATE TABLE statement, i.e.:
CREATE TABLE test (
f1 integer DEFAULT nextval('test_f1_seq'::regclass) NOT NULL
);
But others have an additional ALTER statement:
ALTER TABLE ONLY test2 ALTER COLUMN f1 SET DEFAULT nextval('test2_f1_seq'::regclass);
What is the reason of this? All sequential fields were created with type SERIAL, but in the dump they look different, and I can't guess any rule for this.
The difference must be that in the first case, the sequence is “owned” by the table column.
You can specify this dependency using the OWNED BY clause when you create a sequence. A sequence that is owned by a column will automatically be dropped when the column is.
If a sequence is implicitly created by using serial, it will be owned by the column.

After restoring my database serial removed from column in Postgresql

Postgresql lost the autoincrement feature after a restore. My database was created on Windows 10 (v 10.1) and I restored it to Postgresql on Ubuntu (v 9.6). Now that I posted the question I saw that the versions are different. I didn't use any obscure feature, only tables, functions, and columns with serials. Also, the restore process didn't complain about anything. I checked the dump options but I couldn't find anything that caused the problem.
With Pgadmin right-clicking the table > scripts > create a script on my original table gives this:
CREATE TABLE public.produto
(
produto_id integer NOT NULL DEFAULT nextval('produto_produto_id_seq'::regclass),
...
);
In my server, the restored database. It seems it lost the feature.
CREATE TABLE public.produto
(
produto_id integer NOT NULL,
...
);
You didn't check for errors during restore of the database; there should have been a few.
A dump of a table like yours will look like this in PostgreSQL v10 (this is 10.3 and it looks slightly different in 10.1, but that's irrelevant to this case):
CREATE TABLE public.produto (
produto_id integer NOT NULL
);
ALTER TABLE public.produto OWNER TO laurenz;
CREATE SEQUENCE public.produto_produto_id_seq
AS integer
START WITH 1
INCREMENT BY 1
NO MINVALUE
NO MAXVALUE
CACHE 1;
ALTER TABLE public.produto_produto_id_seq OWNER TO laurenz;
ALTER SEQUENCE public.produto_produto_id_seq
OWNED BY public.produto.produto_id;
ALTER TABLE ONLY public.produto
ALTER COLUMN produto_id
SET DEFAULT nextval('public.produto_produto_id_seq'::regclass);
Now the problem is that AS integer was introduced to CREATE SEQUENCE in PostgreSQL v10, so that statement will fail with a syntax error in 9.6.
What is the consequence?
The table is created like in the first statement.
The third statement creating the sequence fails.
All the following statements that require the sequence will also fail.
Note: It is not supported to downgrade PostgeSQL with dump and restore.
The solution is to manually edit the dump until it works, in particular you'll have to remove the AS integer or AS bigint clause in CREATE SEQUENCE.

PostgreSQL: trivial INSERT fails the first time, succeeds afterwards

I am puzzled by a weird Postgres problem I encounter in the trivial database shown below: If I first insert a tag and explicitly specify its ID and then try to insert another tag without passing an ID, then this second insert fails. If I try a third time (again without ID), the insert succeeds.
DROP DATABASE IF EXISTS mydb;
CREATE DATABASE mydb;
\c mydb
DROP SCHEMA public;
CREATE SCHEMA core;
CREATE TABLE core.tag
(
id serial PRIMARY KEY,
title text NOT NULL
);
-- this works: all columns specified explicitly
INSERT INTO core.tag(id, title) VALUES (1, 'known tag');
-- omitting the tag ID fails with
-- ERROR: duplicate key value violates unique constraint "tag_pkey"
-- DETAIL: Key (id)=(1) already exists.
INSERT INTO core.tag(title) VALUES ('unknown tag');
-- this works again ?!?
INSERT INTO core.tag(title) VALUES ('unknown tag');
The issue only seems to occur on a freshly created database and once it does, it does not seem to happen again. I have never come across anything like this - so far, I have just inserted data with or without explicit ID and AFAICS, nothing ever failed like this...
Does anyone have an idea what's going on here ?!?
Environment: PostgreSQL 9.1.3 on Mac OSX 10.7.5
Of course this fails.
What happens?
When you create the table, a sequence is also created that generates the values for the ID column. The sequence starts with 1 but it is only used if you do not specify a value for the ID column.
Now when you run
INSERT INTO core.tag(id, title) VALUES (1, 'known tag');
you bypass Postgres' automatic assigment of the ID value, the sequence "stays" at one.
Now when you run
INSERT INTO core.tag(title) VALUES ('unknown tag');
Postgres takes the next value from the sequence - which is 1. But that alreay exists so the insert fails. After taking the value from the sequence, the next value is 2, so the subsequent insert without specifying an ID value gets the 2 and succeeds.
The solution is to either never include the ID column in your inserts. Or - if you do - request the ID from the sequence:
INSERT INTO core.tag(id, title) VALUES (nextval('tag_id_seq'), 'known tag');
When a serial column is created it is automatically associated with a sequence which is named <table_name>_<column_name>_seq. And that's the name I used in the above statement.
More details about how the serial "data type" works are in the manual: http://www.postgresql.org/docs/current/static/datatype-numeric.html#DATATYPE-SERIAL