I have two sequences :
sequence_a
sequence_b
they're used to generate primary keys for
table_a
table_b
I need to restart sequence_b so that it's nextval() will be the same as the nextval() for sequence_a.
I can't seem to do this purely in SQL; everything I tried failed. No permutations of ALTER SEQUENCE would accept a lastval(), currval() wasn't defined for this session , and a few other issues.
Does anyone have a clue how I can do this? I can do it manually, but this won't work for my needs. This is part of a database migration, where records from A are being cloned into B.
The sequences will eventually diverge, so it would be acceptable for my needs if sequence_b were a higher number than sequence_a; I just need to ensure that sequence_b starts with some number higher than the highest sequence_a id.
You've got a few options here. The easiest is probably to use setval instead of ALTER SEQUENCE, since you only care about b being equal or higher:
SELECT setval('sequence_b', nextval('sequence_a'));
Related
I have a postgresql (>9.5) table with primary_key id and a unique key col. When I use
INSERT INTO table_a (col) VLUES('xxx') ON CONFLICT(col) DO NOTHING;
to perform a upsert, let's say a row with an id 1 is generated.
If I run the sql again, nothing will happen, but actually the id 2 will be generated and abandoned.
Then if I insert a new record, for example,
INSERT INTO table_a (col) VLUES('yyy') ON CONFLICT(col) DO NOTHING;
Another row with id 3 will be generated and id 2 is wasted!
Is there anyway to avoid this waste?
Presumably id is a serial. Under the hood this causes a nextval() call from a sequence. A number nextval() once returned will never be returned again. And the call of nextval() happens before checking for possible conflicts.
From "9.16. Sequence Manipulation Functions":
nextval
(...)
Important: To avoid blocking concurrent transactions that obtain numbers from the same sequence, a nextval operation is never rolled back; that is, once a value has been fetched it is considered used and will not be returned again. This is true even if the surrounding transaction later aborts, or if the calling query ends up not using the value. For example an INSERT with an ON CONFLICT clause will compute the to-be-inserted tuple, including doing any required nextval calls, before detecting any conflict that would cause it to follow the ON CONFLICT rule instead. Such cases will leave unused "holes" in the sequence of assigned values. Thus, PostgreSQL sequence objects cannot be used to obtain "gapless" sequences.
Concluded that means, that the answer to your question is no, there is no way to avoid this unless you generate the values yourself somehow.
Given an table with id bigint default next_val('foo_sequence')
Can I assume the order of id consisting with the insert order ?
I mean the later inserted id is always greater then earlier inserted ids.
I am trying to calculate and save an increment continuous number of row,
Here is how I did
SELECT count(*) as seq_no from foo where id < some_id;
// get the seq no
UPDATE foo SET seq_no = seq_no_above + 1 WHERE id = some_id;
But it sometimes give duplicate seq_no value,
if the id consists with insert order, it should not have duplicate value.
In the simplest and purest sense, yes. It depends what you mean by "earlier" and "later", though, as you have to consider opening the transaction and closing the transaction. If a transaction has not been committed, then theoretically a record could show up later with an earlier ID.
The IDs are allocated when the insert happens, but the records will not show up until the records are committed. So if commit order is different, you may see some strange behavior depending on how strict your use case is.
I.e.
Open Transaction A
Insert records 1,2
Open Transaction B
Insert records 3,4
Close transaction B
Select * (get 3,4)
Close transaction A
Select * (get 1,2,3,4)
You also have to worry about caching on whether you consider them to be sequential. From the (very good) Postgres docs:
Furthermore, although multiple sessions are guaranteed to allocate
distinct sequence values, the values might be generated out of
sequence when all the sessions are considered. For example, with a
cache setting of 10, session A might reserve values 1..10 and return
nextval=1, then session B might reserve values 11..20 and return
nextval=11 before session A has generated nextval=2. Thus, with a
cache setting of one it is safe to assume that nextval values are
generated sequentially; with a cache setting greater than one you
should only assume that the nextval values are all distinct, not that
they are generated purely sequentially. Also, last_value will reflect
the latest value reserved by any session, whether or not it has yet
been returned by nextval.
One last caveat is someone with appropriate privileges can always reset the sequence to a different value, which obviously would throw a wrench into things.
EDIT:
To address your use case above, you definitely want to use sequences (and likely add NOT NULL / PRIMARY KEY constraints as well, to ensure uniqueness). In pgAdmin, at least, you can do all of this by setting data type serial. Though I have mentioned caveats, for 99% of practical purposes, you get uniqueness and sequential ordering (hence sequences) the way that you want.
In any case, we would need to see example data to confirm why you are seeing duplication (how to create a reproducible example). I presume the duplication you are seeing is in seq_no and not id, which illustrates that the problem is your query. If duplication is in id, then you have other problems, and that would explain duplication in seq_no.
Sequences are much better for transactional definition in the data (they take care of uniqueness for you, perform well in concurrency, and do not cause duplication... plus you get sequential ordering for the most part). For unique keys, they are best used with NOT NULL and PRIMARY KEY or UNIQUE constraints.
But if you need a perfect increment, it is better to do something like the below:
select *, row_number() over (order by value) as id
from foo
;
Postgres window functions are very powerful, but are definitely not the standard to use for inserting data with sequential keys. They are more useful for reporting, analysis, and complex queries after the fact.
I need to migrate a DDL from Postgres to DB2, but I need that it works the same as in Postgres. There is a table that generates values from a sequence, but the values can also be explicitly given.
Postgres
create sequence hist_id_seq;
create table benchmarksql.history (
hist_id integer not null default nextval('hist_id_seq') primary key,
h_c_id integer,
h_c_d_id integer,
h_c_w_id integer,
h_d_id integer,
h_w_id integer,
h_date timestamp,
h_amount decimal(6,2),
h_data varchar(24)
);
(Look at the sequence call in the hist_id column to define the value of the primary key)
The business logic inserts into the table by explicitly providing an ID, and in other cases, it leaves the database to choose the number.
If I change this in DB2 to a GENERATED ALWAYS it will throw errors because there are some provided values. On the other side, if I create the table with GENERATED BY DEFAULT, DB2 will throw an error when trying to insert with the same value (SQL0803N), because the "internal sequence" does not take into account the already inserted values, and it does not retry with a next value.
And, I do not want to restart the sequence each time a provided ID was inserted.
This is the problem in BenchmarkSQL when trying to port it to DB2: https://sourceforge.net/projects/benchmarksql/ (File sqlTableCreates)
How can I implement the same database logic in DB2 as it does in Postgres (and apparently in Oracle)?
You're operating under a misconception: that sources external to the db get to dictate its internal keys. Ideally/conceptually, autogenerated ids will never need to be seen outside of the db, as conceptually there should be unique natural keys for export or reporting. Still, there are times when applications will need to manage some ids, often when setting up related entities (eg, JPA seems to want to work this way).
However, if you add an id value that you generated from a different source, the db won't be able to manage it. How could it? It's not efficient - for one thing, attempting to do so would do one of the following
Be unsafe in the face of multiple clients (attempt to add duplicate keys)
Serialize access to the table (for a potentially slow query, too)
(This usually shows up when people attempt something like: SELECT MAX(id) + 1, which would require locking the entire table for thread safety, likely including statements that don't even touch that column. If you try to find any "first-unused" id - trying to fill gaps - this gets more complicated and problematic)
Neither is ideal, so it's best to not have the problem in the first place. This is usually done by having id columns be autogenerated, but (as pointed out earlier) there are situations where we may need to know what the id will be before we insert the row into the table. Fortunately, there's a standard SQL object for this, SEQUENCE. This provides a db-managed, thread-safe, fast way to get ids. It appears that in PostgreSQL you can use sequences in the DEFAULT clause for a column, but DB2 doesn't allow it. If you don't want to specify an id every time (it should be autogenerated some of the time), you'll need another way; this is the perfect time to use a BEFORE INSERT trigger;
CREATE TRIGGER Add_Generated_Id NO CASCADE BEFORE INSERT ON benchmarksql.history
NEW AS Incoming_Entity
FOR EACH ROW
WHEN Incoming_Entity.id IS NULL
SET id = NEXTVAL FOR hist_id_seq
(something like this - not tested. You didn't specify where in the project this would belong)
So, if you then add a row with something like:
INSERT INTO benchmarksql.history (hist_id, h_data) VALUES(null, 'a')
or
INSERT INTO benchmarksql.history (h_data) VALUES('a')
an id will be generated and attached automatically. Note that ALL ids added to the table must come from the given sequence (as #mustaccio pointed out, this appears to be true even in PostgreSQL), or any UNIQUE CONSTRAINT on the column will start throwing duplicate-key errors. So any time your application needs an id before inserting a row in the table, you'll need some form of
SELECT NEXT VALUE FOR hist_id_seq
FROM sysibm.sysdummy1
... and that's it, pretty much. This is completely thread and concurrency safe, will not maintain/require long-term locks, nor require serialized access to the table.
I have a list of integer values X and I want a primary key for my table Y to come from that list of integer values. I was wondering if this could be done using sequences. In other words: is there a way to tell a PostgreSQL sequence to use this list X to generate primary keys for my table Y?
One way of doing this would be to use a sequence that contains the index of the last used integer from the list X, call setval() on the sequence, get the next value and try to insert it into my table Y. In case of concurrent requests there will be an error, in which case I need to try with the next value from the list X. I would like to know what other (better) ways there are to achieve what I intend to do.
Could work like this:
-- DROP SCHEMA x CASCADE;
CREATE SCHEMA x;
CREATE TABLE x.priv_id(seq_id int primary key, id int);
INSERT INTO x.priv_id
SELECT generate_series(1,100,1), (random() * 1000)::int;
CREATE SEQUENCE x.priv_seq;
SELECT id
FROM x.priv_id
WHERE seq_id = (SELECT nextval('x.priv_seq'));
Major points:
1) Create a lookup table with two numbers
- seq_id is counting from 1 and your primary key.
- id is your numbers in sequence (I substituted random numbers here).
2) Create a helper sequence.
3) Get your numbers with a SELECT like above.
You need the subselect, or all values will be returned at once.
This solution gives all the security nextval() has to offer for concurrency.
Create a unique index on priv_id(id) if you want to make sure your custom id's are unique.
generate_series can be used to produce a list of all numbers every generated by a sequence:
select *
from generate_series(1, (select last_value from my_sequence), 1) as x
Note: This assumes the sequence (here my_sequence) started at 1 and incremented by 1. To change these assumptions, change the parameters appropriately.
The most general way to provide your own set of valid numbers is to store them in a table, and set a foreign key reference to it.
That solves the problem of using your own list, but it doesn't solve the problem of fetching the next number from the list. To do that, you'd need to write your own function. (You can't compel a PostgreSQL sequence to operate on your table instead of using its internals.) It's not particularly hard to do, though.
Another alternative that's often overlooked--perhaps with good reason--is writing your own sequence. PostgreSQL is open source; you can write your own sequencer based on existing sequence code. Compile it, test it, and you're off to the races. (I don't recommend doing that, though.)
I run a small site and use PostgreSQL 8.2.17 (only version available at my host) to store data.
In the last few months there were 3 crashes of the database system on my server and every time it happened 31 ID's from a serial field (primary key) in one of the tables were missing. There are now 93 ID's missing.
Table:
CREATE TABLE "REGISTRY"
(
"ID" serial NOT NULL,
"strUID" character varying(11),
"strXml" text,
"intStatus" integer,
"strUIDOrg" character varying(11),
)
It is very important for me that all the ID values are there. What can I do to to solve this problem?
You can not expect serial column to not have holes.
You can implement gapless key by sacrificing concurrency like this:
create table registry_last_id (value int not null);
insert into registry_last_id values (-1);
create function next_registry_id() returns int language sql volatile
as $$
update registry_last_id set value=value+1 returning value
$$;
create table registry ( id int primary key default next_registry_id(), ... )
But any transaction, which tries to insert something to registry table will block until other insert transaction finishes and writes its data to disk. This will limit you to no more than 125 inserting transactions per second on 7500rpm disk drive.
Also any delete from registry table will create a gap.
This solution is based on article Gapless Sequences for Primary Keys by A. Elein Mustain, which is somewhat outdated.
Are you missing 93 records or do you have 3 "holes" of 31 missing numbers?
A sequence is not transaction safe, it will never rollback. Therefor it is not a system to create a sequence of numbers without holes.
From the manual:
Important: To avoid blocking
concurrent transactions that obtain
numbers from the same sequence, a
nextval operation is never rolled
back; that is, once a value has been
fetched it is considered used, even if
the transaction that did the nextval
later aborts. This means that aborted
transactions might leave unused
"holes" in the sequence of assigned
values. setval operations are never
rolled back, either.
Thanks to the answers from Matthew Wood and Frank Heikens i think i have a solution.
Instead of using serial field I have to create my own sequence and define CACHE parameter to 1. This way postgres will not cache values and each one will be taken directly from the sequence :)
Thanks for all your help :)