I am updating a table with data from another table from another database. Most columns can be moved with no change but some of them require mapping/change after import since they refer to the new database ids. My first attempt was to make insert with returning values and use those values in update on the same table, with additional joins. It does not work. Postgres inserts records and ignores updates, with no error. Simplified example of the case below.
Create table test(
idtst serial ,
name varchar(10),
nameid varchar(20));
with tst as ( -- makes insert, no update
Insert into test(name,nameid) values ('tsta','tsta')
returning idtst,nameid)
update test t
set nameid=t.nameid||t.idtst::varchar(5)
from tst i
where t.idtst=i.idtst;
with tst as ( -- makes insert, no update
Insert into test(name,nameid) values ('tstb','tstb')
returning idtst,nameid)
update test t
set nameid=i.nameid||i.idtst::varchar(5)
from tst i
where t.idtst=i.idtst;
Create temp table tmp_tst(
idtst integer,nameid varchar(20));
with tst as (
Insert into test(name,nameid) values ('tstc','tstc')
returning idtst,nameid)
Insert into tmp_tst Select * from tst; -- insert into tmp
update test t -- makes update
set nameid=i.nameid||i.idtst::varchar(5)
from tmp_tst i
where t.idtst=i.idtst;
Does it depend on postgres version/configuration ?
In test in result there are records
1,tsta,tsta
2,tstb,tstb
3,tstc,tstc3
No errors, no warnings, just psql ignores first two updates. Updates/Inserts on any other table are OK.
What is table i? Where does it come from?
My guess would be, that there is no lines which fullfill t.idtst=i.idtst
Maybe you should look at this:
SELECT *
FROM test as t, i
WHERE t.idtst=i.idtst
I have the following table:
DROP TABLE IF EXISTS TBL_CACL;
CREATE TABLE TBL_CACL (
ID INT GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY,
NAME VARCHAR(250) UNIQUE NOT NULL
);
I am able to query postgres likes this:
SELECT c.relname FROM pg_class c WHERE c.relkind = 'S';
to determine that the default name for the sequence on the table is tbl_cacl_id_seq
I can then read its nextVal like this:
SELECT nextval('tbl_cacl_id_seq');
Directly after creating the table the value is '1'
However, if I insert several rows of data like this:
INSERT INTO TBL_CACL
VALUES
(1, 'CACL_123'),
(2, 'CACL_234'),
(3, 'CACL_345');
and I read nextVal for the table, it returns '2'
I would have thought that tbl_cacl_id_seq would be '4'. Clearly I misunderstanding how the insert is related to the nextVal. Why is the sequence out of sync with the inserts and how do I get them in sequence? Thanks for any advice.
The tbl_cacl_id_seq gets incremented only when the function nextval('tbl_cacl_id_seq') is called. This function call will not happen since you have provided values for ID column during insert (hence no need to get the default value).
Converting below SQL Server procedures and tables to store and generate sequence to postgresql.
Can anyone guide how to do this in Postgres (via table and this function) and not via sequence or nextval or currval
Sequence table
IF NOT EXISTS (SELECT name FROM sys.tables WHERE name = 'testtable')
CREATE TABLE dbo.testtable(Sequence int NOT NULL )
go
IF NOT EXISTS (SELECT * FROM testtable)
INSERT INTO testtable VALUES (-2147483648)
go
Sequence generating proc
CREATE PROCEDURE test_proc
AS
SET NOCOUNT ON
DECLARE #iReturn int
BEGIN TRANSACTION
SELECT #iReturn = Sequence FROM schema.test (TABLOCKX) -- set exclusive table lock
UPDATE schema.test SET Sequence = ( Sequence + 1 )
COMMIT TRANSACTION
SELECT #iReturn
RETURN #iReturn
go
grant execute on schema.test to public
go
Disclaimer: using a sequence is the only scalable and efficient way to generate unique numbers.
Having said that, it is possible to implement your own sequence generator. The only situation where makes any sense is, if you are required to generate gapless numbers. If you do not have such a requirement, use a sequence.
You need one table that stores the values of the sequences. I usually use one table with a row for each "generator" that avoids costly table locks.
create table seq_generator
(
entity varchar(30) not null primary key,
seq_value integer default 0 not null
);
insert into seq_generator (entity) values ('testsequence');
Then create a function to increment the sequence value:
create or replace function next_value(p_entity varchar)
returns integer
as
$$
update seq_generator
set seq_value = seq_value + 1
where entity = lower(p_entity)
returning seq_value;
$$
language sql;
To obtain the next sequence value, e.g. inside an insert:
insert into some_table
(id, ...)
values
(next_value('testsequence'), ...);
Or make it a default value:
create table some_table
(
id integer not null primary key default next_value('testsequence'),
...
);
The UPDATE increments and locks the row in a single statement returning the new value for the sequence. If the calling transaction commits, the update to seq_generator will also be committed. If the calling transaction rolls back, the update will roll back as well.
If a second transaction calls next_value() for the same entity, it has to wait until the first transaction commits or rolls back.
So access to the generator is serialized through this function. Only one transaction at a time can do that.
If you need a second gapless sequence, just insert a new row in the `seq_generator' table.
This will seriously affect performance when you use in an environment that does a lot of concurrent inserts.
The only reason that would justify this is a legal requirement to have a gapless number. In every other case you should really, really use a native Postgres sequence.
Is it possible to do a bulk insert into REdshift using the create table as syntax while defining data type and encoding at the same time? What's the correct syntax?
EG The following gives a syntax error near 'as':
create table my_table (
a int not null encode runlength,
b int not null encode runlength
) distkey(a) sortkey (a, b) as (
select * from other_table
);
I can only get it to work by defining column name only (a or b) and that's a huge limitation...
You can specify the DIST and SORT keys in a CREATE TABLE … AS query like this:
CREATE TABLE new_table
DISTSTYLE KEY
DISTKEY ( my_dist )
SORTKEY ( my_sort )
AS
SELECT *
FROM old_table
;
As per the docs: http://docs.aws.amazon.com/redshift/latest/dg/r_CREATE_TABLE_AS.html I don't believe you can alter the compression encoding from the source table using CREATE TABLE AS.
More details on Redshift CTAS is given here: http://docs.aws.amazon.com/redshift/latest/dg/r_CTAS_usage_notes.html . In a nutshell, no where its mentioned that you can define the encoding in the CTAS statement. But you can define Sort Keys and Hash Keys. The default encoding chosesn by this statement is none.
However if you want to do a bulk insert, why don't you do in two steps.
Create table new_table with your encoding and sort/hash keys
Insert into new_table as select * from old_table
In PostgreSQL, how do I get the last id inserted into a table?
In MS SQL there is SCOPE_IDENTITY().
Please do not advise me to use something like this:
select max(id) from table
( tl;dr : goto option 3: INSERT with RETURNING )
Recall that in postgresql there is no "id" concept for tables, just sequences (which are typically but not necessarily used as default values for surrogate primary keys, with the SERIAL pseudo-type).
If you are interested in getting the id of a newly inserted row, there are several ways:
Option 1: CURRVAL(<sequence name>);.
For example:
INSERT INTO persons (lastname,firstname) VALUES ('Smith', 'John');
SELECT currval('persons_id_seq');
The name of the sequence must be known, it's really arbitrary; in this example we assume that the table persons has an id column created with the SERIAL pseudo-type. To avoid relying on this and to feel more clean, you can use instead pg_get_serial_sequence:
INSERT INTO persons (lastname,firstname) VALUES ('Smith', 'John');
SELECT currval(pg_get_serial_sequence('persons','id'));
Caveat: currval() only works after an INSERT (which has executed nextval() ), in the same session.
Option 2: LASTVAL();
This is similar to the previous, only that you don't need to specify the sequence name: it looks for the most recent modified sequence (always inside your session, same caveat as above).
Both CURRVAL and LASTVAL are totally concurrent safe. The behaviour of sequence in PG is designed so that different session will not interfere, so there is no risk of race conditions (if another session inserts another row between my INSERT and my SELECT, I still get my correct value).
However they do have a subtle potential problem. If the database has some TRIGGER (or RULE) that, on insertion into persons table, makes some extra insertions in other tables... then LASTVAL will probably give us the wrong value. The problem can even happen with CURRVAL, if the extra insertions are done intto the same persons table (this is much less usual, but the risk still exists).
Option 3: INSERT with RETURNING
INSERT INTO persons (lastname,firstname) VALUES ('Smith', 'John') RETURNING id;
This is the most clean, efficient and safe way to get the id. It doesn't have any of the risks of the previous.
Drawbacks? Almost none: you might need to modify the way you call your INSERT statement (in the worst case, perhaps your API or DB layer does not expect an INSERT to return a value); it's not standard SQL (who cares); it's available since Postgresql 8.2 (Dec 2006...)
Conclusion: If you can, go for option 3. Elsewhere, prefer 1.
Note: all these methods are useless if you intend to get the last inserted id globally (not necessarily by your session). For this, you must resort to SELECT max(id) FROM table (of course, this will not read uncommitted inserts from other transactions).
Conversely, you should never use SELECT max(id) FROM table instead one of the 3 options above, to get the id just generated by your INSERT statement, because (apart from performance) this is not concurrent safe: between your INSERT and your SELECT another session might have inserted another record.
See the RETURNING clause of the INSERT statement. Basically, the INSERT doubles as a query and gives you back the value that was inserted.
Leonbloy's answer is quite complete. I would only add the special case in which one needs to get the last inserted value from within a PL/pgSQL function where OPTION 3 doesn't fit exactly.
For example, if we have the following tables:
CREATE TABLE person(
id serial,
lastname character varying (50),
firstname character varying (50),
CONSTRAINT person_pk PRIMARY KEY (id)
);
CREATE TABLE client (
id integer,
CONSTRAINT client_pk PRIMARY KEY (id),
CONSTRAINT fk_client_person FOREIGN KEY (id)
REFERENCES person (id) MATCH SIMPLE
);
If we need to insert a client record we must refer to a person record. But let's say we want to devise a PL/pgSQL function that inserts a new record into client but also takes care of inserting the new person record. For that, we must use a slight variation of leonbloy's OPTION 3:
INSERT INTO person(lastname, firstname)
VALUES (lastn, firstn)
RETURNING id INTO [new_variable];
Note that there are two INTO clauses. Therefore, the PL/pgSQL function would be defined like:
CREATE OR REPLACE FUNCTION new_client(lastn character varying, firstn character varying)
RETURNS integer AS
$BODY$
DECLARE
v_id integer;
BEGIN
-- Inserts the new person record and retrieves the last inserted id
INSERT INTO person(lastname, firstname)
VALUES (lastn, firstn)
RETURNING id INTO v_id;
-- Inserts the new client and references the inserted person
INSERT INTO client(id) VALUES (v_id);
-- Return the new id so we can use it in a select clause or return the new id into the user application
RETURN v_id;
END;
$BODY$
LANGUAGE plpgsql VOLATILE;
Now we can insert the new data using:
SELECT new_client('Smith', 'John');
or
SELECT * FROM new_client('Smith', 'John');
And we get the newly created id.
new_client
integer
----------
1
you can use RETURNING clause in INSERT statement,just like the following
wgzhao=# create table foo(id int,name text);
CREATE TABLE
wgzhao=# insert into foo values(1,'wgzhao') returning id;
id
----
1
(1 row)
INSERT 0 1
wgzhao=# insert into foo values(3,'wgzhao') returning id;
id
----
3
(1 row)
INSERT 0 1
wgzhao=# create table bar(id serial,name text);
CREATE TABLE
wgzhao=# insert into bar(name) values('wgzhao') returning id;
id
----
1
(1 row)
INSERT 0 1
wgzhao=# insert into bar(name) values('wgzhao') returning id;
id
----
2
(1 row)
INSERT 0
The other answers don't show how one might use the value(s) returned by RETURNING. Here's an example where the returned value is inserted into another table.
WITH inserted_id AS (
INSERT INTO tbl1 (col1)
VALUES ('foo') RETURNING id
)
INSERT INTO tbl2 (other_id)
VALUES ((select id from inserted_id));
See the below example
CREATE TABLE users (
-- make the "id" column a primary key; this also creates
-- a UNIQUE constraint and a b+-tree index on the column
id SERIAL PRIMARY KEY,
name TEXT,
age INT4
);
INSERT INTO users (name, age) VALUES ('Mozart', 20);
Then for getting last inserted id use this for table "user" seq column name "id"
SELECT currval(pg_get_serial_sequence('users', 'id'));
SELECT CURRVAL(pg_get_serial_sequence('my_tbl_name','id_col_name'))
You need to supply the table name and column name of course.
This will be for the current session / connection
http://www.postgresql.org/docs/8.3/static/functions-sequence.html
For the ones who need to get the all data record, you can add
returning *
to the end of your query to get the all object including the id.
You can use RETURNING id after insert query.
INSERT INTO distributors (id, name) VALUES (DEFAULT, 'ALI') RETURNING id;
and result:
id
----
1
In the above example id is auto-increment filed.
The better way is to use Insert with returning. Though there are already same answers, I just want to add, if you want to save this to a variable then you can do this
insert into my_table(name) returning id into _my_id;
Postgres has an inbuilt mechanism for the same, which in the same query returns the id or whatever you want the query to return.
here is an example. Consider you have a table created which has 2 columns column1 and column2 and you want column1 to be returned after every insert.
# create table users_table(id serial not null primary key, name character varying);
CREATE TABLE
#insert into users_table(name) VALUES ('Jon Snow') RETURNING id;
id
----
1
(1 row)
# insert into users_table(name) VALUES ('Arya Stark') RETURNING id;
id
----
2
(1 row)
Try this:
select nextval('my_seq_name'); // Returns next value
If this return 1 (or whatever is the start_value for your sequence), then reset the sequence back to the original value, passing the false flag:
select setval('my_seq_name', 1, false);
Otherwise,
select setval('my_seq_name', nextValue - 1, true);
This will restore the sequence value to the original state and "setval" will return with the sequence value you are looking for.
I had this issue with Java and Postgres.
I fixed it by updating a new Connector-J version.
postgresql-9.2-1002.jdbc4.jar
https://jdbc.postgresql.org/download.html:
Version 42.2.12
https://jdbc.postgresql.org/download/postgresql-42.2.12.jar
Based on #ooZman 's answer above, this seems to work for PostgreSQL v12 when you need to INSERT with the next value of a "sequence" (akin to auto_increment) without goofing anything up in your table(s) counter(s). (Note: I haven't tested it in more complex DB cluster configurations though...)
Psuedo Code
$insert_next_id = $return_result->query("select (setval('"your_id_seq"', (select nextval('"your_id_seq"')) - 1, true)) +1");