I'm using PostgreSQL 12.5 on AWS for millions of events. My tables are partitioned by day by day so when the request comes, I use "CREATE TABLE IF NOT EXISTS ... PARTITION OF" syntax for every request.
I expect to skip the command if the table already exists. But PostgreSQL didn't do that. PostgreSQL tried to create tables multiple times instead of skip.
PostgreSQL Error Logs are following
2021-05-30 00:00:00 UTC:IP_ADDRESS(24921):dbname#username:[16666]:ERROR: relation "tablename_20210530" already exists
2021-05-30 00:00:00 UTC:IP_ADDRESS(24921):dbname#username:[16666]:STATEMENT: CREATE TABLE IF NOT EXISTS tablename_20210530 PARTITION OF tablename FOR VALUES FROM ('2021-05-30') TO ('2021-05-31')
The SQL Exception is following;
{
"errorType": "error",
"errorMessage": "relation \"tablename_20210530\" already exists",
"code": "42P07",
"length": 110,
"name": "error",
"severity": "ERROR",
"file": "heap.c",
"line": "1155",
"routine": "heap_create_with_catalog",
"stack": [
"error: relation \"tablename_20210530\" already exists",
" at Parser.parseErrorMessage (/var/task/node_modules/pg-protocol/dist/parser.js:278:15)",
" at Parser.handlePacket (/var/task/node_modules/pg-protocol/dist/parser.js:126:29)",
" at Parser.parse (/var/task/node_modules/pg-protocol/dist/parser.js:39:38)",
" at Socket.<anonymous> (/var/task/node_modules/pg-protocol/dist/index.js:10:42)",
" at Socket.emit (events.js:315:20)",
" at addChunk (internal/streams/readable.js:309:12)",
" at readableAddChunk (internal/streams/readable.js:284:9)",
" at Socket.Readable.push (internal/streams/readable.js:223:10)",
" at TCP.onStreamRead (internal/stream_base_commons.js:188:23)"
]
}
Any idea why this is happening?
PostgreSQL Version:
PostgreSQL 12.5 on x86_64-pc-linux-gnu, compiled by gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-11), 64-bit
Table Scheme:
create table tablename
(
id bigint default nextval('tablename_partition_id_seq'::regclass) not null,
col1 timestamp(0) default CURRENT_TIMESTAMP not null,
col2 varchar(255) not null,
constraint tablename_partition_pkey
primary key (id, col1)
)
partition by RANGE ("col1");
create table tablename_20210530
partition of tablename
(
constraint tablename_20210530_pkey
primary key (id, col1)
)
FOR VALUES FROM ('2021-05-30 00:00:00') TO ('2021-05-31 00:00:00');
SQL Statement:
CREATE TABLE IF NOT EXISTS tablename_20210530 PARTITION OF tablename FOR VALUES FROM ('2021-05-30') TO ('2021-05-31');
My code runs the create statement before the insert statement under the thousands of requests. Because every request has a different "col1" value. And I can not control that. so the create table statement is running every time.
Related
I'm trying to implement auto-partitioning of a table
CREATE TABLE incoming_ais_messages (
id uuid NOT NULL,
"source" int4 NOT NULL,
ais_channel varchar(8) NOT NULL,
is_read bool NOT NULL,
"time_stamp" timestamptz NOT null,
address_type varchar(32) NOT NULL,
"text" varchar NOT NULL,
CONSTRAINT incoming_ais_messages_pkey PRIMARY KEY (id,time_stamp)
) partition by range ("time_stamp");
For that I use a function:
create or replace function create_partition() returns trigger as $auto_partition$
begin
raise notice 'create_partion called';
execute 'create table if not exists incoming_ais_messages_partition_' || to_char(now()::date, 'yyyy_mm_dd') || ' partition of incoming_ais_messages
for values from (''' || to_char(now()::date, 'yyyy-mm-dd') || ''') to (''' || to_char((now() + interval '1 day')::date, 'yyyy-mm-dd') || ''');';
return new;
end;
$auto_partition$ language plpgsql;
And a trigger that should call it before any inserts:
create trigger auto_partition
before insert on incoming_ais_messages
for each row
execute procedure create_partition();
However when I insert something like:
INSERT INTO incoming_ais_messages (id, "source", ais_channel, is_read, "time_stamp", address_type, "text")
VALUES('123e4567-e89b-12d3-a456-426614174000'::uuid, 0, 'A', false, now(), 'DIRECT', 'text');
I get ther error:
SQL Error [23514]: ERROR: no partition of relation "incoming_ais_messages" found for row
Detail: Partition key of the failing row contains (time_stamp) = (2022-07-21 18:01:41.787604+03).
After that I created the partition manually:
create table if not exists incoming_ais_messages_partition_1970_01_01 partition of incoming_ais_messages
for values from (now()::date) to ((now() + interval '1 day')::date);
executed the same insert statement and got the error:
SQL Error [55006]: ERROR: cannot CREATE TABLE .. PARTITION OF "incoming_ais_messages" because it is being used by active queries in this session
Where: SQL statement "create table if not exists incoming_ais_messages_partition_2022_07_21 partition of incoming_ais_messages
for values from ('2022-07-21') to ('2022-07-22');"
PL/pgSQL function create_partition() line 4 at EXECUTE
Would be great to know what is wrong here. My solution is based on the approach described here https://evilmartians.com/chronicles/a-slice-of-life-table-partitioning-in-postgresql-databases
(Section: Bonus: how to create partitions)
PostgreSQL wants to know which partition the new rows will go into before it calls BEFORE ROW triggers, so the error is thrown before the CREATE gets a chance to run. (Note that the blog example is using a trigger on one table to create partition for a different table).
Doing what you want is possible (timescaledb extension does it, and you could research how if you want), but do yourself a favor and just pre-create a lot of partitions, and add a note to your calendar to add more in the future (as well as dropping old ones). Or write a cron job to do it.
I got this error in hasura console:
{ "internal": { "statement": "ALTER TABLE users\nADD id Integer auto-increment;", "prepared": false, "error": { "exec_status": "FatalError", "hint": null, "message": "syntax error at or near \"auto\"", "status_code": "42601", "description": null }, "arguments": [] }, "path": "$[0]", "error": "query execution failed", "code": "postgres-error" }
After checking my table, I find that I don't give the id column to the integer (auto-increment) datatype now id column is attack with another table so I can not delete it so I want to change id column to auto-increment and I use Postgres database with Hasura and graphql
This is my id look like
id - integer, primary key, unique, default: nextval('users_id_seq'::regclass)
If you have not a SEQUENCE firstly you must create sequence for table:
CREATE
SEQUENCE test_table_id_seq
INCREMENT BY 1
MINVALUE 1
MAXVALUE 2147483647
START 1
CACHE 1
NO CYCLE;
After than you can add this sequence to your table:
ALTER TABLE
test_table
ALTER COLUMN
id SET DEFAULT nextval('test_table_id_seq'::regclass);
After than, you must update your sequence value to max(id) of table:
SELECT setval('test_table_id_seq', (SELECT MAX(id)+1 FROM test_table));
Below is the create table statement which has successfully created the table in the PostgreSQL database.
CREATE TABLE
SYNC_REPOSITORIES (
REPOSITORY_ID BIGINT NOT NULL GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
REPOSITORY_NAME VARCHAR(256) NOT NULL,
REPOSITORY_KIND VARCHAR(256) NOT NULL,
LOGIN_NAME VARCHAR(256),
LOGIN_INFO VARCHAR(2000),
CONSTRAINT REPO_NAME_U UNIQUE (REPOSITORY_NAME));
Mybatis mapper.xml
for insert
<insert id="insertRepository" parameterType="RepositoryDO" useGeneratedKeys="true" keyProperty="id" keyColumn="REPOSITORY_ID">
INSERT INTO SYNC_REPOSITORIES (
REPOSITORY_NAME,
REPOSITORY_KIND,
LOGIN_NAME,
LOGIN_INFO)
VALUES (
#{name},
#{kind},
#{loginName,jdbcType=VARCHAR},
#{loginInfo,jdbcType=VARCHAR})
</insert>
Since REPOSITORY_ID is auto generated value, I have not added it in the insert statement.
When this is executed via sqlSessionFactory , it returns an error saying
The error occurred while setting parameters
SQL: INSERT INTO SYNC_REPOSITORIES ( REPOSITORY_NAME, REPOSITORY_KIND, LOGIN_NAME, LOGIN_INFO) VALUES ( ?, ?, ?, ?)
### Cause: org.postgresql.util.PSQLException: ERROR: column "REPOSITORY_ID" does not exist
Position: 194
; bad SQL grammar []; nested exception is org.postgresql.util.PSQLException: ERROR: column "REPOSITORY_ID" does not exist
Position: 194
The same mapper xml works for MSSQL and also ORACLE database. I am new to PostgreSQL.
I am trying to restore database using psql. But it seems like psql always fail when it encounter CONSTRAINT. After I look into the dump. I found out that the child table, the table that hold FOREIGN KEY, is created before the parent table.
Here is the snippet...
DROP TABLE IF EXISTS "answer";
DROP SEQUENCE IF EXISTS answer_id_seq;
CREATE SEQUENCE answer_id_seq INCREMENT 1 MINVALUE 1 MAXVALUE 2147483647 START 71 CACHE 1;
CREATE TABLE "public"."answer" (
"id" integer DEFAULT nextval('answer_id_seq') NOT NULL,
"text" character varying NOT NULL,
"weight" double precision NOT NULL,
"questionId" integer NOT NULL,
"baseCreated" timestamp DEFAULT now() NOT NULL,
"baseUpdated" timestamp DEFAULT now() NOT NULL,
CONSTRAINT "PK_9232db17b63fb1e94f97e5c224f" PRIMARY KEY ("id"),
CONSTRAINT "FK_a4013f10cd6924793fbd5f0d637" FOREIGN KEY ("questionId") REFERENCES question(id) ON DELETE CASCADE NOT DEFERRABLE
) WITH (oids = false);
psql command.
psql -h 0.0.0.0 -p 5432 -U foobar -1 foobar < foobar.sql
And the error.
NOTICE: table "answer" does not exist, skipping
DROP TABLE
NOTICE: sequence "answer_id_seq" does not exist, skipping
DROP SEQUENCE
CREATE SEQUENCE
ERROR: relation "question" does not exist
ERROR: relation "answer" does not exist
ERROR: relation "answer" does not exist
LINE 1: INSERT INTO "answer" ("id", "text", "weight", "questionId", ...
I am using Postgres 11 and SymmetricDS 3.9.14.
I have a database table with primary key of UUID type. It seems like SymmetricDS is not able to cast 'UUID' correctly. One of SymmetricDS replication query is failing with the below error
JdbcSqlTemplate - SQL caused exception: [select "id" from "dbo"."groups" where "id"=?]
sql args: [ ]
org.postgresql.util.PSQLException: ERROR: invalid input syntax for type uuid: " "
my insert statement :-
INSERT INTO dbo.groups(
id, sortorder, name, hidden, sessionid, creationtime, modificationtime, regionid)
VALUES ('5A171D3F-F6A6-4D09-AE89-73B5793DA171', 1, 'abc', false, null,'2018-11-20 20:25:49.663', null, null);
my database table is :-
CREATE TABLE dbo.groups
(
id uuid NOT NULL,
sortorder integer NOT NULL DEFAULT 0,
name character varying(80) COLLATE pg_catalog."default",
hidden boolean NOT NULL DEFAULT false,
sessionid uuid,
creationtime timestamp without time zone DEFAULT CURRENT_TIMESTAMP,
modificationtime timestamp without time zone,
regionid uuid,
CONSTRAINT "PK_dbo.Groups" PRIMARY KEY (id)
)
EDIT:
My source database is MS SQL Server and target database is Postgres
The value of UUID in this case is a space or tab as it could be seen from the error message
invalid input syntax for type uuid: " "
Try finding why instead of the concrete UUID value this space/tab value is passed