I am trying to restore database using psql. But it seems like psql always fail when it encounter CONSTRAINT. After I look into the dump. I found out that the child table, the table that hold FOREIGN KEY, is created before the parent table.
Here is the snippet...
DROP TABLE IF EXISTS "answer";
DROP SEQUENCE IF EXISTS answer_id_seq;
CREATE SEQUENCE answer_id_seq INCREMENT 1 MINVALUE 1 MAXVALUE 2147483647 START 71 CACHE 1;
CREATE TABLE "public"."answer" (
"id" integer DEFAULT nextval('answer_id_seq') NOT NULL,
"text" character varying NOT NULL,
"weight" double precision NOT NULL,
"questionId" integer NOT NULL,
"baseCreated" timestamp DEFAULT now() NOT NULL,
"baseUpdated" timestamp DEFAULT now() NOT NULL,
CONSTRAINT "PK_9232db17b63fb1e94f97e5c224f" PRIMARY KEY ("id"),
CONSTRAINT "FK_a4013f10cd6924793fbd5f0d637" FOREIGN KEY ("questionId") REFERENCES question(id) ON DELETE CASCADE NOT DEFERRABLE
) WITH (oids = false);
psql command.
psql -h 0.0.0.0 -p 5432 -U foobar -1 foobar < foobar.sql
And the error.
NOTICE: table "answer" does not exist, skipping
DROP TABLE
NOTICE: sequence "answer_id_seq" does not exist, skipping
DROP SEQUENCE
CREATE SEQUENCE
ERROR: relation "question" does not exist
ERROR: relation "answer" does not exist
ERROR: relation "answer" does not exist
LINE 1: INSERT INTO "answer" ("id", "text", "weight", "questionId", ...
Related
I have an issue when trying to modify and existing PostgreSQL (version 13.3) table to support partitioning it gets stuck when inserting the new data from the old table because the inserted timestamp in some cases may not be unique, so it fails on execution.
The partition forces me to create the primary to be the range (timestamp) value. You can see the new table definition below:
CREATE TABLE "UserFavorites_master" (
"Id" int4 NOT NULL GENERATED BY DEFAULT AS IDENTITY,
"UserId" int4 NOT NULL,
"CardId" int4 NOT NULL,
"CreationDate" timestamp NOT NULL,
CONSTRAINT "PK_UserFavorites_CreationDate" PRIMARY KEY ("CreationDate")
) partition by range ("CreationDate");
The original table didn't have a constraint on timestamp to either be unique or a primary key nor would we particularly want that but that seems to be a requirement of partitioning. Looking for alternatives or good ideas to solve the issue.
You can see the full code below:
alter table "UserFavorites" rename to "UserFavorites_old";
CREATE TABLE "UserFavorites_master" (
"Id" int4 NOT NULL GENERATED BY DEFAULT AS IDENTITY,
"UserId" int4 NOT NULL,
"CardId" int4 NOT NULL,
"CreationDate" timestamp NOT NULL,
CONSTRAINT "PK_UserFavorites_CreationDate" PRIMARY KEY ("CreationDate")
) partition by range ("CreationDate");
-- Frome Reference: https://stackoverflow.com/a/53600145/1190540
create or replace function createPartitionIfNotExists(forDate timestamp) returns void
as $body$
declare yearStart date := date_trunc('year', forDate);
declare yearEndExclusive date := yearStart + interval '1 year';
declare tableName text := 'UserFavorites_Partition_' || to_char(forDate, 'YYYY');
begin
if to_regclass(tableName) is null then
execute format('create table %I partition of "UserFavorites_master" for values from (%L) to (%L)', tableName, yearStart, yearEndExclusive);
-- Unfortunatelly Postgres forces us to define index for each table individually:
--execute format('create unique index on %I (%I)', tableName, 'UserId'::text);
end if;
end;
$body$ language plpgsql;
do
$$
declare rec record;
begin
loop
for rec in 2015..2030 loop
-- ... and create a partition for them
perform createPartitionIfNotExists(to_date(rec::varchar,'yyyy'));
end loop;
end
$$;
create or replace view "UserFavorites" as select * from "UserFavorites_master";
insert into "UserFavorites" ("Id", "UserId", "CardId", "CreationDate") select * from "UserFavorites_old";
It fails on the Last line with the following error:
SQL Error [23505]: ERROR: duplicate key value violates unique constraint "UserFavorites_Partition_2020_pkey"
Detail: Key ("CreationDate")=(2020-11-02 09:38:54.997) already exists.
ERROR: duplicate key value violates unique constraint "UserFavorites_Partition_2020_pkey"
Detail: Key ("CreationDate")=(2020-11-02 09:38:54.997) already exists.
ERROR: duplicate key value violates unique constraint "UserFavorites_Partition_2020_pkey"
Detail: Key ("CreationDate")=(2020-11-02 09:38:54.997) already exists.
No, partitioning doesn't force you to create a primary key. Just omit that line, and your example should work.
However, you definitely always should have a primary key on your tables. Otherwise, you can end up with identical rows, which is a major headache in a relational database. You might have to clean up your data.
#Laurenz Albe is correct, it seems I also have the ability to specify multiple keys though it may affect performance as referenced here Multiple Keys Performance, even indexing the creation date of the partition seemed to make the performance worse.
You can see a reference to multiple keys below, you mileage may vary.
CREATE TABLE "UserFavorites_master" (
"Id" int4 NOT NULL GENERATED BY DEFAULT AS IDENTITY,
"UserId" int4 NOT NULL,
"CardId" int4 NOT NULL,
"CreationDate" timestamp NOT NULL,
CONSTRAINT "PK_UserFavorites" PRIMARY KEY ("Id", "CreationDate")
) partition by range ("CreationDate");
#Error description:
It's possible to create a table that has a foreign key into a hypertable provided the foreign key is defined when the table is created
#To Reproduce, there are next tables:
CREATE TABLE ids (
measurement_id int DEFAULT 0,
description text DEFAULT 0,
m_id bigserial NOT NULL,
service_id int DEFAULT NULL,
time bigint NOT NULL DEFAULT cast((EXTRACT(EPOCH FROM now() AT TIME ZONE 'UTC') * 1000) as bigint),
user_id int DEFAULT NULL,
end_time DOUBLE PRECISION DEFAULT 0,
start_time int NOT NULL DEFAULT 0
);
CREATE INDEX ON ids (time DESC, user_id);
CREATE INDEX ON ids (time DESC, service_id);
SELECT create_hypertable('ids', 'start_time', chunk_time_interval => 604800016);
---------
CREATE TABLE IF NOT EXISTS metrics (
id bigserial NOT NULL,
duration real DEFAULT NULL,
metric integer DEFAULT 0,
m_id bigint NOT NULL,
time bigint NOT NULL DEFAULT 0
);
ALTER TABLE metrics ADD PRIMARY KEY (time, m_id);
CREATE INDEX ON metrics (time DESC);
CREATE INDEX ON metrics (time DESC, measurement );
CREATE INDEX ON metrics (time DESC, m_id );
grant all privileges on ids, metrics to your_db_user;
SELECT create_hypertable('metrics', 'time' , chunk_time_interval => 604800016);
SELECT table_catalog, table_schema, table_name, privilege_type FROM information_schema.table_privileges WHERE grantee = 'your_db_user';
---------
DROP TABLE IF EXISTS resource;
CREATE TABLE resource(
id int NOT NULL,
cpu text DEFAULT 0,
storing text DEFAULT 0,
memory text DEFAULT 0
);
ALTER TABLE resource ADD PRIMARY KEY (id);
CREATE SEQUENCE resource_id_seq
INCREMENT 1
MINVALUE 1
MAXVALUE 2147483647
START 1
CACHE 1;
ALTER TABLE resource_id_seq
OWNER TO your_db_user;
ALTER TABLE resource ALTER COLUMN id SET DEFAULT nextval('resource_id_seq'::regclass);
---------
CREATE EXTENSION IF NOT EXISTS "uuid-ossp";
DROP TABLE IF EXISTS ns;
CREATE TABLE ns(
id bigint NOT NULL,
uuid uuid NOT NULL DEFAULT uuid_generate_v4 (),
availability double precision,
faultTolerance boolean,
activated boolean,
UNIQUE (id, uuid),
PRIMARY KEY(id),
CONSTRAINT fk_resource
FOREIGN KEY(id)
REFERENCES resource(id)
ON DELETE CASCADE
);
CREATE SEQUENCE ns_id_seq
INCREMENT 1
MINVALUE 1
MAXVALUE 9223372036854775807
START 1
CACHE 1;
ALTER TABLE ns_id_seq
OWNER TO your_db_user;
ALTER TABLE ns ALTER COLUMN id SET DEFAULT nextval('ns_id_seq'::regclass);
---------
DROP TABLE IF EXISTS authentication;
CREATE TABLE authentication(
id integer NOT NULL,
username character varying(255) NOT NULL,
password character varying(255) NOT NULL,
host character varying(255) NOT NULL,
port character varying(10) NOT NULL,
PRIMARY KEY(id)
);
CREATE SEQUENCE auth_id_seq
INCREMENT 1
MINVALUE 1
MAXVALUE 2147483647
START 1
CACHE 1;
ALTER TABLE auth_id_seq
OWNER TO your_db_user;
ALTER TABLE authentication ALTER COLUMN id SET DEFAULT nextval('auth_id_seq'::regclass);
---------
DROP TABLE IF EXISTS job;
CREATE TABLE job(
id int NOT NULL,
interval integer NOT NULL,
auth_id integer REFERENCES authentication (id),
ns_id integer REFERENCES ns (id),
UNIQUE (auth_id, ns_id),
PRIMARY KEY(id)
);
ALTER TABLE job
ADD CONSTRAINT fk_auth_id
FOREIGN KEY (id) REFERENCES authentication (id)
ON DELETE CASCADE
DEFERRABLE INITIALLY DEFERRED;
ALTER TABLE job
ADD CONSTRAINT fk_ns_id
FOREIGN KEY (id) REFERENCES ns (id)
ON DELETE CASCADE
DEFERRABLE INITIALLY DEFERRED;
CREATE SEQUENCE job_id_seq
INCREMENT 1
MINVALUE 1
MAXVALUE 2147483647
START 1
CACHE 1;
ALTER TABLE job_id_seq
OWNER TO your_db_user;
ALTER TABLE job ALTER COLUMN id SET DEFAULT nextval('job_id_seq'::regclass);
---------
DROP TABLE IF EXISTS job_metric;
CREATE TABLE job_metric (
id int NOT NULL,
j_id int NOT NULL REFERENCES job (id),
mj_id bigint NOT NULL,
jm_time bigint NOT NULL
);
CREATE INDEX ON job_metric (jm_time DESC);
CREATE INDEX ON job_metric (jm_time DESC, id);
CREATE INDEX ON job_metric (jm_time DESC, mj_id);
ALTER TABLE job_metric ADD PRIMARY KEY (jm_time, id);
grant all privileges on job_metric to your_db_user;
SELECT create_hypertable('job_metric', 'jm_time' , chunk_time_interval => 604800016);
CREATE SEQUENCE mjob_metric_id_seq
INCREMENT 1
MINVALUE 1
MAXVALUE 2147483647
START 1
CACHE 1;
ALTER TABLE mjob_metric_id_seq
OWNER TO your_db_user;
ALTER TABLE job_metric ALTER COLUMN id SET DEFAULT nextval('mjob_metric_id_seq'::regclass);
---------
After creating the tables, I have used the solution proposed by #Laurenz in a database with PostgreSQL 12.6 using the extension of timescaledb 1.7.5 as follows:
#To fill the table with the appropriate values:
UPDATE job_metric AS jm_point
SET jm_time = qm.time
FROM metrics AS qm
WHERE qm.m_id = jm_point.mj_id;
#Then set it NOT NULL:
ALTER TABLE job_metric ALTER jm_time SET NOT NULL;
#To define your foreign key:
ALTER TABLE job_metric
ADD FOREIGN KEY (mj_id, jm_time)
REFERENCES metrics (time, m_id) MATCH FULL;
#Response of the last reference table to enable foreign key: Query returned successfully in 40 msec.
Expected behavior:
The idea is to use the table job_metric in an even many-to-many relationship to access the information of job and metrics tables.
Actual behavior and error:
Tables are created and FKs were created but cannot be used when data is inserted at job_metric as is detailed in the following:
INSERT INTO job_metric (j_id, mj_id, jm_time)
VALUES(13, 185063, 1621957192266);
ERROR: foreign keys to hypertables are not supported CONTEXT: SQL
statement " ALTER TABLE _timescaledb_internal._hyper_5_5_chunk ADD
CONSTRAINT "5_13_job_metric_j_id_mj_id_jm_time_fkey" FOREIGN KEY
(j_id, mj_id, jm_time) REFERENCES qmetrics("time", m_id) MATCH FULL "
PL/pgSQL function
_timescaledb_internal.chunk_constraint_add_table_constraint(_timescaledb_catalog.chunk_constraint)
line 42 at EXECUTE SQL state: 0A000
***According to https://docs.timescale.com/timescaledb/latest/overview/limitations/##distributed-hypertable-limitations, it looks like the above error is part of the hypertable limitations:
Foreign key constraints referencing a hypertable are not supported.
#Request:
Given the above information and errors, does anyone know any solution at the DB level to establish the relationships (many-to-many or one-to-many) using timescaledb extension and mainly hypertables?
Actually, I have obtained similar of above error when I had attempted to create many-to-many relation among the tables metrics and job_metric using the Django Rest Framework:
class Job_Metrics(models.Model):
job = models.OneToOneField(Job, on_delete=models.CASCADE)
metrics = models.ManyToManyField(Metrics)
time = models.IntegerField(default=0)
Running the application metrics pointing out directly metrics_db:
$ python3 manage.py migrate metrics --database=metrics_db
Operations to perform: Apply all migrations: metrics Running migrations: Applying
metrics.0002_job...Traceback (most recent call last): File
"/var/myproject/myprojectenv/lib/python3.8/site-packages/django/db/backends/utils.py",
line 84, in _execute return self.cursor.execute(sql, params)
psycopg2.errors.FeatureNotSupported: foreign keys to hypertables are
not supported
If someone knows a solution or has an idea to deal with the above error at the REST API level, please could you share your idea with the aim to access data associated tables (metrics and jobs) and modify them together when is required to delete e.g., a job_metric. So far, using hypertables amendments of timescaledb extension seems to be not a viable solution.
I am facing an issue when correcting an existing table to use serial column on the primary key. In order to simulate the issue I created a new table:
CREATE TABLE markets."TestSequence" (
"Id" integer NOT NULL,
"Name" text COLLATE pg_catalog."default",
CONSTRAINT "PK_TestSequence" PRIMARY KEY ("Id")
);
Then I ran the query that is causing problem:
ALTER TABLE markets."TestSequence" ALTER COLUMN "Id" TYPE integer;
ALTER TABLE markets."TestSequence" ALTER COLUMN "Id" SET NOT NULL;
CREATE SEQUENCE "TestSequence_Id_seq" AS integer START WITH 1 INCREMENT BY 1 NO MINVALUE NO MAXVALUE NO CYCLE;
ALTER TABLE markets."TestSequence" ALTER COLUMN "Id" SET DEFAULT (nextval('"TestSequence_Id_seq"'));
ALTER SEQUENCE "TestSequence_Id_seq" OWNED BY "TestSequence"."Id";
I get the following error:
ERROR: relation "TestSequence" does not exist
SQL state: 42P01
According to the doc OWNED BY does not take any schema prefix. So I tried to create the table without schema and it works fine.
CREATE TABLE "TestSequence" (
"Id" integer NOT NULL,
"Name" text COLLATE pg_catalog."default",
CONSTRAINT "PK_TestSequence" PRIMARY KEY ("Id")
);
and run the corresponding alter queries:
ALTER TABLE "TestSequence" ALTER COLUMN "Id" TYPE integer;
ALTER TABLE "TestSequence" ALTER COLUMN "Id" SET NOT NULL;
CREATE SEQUENCE "TestSequence_Id_seq" AS integer START WITH 1 INCREMENT BY 1 NO MINVALUE NO MAXVALUE NO CYCLE;
ALTER TABLE "TestSequence" ALTER COLUMN "Id" SET DEFAULT (nextval('"TestSequence_Id_seq"'));
ALTER SEQUENCE "TestSequence_Id_seq" OWNED BY "TestSequence"."Id";
How can I make this work for relations with schema?
The doc you have linked says, for owned by
The specified table must have the same owner and be in the same schema
as the sequence.
You haven't specified a schema for the sequence, so it is created in public by default, which is not the same as the table schema.
Try creating the sequence as
CREATE SEQUENCE markets."TestSequence_Id_seq" AS integer ...
That being said, nothing prevents you from specifying the schema of both the sequence and the table
ALTER SEQUENCE markets."TestSequence_Id_seq" OWNED BY markets."TestSequence"."Id";
A table in production environment was created like this:
-- DDL generated by Postico 1.5.10
-- Not all database features are supported. Do not use for backup.
-- Table Definition ----------------------------------------------
CREATE TABLE "TempUser" (
id bigint PRIMARY KEY,
"Uuid" text NOT NULL,
"dateCreated" timestamp without time zone NOT NULL,
"lastRequestDate" timestamp without time zone NOT NULL,
"lastRequestLocation" text,
"numOfRequest" bigint NOT NULL,
comment text NOT NULL,
"lastRequestDescription" text NOT NULL,
"bookedSeats" text NOT NULL,
"bookUntil" timestamp without time zone,
"lastEventVisited" text
);
-- Indices -------------------------------------------------------
CREATE UNIQUE INDEX "pk:TempUser.id" ON "TempUser"(id int8_ops);
Why I remove table in dev environment and create like this in Postgres console, I got following error after try writing programmatically:
[ ERROR ] PostgreSQLError.server.error.ExecConstraints: POST /registerTempUser null value in column "id" violates not-null constraint (ErrorMiddleware.swift:26)
[ DEBUG ] Possible causes for PostgreSQLError.server.error.ExecConstraints: Failing row contains (null, 015574220836500, 2020-07-09 19:42:36.803846, 2020-07-09 19:42:36.803846, null, 1, , POST /registerTempUser HTTP/1.1
Host: localhost:8080
Content-Typ..., [], null, null). (ErrorMiddleware.swift:26)
Why, what is wrong?
Tried set sequence thing:
select setval(pg_get_serial_sequence('"TempUser"', 'id'), 1);
select currval(pg_get_serial_sequence('"TempUser"', 'id'));
nothing changed.
I am trying to implement triggers in hsql after update
where I have one table called component table and on update in that table I want to log it in another table using after insert trigger, for which I am doing
CREATE TABLE IF NOT EXISTS "component"(
"id" INTEGER IDENTITY,
"name" VARCHAR(100),
"configuration" LONGVARCHAR,
"owner_id" INTEGER );
CREATE TABLE IF NOT EXISTS "component_audit"(
"id" INTEGER IDENTITY,
"component_id" INTEGER ,
"action" VARCHAR(20),
"activity_time" BIGINT,
"user_id" INTEGER,
FOREIGN KEY ("component_id") REFERENCES "component"("id") ON UPDATE RESTRICT ON DELETE CASCADE
);
CREATE TRIGGER trig AFTER INSERT ON "component"
REFERENCING NEW ROW AS newrow
FOR EACH ROW
INSERT INTO "component_audit" ("id","component_id","action","activity_time","user_id")
VALUES (DEFAULT, 1, newrow.id, 123, 1);
On running HSQL throws error
Caused by: org.hsqldb.HsqlException: user lacks privilege or object
not found: NEWROW.ID
Its due to my id column being in "id" because I needed it in small caps (by DEFAULT HSQLDB is upper case)
how do I pass my variable subsitution ?
Just use the same naming as in your CREATE TABLE statement.
CREATE TRIGGER trig AFTER INSERT ON "component"
REFERENCING NEW ROW AS newrow
FOR EACH ROW
INSERT INTO "component_audit" ("id","component_id","action","activity_time","user_id")
VALUES (DEFAULT, 1, newrow."id", 123, 1);