I have been searching around stack overflow for some relevant problems, but I did not find any.
I have a table in sql on this format (call this file for create_table.sql):
CREATE TABLE object (
id BIGSERIAL PRIMARY KEY,
name_c VARCHAR(10) NOT NULL,
create_timestamp TIMESTAMP NOT NULL,
change_timestamp TIMESTAMP NOT NULL,
full_id VARCHAR(10),
mod VARCHAR(10) NOT NULL CONSTRAINT mod_enum CHECK (mod IN ('original', 'old', 'delete')),
status VARCHAR(10) NOT NULL CONSTRAINT status_enum CHECK (status IN ('temp', 'good', 'bad')),
vers VARCHAR(10) NOT NULL REFERENCES vers (full_id),
frame_id BIGINT NOT NULL REFERENCES frame (id),
name VARCHAR(10),
definition VARCHAR(10),
order_ref BIGINT REFERENCES order_ref (id),
UNIQUE (id, name_c)
);
This table is stored in google cloud. I have about 200000 insert statement, where I use a "insert block" method. Look like this (call this file for object_name.sql):
INSERT INTO object(
name,
create_timestamp,
change_timestamp,
full_id,
mod,
status,
vers,
frame_id,
name)
VALUES
('Element', current_timestamp, current_timestamp, 'Element:1', 'current', 'temp', 'V1', (SELECT id FROM frame WHERE frame_id='Frame:data'), 'Description to element 1'),
('Element', current_timestamp, current_timestamp, 'Element:2', 'current', 'temp', 'V1', (SELECT id FROM frame WHERE frame_id='Frame:data'), 'Description to element 2'),
...
...
('Element', current_timestamp, current_timestamp, 'Element:200000', 'current', 'temp', 'V1', (SELECT id FROM frame WHERE frame_id='Frame:data'), 'Description to object 200000');
I have a bash script where a postgres command is used to upload the data in object_name.sql to the table in google cloud:
PGPASSWORD=password psql -d database --username username --port 1234 --host 11.111.111 << EOF
BEGIN;
\i object_name.sql
COMMIT;
EOF
(Source: single transaction)
When I run this, I get this error:
BEGIN
psql:object_name.sql:60002: SSL SYSCALL error: EOF detected
psql:object_name.sql:60002: connection to server was lost
The current "solution" I have done now, is to chunk the file so each file can only have max 10000 insert statements. Running the psql command on these files works, but they take around 7 min.
Instead of having one file with 200000 insert statements, I divided them into 12 files where each file had max 10000 insert statements.
My question:
1. Is there a limit how large a file can contain?
2. I also saw this post about how to speed up insert, but I could not get COPY to work.
Hope someone out there have time to help me 🙂
Related
#Error description:
It's possible to create a table that has a foreign key into a hypertable provided the foreign key is defined when the table is created
#To Reproduce, there are next tables:
CREATE TABLE ids (
measurement_id int DEFAULT 0,
description text DEFAULT 0,
m_id bigserial NOT NULL,
service_id int DEFAULT NULL,
time bigint NOT NULL DEFAULT cast((EXTRACT(EPOCH FROM now() AT TIME ZONE 'UTC') * 1000) as bigint),
user_id int DEFAULT NULL,
end_time DOUBLE PRECISION DEFAULT 0,
start_time int NOT NULL DEFAULT 0
);
CREATE INDEX ON ids (time DESC, user_id);
CREATE INDEX ON ids (time DESC, service_id);
SELECT create_hypertable('ids', 'start_time', chunk_time_interval => 604800016);
---------
CREATE TABLE IF NOT EXISTS metrics (
id bigserial NOT NULL,
duration real DEFAULT NULL,
metric integer DEFAULT 0,
m_id bigint NOT NULL,
time bigint NOT NULL DEFAULT 0
);
ALTER TABLE metrics ADD PRIMARY KEY (time, m_id);
CREATE INDEX ON metrics (time DESC);
CREATE INDEX ON metrics (time DESC, measurement );
CREATE INDEX ON metrics (time DESC, m_id );
grant all privileges on ids, metrics to your_db_user;
SELECT create_hypertable('metrics', 'time' , chunk_time_interval => 604800016);
SELECT table_catalog, table_schema, table_name, privilege_type FROM information_schema.table_privileges WHERE grantee = 'your_db_user';
---------
DROP TABLE IF EXISTS resource;
CREATE TABLE resource(
id int NOT NULL,
cpu text DEFAULT 0,
storing text DEFAULT 0,
memory text DEFAULT 0
);
ALTER TABLE resource ADD PRIMARY KEY (id);
CREATE SEQUENCE resource_id_seq
INCREMENT 1
MINVALUE 1
MAXVALUE 2147483647
START 1
CACHE 1;
ALTER TABLE resource_id_seq
OWNER TO your_db_user;
ALTER TABLE resource ALTER COLUMN id SET DEFAULT nextval('resource_id_seq'::regclass);
---------
CREATE EXTENSION IF NOT EXISTS "uuid-ossp";
DROP TABLE IF EXISTS ns;
CREATE TABLE ns(
id bigint NOT NULL,
uuid uuid NOT NULL DEFAULT uuid_generate_v4 (),
availability double precision,
faultTolerance boolean,
activated boolean,
UNIQUE (id, uuid),
PRIMARY KEY(id),
CONSTRAINT fk_resource
FOREIGN KEY(id)
REFERENCES resource(id)
ON DELETE CASCADE
);
CREATE SEQUENCE ns_id_seq
INCREMENT 1
MINVALUE 1
MAXVALUE 9223372036854775807
START 1
CACHE 1;
ALTER TABLE ns_id_seq
OWNER TO your_db_user;
ALTER TABLE ns ALTER COLUMN id SET DEFAULT nextval('ns_id_seq'::regclass);
---------
DROP TABLE IF EXISTS authentication;
CREATE TABLE authentication(
id integer NOT NULL,
username character varying(255) NOT NULL,
password character varying(255) NOT NULL,
host character varying(255) NOT NULL,
port character varying(10) NOT NULL,
PRIMARY KEY(id)
);
CREATE SEQUENCE auth_id_seq
INCREMENT 1
MINVALUE 1
MAXVALUE 2147483647
START 1
CACHE 1;
ALTER TABLE auth_id_seq
OWNER TO your_db_user;
ALTER TABLE authentication ALTER COLUMN id SET DEFAULT nextval('auth_id_seq'::regclass);
---------
DROP TABLE IF EXISTS job;
CREATE TABLE job(
id int NOT NULL,
interval integer NOT NULL,
auth_id integer REFERENCES authentication (id),
ns_id integer REFERENCES ns (id),
UNIQUE (auth_id, ns_id),
PRIMARY KEY(id)
);
ALTER TABLE job
ADD CONSTRAINT fk_auth_id
FOREIGN KEY (id) REFERENCES authentication (id)
ON DELETE CASCADE
DEFERRABLE INITIALLY DEFERRED;
ALTER TABLE job
ADD CONSTRAINT fk_ns_id
FOREIGN KEY (id) REFERENCES ns (id)
ON DELETE CASCADE
DEFERRABLE INITIALLY DEFERRED;
CREATE SEQUENCE job_id_seq
INCREMENT 1
MINVALUE 1
MAXVALUE 2147483647
START 1
CACHE 1;
ALTER TABLE job_id_seq
OWNER TO your_db_user;
ALTER TABLE job ALTER COLUMN id SET DEFAULT nextval('job_id_seq'::regclass);
---------
DROP TABLE IF EXISTS job_metric;
CREATE TABLE job_metric (
id int NOT NULL,
j_id int NOT NULL REFERENCES job (id),
mj_id bigint NOT NULL,
jm_time bigint NOT NULL
);
CREATE INDEX ON job_metric (jm_time DESC);
CREATE INDEX ON job_metric (jm_time DESC, id);
CREATE INDEX ON job_metric (jm_time DESC, mj_id);
ALTER TABLE job_metric ADD PRIMARY KEY (jm_time, id);
grant all privileges on job_metric to your_db_user;
SELECT create_hypertable('job_metric', 'jm_time' , chunk_time_interval => 604800016);
CREATE SEQUENCE mjob_metric_id_seq
INCREMENT 1
MINVALUE 1
MAXVALUE 2147483647
START 1
CACHE 1;
ALTER TABLE mjob_metric_id_seq
OWNER TO your_db_user;
ALTER TABLE job_metric ALTER COLUMN id SET DEFAULT nextval('mjob_metric_id_seq'::regclass);
---------
After creating the tables, I have used the solution proposed by #Laurenz in a database with PostgreSQL 12.6 using the extension of timescaledb 1.7.5 as follows:
#To fill the table with the appropriate values:
UPDATE job_metric AS jm_point
SET jm_time = qm.time
FROM metrics AS qm
WHERE qm.m_id = jm_point.mj_id;
#Then set it NOT NULL:
ALTER TABLE job_metric ALTER jm_time SET NOT NULL;
#To define your foreign key:
ALTER TABLE job_metric
ADD FOREIGN KEY (mj_id, jm_time)
REFERENCES metrics (time, m_id) MATCH FULL;
#Response of the last reference table to enable foreign key: Query returned successfully in 40 msec.
Expected behavior:
The idea is to use the table job_metric in an even many-to-many relationship to access the information of job and metrics tables.
Actual behavior and error:
Tables are created and FKs were created but cannot be used when data is inserted at job_metric as is detailed in the following:
INSERT INTO job_metric (j_id, mj_id, jm_time)
VALUES(13, 185063, 1621957192266);
ERROR: foreign keys to hypertables are not supported CONTEXT: SQL
statement " ALTER TABLE _timescaledb_internal._hyper_5_5_chunk ADD
CONSTRAINT "5_13_job_metric_j_id_mj_id_jm_time_fkey" FOREIGN KEY
(j_id, mj_id, jm_time) REFERENCES qmetrics("time", m_id) MATCH FULL "
PL/pgSQL function
_timescaledb_internal.chunk_constraint_add_table_constraint(_timescaledb_catalog.chunk_constraint)
line 42 at EXECUTE SQL state: 0A000
***According to https://docs.timescale.com/timescaledb/latest/overview/limitations/##distributed-hypertable-limitations, it looks like the above error is part of the hypertable limitations:
Foreign key constraints referencing a hypertable are not supported.
#Request:
Given the above information and errors, does anyone know any solution at the DB level to establish the relationships (many-to-many or one-to-many) using timescaledb extension and mainly hypertables?
Actually, I have obtained similar of above error when I had attempted to create many-to-many relation among the tables metrics and job_metric using the Django Rest Framework:
class Job_Metrics(models.Model):
job = models.OneToOneField(Job, on_delete=models.CASCADE)
metrics = models.ManyToManyField(Metrics)
time = models.IntegerField(default=0)
Running the application metrics pointing out directly metrics_db:
$ python3 manage.py migrate metrics --database=metrics_db
Operations to perform: Apply all migrations: metrics Running migrations: Applying
metrics.0002_job...Traceback (most recent call last): File
"/var/myproject/myprojectenv/lib/python3.8/site-packages/django/db/backends/utils.py",
line 84, in _execute return self.cursor.execute(sql, params)
psycopg2.errors.FeatureNotSupported: foreign keys to hypertables are
not supported
If someone knows a solution or has an idea to deal with the above error at the REST API level, please could you share your idea with the aim to access data associated tables (metrics and jobs) and modify them together when is required to delete e.g., a job_metric. So far, using hypertables amendments of timescaledb extension seems to be not a viable solution.
In my cakephp 3.7.5 app I have a test database in which one of the tables is partitioned as follow:
-- Table: public.impressions
-- DROP TABLE public.impressions;
CREATE TABLE public.impressions
(
id bigint NOT NULL DEFAULT nextval('impressions_id_seq'::regclass),
company_id bigint NOT NULL,
ad_id bigint,
seen_at timestamp without time zone NOT NULL,
site_url text COLLATE pg_catalog."default" NOT NULL,
cost_per_impression_usd numeric(20,10),
user_ip inet NOT NULL,
user_data jsonb,
CONSTRAINT impressions_pkey PRIMARY KEY (id),
CONSTRAINT impressions_company_id_fkey FOREIGN KEY (ad_id, company_id)
REFERENCES public.ads (id, company_id) MATCH SIMPLE
ON UPDATE NO ACTION
ON DELETE NO ACTION
)
WITH (
OIDS = FALSE
)
TABLESPACE pg_default;
ALTER TABLE public.impressions
OWNER to postgres;
-- Rule: impressions_one ON public.impressions
-- DROP Rule impressions_one ON public.impressions;
CREATE OR REPLACE RULE impressions_one AS
ON INSERT TO public.impressions
WHERE new.company_id < 25
DO INSTEAD
INSERT INTO impressions_one (id, company_id, ad_id, seen_at, site_url, cost_per_impression_usd, user_ip, user_data)
VALUES (new.id, new.company_id, new.ad_id, new.seen_at, new.site_url, new.cost_per_impression_usd, new.user_ip, new.user_data);
-- Rule: impressions_three ON public.impressions
-- DROP Rule impressions_three ON public.impressions;
CREATE OR REPLACE RULE impressions_three AS
ON INSERT TO public.impressions
WHERE new.company_id >= 50
DO INSTEAD
INSERT INTO impressions_three (id, company_id, ad_id, seen_at, site_url, cost_per_impression_usd, user_ip, user_data)
VALUES (new.id, new.company_id, new.ad_id, new.seen_at, new.site_url, new.cost_per_impression_usd, new.user_ip, new.user_data);
-- Rule: impressions_two ON public.impressions
-- DROP Rule impressions_two ON public.impressions;
CREATE OR REPLACE RULE impressions_two AS
ON INSERT TO public.impressions
WHERE new.company_id >= 25 AND new.company_id < 50
DO INSTEAD
INSERT INTO impressions_two (id, company_id, ad_id, seen_at, site_url, cost_per_impression_usd, user_ip, user_data)
VALUES (new.id, new.company_id, new.ad_id, new.seen_at, new.site_url, new.cost_per_impression_usd, new.user_ip, new.user_data);
All the rules are working correctly and the values are inserted properly in their partitions, but only when I do an INSERT INTO sql command.
The problem with my partitioned table and cakephp is that cakephp seems to retrive the inserted entity using the RETURNING option after sql INSERT
This is from cakephp debug
INSERT INTO impressions (company_id, seen_at, site_url,
cost_per_impression_usd, user_ip, user_data) VALUES (:c0, :c1, :c2,
:c3, :c4, :c5) RETURNING *
and of course when I run this query in postgres raises the same exception as the cakephp
ERROR: cannot perform INSERT RETURNING on relation «impressions»
HINT: You need an unconditional ON INSERT DO INSTEAD rule with a
RETURNING clause. SQL state: 0A000
the problem is that postgresql does not allow this kind of scenario with returning data, but this is the cakephp way. I did it with symfony, same database, and all worked fine.
Is this cakephp beahvior changeable or configurable in a way that when I do the save for an entity it skip the returning option for the insert. something like:
$this->Impressions->save($impression,['skip_returning_stm'=>true]);
I need the table partitioned and the cakephp framework, I only tried on symfony in order to discard framework compatibility.
Thanks in advance!!!
This is the query , running this in C#.
n getting above error
"DROP TABLE IF EXISTS `NATIONAL_ID_ISSUANCE_CENTER`;
CREATE TABLE `NATIONAL_ID_ISSUANCE_CENTER` (
`ID` INTEGER NOT NULL AUTO_INCREMENT,
`NAME` VARCHAR(100),
`APPLICATION_ID` INTEGER,
`STATUS` INTEGER,
`CREATED_BY` INTEGER,
`UPDATED_BY` INTEGER,
`CREATED_DATE` DATETIME,
`UPDATED_DATE` DATETIME,
`THIRD_PARTY_ID` INTEGER,
`PROVINCE_ID` INTEGER,
INDEX (`APPLICATION_ID`),
PRIMARY KEY (`ID`),
INDEX (`PROVINCE_ID`),
INDEX (`THIRD_PARTY_ID`)
)"
You can't put an IF statement inside Drop and Create statements. Anytime you want to drop a table that you're not sure exists, use the following:
IF(OBJECT_ID('[Database].[Schema].[TableName]') is not null)
BEGIN
DROP TABLE [Database].[Schema].[TableName];
END;
Please note you should replace [Database], [Schema], and [TableName] with the appropriate database, schema, and table names, respectively.
-- MySQL Workbench Forward Engineering
SET #OLD_UNIQUE_CHECKS=##UNIQUE_CHECKS, UNIQUE_CHECKS=0;
SET #OLD_FOREIGN_KEY_CHECKS=##FOREIGN_KEY_CHECKS, FOREIGN_KEY_CHECKS=0;
SET #OLD_SQL_MODE=##SQL_MODE, SQL_MODE='TRADITIONAL,ALLOW_INVALID_DATES';
-- -----------------------------------------------------
-- Schema motivian
-- -----------------------------------------------------
-- -----------------------------------------------------
-- Schema motivian
-- -----------------------------------------------------
CREATE SCHEMA IF NOT EXISTS `motivian` DEFAULT CHARACTER SET utf8 COLLATE utf8_general_ci ;
USE `motivian` ;
-- -----------------------------------------------------
-- Table `motivian`.`user`
-- -----------------------------------------------------
DROP TABLE IF EXISTS `motivian`.`user` ;
CREATE TABLE IF NOT EXISTS `motivian`.`user` (
`user_id` INT UNSIGNED NOT NULL AUTO_INCREMENT,
`username` VARCHAR(100) NOT NULL,
`email` VARCHAR(100) NULL,
`password` VARCHAR(100) NOT NULL,
UNIQUE INDEX `username_UNIQUE` (`username` ASC),
PRIMARY KEY (`user_id`));
-- -----------------------------------------------------
-- Table `motivian`.`calculation`
-- -----------------------------------------------------
DROP TABLE IF EXISTS `motivian`.`calculation` ;
CREATE TABLE IF NOT EXISTS `motivian`.`calculation` (
`category_id` INT UNSIGNED NOT NULL AUTO_INCREMENT,
`calc` VARCHAR(100) NOT NULL,
`user_id` INT NULL,
`date_created` TIMESTAMP NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
PRIMARY KEY (`category_id`),
CONSTRAINT `user_id`
FOREIGN KEY ()
REFERENCES `motivian`.`user` ()
ON DELETE NO ACTION
ON UPDATE NO ACTION);
SET SQL_MODE=#OLD_SQL_MODE;
SET FOREIGN_KEY_CHECKS=#OLD_FOREIGN_KEY_CHECKS;
SET UNIQUE_CHECKS=#OLD_UNIQUE_CHECKS;
This is auto generated with mysql workbench but the constraint could not be applied. I tried like this:
CONSTRAINT user_id
FOREIGN KEY (user_id)
REFERENCESmotivian.user(user_id)
ON DELETE NO ACTION
ON UPDATE NO ACTION);
but also don't work.....
This is the error:
Executing SQL script in server
ERROR: Error 1064: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')
REFERENCES `motivian`.`user` ()
ON DELETE NO ACTION
ON UPDATE NO A' at line 8
SQL Code:
CREATE TABLE IF NOT EXISTS `motivian`.`calculation` (
`category_id` INT UNSIGNED NOT NULL AUTO_INCREMENT,
`calc` VARCHAR(100) NOT NULL,
`user_id` INT NULL,
`date_created` TIMESTAMP NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
PRIMARY KEY (`category_id`),
CONSTRAINT `user_id`
FOREIGN KEY ()
REFERENCES `motivian`.`user` ()
ON DELETE NO ACTION
ON UPDATE NO ACTION)
SQL script execution finished: statements: 8 succeeded, 1 failed
Fetching back view definitions in final form.
Nothing to fetch
Your FK definition is incomplete. Youd did not specify any column. Select your foreign key in the table editor and specify column pairs in the list right to it:
I try to create tables from file on my schema, but i receive error message. Users table created on public schema was successfully, but other table i created on test1 schema was wrong. I don't know why, help me please.
Thank for advance.
CREATE SCHEMA test1
--Create sequence
CREATE SEQUENCE test1.table_id_seq START 1;
--Create function to auto generate ID
CREATE OR REPLACE FUNCTION test1.next_id(OUT result bigint) AS $$
DECLARE
our_epoch bigint := 1314220021721;
seq_id bigint;
now_millis bigint;
shard_id int := 1;
BEGIN
SELECT nextval('test1.table_id_seq') % 1024 INTO seq_id;
SELECT FLOOR(EXTRACT(EPOCH FROM clock_timestamp()) * 1000) INTO now_millis;
result := (now_millis - our_epoch) << 23;
result := result | (shard_id << 10);
result := result | (seq_id);
END;
$$ LANGUAGE PLPGSQL;
--Talbe ----users----
CREATE TABLE users
(
id bigserial NOT NULL PRIMARY KEY,
username varchar(30) NOT NULL UNIQUE,
password varchar NOT NULL,
first_name varchar(10),
last_name varchar(10),
profile_picture varchar,
create_time timestamp (0) without time zone
);
--Table ----photos----
CREATE TABLE test.photos
(
id bigint NOT NULL PRIMARY KEY DEFAULT test1.next_id(),
caption text,
low_resolution varchar,
hight_resolution varchar,
thumnail varchar,
user_id bigint NOT NULL REFERENCES users(id),
create_time timestamp (0) without time zone
--Table ----comments----
CREATE TABLE test1.comments
(
id bigint NOT NULL PRIMARY KEY DEFAULT test1.next_id(),
create_time timestamp (0) without time zone,
text text,
user_id bigint REFERENCES users(id),
photo_id bigint REFERENCES test1.photos(id)
);
--Table ----likes----
CREATE TABLE test1.likes
(
photo_id bigint REFERENCES test1.photos(id),
user_id bigint REFERENCES users(id),
);
--Table ----follows----
CREATE TABLE test1.follows
(
user_id bigint NOT NULL REFERENCES users(id),
target_id bigint NOT NULL REFERENCES users(id),
);
CREATE TABLE teset1.feeds
(
user_id bigint NOT NULL REFERENCES users(id),
photo_id bigint NOT NULL REFERENCES test1.photos(id),
create_time timestamp (0) without time zone,
);
Well, it would have been helpful for you to have shown us what errors you were getting and what you tried to do to fix them. But here are some obvious problems with the SQL you posted:
Missing semicolon after "CREATE SCHEMA test1 "
Missing closing paren and semicolon at the end of CREATE TABLE test.photos ...
Dump neglects to create the "test" schema where the "photos" table wants to go.
Several foreign key references of test1.photos.id, but "photos" was created in the "test" schema, not "test1".
Extraneous trailing commas after the last column definitions for several tables
Typo: "teset1.feeds", should be "test.feeds"
TL;DR Whoever created this dump file wasn't paying very close attention.
After fixing all the problems above, I managed to load your SQL.