Let's say I have a partitioned table A.
create table A (
col1 timestamp,
col2 int
)
partition by col2;
create table partition1 partition of A from values (minvalue) to (y);
create table partition1 partition of A from values (y) to (maxvalue);
copy A from '/some/csv/file'
The above code gives me a paritioned table A with the data populated. I want to create another table using -
create table B as (
select *,
col2 * 3 as col3 -- Add a new column
from A
);
Can I save A as a partitioned CSV/'insert_format' file?
Is it possible that B is also paritioned the same way A is?
Related
Hi I'm trying to automate a workflow in Airflow where I am going to be appending rows to an external Spectrum table daily and I need to alter the numRows on the spectrum table by extracting the count of the existing table + the new count of rows I am appending.
CREATE EXTERNAL TABLE spectrum.my_external_table
(
id INTEGER,
barkdata_timestamp timestamp,
created_at timestamp,
updated_at timestamp
)
PARTITIONED BY (asofdate timestamp)
STORED AS PARQUET
LOCATION 's3://<SOME BUCKET>/manifest'
table properties ('numRows'= '<some number>';
ALTER TABLE spectrum.my_external_table
ADD PARTITION (asofdate='2021-03-03 00:00:00') LOCATION 's3://<SOME BUCKET>/asofdate=2021-03-03 00:00:00/';
ALTER TABLE spectrum.couponable_coupon
SET TABLE PROPERTIES ('numRows'='<HELP HERE should be count(*) from my_external_table + count(*) from table_I_unloaded_to_s3 where asofdate='2021-03-03 00:00:00'>');
I'm trying to use the performance of COPY FROM command in PostgreSQL to get all data of 1 table of a CSV file (CSV -> table1) and I need to insert other data, but, in a new table. I will need of a primary key of first table to put as a foreign key in second table.
Example:
I need to insert 1,000,000 of names in table1 and 500,000 of names in table2, but, all names in table2 reference to 1 tuple in table1.
CREATE TABLE table1 (
table1Id bigserial NOT NULL,
Name varchar(100) NULL,
CONSTRAINT table1Id PRIMARY KEY (table1Id)
);
CREATE TABLE table2 (
table2Id bigserial NOT NULL,
Other_name varchar(100) NOT NULL
table1_table1Id int8 NOT NULL,
CONSTRAINT table2_pk PRIMARY KEY (table2Id)
);
Command COPY does not allow table manipulations while copying data (such as look up to other table for fetching proper foreign keys to insert). To insert into table2 ids for corresponding rows from table1 you need to drop NOT NULL constraint for that field, COPY data and then UPDATE that fields separately.
Assuming table1 and table2 tables can be joined by table1.Name = table2.Other_name, the code is:
Before COPY:
ALTER TABLE table2 ALTER COLUMN table1_table1Id DROP NOT NULL;
After COPY:
UPDATE table2 SET table2.table1_table1Id = table1.table1Id
FROM table1
WHERE table1.Name = table2.Other_name;
ALTER TABLE table2 ALTER COLUMN table1_table1Id SET NOT NULL;
It is possible to mantain an unique sequence on a partitioned table?
I am using Postgresql 10
This is with 9.6. I'm not sure why you would want to do that though - could you explain more about why you need ids to be unique across partitions?
CREATE SCHEMA so46793511;
set search_path=so46793511;
CREATE SEQUENCE partitioned_seq;
CREATE TABLE partitioned (
id integer default nextval('partitioned_seq'),
val text
);
CREATE TABLE partitioned_1 ( LIKE partitioned ) INHERITS (partitioned);
CREATE TABLE partitioned_2 ( LIKE partitioned ) INHERITS (partitioned);
INSERT INTO partitioned_1 (val) VALUES ('a'), ('b'), ('c');
INSERT INTO partitioned_2 (val) VALUES ('a'), ('b'), ('c');
SELECT * FROM partitioned;
DROP SCHEMA so46793511 CASCADE;
NB: this doesn't include anything fancy to auto-pick partitions when inserting.
I have two tables master / detail their DDL are:
Table1:
tbl1_id integer,
tbl1_sm_id integer,
tbl1_name varchar(30)
table2:
tbl2_id integer,
tbl1_id integer,
tbl2_name varchar(30)
table2.tbl1_id is the foreign key between the two tables, tbl1_id and tbl2_id are generated automatically from triggers.
I need to copy all records from these two tables where tbl1_sm_id = 2 into new value where tbl1_sm_id = 3.
I tried this SQL inside stored procedure:
For
select tbl1_id from table1
where tbl1_sm_id = 2
into :v_tbl1_id
do begin
insert into table1 (tbl1_sm_id, tbl1_name)
select 3, tbl1_name returning tbl1_id into :v_tbl1_id;
insert into table2 (tbl1_id, tbl2_name)
select :v_tbl1_id, tbl2_name
from table2 where tbl1_id = :v_tbl1_id;
end
Is this correct? Is there another way to copy the two tables?
I have the following table:
RecordID
Name
Col1
Col2
....
ColN
The RecordID is BIGINT PRIMARY KEY CLUSTERED IDENTITY(1,1) and RecordID and Name are initialized. The other columns are NULLs.
I have a function which returns information about the other columns by Name.
To initialized my table I use the following algorithm:
Create a LOOP
Get a row, select its Name value
Execute the function using the selected name, and store its result
in temp variables
Insert the temp variables in the table
Move to the next record
Is there a way to do this without looping?
Cross apply was basically built for this
SELECT D.deptid, D.deptname, D.deptmgrid
,ST.empid, ST.empname, ST.mgrid
FROM Departments AS D
CROSS APPLY fn_getsubtree(D.deptmgrid) AS ST;
Using APPLY
UPDATE some_table
SET some_row = another_row,
some_row2 = another_row/2
FROM some_table st
CROSS APPLY
(SELECT TOP 1 another_row FROM another_table at WHERE at.shared_id=st.shared_id)
WHERE ...
using cross apply in an update statement
You can simply say the following if you already have the records in the table.
UPDATE MyTable
SET
col1 = dbo.col1Method(Name),
col2 = dbo.col2Method(Name),
...
While inserting new records, assuming RecordID is auto-generated, you can say
INSERT INTO MyTable(Name, Col1, Col2, ...)
VALUES(#Name, dbo.col1Method(#Name), dbo.col2Method(#name), ...)
where #Name contains the value for the Name column.