I have a table populated by CSV raw data
| NNAME | DateDriven | username |
|--------------------------------|
| Thunder| 1-1-1999 | mickey |
|--------------------------------|
And an existing MSSQL database
> Tables
Drivers
| ------------- |
| ID | username |
|---------------|
| 1 | mickey |
| 2 | jonny |
| 3 | ryan |
-----------------
Cars
-----------------------------
| ID | NNAME | DateDriven |
|---------------------------|
| | | |
-----------------------------
Car_Drivers Table
-----------------------
| Cars_ID | Driver_ID |
|---------------------|
| | |
-----------------------
How can I take the cvs table data and insert it into the above? I am very lost!
CARS IDs are identity(1,1). Table Car_Drivers has a composite primary key off two foreign keys.
What I think I need to do is create a join to convert username to ID but I am getting lost completing the insert query.
Desired outcome
Cars Table
-----------------------------
| ID | NNAME | DateDriven |
|---------------------------|
| 1 | Thunder | 1-1-1999 |
-----------------------------
Car_Drivers Table
-----------------------
| Cars_ID | Driver_ID |
|---------------------|
| 1 | 1 |
-----------------------
The following ought to do what you need. The problem is that you need to keep some temporary data around as rows are inserted into Cars, but some of the data is from a different table. Merge provides the answer:
-- Create the test data.
declare #CSVData as Table ( NName NVarChar(16), DateDriven Char(8), Username NVarChar(16));
insert into #CSVData ( NName, DateDriven, Username ) values
( N'Thunder', '1-1-1999', N'mickey' );
select * from #CSVData;
declare #Drivers as Table ( Id SmallInt Identity, Username NVarChar(16) );
insert into #Drivers ( Username ) values
( N'mickey' ), ( N'jonny' ), ( N'ryan' );
select * from #Drivers;
declare #Cars as Table ( Id SmallInt Identity, NName NVarChar(16), DateDriven Char(8) );
declare #CarDrivers as Table ( Cars_Id SmallInt, Driver_Id SmallInt );
-- Temporary data needed for the #CarDrivers table.
declare #NewCars as Table ( Username NVarChar(16), Cars_Id SmallInt );
-- Merge the new data into #Cars .
-- MERGE allows the use of OUTPUT with references to columns not inserted,
-- e.g. Username .
merge into #Cars
using ( select NName, DateDriven, Username from #CSVData ) as CSVData
on 1 = 0
when not matched by target then
insert ( NName, DateDriven ) values ( CSVData.NName, CSVData.DateDriven )
output CSVData.Username, Inserted.Id into #NewCars;
-- Display the results.
select * from #Cars;
-- Display the temporary data.
select * from #NewCars;
-- Add the connections.
insert into #CarDrivers ( Cars_Id, Driver_Id )
select NewCars.Cars_Id, Drivers.Id
from #NewCars as NewCars inner join
#Drivers as Drivers on Drivers.Username = NewCars.Username;
-- Display the results.
select * from #CarDrivers;
DBFiddle.
Related
I have a data insert goal in Postgres. My table's columns:
id, col_1, col_2, col_3, col_4, col_5, col_6, col_7
Id column is auto incremented.
My Python insert code
copy_table_query = "COPY my_table (col_1, col_2, col_3, col_4, col_5, col_6, col_7) FROM STDIN WITH (DELIMITER '\t');"
curs.copy_expert(copy_table_query, data)
But it tries to insert col_1 into id and of course it fails with psycopg2.errors.InvalidTextRepresentation: invalid input syntax for type bigint. Because col_1 is string.
How can I let Postgres generate ids while I just insert data from CSV?
Example that shows it works. There is really no need to use copy_expert you can use [copy_from](https://www.psycopg.org/docs/cursor.html#cursor.copy_from. By default the separator is tab. You specify the columns with the columns parameter.
cat csv_test.csv
test f
test2 t
test3 t
\d csv_test
Table "public.csv_test"
Column | Type | Collation | Nullable | Default
--------+-------------------+-----------+----------+--------------------------------------
id | integer | | not null | nextval('csv_test_id_seq'::regclass)
col1 | character varying | | |
col2 | boolean | | |
with open('csv_test.csv') as csv_file:
cur.copy_from(csv_file, 'csv_test', columns=['col1', 'col2'])
con.commit()
select * from csv_test ;
id | col1 | col2
----+-------+------
1 | test | f
2 | test2 | t
3 | test3 | t
I need to turn a 1:n relationship into a 1:1 relationship with the data remaining the same.
I want to know if is it possible to achieve this with a single pure sql (no plpgsql, no external language).
Below there are more details, a MWE and some extra context.
To illustrate, if I have
+------+--------+ +------+----------+--------+
| id | name | | id | foo_id | name |
|------+--------| |------+----------+--------|
| 1 | foo1 | | 1 | 1 | baz1 |
| 2 | foo2 | | 2 | 1 | baz2 |
| 3 | foo3 | | 3 | 2 | baz3 |
+------+--------+ | 4 | 2 | baz4 |
| 5 | 3 | baz5 |
+------+----------+--------+
I want to get to
+------+--------+ +------+----------+--------+
| id | name | | id | foo_id | name |
|------+--------| |------+----------+--------|
| 4 | foo1 | | 1 | 4 | baz1 |
| 5 | foo1 | | 2 | 5 | baz2 |
| 6 | foo2 | | 3 | 6 | baz3 |
| 7 | foo2 | | 4 | 7 | baz4 |
| 8 | foo3 | | 5 | 8 | baz5 |
+------+--------+ +------+----------+--------+
Here is some code to set up the tables if needed:
drop table if exists baz;
drop table if exists foo;
create table foo(
id serial primary key,
name varchar
);
insert into foo (name) values
('foo1'),
('foo2'),
('foo3');
create table baz(
id serial primary key,
foo_id integer references foo (id),
name varchar
);
insert into baz (foo_id, name) values
(1, 'baz1'),
(1, 'baz2'),
(2, 'baz3'),
(2, 'baz4'),
(3, 'baz5');
I managed to work out the following query that updates only one entry (ie, the
pair <baz id, foo id> has to be provided):
with
existing_foo_values as (
select name from foo where id = 1
),
new_id as (
insert into foo(name)
select name from existing_foo_values
returning id
)
update baz
set foo_id = (select id from new_id)
where id = 1;
The real case scenario (a db migration in a nodejs environment) was solved using
something similar to
const existingPairs = await runQuery(`
select id, foo_id from baz
`);
await Promise.all(existingPairs.map(({
id, foo_id
}) => runQuery(`
with
existing_foo_values as (
select name from foo where id = ${foo_id}
),
new_id as (
insert into foo(name)
select name from existing_foo_values
returning id
)
update baz
set foo_id = (select id from new_id)
where id = ${id};
`)));
// Then delete all the orphan entries from `foo`
Here's a solution that works by first putting together what we want foo to look like (using values from the sequence), and then making the necessary changes to the two tables based on that.
WITH new_ids AS (
SELECT nextval('foo_id_seq') as foo_id, baz.id as baz_id, foo.name as foo_name
FROM foo
JOIN baz ON (foo.id = baz.foo_id)
),
inserts AS (
INSERT INTO foo (id, name)
SELECT foo_id, foo_name
FROM new_ids
),
updates AS (
UPDATE baz
SET foo_id = new_ids.foo_id
FROM new_ids
WHERE new_ids.baz_id = baz.id
)
DELETE FROM foo
WHERE id < (SELECT min(foo_id) FROM new_ids);
I have a couple of tables in Postgres database. I have joined and merges the tables. However, I would like to have common values in a specific column to appear together in the final table (In the end, I would like to perform groupby and maximum value calculation on the table).
The schema of the test tables looks like this:
Schema (PostgreSQL v11)
CREATE TABLE table1 (
id CHARACTER VARYING NOT NULL,
seq CHARACTER VARYING NOT NULL
);
INSERT INTO table1 (id, seq) VALUES
('UA502', 'abcdef'), ('UA503', 'ghijk'),('UA504', 'lmnop')
;
CREATE TABLE table2 (
id CHARACTER VARYING NOT NULL,
score FLOAT
);
INSERT INTO table2 (id, score) VALUES
('UA502', 2.2), ('UA503', 2.6),('UA504', 2.8)
;
CREATE TABLE table3 (
id CHARACTER VARYING NOT NULL,
seq CHARACTER VARYING NOT NULL
);
INSERT INTO table3 (id, seq) VALUES
('UA502', 'qrst'), ('UA503', 'uvwx'),('UA504', 'yzab')
;
CREATE TABLE table4 (
id CHARACTER VARYING NOT NULL,
score FLOAT
);
INSERT INTO table4 (id, score) VALUES
('UA502', 8.2), ('UA503', 8.6),('UA504', 8.8);
;
I performed join and union and oepration of the tables to get the desired columns.
Query #1
SELECT table1.id, table1.seq, table2.score
FROM table1 INNER JOIN table2 ON table1.id = table2.id
UNION
SELECT table3.id, table3.seq, table4.score
FROM table3 INNER JOIN table4 ON table3.id = table4.id
;
The output looks like this:
| id | seq | score |
| ----- | ------ | ----- |
| UA502 | qrst | 8.2 |
| UA502 | abcdef | 2.2 |
| UA504 | yzab | 8.8 |
| UA503 | uvwx | 8.6 |
| UA504 | lmnop | 2.8 |
| UA503 | ghijk | 2.6 |
However, the desired output should be:
| id | seq | score |
| ----- | ------ | ----- |
| UA502 | qrst | 8.2 |
| UA502 | abcdef | 2.2 |
| UA504 | yzab | 8.8 |
| UA504 | lmnop | 2.8 |
| UA503 | uvwx | 8.6 |
| UA503 | ghijk | 2.6 |
View on DB Fiddle
How should I modify my query to get the desired output?
I want to transpose the rows to columns using Pivot function in Oracle and/or SQL Server using Pivot function. My use case is very similar to this Efficiently convert rows to columns in sql server
However, I am organizing data by specific data type (below StringValue and NumericValue is shown).
This is my example:
----------------------------------------------------------------------
| Id | Person_ID | ColumnName | StringValue | NumericValue |
----------------------------------------------------------------------
| 1 | 1 | FirstName | John | (null) |
| 2 | 1 | Amount | (null) | 100 |
| 3 | 1 | PostalCode | (null) | 112334 |
| 4 | 1 | LastName | Smith | (null) |
| 5 | 1 | AccountNumber | (null) | 123456 |
----------------------------------------------------------------------
This is my result:
---------------------------------------------------------------------
| FirstName |Amount| PostalCode | LastName | AccountNumber |
---------------------------------------------------------------------
| John | 100 | 112334 | Smith | 123456 |
---------------------------------------------------------------------
How can I build the SQL Query?
I have already tried using MAX(DECODE()) and CASE statement in Oracle. However the performance is very poor. Looking to see if Pivot function in Oracle and/or SQL server can do this faster. Or should I go to single column value?
Below code will satisfy your requirement
Create table #test
(id int,
person_id int,
ColumnName varchar(50),
StringValue varchar(50),
numericValue varchar(50)
)
insert into #test values (1,1,'FirstName','John',null)
insert into #test values (2,1,'Amount',null,'100')
insert into #test values (3,1,'PostalCode',null,'112334')
insert into #test values (4,1,'LastName','Smith',null)
insert into #test values (5,1,'AccountNumber',null,'123456')
--select * from #test
Declare #Para varchar(max)='',
#Para1 varchar(max)='',
#main varchar(max)=''
select #Para += ','+QUOTENAME(ColumnName)
from (select distinct ColumnName from #test) as P
set #Para1= stuff(#para ,1,1,'')
print #Para1
set #main ='select * from (
select coalesce(StringValue,numericValue) as Val,ColumnName from #test) as Main
pivot
(
min(val) for ColumnName in ('+#Para1+')
) as pvt'
Exec(#main)
If I want to create a table with column create_by automatically filled with the user who create the entry, what is the DDL look like?
Wonder whether Postgres can do this similar to create_at
e.g. create_at TIMESTAMP NOT NULL DEFAULT NOW()
kind of thing.
SQL Fiddle
PostgreSQL 9.6 Schema Setup:
CREATE TABLE foo
(
id serial primary key
, "bar" varchar(1)
, created_by text NOT NULL DEFAULT current_user
, created_at timestamp DEFAULT current_timestamp
)
;
INSERT INTO foo
("bar")
VALUES
('a'),
('b'),
('c')
;
Query 1:
select *
from foo
Results:
| id | bar | created_by | created_at |
|----|-----|---------------|-----------------------------|
| 1 | a | user_17_3a66a | 2017-11-04T05:05:18.161681Z |
| 2 | b | user_17_3a66a | 2017-11-04T05:05:18.161681Z |
| 3 | c | user_17_3a66a | 2017-11-04T05:05:18.161681Z |