Try to answer this question then found out, I cannot solve it.
Basic idea: propagate twice, from country_id propagate to state_id, state_id propagate to city_id twice.Then the country_id need be joined twice. when we do array_agg on state level, we need explicitly join country_id, during city level we also need using join country_id.
Reference link: https://github.com/hettie-d/NORM/tree/master/sql
Basic idea, input one country_id, all the relevant country, state, city level information will be transformed to json format.
Prepare. I use country_id, state_id, city_id, since they are more descriptive.
begin;
create table public.country(country_id bigint primary key , name text, leader text);
create table public.states(state_id bigint primary key, name text, population bigint,country_id bigint REFERENCES public.country (country_id));
create table public.cities(city_id bigint,name text,state_id bigint REFERENCES public.states (state_id));
insert into public.country values ( 1, 'India', 'Narendra Modi');
insert into public.country values ( 2 , 'USA', 'Joe Biden');
insert into public.country values ( 3 , 'Australia', 'Scott Morrison');
insert into public.states values( 1 ,'California' , 39500000 , 2);
insert into public.states values( 2 , 'Washington' , 7610000 ,2 );
insert into public.states values( 4 , 'Karnataka' , 64100000,1);
insert into public.states values( 5 , 'Rajasthan' , 68900000,1 );
insert into public.states values( 6 , 'Maharashtra' , 125700000,1 );
insert into public.cities values( 1 , 'Mumbai' , 6 );
insert into public.cities values( 2 , 'Pune' , 6 );
insert into public.cities values( 3 , 'San Francisco' , 1 );
commit;
--- create composite types.
begin;
create type city_record as(city_name text);
create type state_record as (state_name text, population bigint,cities city_record[]);
create type country_record as (country_name text, leader text, states state_record[]);
commit;
array transport
create or replace
function array_transport (all_items anyarray) returns setof text
returns null on null input
language plpgsql as
$body$
declare
item record;
begin
foreach item in array all_items
loop
return next(to_json(item)::text);
end loop;
end;
$body$;
--the main function country_select_json
create or replace function country_select_json (_country_id bigint)
returns country_record[]
as
$$
declare
_result text;
begin
select array_agg(single_item)
from (select
array_agg(row(
co.name,
co.leader,
(select array_agg(row
(s.name,
s.population,
(select array_agg
(row
(c.name)::city_record)
from cities c
join states s using (state_id)
where s.country_id = co.country_id)
)::state_record) from states s where s.country_id = co.country_id
)
)::country_record)
as single_item
from country co
where co.country_id = _country_id)y into _result;
-- raise info 'state_record test: %', _result;
return (_result);
end
$$ language plpgsql;
run
select * from array_transport(country_select_json(1));
{"country_name":"India","leader":"Narendra Modi","states":[{"state_name":"Karnataka","population":64100000,"cities":[{"city_name":"Mumbai"},{"city_name":"Pune"}]},{"state_name":"Rajasthan","population":68900000,"cities":[{"city_name":"Mumbai"},{"city_name":"Pune"}]},{"state_name":"Maharashtra","population":125700000,"cities":[{"city_name":"Mumbai"},{"city_name":"Pune"}]}]}
(1 row)
country level, state level ok, but the city level is wrong. How to solve this problem.
Expected Result:
{"country_name":"India","leader":"Narendra Modi","states":[{"state_name":"Karnataka","population":64100000,"cities":[NULL]},{"state_name":"Rajasthan","population":68900000,"cities":[NULL]},{"state_name":"Maharashtra","population":125700000,"cities":[{"city_name":"Mumbai"},{"city_name":"Pune"}]}]}
update 2022-03-04.
(select array_agg
(c.name) as city_name
from cities c
join states s using (state_id)
where s.country_id = co.country_id)
Now I know the problem: because the propagate is first from city, then to state then country. Once function input the country_id then all the country related city name will be pulled together.
Does this query answer your needs ? (Result here)
with ci as (select cities.state_id,jsonb_agg(jsonb_build_object('city_name',cities.name)) as cities from cities group by state_id)
select jsonb_pretty(jsonb_build_object(
'country_name',c.name,
'leader',c.leader,
'states', jsonb_agg(jsonb_build_object(
'state_name',s.name,
'population',s.population,
'cities',ci.cities
))
))
from country c left join states s on s.country_id = c.country_id
left join ci on ci.state_id = s.state_id
where c.country_id = 1
group by c.name,c.leader
// Result
{
"country_name": "India",
"leader": "Narendra Modi",
"states": [
{
"state_name": "Maharashtra",
"population": 125700000,
"cities": [
{
"city_name": "Mumbai"
},
{
"city_name": "Pune"
}
]
},
{
"state_name": "Rajasthan",
"population": 68900000,
"cities": null
},
{
"state_name": "Karnataka",
"population": 64100000,
"cities": null
}
]
Related
I have a table where insertion is in this form.
Table
I want the verion to get update by 1 whenever there is a new row with same name and id.
Required output
I tried using a function and trigger.
CREATE OR REPLACE FUNCTION update_ver()
RETURNS TRIGGER
LANGUAGE PLPGSQL
AS
$$
BEGIN
update version
set ver = ver + 1
where new.name = 'A' and new.id ='1';
RETURN new;
END;
$$
-- create table
CREATE TABLE mytable (
"name" varchar NULL,
id int4 NULL,
phone varchar NULL,
email varchar NULL,
ver int4 NULL
);
-- create trigger function
CREATE OR REPLACE FUNCTION before_insert()
RETURNS TRIGGER
LANGUAGE PLPGSQL
AS
$$
begin
new.ver = (select coalesce(max(ver), 0) + 1 from mytable where name = new.name and id = new.id);
return new;
end;
$$
-- set trigger function to table
create trigger trg_before_insert before
insert
on
mytable for each row execute function before_insert();
-- inserting sample data
INSERT INTO mytable ("name", id, phone, email) VALUES('A', 1, '123', '123#email.com');
INSERT INTO mytable ("name", id, phone, email) VALUES('B', 2, '345', '345#email.com');
INSERT INTO mytable ("name", id, phone, email) VALUES('A', 1, '456', '456#email.com');
-- select data and view
select * from mytable;
Result:
name|id|phone|email |ver|
----+--+-----+-------------+---+
A | 1|123 |123#email.com| 1|
B | 2|345 |345#email.com| 1|
A | 1|456 |456#email.com| 2|
I am trying to create a PGSQL function that uses HTTP to get some json, format it, and insert it into a table using the donation_id key as a row constraint to prevent duplicates.
I have tried this function:
BEGIN
INSERT INTO donations(
donation_id, amount, avatar_image_url, created_date_utc, display_name, donor_id, event_id, incentive_id, message, participant_id, team_id)
ON CONFLICT (donation_id) DO NOTHING
SELECT elms::jsonb->>'donationID' AS donation_id ,
(elms::jsonb->>'amount')::float8 AS amount ,
elms::jsonb->>'avatarImageURL' AS avatar_image_url ,
(elms::jsonb->>'createdDateUTC')::timestamptz AS created_date_utc ,
elms::jsonb->>'displayName' AS display_name ,
elms::jsonb->>'donorID' AS donor_id ,
(elms::jsonb->>'eventID')::int AS event_id ,
elms::jsonb->>'incentiveID' AS incentive_id ,
elms::jsonb->>'message' AS message ,
(elms::jsonb->>'participantID')::int AS participant_id ,
(elms::jsonb->>'teamID')::int AS team_id
FROM (
select jsonb_array_elements(content::jsonb) AS elms
from http_get('https://extralife.donordrive.com/api/teams/59881/donations/')) as alias;
END;
I'm not quite understanding what I am doing wrong with the ON CONFLICT part of the query, just that it is apparently not valid syntax. I appreciate the insight as I'm not quite grasping the explainer written in docs.
Assuming a test table:
drop table if exists donations;
create table donations (
donation_id text primary key,
amount float8,
avatar_image_url text,
created_date_utc timestamptz,
display_name text,
donor_id text,
event_id int,
incentive_id text,
message text,
participant_id int,
team_id int);
It will work once you move the ON CONFLICT (donation_id) DO NOTHING to the end of the query:
INSERT INTO donations(donation_id, amount, avatar_image_url, created_date_utc,
display_name, donor_id, event_id, incentive_id, message, participant_id,
team_id)
SELECT elms::jsonb->>'donationID' AS donation_id ,
(elms::jsonb->>'amount')::float8 AS amount ,
elms::jsonb->>'avatarImageURL' AS avatar_image_url ,
(elms::jsonb->>'createdDateUTC')::timestamptz AS created_date_utc ,
elms::jsonb->>'displayName' AS display_name ,
elms::jsonb->>'donorID' AS donor_id ,
(elms::jsonb->>'eventID')::int AS event_id ,
elms::jsonb->>'incentiveID' AS incentive_id ,
elms::jsonb->>'message' AS message ,
(elms::jsonb->>'participantID')::int AS participant_id ,
(elms::jsonb->>'teamID')::int AS team_id
FROM ( select jsonb_array_elements('[
{
"displayName": "Christine",
"donorID": "A05C2C1E5DE15CDC",
"links": {
"recipient": "https://assets.yourdomain.com/somelink"
},
"eventID": 552,
"createdDateUTC": "2022-09-18T14:08:35.227+0000",
"recipientName": "Have A Drink Show",
"participantID": 494574,
"amount": 50,
"avatarImageURL": "https://assets.yourdomain.com/asset.gif",
"teamID": 59881,
"donationID": "FDBB61C5C8FFB3AE"
}
]'::jsonb) AS elms) as alias
ON CONFLICT (donation_id) DO NOTHING;
Demo.
I have the following database schema (oversimplified):
create sequence partners_partner_id_seq;
create table partners
(
partner_id integer default nextval('partners_partner_id_seq'::regclass) not null primary key,
name varchar(255) default NULL::character varying,
company_id varchar(20) default NULL::character varying,
vat_id varchar(50) default NULL::character varying,
is_deleted boolean default false not null
);
INSERT INTO partners(name, company_id, vat_id) VALUES('test1','1010109191191', 'BG1010109191192');
INSERT INTO partners(name, company_id, vat_id) VALUES('test2','1010109191191', 'BG1010109191192');
INSERT INTO partners(name, company_id, vat_id) VALUES('test3','3214567890102', 'BG1010109191192');
INSERT INTO partners(name, company_id, vat_id) VALUES('test4','9999999999999', 'GE9999999999999');
I am trying to figure out how to return test1, test2 (because the company_id column value duplicates vertically) and test3 (because the vat_id column value duplicates vertically as well).
To put it in other words - I need to find duplicating company_id and vat_id records and group them together, so that test1, test2 and test3 would be together, because they duplicate by company_id and vat_id.
So far I have the following query:
SELECT *
FROM (
SELECT *, LEAD(row, 1) OVER () AS nextrow
FROM (
SELECT *, ROW_NUMBER() OVER (w) AS row
FROM partners
WHERE is_deleted = false
AND ((company_id != '' AND company_id IS NOT null) OR (vat_id != '' AND vat_id IS NOT NULL))
WINDOW w AS (PARTITION BY company_id, vat_id ORDER BY partner_id DESC)
) x
) y
WHERE (row > 1 OR nextrow > 1)
AND is_deleted = false
This successfully shows all company_id duplicates, but does not appear to show vat_id ones - test3 row is missing. Is this possible to be done within one query?
Here is a db-fiddle with the schema, data and predefined query reproducing my result.
You can do this with recursion, but depending on the size of your data you may want to iterate, instead.
The trick is to make the name just another match key instead of treating it differently than the company_id and vat_id:
create table partners (
partner_id integer generated always as identity primary key,
name text,
company_id text,
vat_id text,
is_deleted boolean not null default false
);
insert into partners (name, company_id, vat_id) values
('test1','1010109191191', 'BG1010109191192'),
('test2','1010109191191', 'BG1010109191192'),
('test3','3214567890102', 'BG1010109191192'),
('test4','9999999999999', 'GE9999999999999'),
('test5','3214567890102', 'BG8888888888888'),
('test6','2983489023408', 'BG8888888888888')
;
I added a couple of test cases and left in the lone partner.
with recursive keys as (
select partner_id,
array['n_'||name, 'c_'||company_id, 'v_'||vat_id] as matcher,
array[partner_id] as matchlist,
1 as size
from partners
), matchers as (
select *
from keys
union all
select p.partner_id, c.matcher,
p.matchlist||c.partner_id as matchlist,
p.size + 1
from matchers p
join keys c
on c.matcher && p.matcher
and not p.matchlist #> array[c.partner_id]
), largest as (
select distinct sort(matchlist) as matchlist
from matchers m
where not exists (select 1
from matchers
where matchlist #> m.matchlist
and size > m.size)
-- and size > 1
)
select *
from largest
;
matchlist
{1,2,3,5,6}
{4}
fiddle
EDIT UPDATE
Since recursion did not perform, here is an iterative example in plpgsql that uses a temporary table:
create temporary table match1 (
partner_id int not null,
group_id int not null,
matchkey uuid not null
);
create index on match1 (matchkey);
create index on match1 (group_id);
insert into match1
select partner_id, partner_id, md5('n_'||name)::uuid from partners
union all
select partner_id, partner_id, md5('c_'||company_id)::uuid from partners
union all
select partner_id, partner_id, md5('v_'||vat_id)::uuid from partners;
do $$
declare _cnt bigint;
begin
loop
with consolidate as (
select group_id,
min(group_id) over (partition by matchkey) as new_group_id
from match1
), minimize as (
select group_id, min(new_group_id) as new_group_id
from consolidate
group by group_id
), doupdate as (
update match1
set group_id = m.new_group_id
from minimize m
where m.group_id = match1.group_id
and m.new_group_id != match1.group_id
returning *
)
select count(*) into _cnt from doupdate;
if _cnt = 0 then
exit;
end if;
end loop;
end;
$$;
updated fiddle
I have the following two tables:
CREATE TABLE tableone (
id integer NOT NULL GENERATED BY DEFAULT AS IDENTITY ( INCREMENT 1 START 1 MINVALUE 1 MAXVALUE 2147483647 CACHE 1 ),
sampletextone text
);
CREATE TABLE tabletwo (
id integer NOT NULL GENERATED BY DEFAULT AS IDENTITY ( INCREMENT 1 START 1 MINVALUE 1 MAXVALUE 2147483647 CACHE 1 ),
tableone_id int,
sampletexttwo text
);
With the following JSON parameter sent to the procedure:
[
{
"sampletextone": "table one text 1",
"tabletwo":
[
{
"sampletexttwo": "table two sample one text 1"
},
{
"sampletexttwo": "table two sample one text 2"
}
]
},
{
"sampletextone": "table one text 2",
"tabletwo":
[
{
"sampletexttwo": "table two sample one text 3"
},
{
"sampletexttwo": "table two sample one text 4"
}
]
}
]
I have the following stored procedure
CREATE OR REPLACE procedure testproc(jsonparam json)
AS
$BODY$
WITH ins1 AS (INSERT INTO "tableone" ("sampletextone")
SELECT prop->>'sampletextone'
FROM json_array_elements(jsonparam) prop
Returning "id"
)
INSERT INTO "tabletwo" ("tableone_id", "sampletexttwo")
SELECT ins1."id", 'should be all sample text two'
FROM ins1
--JOIN json_array_elements(jsonparam) prop;
$BODY$
LANGUAGE sql;
I am trying to join the JSON param again to insert all four rows in tabletwo with the correct foreign keys from tableone. I am not sure what would best way of joining the relevant data again.
Online example
UPDATE:
CREATE OR REPLACE procedure testproc(jsonparam json)
AS
$BODY$
WITH ins1 AS (INSERT INTO "tableone" ("sampletextone")
SELECT prop->>'sampletextone'
FROM json_array_elements(jsonparam) prop
Returning "id", "sampletextone"
)
INSERT INTO "tabletwo" ("tableone_id", "sampletexttwo")
SELECT ins1."id", json_extract_path(prop, 'tabletwo', 'sampletexttwo')
FROM ins1
JOIN json_array_elements(jsonparam) prop ON prop->>'sampletextone' =
ins1."sampletextone"
$BODY$
LANGUAGE sql;
With the updated stored procedure only 2 rows created in tabletwo with the correct foreign keys instead of 4 and sampletexttwo is null
I have found the answer
CREATE OR REPLACE procedure testproc(jsonparam json)
AS
$BODY$
WITH ins1 AS (INSERT INTO "tableone" ("sampletextone")
SELECT prop->>'sampletextone'
FROM json_array_elements(jsonparam) prop
Returning "id", "sampletextone"
)
INSERT INTO "tabletwo" ("tableone_id", "sampletexttwo")
SELECT ins1."id", json_array_elements(json_extract_path(prop, 'tabletwo'))->>'sampletexttwo'
FROM ins1
JOIN json_array_elements(jsonparam) prop ON prop->>'sampletextone' =
ins1."sampletextone"
$BODY$
LANGUAGE sql;
If there is a better way of doing it, please let me know
so I'm working on a project which requires my query to insert into one main table and its detail table (which will be sent into the DB as a list) in one transaction so that it'll roll-back if one of the insert functions are failed.
Let's say I have these tables:
CREATE TABLE transaction(
id BIGSERIAL PRIMARY KEY NOT NULL,
user_id BIGINT FOREIGN KEY NOT NULL,
total_item INT NOT NULL DEFAULT 0,
total_purchase BIGINT NOT NULL DEFAULT 0
)
CREATE TABLE transaction_detail(
id BIGSERIAL PRIMARY KEY NOT NULL,
transaction_id BIGINT FOREIGN KEY NOT NULL,
product_id BIGINT FOREIGN KEY NOT NULL,
product_price INT NOT NULL DEFAULT 0,
purchase_amount INT NOT NULL DEFAULT 0
)
And I have this function:
CREATE OR REPLACE FUNCTION create_transaction(order JSONB, product_list JSONB)
Function Parameters:
order : An object which will be inserted into the transaction table
product_list : List of Product object which will be inserted into the transaction_detail table
My current query looks something like this:
CREATE OR REPLACE FUNCTION insert_order(tx JSONB, product_list JSONB)
RETURNS BIGINT
AS $$
WITH result AS (
INSERT INTO transaction(
user_id,
total_item,
total_purchase,
) VALUES (
(tx ->> 'user_id') :: BIGINT,
(tx ->> 'total_item') :: INT,
(tx ->> 'total_purchase') :: INT,
)
RETURNING id AS transaction_id
)
FOR row IN product_list LOOP
INSERT INTO transaction_detail(
transaction_id,
product_id,
product_price,
purchase_amount,
) VALUES (
transaction_id,
(row ->> 'product_id') :: BIGINT,
(row ->> 'product_price') :: INT,
(row ->> 'purchase_amount') :: INT,
)
END LOOP;
$$ LANGUAGE SQL SECURITY DEFINER;
JSON files:
tx.json
[
"user_id" : "1",
"total_item" : "2",
"total_purchase" : "2000",
]
product_list.json
[
{
"product_id" : "1",
"product_price" : "500",
"purchase_amount" : "2"
},
{
"product_id" : "2",
"product_price" : "1000",
"purchase_amount" : "1"
}
]
I know something is wrong with my query although I can't put a finger on it.
Any pointer is much appreciated.
Assuming that the data passed as product_list is an array, you can do something like this:
CREATE OR REPLACE FUNCTION insert_order(p_order JSONB, p_product_list JSONB)
RETURNS BIGINT
AS $$
WITH result AS (
INSERT INTO "transaction"(
user_id,
total_item,
total_purchase
) VALUES (
(p_order ->> 'user_id') :: BIGINT,
(p_order ->> 'total_item') :: INT,
(p_order ->> 'total_purchase') :: INT
)
RETURNING id AS transaction_id
), details as (
INSERT INTO transaction_detail(
transaction_id,
product_id,
product_price,
purchase_amount
)
select r.transaction_id,
(pl.data ->> 'product_id')::bigint,
(pl.data ->> 'product_price')::int,
(pl.data ->> 'purchase_amount')::int
from result r,
jsonb_array_elements(p_product_list) as pl(data)
)
select transaction_id
from result;
$$
LANGUAGE SQL SECURITY DEFINER;
I renamed the parameters to avoid name clashes with reserved keywords. By prefixing the parameter names you also avoid name clashes with column or table names. order is a reserved keyword and can only be used when quoted, e.g. "order". transaction is a keyword however it's not reserved, but it's better to quote it nonetheless.
The insert into the transaction details needs to be an INSERT...SELECT selecting from the result to get the generated transaction id and by unnesting the array elements in the product list JSON value.
The final select of the CTE then returns the generated transaction id.
You can call the function like this:
select insert_order('{"user_id": 42, "total_item": 1, "total_purchase": 100}'::jsonb,
'[ {"product_id": 1, "product_price": 10, "purchase_amount": 1},
{"product_id": 2, "product_price": 20, "purchase_amount": 2},
{"product_id": 3, "product_price": 30, "purchase_amount": 3} ]'::jsonb);