Bulk insert/update from Json param in Postgresql using ON CONFLICT (Id), but Id property must be acquired within the function as not included in param - postgresql

Question

Table definition:
CREATE TABLE public."FeatureToggles"
(
"Id" integer NOT NULL GENERATED BY DEFAULT AS IDENTITY ( INCREMENT 1 START 1 MINVALUE 1 MAXVALUE 2147483647 CACHE 1 ),
"IsDeleted" boolean NOT NULL,
"IsImported" boolean NOT NULL,
"TextProp" character varying(35),
CONSTRAINT "PK_FeatureToggles" PRIMARY KEY ("Id")
)
CREATE TABLE public."Additions"
(
"Id" integer NOT NULL GENERATED BY DEFAULT AS IDENTITY ( INCREMENT 1 START 1 MINVALUE 1 MAXVALUE 2147483647 CACHE 1 ),
"FeatureToggleId" int NOT NULL,
"IsDeleted" boolean NOT NULL,
"Url" character varying(35) NULL,
CONSTRAINT "PK_FeatureToggles" PRIMARY KEY ("Id")
CONSTRAINT "FK_Additions_FeatureToggles_FeatureToggleId" FOREIGN KEY ("FeatureToggleId")
REFERENCES public."FeatureToggles" ("Id") MATCH SIMPLE
ON UPDATE NO ACTION
ON DELETE CASCADE,
)
Insert one record into table:
INSERT INTO public."FeatureToggles" ("IsDeleted", "TextProp", "IsImported") VALUES(false, 'X', true);
Function:
CREATE OR REPLACE FUNCTION testfunctionname(jsonparam json)
RETURNS void AS
$BODY$
INSERT INTO "FeatureToggles" ("Id", "IsDeleted", "IsImported", "TextProp")
SELECT (COALESCE(SELECT "Id" FROM "FeatureToggles" WHERE "TextProp" = (prop->>'TextProp')::character varying(35)), 0),
(prop->>'IsDeleted')::boolean,
true,
(prop->>'TextProp')::character varying(35)
json_array_elements(jsonparam) prop
ON CONFLICT ("Id") DO
UPDATE SET
"IsDeleted" = EXCLUDED."IsDeleted"
INSERT INTO "Additions" ("FeatureToggleId", "IsDeleted", "Url")
SELECT (SELECT "Id" FROM "FeatureToggles" WHERE "TextProp" = (prop->>'TextProp')::character varying(35)),
(prop->>'IsDeleted')::boolean,
(prop->>'Additions')::character varying(35)
json_array_elements(jsonparam) prop
DELETE FROM "FeatureToggles" WHERE "IsImported" = true AND "TextProp" IS NOT IN (SELECT DISTINCT (prop->>'TextProp')::character varying(35)szi
json_array_elements(jsonparam) prop)
$BODY$
LANGUAGE sql
Sample JSON:
[
{
"IsDeleted": true,
"TextProp": "X",
"Additions":
[
"Test1",
"Test2"
]
},
{
"IsDeleted": false,
"TextProp": "Y",
"Additions":
[
"Test3",
"Test4"
]
}
]
Calling the function with this JSON param should update the one and only row in the FeatureToggles table to IsDeleted true and insert a new row into the FeatureToggles table with Id equals to 2, IsDeleted false and TextProp is Y. Also it should insert all Additions given in the JSON param into the corresponding table and with the correct foreign keys.
I ran into problems with populating the Id properties from the existing table and also inserting Additions into the other table.
It would be a great if the function would delete any rows in the FeatureToggle and the corresponding Additions table too if it does exists in table already, IsImported property is true, but is not in the JSON param.
Example if we change the insert script to:
INSERT INTO public."FeatureToggles" ("IsDeleted", "TextProp", "IsImported") VALUES(false, 'X', true);
INSERT INTO public."FeatureToggles" ("IsDeleted", "TextProp", "IsImported") VALUES(false, 'X222', true);
After calling the function with the same JSON param, the row with X222 should be deleted because it is marked as imported, but has no matching item (matched by TextProp property) within the new param list.
Any help would be much appreciated as this function needs to handle tens of thousands of records as parameter on each call.

Answer 1

You have several errors in your function (and your DDL)
Most importantly, json_array_elements() is a set returning function, so you need a FROM clause in order to generate multiple rows.
You also need to terminate each SQL statement in the function with ; and IS NOT IN is invalid - you need NOT IN
So the function should be something like this:
CREATE OR REPLACE FUNCTION testfunctionname(jsonparam json)
RETURNS void AS
$BODY$
INSERT INTO "FeatureToggles" ("Id", "IsDeleted", "IsImported", "TextProp")
SELECT coalesce(ft."Id", 0),
(prop->>'IsDeleted')::boolean,
true,
prop->>'TextProp'
FROM json_array_elements(jsonparam) prop
LEFT JOIN "FeatureToggles" ft on ft."TextProp" = (prop->>'TextProp')
ON CONFLICT ("Id") DO
UPDATE SET
"IsDeleted" = EXCLUDED."IsDeleted";
INSERT INTO "Additions" ("FeatureToggleId", "IsDeleted", "Url")
SELECT coalesce(ft."Id", 0),
(prop->>'IsDeleted')::boolean,
prop->>'Additions'
FROM json_array_elements(jsonparam) prop
JOIN "FeatureToggles" ft on ft."TextProp" = (prop->>'TextProp');
DELETE FROM "FeatureToggles"
WHERE "IsImported" = true
AND "TextProp" NOT IN (SELECT DISTINCT prop->>'TextProp' szi
FROM json_array_elements(jsonparam) prop);
$BODY$
LANGUAGE sql;
Note that ->> returns a text value, so there is no need to cast the result of those expression if the target column is text or varchar.
I also changed the scalar sub-queries to JOINs. The first insert is equivalent to an outer join - although I think that is wrong (but that's what your current code tries to do). Because if the join doesn't return anything, the INSERT will try to create a row with "Id" = 0 - bypassing the sequence generation. Using on conflict() with an auto-generated ID rarely makes sense. Maybe you want a unique index on TextProp?
I would probably implement that as a procedure rather than a function though.
Online example