I'm trying to migrate a Tomcat app from using Postgres 9.5 to SQL Server 2016 and I've got a problem statement I can't seem to duplicate.
It's basically an upsert but one of the complications is the request supplies arguments to do the update, but when there is conflict I need to use some of the existing values from conflicting rows to insert/update.
The primary keys in the table can sometimes cause a conflict, which requires updating rows and deleting the old ones.
The table schema in MS SQL looks like:
CREATE TABLE [dbo].[signup](
[site_key] [varchar](32) NOT NULL,
[list_id] [bigint] NOT NULL,
[email_address] [varchar](256) NOT NULL,
[customer_id] [bigint] NULL,
[attribute1] [varchar](64) NULL,
[date1] [datetime] NOT NULL,
[date2] [datetime] NULL,
CONSTRAINT [pk_signup] PRIMARY KEY CLUSTERED
(
[site_key] ASC,
[list_id] ASC,
[email_address] ASC
)WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON) ON [PRIMARY]
) ON [PRIMARY]
The old Postgres SQL looked like this:
WITH updated_rows AS (
INSERT INTO signup
(site_key, list_id, email_address, customer_id, attribute1, date1, date2)
SELECT site_key, list_id, :emailAddress, customer_id, attribute1, date1, date2
FROM signup WHERE customer_id = :customerId and email_address <> :emailAddress
ON CONFLICT (site_key, list_id, email_address) DO UPDATE SET customer_id = excluded.customer_id
RETURNING site_key, customer_id, email_address, list_id
)
DELETE FROM signup AS signup_delete USING updated_rows
WHERE
signup_delete.site_key = updated_rows.site_key
AND signup_delete.customer_id = updated_rows.customer_id
AND signup_delete.list_id = updated_rows.list_id
AND signup_delete.email_address <> :emailAddress;
Two arguments are supplied, customer id and email address, shown here as Spring NamedParameterJdbcTemplate values :customerId and :emailAddress
It's trying to change the email address of the customer id to be the supplied one, but sometimes the supplied email address already exists in the primary key constraint.
In which case it needs to change the existing customer id to be supplied one, and remove the rows with that don't match the new email address.
I also need to try and maintain isolation so that nothing can change the data whilst I'm updating.
I'm trying to do it with a MERGE statement but I can't seem to get it to work, it's complaining I cant use values that aren't in the clause scope, but I think I've probably got other issues here too.
This is what I had so far. It doesn't even address the deleting part - only the upserting, but I can't even get this part to work. I was planning to use the OUTPUT from this as input to something to delete the rows similar to the postgres version.
WITH source AS (
SELECT cs.[site_key] as existing_site_key,
cs.list_id as existing_list_id,
cs.email_address as existing_email,
cs.customer_id as existing_customer_id,
cs.attribute1 as existing_attribute1,
cs.date1 as existing_date1,
cs.date2 as existing_date2,
cs2.email_address as conflicting_email,
cs2.customer_id AS conflicting_customer_id
FROM [dbo].[signup] cs
LEFT JOIN [dbo].[signup] cs2 ON cs2.email_address = :emailAddress
AND cs.site_key = cs2.site_key
AND cs.list_id = cs2.list_id
WHERE cs.customer_id = :customerId
)
MERGE signup WITH (HOLDLOCK) AS target
USING source
ON ( source.conflicting_customer_id is not null )
WHEN MATCHED AND source.existing_site_key = target.site_key AND source.existing_list_id = target.list_id AND source.conflicting_email = target.email_address THEN UPDATE
SET customer_id = :customerId
WHEN NOT MATCHED BY target AND source.existing_site_key = target.site_key AND source.existing_list_id = target.list_id AND source.conflicting_customer_id = :customerId THEN INSERT
(site_key, list_id, email_address, customer_id, attribute1, date1, date2) VALUES
(source.existing_site_key, source.existing_list_id, :emailAddress, source.customer_id, source.existing_attribute1, source.existing_date1, source.existing_date2)
Thanks,
mikee
Related
have a view like this:
Table
The record "NDocumento" is populated only in the first row of a transaction by design. These rows are grouped by the column "NMov" which is the ID.
Since this is a view, I would like to populate each empty "NDocumento" record with the corresponding value contained in the first transaction through a SELECT statement.
As you can see by the picture this is MS-SQL Server 2008, so the lack of LAG makes the game harder.
I would immensely appreciate any help,
thanks
Try this:
SELECT
T1.NDocumento
, T2.NMov
, T2.NRiga
-- , T2. Rest of the fields
FROM NDocumentoTable T1
JOIN NDocumentoTable T2 ON T2.NMov = T1.NMov
WHERE T1.NRiga = 1
I used LAG() over the partition of NMov,Causale by based on your data. You cna change the partition with your requirement. The logic is you get the previous value if the NDocument is empty for the given partition.
CREATE TABLE myTable_1
(
NMov int
,NRiga int
,CodiceAngrafica varchar(100)
,Causale varchar(100)
,DateRegistration date
,DateDocumented date
,NDocument varchar(100)
)
INSERT INTO myTable_1 VALUES (5133, 1, '', 'V05', '01/14/2021', '01/14/2021', 'VI-2100001')
,(5133, 2, '', 'V05', null, null, '')
,(5134, 1, '', 'V05', '01/14/2021', '01/14/2021', 'VI-2100002')
,(5134, 2, '', 'V05', null, null, '')
SELECT
NMov
,NRiga
,CASE WHEN ISNULL(NDocument,'') = ''
THEN LAG(NDocument) OVER (PARTITION BY NMov,Causale ORDER BY NMov)
ELSE NDocument END AS [NDocument]
FROM myTable_1
PostgreSQL 11.1 PgAdmin 4.1
This works some of the time:
BEGIN;
SET CONSTRAINTS ALL DEFERRED;
WITH _in(trx, lastname, firstname, birthdate, old_disp, old_medname, old_sig, old_form, new_disp, new_medname, new_sig, new_form, new_refills) AS (
VALUES ('2001-06-07 00:00:00'::timestamp,
UPPER(TRIM('JONES')), UPPER(TRIM('TOM')), '1952-12-30'::date,
64::integer,
LOWER(TRIM('adipex 37.5mg tab')), LOWER(TRIM('one tab po qd')), LOWER(TRIM('tab')),
63::integer,
LOWER(TRIM('adipex 37.5mg tab')), LOWER(TRIM('one tab po qd')), LOWER(TRIM('tab')),
33::integer
)
),
_s AS ( -- RESOLVE ALL SURROGATE KEYS.
SELECT n.*, d1.recid as old_medication_recid, d2.recid as new_medication_recid, pt.recid as patient_recid
FROM _in n
JOIN medications d1 ON (n.old_medname, n.old_sig, n.old_form) = (d1.medname, d1.sig, d1.form)
JOIN medications d2 ON (n.new_medname, n.new_sig, n.new_form) = (d2.medname, d2.sig, d2.form)
JOIN patients pt ON (pt.lastname, pt.firstname, pt.birthdate) = (n.lastname, n.firstname, n.birthdate)
),
_t AS ( -- REMOVE CONFLICTING RECORD, IF ANY.
DELETE FROM rx r
USING _s n
WHERE (r.trx::date, r.disp, r.patient_recid, r.medication_recid)=(n.trx::date, n.new_disp, n.patient_recid, n.new_medication_recid)
RETURNING r.*
),
_u AS( -- GET NEW SURROGATE KEY.
SELECT COALESCE(_t.recid, r.recid) as target_recid, r.recid as old_recid
FROM _s n
JOIN rx r ON (r.trx, r.disp, r.patient_recid, r.medication_recid) = (n.trx, n.old_disp, n.patient_recid, n.old_medication_recid)
LEFT JOIN _t ON (_t.trx::date, _t.disp, _t.patient_recid, _t.medication_recid) = (n.trx::date, n.new_disp, n.patient_recid, n.new_medication_recid)
)
UPDATE rx r -- UPDATE ORIGINAL RECORD WITH NEW VALUES.
SET disp = n.new_disp, medication_recid = n.new_medication_recid, refills = n.new_refills, recid = _u.target_recid
FROM _s n, _u
WHERE r.recid = _u.old_recid
RETURNING r.*;
COMMIT;
Where table rx is defined as:
CREATE TABLE phoenix.rx
(
recid integer NOT NULL DEFAULT nextval('rx_recid_seq'::regclass),
trx timestamp without time zone NOT NULL,
disp integer NOT NULL,
refills integer,
tprinted timestamp without time zone,
tstop timestamp without time zone,
modified timestamp without time zone DEFAULT now(),
patient_recid integer NOT NULL,
medication_recid integer NOT NULL,
dposted date NOT NULL,
CONSTRAINT pk_rx_recid PRIMARY KEY (recid),
CONSTRAINT rx_unique UNIQUE (dposted, disp, patient_recid, medication_recid)
DEFERRABLE,
CONSTRAINT rx_medication_fk FOREIGN KEY (medication_recid)
REFERENCES phoenix.medications (recid) MATCH SIMPLE
ON UPDATE CASCADE
ON DELETE RESTRICT
DEFERRABLE,
CONSTRAINT rx_patients FOREIGN KEY (patient_recid)
REFERENCES phoenix.patients (recid) MATCH SIMPLE
ON UPDATE CASCADE
ON DELETE RESTRICT
)
After many hours, it is found that the "Delete.." of a conflicting record works as expected, but the "COALESCE" STATEMENT seems to fail when deciding on the new surrogate key (primary key) of rx.recid -- it does not seem to receive the result of the delete. (Or maybe the timing is wrong???)
Any help would be most appreciated.
TIA
This is documented:
The sub-statements in WITH are executed concurrently with each other and with the main query. Therefore, when using data-modifying statements in WITH, the order in which the specified updates actually happen is unpredictable. All the statements are executed with the same snapshot (see Chapter 13, so they cannot “see” one another's effects on the target tables.
Don't use the same table twice in a statement with a CTE if it occurs in a DML statement. Rather, use DELETE ... RETURNING and use the returned values in the other parts of the statement.
If you cannot rewrite the statement like that, use more than one statement instead of putting everything into a single CTE.
#LaurenzAlbe is totally correct in his answer. Below is a working solution to my problem. There are a few things to note:
The unique constraint is formed on a column in rx defined as a date and created by a trigger on update/insert that casts the timestamp of trx to a date: as in trx::date. For reasons I am not clear on, using r.trx::date in place of r.dposted leads to many records being identified and not the one record I want. Not sure why???. So the first fix was to use r.dposted, not r.trx::date.
Although the cte's are designed to be independent of each other, by using "RETURNING..." and incorporating the cte's in a step-wise fashion, one can be built upon another to obtain a final result set.
The working code is:
WITH _in(trx, lastname, firstname, birthdate, old_disp, old_medname, old_sig, old_form, new_disp, new_medname, new_sig, new_form, new_refills) AS (
VALUES ('2001-06-07 00:00:00'::timestamp,
UPPER(TRIM('smith')), UPPER(TRIM('john')), '1957-12-30'::date,
28::integer,
LOWER(TRIM('test')), LOWER(TRIM('i am sig')), LOWER(TRIM('tab')),
28::integer,
LOWER(TRIM('test 1')), LOWER(TRIM('i am sig')), LOWER(TRIM('tab')),
8::integer
)
),
_m AS (
SELECT n.*, d1.recid as old_medication_recid, d2.recid as new_medication_recid, pt.recid as patient_recid
FROM _in n
JOIN patients pt ON (pt.lastname, pt.firstname, pt.birthdate) = (n.lastname, n.firstname, n.birthdate)
JOIN medications d1 ON (n.old_medname, n.old_sig, n.old_form) = (d1.medname, d1.sig, d1.form)
LEFT JOIN medications d2 ON (n.new_medname, n.new_sig, n.new_form) = (d2.medname, d2.sig, d2.form)
),
_t AS ( -- REMOVE CONFLICTING RECORD, IF ANY.
DELETE FROM rx r
USING _m
WHERE (r.dposted, r.disp, r.patient_recid, r.medication_recid) = (_m.trx::date,_m.new_disp, _m.patient_recid, _m.new_medication_recid)
RETURNING r.*
),
_s AS ( -- GET NEW SURROGATE KEY
SELECT _m.*, r1.recid as old_recid, r2.recid as new_recid, COALESCE(r2.recid, r1.recid) as target_recid
FROM _m
JOIN rx r1 ON (r1.dposted, r1.disp, r1.patient_recid, r1.medication_recid) = (_m.trx::date,_m.old_disp, _m.patient_recid, _m.old_medication_recid)
LEFT JOIN rx r2 ON (r2.dposted, r2.disp, r2.patient_recid, r2.medication_recid) = (_m.trx::date,_m.new_disp, _m.patient_recid, _m.new_medication_recid)
LEFT JOIN _t ON (_t.recid = r2.recid)
)
UPDATE rx -- UPDATE ORIGINAL RECORD WITH NEW VALUES.
SET disp = _s.new_disp, medication_recid = _s.new_medication_recid, refills = _s.new_refills, recid = _s.target_recid
FROM _s
WHERE rx.recid = _s.old_recid
RETURNING rx.*;
COMMIT;
Hope this helps somebody.
I've created below code to implement SCD type 2 using merge, when i run the code i'm getting primary key violations on csname field. I have the below values as part of primary key, not sure whether merge SQL does support for varchar or not.
if I run the normal inner join SQL on the same key then i'm getting the matching records as well.
Any help much appreciated
csname
ER - Building Complaints
TR - Building Applications
CREATE PROCEDURE dbo.load_target
AS
BEGIN
INSERT INTO [TR_DW].[enum].[Rt]([csname],[enddatetime],[EffectiveToDate],[EffectiveFromDate],[CurrentRecord])
SELECT[csname],[enddatetime],[EffectiveToDate],[EffectiveFromDate],[CurrentRecord]
FROM
(
MERGE [TR_DW].[enum].[Rt] RtCSQSuTT
USING [TR].[enum].[Rt] RtCSQSuST
ON (RtCSQSuTT.csname = RtCSQSuST.csname)
WHEN NOT MATCHED THEN
INSERT ([csname],[enddatetime],[EffectiveToDate],[EffectiveFromDate],[CurrentRecord])
VALUES ([csname],[enddatetime],'12/31/9999', getdate(), 'Y')
WHEN MATCHED AND RtCSQSuTT.[CurrentRecord] = 'Y' AND
(ISNULL(RtCSQSuTT.[enddatetime], '') != ISNULL(RtCSQSuST.[enddatetime], ''))THEN
UPDATE SET
RtCSQSuTT.[CurrentRecord] = 'N',
RtCSQSuTT.[EffectiveFromDate] = GETDATE() - 1,
RtCSQSuTT.[EffectiveToDate] = GETDATE()
OUTPUT $Action Action_Taken,RtCSQSuST.[csqname],RtCSQSuST.[enddatetime],'12/31/9999' AS[EffectiveToDate],GETDATE() AS[EffectiveFromDate],'Y' AS[CurrentRecord]
)AS MERGE_OUT21
WHERE MERGE_OUT21.Action_Taken = 'UPDATE';
END
GO
I am a newbie here.
I am using PostgreSQL to manipulate lots of data in my specific field of research. Unfortunately, I am encountering a problem that is not allowing me to continue my analysis. I tried to simplify my problem to clearly illustrate it.
Let's suppose I have a table called "Buyers" with those data:
table_buyers
The buyers can make ONLY ONE purchase in each store or none. There are three stores and there a table for each one. Just like below:
table_store1
table_store2
table_store3
To create the tables, I am using the following code:
CREATE TABLE public.buyer
(
ID integer NOT NULL PRIMARY KEY,
name text NOT NULL,
phone text NOT NULL
)
WITH (
OIDS = FALSE
)
;
CREATE TABLE public.Store1
(
ID_buyer integer NOT NULL PRIMARY KEY,
total_order numeric NOT NULL,
total_itens integer NOT NULL
)
WITH (
OIDS = FALSE
)
;
CREATE TABLE public.Store2
(
ID_buyer integer NOT NULL PRIMARY KEY,
total_order numeric NOT NULL,
total_itens integer NOT NULL
)
WITH (
OIDS = FALSE
)
;
CREATE TABLE public.Store3
(
ID_buyer integer NOT NULL PRIMARY KEY,
total_order numeric NOT NULL,
total_itens integer NOT NULL
)
WITH (
OIDS = FALSE
)
;
To add the information on the tables, I am using the following code:
INSERT INTO buyer (ID, name, phone) VALUES
(1, 'Alex', 88888888),
(2, 'Igor', 77777777),
(3, 'Mike', 66666666);
INSERT INTO Store1 (ID_buyer, total_order, total_itens) VALUES
(1, 87.45, 8),
(2, 14.00, 3),
(3, 12.40, 4);
INSERT INTO Store2 (ID_buyer, total_order, total_itens) VALUES
(1, 785.12, 7),
(2, 9874.21, 25);
INSERT INTO Store3 (ID_buyer, total_order, total_itens) VALUES
(2, 45.87, 1);
As all the tables are interconnected by buyer's ID, I wish I could have a query that generates an output just like this:
desired output table.
Please, note that if the buyer did not buy anything in a store, I must print '0'.
I know this is an easy task, but unfortunately, I have been failing on accomplish it.
Using the 'AND' logical operator, I tried the following code to accomplish this task:
SELECT
buyer.id,
buyer.name,
store1.total_order,
store2.total_order,
store3.total_order
FROM
public.buyer,
public.store1,
public.store2,
public.store3
WHERE
buyer.id = store1.id_buyer AND
buyer.id = store2.id_buyer AND
buyer.id = store3.id_buyer;
But, obviously, it just returned 'Igor' as this was the only buyer that have bought items on all three stores (print screen).
Then, I tried the 'OR' logical operator, just like the following code:
SELECT
buyer.id,
buyer.name,
store1.total_order,
store2.total_order,
store3.total_order
FROM
public.buyer,
public.store1,
public.store2,
public.store3
WHERE
buyer.id = store1.id_buyer OR
buyer.id = store2.id_buyer OR
buyer.id = store3.id_buyer;
But then, it returns 12 lines with wrong values (print screen).
Clearly, my mistake is about not considering that 'Buyers' don't have to on all three stores on my code. I just can't correct it on my own, can you please help me?
I appreciate a lot for an answer that can light up my way. Thanks a lot!
Tips about how I can search for this issue are very welcome as well!
Ok. I doubt that this is the final answer for you, but its a start
SELECT
buyer.id,
buyer.name,
COALESCE( gb_store1.total_orders, 0 ) as store1_total,
COALESCE( gb_store2.total_orders, 0 ) as store2_total,
COALESCE( gb_store3.total_orders, 0 ) as store3_total
FROM
public.buyer,
LEFT OUTER JOIN ( SELECT ID_buyer,
SUM( total_orders ) as total_orders,
SUM( total_itens ) as total_itens
FROM public.store1
GROUP BY ID_buyer ) gb_store1 ON gb_store1.id_buyer = buyer.id ,
LEFT OUTER JOIN ( SELECT ID_buyer,
SUM( total_orders ) as total_orders,
SUM( total_itens ) as total_itens
FROM public.store2
GROUP BY ID_buyer ) gb_store2 ON gb_store2.id_buyer = buyer.id ,
LEFT OUTER JOIN ( SELECT ID_buyer,
SUM( total_orders ) as total_orders,
SUM( total_itens ) as total_itens
FROM public.store3
GROUP BY ID_buyer ) gb_store3 ON gb_store3.id_buyer = buyer.id ;
So, this query has a couple elements should focus on. The subselects/groupby allow you to total within your subtables by ID_buyer. The LEFT OUTER JOIN make its so your query can still return a result, even if a subselect finds no matching record. Finally, the COALESCE allows you to return 0 when one of your totals is NULL (because the subselect found no match).
Hope this helps.
I'm trying to insert records on my trying to implement an SCD2 on Redshift
but get an error.
The target table's DDL is
CREATE TABLE ditemp.ts_scd2_test (
id INT
,md5 CHAR(32)
,record_id BIGINT IDENTITY
,from_timestamp TIMESTAMP
,to_timestamp TIMESTAMP
,file_id BIGINT
,party_id BIGINT
)
This is the insert statement:
INSERT
INTO ditemp.TS_SCD2_TEST(id, md5, from_timestamp, to_timestamp)
SELECT TS_SCD2_TEST_STAGING.id
,TS_SCD2_TEST_STAGING.md5
,from_timestamp
,to_timestamp
FROM (
SELECT '20150901 16:34:02' AS from_timestamp
,CASE
WHEN last_record IS NULL
THEN '20150901 16:34:02'
ELSE '39991231 11:11:11.000'
END AS to_timestamp
,CASE
WHEN rownum != 1
AND atom.id IS NOT NULL
THEN 1
WHEN atom.id IS NULL
THEN 1
ELSE 0
END AS transfer
,stage.*
FROM (
SELECT id
FROM ditemp.TS_SCD2_TEST_STAGING
WHERE file_id = 2
GROUP BY id
HAVING count(*) > 1
) AS scd2_count_ge_1
INNER JOIN (
SELECT row_number() OVER (
PARTITION BY id ORDER BY record_id
) AS rownum
,stage.*
FROM ditemp.TS_SCD2_TEST_STAGING AS stage
WHERE file_id IN (2)
) AS stage
ON (scd2_count_ge_1.id = stage.id)
LEFT JOIN (
SELECT max(rownum) AS last_record
,id
FROM (
SELECT row_number() OVER (
PARTITION BY id ORDER BY record_id
) AS rownum
,stage.*
FROM ditemp.TS_SCD2_TEST_STAGING AS stage
)
GROUP BY id
) AS last_record
ON (
stage.id = last_record.id
AND stage.rownum = last_record.last_record
)
LEFT JOIN ditemp.TS_SCD2_TEST AS atom
ON (
stage.id = atom.id
AND stage.md5 = atom.md5
AND atom.to_timestamp > '20150901 16:34:02'
)
) AS TS_SCD2_TEST_STAGING
WHERE transfer = 1
and to short things up, I am trying to insert 20150901 16:34:02 to from_timestamp and 39991231 11:11:11.000 to to_timestamp.
and get
ERROR: 42804: column "from_timestamp" is of type timestamp without time zone but expression is of type character varying
Can anyone please suggest how to solve this issue?
Postgres isn't recognizing 20150901 16:34:02 (your input) as a valid time/date format, so it assumes it's a string.
Use a standard date format instead, preferably ISO-8601. 2015-09-01T16:34:02
SQLFiddle example
Just in case someone ends up here trying to insert into a postgresql a timestamp or a timestampz from a variable in groovy or Java from a prepared statement and getting the same error (as I did), I managed to do it by setting the property stringtype to "unspecified". According to the documentation:
Specify the type to use when binding PreparedStatement parameters set
via setString(). If stringtype is set to VARCHAR (the default), such
parameters will be sent to the server as varchar parameters. If
stringtype is set to unspecified, parameters will be sent to the
server as untyped values, and the server will attempt to infer an
appropriate type. This is useful if you have an existing application
that uses setString() to set parameters that are actually some other
type, such as integers, and you are unable to change the application
to use an appropriate method such as setInt().
Properties props = [user : "user", password: "password",
driver:"org.postgresql.Driver", stringtype:"unspecified"]
def sql = Sql.newInstance("url", props)
With this property set, you can insert a timestamp as a string variable without the error raised in the question title. For instance:
String myTimestamp= Instant.now().toString()
sql.execute("""INSERT INTO MyTable (MyTimestamp) VALUES (?)""",
[myTimestamp.toString()]
This way, the type of the timestamp (from a String) is inferred correctly by postgresql. I hope this helps.
Inside apache-tomcat-9.0.7/conf/server.xml
Add "?stringtype=unspecified" to the end of url address.
For example:
<GlobalNamingResources>
<Resource name="jdbc/??" auth="Container" type="javax.sql.DataSource"
...
url="jdbc:postgresql://127.0.0.1:5432/Local_DB?stringtype=unspecified"/>
</GlobalNamingResources>