I would like to use Upsolver MERGE command in my new transformations to populates S3/Athena and Snowflake tables. Since Snowflake is supporting Upsert command, while defining my transformation job, do I rely on Snowflake functionality and use Upsolver INSER statement or define Upsolver MERGE transformation in the same way I do it for Athena , i.e.
CREATE JOB my_job_upsert
START_FROM = BEGINNING
ADD_MISSING_COLUMNS = TRUE
RUN_INTERVAL = 1 MINUTE
AS MERGE INTO default_glue_catalog.upsolver_samples.test_upsert_with_merge AS target
/*
Use the SELECT statement below to choose your columns and performed the desired transformations.
In this example, we aggregate the sample orders data by customer and filter it to only include repeat purchasers.
*/
USING (SELECT field1 AS email,
COUNT(DISTINCT field2) AS count
MIN(field3) AS min_number,
MAX(date) AS last_date
FROM default_glue_catalog.upsolver_samples.test_raw_data
WHERE $commit_time BETWEEN run_start_time() AND run_end_time()
GROUP BY 1
HAVING COUNT(DISTINCT field2) > 1) source
ON (target.email = source.email)--primary key
WHEN MATCHED THEN REPLACE -- Update if primary keys match
WHEN NOT MATCHED THEN INSERT MAP_COLUMNS_BY_NAME; -- Insert if primary key is unique (new record)
It would be nice to know in general if MERGE command syntax is consistent across various target platforms.
I already build Athena transformation and it works as expected
You can use the same way you did for Athena. Upsolver INSERT command will insert new keys (APPEND) and if the table had primary key defined, INSERT command will update the existing keys (Upsert) as its default behavior.
MERGE as its definition is for UPSERT and can handle Deletes as well. And the syntax is consistent across all database/data warehouse/catalog targets
I have two other questions about this but this one is more focused. The Merge statement's basic structure is given below. In the interest of time and length, I have summarized some of the details Originally, I was going to try to merge 8 into 1. That has been a fruitless venture so I have narrowed it down. For this one, I am trying to combine industry21 and industry4 knowing that there are 8 rows of primary key duplication. The primary key is a compound one comprised of 9 fields (not sure if that is relevant). They are 13 foreign keys.
From looking at the code below, can someone see why I would get an incorrect syntax on the = sign on the first bold item. Next on the same bold line, why when I hover over the . between source and periodtype on the same line do I get an error about "Merge Statement must be terminated in a column"
Merge industry4 AS TARGET
USING industry21 as SOURCE
ON (target.primarykey1 = Source.primarykey1) or (target.primarykey2 =
Source.primarykey2) or..... (target.primarykey9 = Source.primarykey9)
--When records are matched, update
--the records if there is any change
WHEN MATCHED AND
TARGET.foreignkey1 <> SOURCE.foreignkey1
OR .... TARGET.foreignkey13 <> SOURCE.foreignkey2
THEN
UPDATE SET TARGET.pk1 = SOURCE.pk1,
TARGET.pk2 = SOURCE.pk2
**TARGET.periodtype(pk5) = SOURCE.periodtype (pk5)**
.....
TARGET.fk13 = SOURCE.fk13
--When no records are matched, insert
--the incoming records from source
--table to target table
WHEN NOT MATCHED BY TARGET THEN
INSERT (fieids 1-22)
VALUES (SOURCE.pk1, SOURCE.pk2......SOURCE.fk13)
WHEN NOT MATCHED BY SOURCE THEN
DELETE
I was able to accomplish the task using the following code. It is not related to my original question but apparently merge statements are not the best nor popular way to do this.
SELECT x.*
INTO [NEW_TABLE]
FROM (SELECT * FROM TABLE1
UNION
SELECT * FROM TABLE2) x
Hello I'm writing an sql query But i am getting a syntax error on the line with the GROUP BY. What can possibly be the problem, help if you can please.
UPDATE intersection_points i
SET nbr_victimes = sum(tue+bl+bg)
FROM accident_ma a ,intersection_points i
WHERE (ST_DWithin(i.st_intersection,a.geom_acc, 10000) group by st_intersection)) ;
GROUP BY is its own clause, it's not part of a WHERE clause.
This is what you have:
WHERE (
ST_DWithin(i.st_intersection,a.geom_acc, 10000)
group by st_intersection
)
This is what you need:
WHERE ST_DWithin(i.st_intersection,a.geom_acc, 10000)
group by st_intersection
Edit: In response to comments, it sounds like your JOIN is a bit more complex than the UPDATE ... FROM syntax would need. Take a look at the "Notes" section on this page:
When a FROM clause is present, what essentially happens is that the target table is joined to the tables mentioned in the from_list, and each output row of the join represents an update operation for the target table. When using FROM you should ensure that the join produces at most one output row for each row to be modified. In other words, a target row shouldn't join to more than one row from the other table(s). If it does, then only one of the join rows will be used to update the target row, but which one will be used is not readily predictable.
Because of this indeterminacy, referencing other tables only within sub-selects is safer, though often harder to read and slower than using a join.
Normally this would involve changing the syntax to something like:
UDPATE SomeTable
SET SomeColumn = 'Some Value'
WHERE AnotherColumn =
(SELECT AnotherColumn
FROM AnotherTable
-- etc.)
However, the use of ST_DWithin() in this query may complicate that quite a bit. Without much deeper knowledge of the table structures, relationships, and overall intent of this update there probably isn't much more help I can give. Essentially you're going to need to clarify for the database exactly what records need to be updated and how to update them, which may involve changing your query to this latter sub-select syntax in some way.
I don' t understand your data structure. I create the following tables from your query. Please check table structure.
if table's structure is this
your query must be
UPDATE intersection_points SET nbr_victimes = (SELECT SUM(a.tue+a.bl+a.bg) FROM accident_ma a WHERE st_dwithin(st_intersection, a.geom_acc, 1000));
I have three tables and only one that I directly control and am doing a MERGE between them. See my abbreviated but working example here (sqlfiddle example).
I am doing a MERGE between table 1 and Table 2 to Table 3. Table 1 has duplicate data which the MERGE (erroneously) can handle on the first run (insert) but fails with this message on the second run (update).
The MERGE statement attempted to UPDATE or DELETE the same row more
than once.
My question is, can the MERGE be written to either use an EXCEPT such as
SELECT AdFull FROM [dbo].[Users] WHERE AdFull IS NOT NULL
EXCEPT
SELECT AdFull FROM [dbo].[Users]
WHERE AdFull IS NOT NULL
GROUP BY AdFull
HAVING COUNT(*) = 1
or a different Join to only show users that are not duplicated? Or even a way to select a specific one of the duplicates?
Answered Questions
MERGE is a working Insert due to the nature of Fiddle. But due (AFAIK) to the stateless nature of fiddle one never sees the error in Fiddle on a second run, because a merge never happens with the data, only inserts.
Ignore Rows: Actually I would eventually like to use an individual duplicate row via divining of one based on a condition. The actual data table I am dealing with away from the fiddle example has more columns and it would be nice to maybe select a specific row in a duplicate set due to a specific condition.
The example doesn't bare it out, but yes the duplicates are due to the computed AdFull column. Think of a system adding a temp employee, that user gets a row. Then the temp employee gets hired on as fulltime, keeps the ad account but then gets another row in the user table. Yes I know it shouldn't happen. So that is how a duplicate comes about.
(Duplicate values Table 3) Table three is a result table that can be cleaned out for any duplicates to start this process afresh.
In your MERGE statement can you do something similar this?
MERGE INTO [dbo].Table3 AS T3
USING
(
SELECT
AdFull,
MAX(StartedOn)
FROM [dbo].Table2 AS [ad]
GROUP BY AdFull
) AS T2
ON (T2.AdFull = T3.AdFull)
WHEN MATCHED THEN UPDATE blah
WHEN NOT MATCHED THEN INSERT blah
Using the MAX aggregate with a GROUP BY should give you only the information from when the temp was hired on. Then if the AdFull matches you can simply UPDATE Table3 with the most recent information and if there is no match then INSERT a new row.
UPDATE: If I fail to mention that MERGE should be used with caution I will take flak from #AaronBertrand.
I'm getting the following error in SSRS:
An error occurred while the query design method was being saved.
An item with the same key has already been added
What does an "item" denote, though? I even tried editing the RDL and deleting all references to the Stored Procedure I need to use called prc_RPT_Select_BI_Completes_Data_View.
Could this possibly have to do with the fact that the Stored Procedure uses Dynamic SQL (the N' notation)?
In the stored procedure I have:
SET #SQL += N'
SELECT bi.SupplierID as ''Supplier ID''
,bi.SupplierName as ''Supplier Name''
,bi.PID as ''PID''
,bi.RespondentID as ''Respondent ID''
,lk_slt.Name as ''Entry Link Type''
,ts.SurveyNumber as ''Initial Survey ID'''
It appears that SSRS has an issue(at leastin version 2008) - I'm studying this website that explains it
Where it says if you have two columns(from 2 diff. tables) with the same name, then it'll cause that problem.
From source:
SELECT a.Field1, a.Field2, a.Field3, b.Field1, b.field99 FROM TableA a
JOIN TableB b on a.Field1 = b.Field1
SQL handled it just fine, since I had prefixed each with an alias
(table) name. But SSRS uses only the column name as the key, not table
+ column, so it was choking.
The fix was easy, either rename the second column, i.e. b.Field1 AS
Field01 or just omit the field all together, which is what I did.
I face the same issue. After debug I fixed the same. if the column name in your SQL query has multiple times then this issue occur. Hence use alias in SQL query to differ the column name.
Ex:
The below query will work proper in sql query but create issue in SSRS report:
Select P.ID, P.FirstName, P.LastName, D.ID, D.City, D.Area, D.Address
From PersonalDetails P
Left Join CommunicationDetails D On P.ID = D.PersonalDetailsID
Reason : ID has mentioned twice (Multiple Times)
Correct Query:
Select P.ID As PersonalDetailsID, P.FirstName, P.LastName, D.ID, D.City, D.Area, D.Address
From PersonalDetails P
Left Join CommunicationDetails D On P.ID = D.PersonalDetailsID
I have experience this issue in past. Based on that I can say that generally we get this issue if your dataset has multiple fieldnames that points to same field source.
Take a look into following posts for detail error description
https://bi-rootdata.blogspot.com/2012/09/an-error-occurred-during-report.html
https://bi-rootdata.blogspot.com/2012/09/an-item-with-same-key-has-already-been.html
In your case, you should check your all field names returned by Sp prc_RPT_Select_BI_Completes_Data_View and make sure that all fields has unique name.
I had the same error in a report query. I had columns from different tables with the same name and the prefix for each table (eg: select a.description, b.description, c.description) that runs ok in Oracle, but for the report you must have an unique alias for each column so simply add alias to the fields with the same name (select a.description a_description, b.description b_description and so on)
Sorry, it is a response to an old thread, but might still be useful.
In addition to above responses,
This generally happens when two columns with same name, even from different tables are included in the same query.
for example if we joining two tables city and state where tables have column name
e.g. city.name and state.name.
when such a query is added to the dataset, SSRS removes the table name or the table alias and only keeps the name, whih eventually appears twice in the query and errors as duplicate key.
The best way to avoid it is to use alias such as calling the column names
city.name as c_name
state.name as s_name.
This will resolve the issue.
I got this error message with vs2015 enterprise, ssdt 14.1.xxx, ssrs. For me I think it was something different than described above with a 2 column, same name problem. I added this report, then deleted the report, then when I tried to add the query back in the ssrs wizard I got this message, " An error occurred while the query design method was being saved :invalid object name: tablename" . where tablename was the table on the query the wizard was reading. I tried cleaning the project, I tried rebuilding the project. In my opinion Microsoft isn't completing cleaning out the report when you delete it and as long as you try to add the original query back it won't add. The way I was able to fix it was to create the ssrs report in a whole new project (obviously nothing wrong with the query) and save it off to the side. Then I reopened my original ssrs project, right clicked on Reports, then Add, then add Existing Item. The report added back in just fine with no name conflict.
I just got this error and i came to know that it is about the local variable alias
at the end of the stored procedure i had like
select #localvariable1,#localvariable2
it was working fine in sql but when i ran this in ssrs
it was always throwing error but after I gave alias it is fixed
select #localvariable1 as A,#localvariable2 as B
SSRS will not accept duplicated columns so ensure that your query or store procedure is returning unique column names.
If you are using SPs and if the sps have multiple Select statements (within if conditions) all those selects needs to be handled with unique field names.