I have following dataset:
org system_id punch_start_tb1 punch_start_tb2
CG 100242 2022-08-16T00:08:00Z 2022-08-16T03:08:00Z
LA 250595 2022-08-16T00:00:00Z 2022-08-16T03:00:00Z
LB 300133 2022-08-15T04:00:00Z 2022-08-16T04:00:00Z
LB 300133 2022-08-16T04:00:00Z 2022-08-15T04:00:00Z
MO 400037 2022-08-15T14:00:00Z 2022-08-15T23:00:00Z
MO 400037 2022-08-15T23:00:00Z 2022-08-15T14:00:00Z
I am trying to filter out data so that it only populates the outcome when Count of "system_id" = 1.
So, the expected outcome would be only following two rows:
org system_id punch_start_tb1 punch_start_tb2
CG 100242 2022-08-16T00:08:00Z 2022-08-16T03:08:00Z
LA 250595 2022-08-16T00:00:00Z 2022-08-16T03:00:00Z
I tried with Group by and Having clause, but I did not have a success.
You can try below
SELECT * FROM
(
SELECT org,system_id,punch_start_tbl,punch_start_tb2
,ROW_NUMBER()OVER(PARTITION BY system_id ORDER BY system_id)RN
FROM <TableName>
)X
WHERE RN = 1
CTE returns org with only one record then join with main table on org column.
;WITH CTE AS (
select org
from <table_name>
group by org
Having count(1) = 1
)
select t.*
from cte
inner join <table_name> t on cte.org = t.org
You can try this (use min because we have only one row):
select MIN(org), system_id, MIN(punch_start_tb1), MIN(punch_start_tb2)
from <table_name>
group by system_id
Having count(1) = 1
or use answer #Meyssam Toluie with group by by system_id
I have report like this:
I tried to remove the duplicate values in documentno as paymentno column using :
DELETE FROM c_payment pa USING (
SELECT MIN(ctid) as ctid, pa.documentno
FROM c_payment pa
GROUP BY pa.documentno
HAVING COUNT(*) > 1
) b
WHERE pa.documentno = b.documentno
AND pa.ctid <> b.ctid) documentno
but it's still not working, any other advice on what method should I use?
You should remove the last closing brace together with the word documentno.
By using sequence number logic will work for my case
ROW_NUMBER() OVER( PARTITION BY pym.c_payment_id ORDER BY pym.c_bpartner_id)
as seq_no, coalesce (ci.documentno, '-') as inv_no ,
instead of DELETE with or without subquery
DELETE FROM c_payment pa USING (
SELECT MIN(ctid) as ctid, pa.documentno
FROM c_payment pa
GROUP BY pa.documentno
HAVING COUNT(*) > 1
) b
WHERE pa.documentno = b.documentno
AND pa.ctid <> b.ctid) documentno
I am using TSQL, SSMS v.17.9.1 The underlying db is Microsoft SQL Server 2014 SP3
For display purposes, I want to concatenate the results of two queries:
SELECT TOP 1 colA as 'myCol1' FROM tableA
--
SELECT TOP 1 colB as 'myCol2' FROM tableB
and display the results from the queries in one row in SSMS.
(The TOP 1 directive would hopefully guarantee the same number of results from each query, which would assist displaying them together. If this could be generalized to TOP 10 per query that would help also)
This should work for any number of rows, it assumes you want to pair ordered by the values in the column displayed
With
TableA_CTE AS
(
SELECT TOP 1 colA as myCol1
,Row_Number() OVER (ORDER BY ColA DESC) AS RowOrder
FROM tableA
),
TableB_CTE AS
(
SELECT TOP 1 colB as myCol2
,Row_Number() OVER (ORDER BY ColB DESC) AS RowOrder
FROM tableB
)
SELECT A.myCol1, B.MyCol2
FROM TableA_CTE AS A
INNER JOIN TableB_CTE AS B
ON A.RowOrder = B.RowOrder
There are currently two issues with the accepted answer:
I) a missing comma before the line: "Table B As"
II) TSQL seems to find it recursive as written, so I re-wrote it in a non-recursive way:
This is a re-working of the accepted answer that actually works in T-SQL:
USE [Database_1];
With
CTE_A AS
(
SELECT TOP 1 [Col1] as myCol1
,Row_Number() OVER (ORDER BY [Col2] desc) AS RowOrder
FROM [TableA]
)
,
CTE_B AS
(
SELECT TOP 1 [Col2] as myCol2
,Row_Number() OVER (ORDER BY [Col2] desc) AS RowOrder
FROM [TableB]
)
SELECT A.myCol1, B.myCol2
FROM CTE_A AS A
INNER JOIN CTE_B AS B
ON ( A.RowOrder = B.RowOrder)
Using ms-sql 2008 r2; am sure this is very straightforward. I am trying to identify where a unique value {ISIN} has been linked to more than 1 Identifier. An example output would be:
isin entity_id
XS0276697439 000BYT-E
XS0276697439 000BYV-E
This is actually an error and I want to look for other instances where there may be more than one entity_id linked to a unique ISIN.
This is my current working but it's obviously not correct:
select isin, entity_id from edm_security_entity_map
where isin is not null
--and isin = ('XS0276697439')
group by isin, entity_id
having COUNT(entity_id) > 1
order by isin asc
Thanks for your help.
Elliot,
I don't have a copy of SQL in front of me right now, so apologies if my syntax isn't spot on.
I'd start by finding the duplicates:
select
x.isin
,count(*)
from edm_security_entity_map as x
group by x.isin
having count(*) > 1
Then join that back to the full table to find where those duplicates come from:
;with DuplicateList as
(
select
x.isin
--,count(*) -- not used elsewhere
from edm_security_entity_map as x
group by x.isin
having count(*) > 1
)
select
map.isin
,map.entity_id
from edm_security_entity_map as map
inner join DuplicateList as dup
on dup.isin = map.isin;
HTH,
Michael
So you're saying that if isin-1 has a row for both entity-1 and entity-2 that's an error but isin-3, say, linked to entity-3 in two separe rows is OK? The ugly-but-readable solution to that is to pre-pend another CTE on the previous solution
;with UniqueValues as
(select distinct
y.isin
,y.entity_id
from edm_security_entity_map as y
)
,DuplicateList as
(
select
x.isin
--,count(*) -- not used elsewhere
from UniqueValues as x
group by x.isin
having count(*) > 1
)
select
map.isin
,map.entity_id
from edm_security_entity_map as map -- or from UniqueValues, depening on your objective.
inner join DuplicateList as dup
on dup.isin = map.isin;
There are better solutions with additional GROUP BY clauses in the final query. If this is going into production I'd be recommending that. Or if your table has a bajillion rows. If you just need to do some analysis the above should suffice, I hope.
This query for creating a list of Candidate duplicates is easy enough:
SELECT Count(*), Can_FName, Can_HPhone, Can_EMail
FROM Can
GROUP BY Can_FName, Can_HPhone, Can_EMail
HAVING Count(*) > 1
But if the actual rule I want to check against is FName and (HPhone OR Email) - how can I adjust the GROUP BY to work with this?
I'm fairly certain I'm going to end up with a UNION SELECT here (i.e. do FName, HPhone on one and FName, EMail on the other and combine the results) - but I'd love to know if anyone knows an easier way to do it.
Thank you in advance for any help.
Scott in Maine
Before I can advise anything, I need to know the answer to this question:
name phone email
John 555-00-00 john#example.com
John 555-00-01 john#example.com
John 555-00-01 john-other#example.com
What COUNT(*) you want for this data?
Update:
If you just want to know that a record has any duplicates, use this:
WITH q AS (
SELECT 1 AS id, 'John' AS name, '555-00-00' AS phone, 'john#example.com' AS email
UNION ALL
SELECT 2 AS id, 'John', '555-00-01', 'john#example.com'
UNION ALL
SELECT 3 AS id, 'John', '555-00-01', 'john-other#example.com'
UNION ALL
SELECT 4 AS id, 'James', '555-00-00', 'james#example.com'
UNION ALL
SELECT 5 AS id, 'James', '555-00-01', 'james-other#example.com'
)
SELECT *
FROM q qo
WHERE EXISTS
(
SELECT NULL
FROM q qi
WHERE qi.id <> qo.id
AND qi.name = qo.name
AND (qi.phone = qo.phone OR qi.email = qo.email)
)
It's more efficient, but doesn't tell you where the duplicate chain started.
This query select all entries along with the special field, chainid, that indicates where the duplicate chain started.
WITH q AS (
SELECT 1 AS id, 'John' AS name, '555-00-00' AS phone, 'john#example.com' AS email
UNION ALL
SELECT 2 AS id, 'John', '555-00-01', 'john#example.com'
UNION ALL
SELECT 3 AS id, 'John', '555-00-01', 'john-other#example.com'
UNION ALL
SELECT 4 AS id, 'James', '555-00-00', 'james#example.com'
UNION ALL
SELECT 5 AS id, 'James', '555-00-01', 'james-other#example.com'
),
dup AS (
SELECT id AS chainid, id, name, phone, email, 1 as d
FROM q
UNION ALL
SELECT chainid, qo.id, qo.name, qo.phone, qo.email, d + 1
FROM dup
JOIN q qo
ON qo.name = dup.name
AND (qo.phone = dup.phone OR qo.email = dup.email)
AND qo.id > dup.id
),
chains AS
(
SELECT *
FROM dup do
WHERE chainid NOT IN
(
SELECT id
FROM dup di
WHERE di.chainid < do.chainid
)
)
SELECT *
FROM chains
ORDER BY
chainid
None of these answers is correct. Quassnoi's is a decent approach, but you will notice one fatal flaw in the expressions "qo.id > dup.id" and "di.chainid < do.chainid": comparisons made by ID! This is ALWAYS bad practice because it depends on some inherent ordering in the IDs. IDs should NEVER be given any implicit meaning and should ONLY participate in equality or null testing. You can easily break Quassnoi's solution in this example by simply reordering the IDs in the data.
The essential problem is a disjunctive condition with a grouping, which leads to the possibility of two records being related through an intermediate, though they are not directly relatable.
e.g., you stated these records should all be grouped:
(1) John 555-00-00 john#example.com
(2) John 555-00-01 john#example.com
(3) John 555-00-01 john-other#example.com
You can see that #1 and #2 are relatable, as are #2 and #3, but clearly #1 and #3 are not directly relatable as a group.
This establishes that a recursive or iterative solution is the ONLY possible solution.
So, recursion is not viable since you can easily end up in a looping situation. This is what Quassnoi was trying to avoid with his ID comparisons, but in doing so he broke the algorithm. You could try limiting the levels of recursion, but you may not then complete all relations, and you will still potentially be following loops back upon yourself, leading to excessive data size and prohibitive inefficiency.
The best solution is ITERATIVE: Start a result set by tagging each ID as a unique group ID, and then spin through the result set and update it, combining IDs into the same unique group ID as they match on the disjunctive condition. Repeat the process on the updated set each time until no further updates can be made.
I will create example code for this soon.
GROUP BY doesn't support OR - it's implicitly AND and must include every non-aggregator in the select list.
I assume you also have a unique ID integer as the primary key on this table. If you don't, it's a good idea to have one, for this purpose and many others.
Find those duplicates by a self-join:
select
c1.ID
, c1.Can_FName
, c1.Can_HPhone
, c1.Can_Email
, c2.ID
, c2.Can_FName
, c2.Can_HPhone
, c2.Can_Email
from
(
select
min(ID),
Can_FName,
Can_HPhone,
Can_Email
from Can
group by
Can_FName,
Can_HPhone,
Can_Email
) c1
inner join Can c2 on c1.ID < c2.ID
where
c1.Can_FName = c2.Can_FName
and (c1.Can_HPhone = c2.Can_HPhone OR c1.Can_Email = c2.Can_Email)
order by
c1.ID
The query gives you N-1 rows for each N duplicate combinations - if you want just a count along with each unique combination, count the rows grouped by the "left" side:
select count(1) + 1,
, c1.Can_FName
, c1.Can_HPhone
, c1.Can_Email
from
(
select
min(ID),
Can_FName,
Can_HPhone,
Can_Email
from Can
group by
Can_FName,
Can_HPhone,
Can_Email
) c1
inner join Can c2 on c1.ID < c2.ID
where
c1.Can_FName = c2.Can_FName
and (c1.Can_HPhone = c2.Can_HPhone OR c1.Can_Email = c2.Can_Email)
group by
c1.Can_FName
, c1.Can_HPhone
, c1.Can_Email
Granted, this is more involved than a union - but I think it illustrates a good way of thinking about duplicates.
Project the desired transformation first from a derived table, then do the aggregation:
SELECT COUNT(*)
, CAN_FName
, Can_HPhoneOrEMail
FROM (
SELECT Can_FName
, ISNULL(Can_HPhone,'') + ISNULL(Can_EMail,'') AS Can_HPhoneOrEMail
FROM Can) AS Can_Transformed
GROUP BY Can_FName, Can_HPhoneOrEMail
HAVING Count(*) > 1
Adjust your 'OR' operation as needed in the derived table project list.
I know this answer will be criticised for the use of the temp table, but it will work anyway:
-- create temp table to give the table a unique key
create table #tmp(
ID int identity,
can_Fname varchar(200) null, -- real type and len here
can_HPhone varchar(200) null, -- real type and len here
can_Email varchar(200) null, -- real type and len here
)
-- just copy the rows where a duplicate fname exits
-- (better performance specially for a big table)
insert into #tmp
select can_fname,can_hphone,can_email
from Can
where can_fname exists in (select can_fname from Can
group by can_fname having count(*)>1)
-- select the rows that have the same fname and
-- at least the same phone or email
select can_Fname, can_Hphone, can_Email
from #tmp a where exists
(select * from #tmp b where
a.ID<>b.ID and A.can_fname = b.can_fname
and (isnull(a.can_HPhone,'')=isnull(b.can_HPhone,'')
or (isnull(a.can_email,'')=isnull(b.can_email,'') )
Try this:
SELECT Can_FName, COUNT(*)
FROM (
SELECT
rank() over(partition by Can_FName order by Can_FName,Can_HPhone) rnk_p,
rank() over(partition by Can_FName order by Can_FName,Can_EMail) rnk_m,
Can_FName
FROM Can
) X
WHERE rnk_p=1 or rnk_m =1
GROUP BY Can_FName
HAVING COUNT(*)>1