If database d1 has tables T1,T2,T3,T4 all with the field "Date1".
What is the best way to get a count of all records across all tables with a date older than 3 days ago?
I know one could do unions, I assume there is no nifty syntax that would omit all tables [like a 'parent' object in C++].
Here best may mean more efficient, or just a pleasing syntax in T-SQL.
This is for SSMS 17.7. Microsoft SQL Server 2014 (SP2)
If you know the table names in advance, a simple query on union all will probably be the simplest way:
SELECT COUNT(*)
FROM
(
SELECT Date1
FROM T1
UNION ALL
SELECT Date1
FROM T2
SELECT Date1
FROM T3
SELECT Date1
FROM T4
) As t
WHERE Date1 <= DATEADD(DAY, -3, GETDATE())
If you don't know the table names in advance, you can use information_schema.columns to build the union query dynamically.
Well, you're interested in a parent object, that would be a view, then. You can reuse it for a variety of queries. Alternatively, add more columns if you need them:
CREATE VIEW parent AS
SELECT Date1 FROM t1 UNION ALL
SELECT Date1 FROM t2 UNION ALL
SELECT Date1 FROM t3 UNION ALL
SELECT Date1 FROM t4;
And now, that can be queried in the way you want
SELECT COUNT(*) FROM parent WHERE Date1 <= DATEADD(DAY, -3, GETDATE())
Without UNION?
Since a COUNT without a GROUP BY returns 1 value, why not use CROSS JOIN for once?
SELECT
t1.Cnt AS [T1],
t2.Cnt AS [T2],
t3.Cnt AS [T3],
t4.Cnt AS [T4],
(t1.Cnt + t2.Cnt + t3.Cnt + t4.Cnt) AS [T1234]
FROM
(SELECT COUNT(*) AS Cnt FROM T1 WHERE [Date1] < CAST(GetDate()-3 AS DATE)) AS t1
CROSS JOIN
(SELECT COUNT(*) AS Cnt FROM T2 WHERE [Date1] < CAST(GetDate()-3 AS DATE)) AS t2
CROSS JOIN
(SELECT COUNT(*) AS Cnt FROM T3 WHERE [Date1] < CAST(GetDate()-3 AS DATE)) AS t3
CROSS JOIN
(SELECT COUNT(*) AS Cnt FROM T4 WHERE [Date1] < CAST(GetDate()-3 AS DATE)) AS t4
Or a CROSS APPLY
SELECT
t1.Cnt AS [T1],
t2.Cnt AS [T2],
t3.Cnt AS [T3],
t4.Cnt AS [T4],
(t1.Cnt + t2.Cnt + t3.Cnt + t4.Cnt) AS [T1234]
FROM (SELECT CAST(GetDate()-3 AS DATE) as Dt) d
CROSS APPLY (SELECT COUNT(*) AS Cnt FROM T1 WHERE [Date1] < d.Dt) AS t1
CROSS APPLY (SELECT COUNT(*) AS Cnt FROM T2 WHERE [Date1] < d.Dt) AS t2
CROSS APPLY (SELECT COUNT(*) AS Cnt FROM T3 WHERE [Date1] < d.Dt) AS t3
CROSS APPLY (SELECT COUNT(*) AS Cnt FROM T4 WHERE [Date1] < d.Dt) AS t4
Example snippet for Sql Server:
declare #T1 table (id int primary key identity(1,1), [Date1] date);
declare #T2 table (id int primary key identity(1,1), [Date1] date);
declare #T3 table (id int primary key identity(1,1), [Date1] date);
declare #T4 table (id int primary key identity(1,1), [Date1] date);
insert into #T1 ([Date1]) values (getdate()-6),(getdate()-5),(getdate()-4),(getdate()-3),(getdate()-2),(getdate()-1),(getdate()-0);
insert into #T2 ([Date1]) select top 6 [Date1] from #T1 order by [Date1] desc;
insert into #T3 ([Date1]) select top 5 [Date1] from #T1 order by [Date1] desc;
insert into #T4 ([Date1]) select top 4 [Date1] from #T1 order by [Date1] desc;
SELECT
t1.Cnt AS [T1],
t2.Cnt AS [T2],
t3.Cnt AS [T3],
t4.Cnt AS [T4],
(t1.Cnt + t2.Cnt + t3.Cnt + t4.Cnt) AS [T1234]
FROM
(SELECT COUNT(*) AS Cnt FROM #T1 WHERE [Date1] < CAST(GetDate()-3 AS DATE)) AS t1
CROSS JOIN
(SELECT COUNT(*) AS Cnt FROM #T2 WHERE [Date1] < CAST(GetDate()-3 AS DATE)) AS t2
CROSS JOIN
(SELECT COUNT(*) AS Cnt FROM #T3 WHERE [Date1] < CAST(GetDate()-3 AS DATE)) AS t3
CROSS JOIN
(SELECT COUNT(*) AS Cnt FROM #T4 WHERE [Date1] < CAST(GetDate()-3 AS DATE)) AS t4
Returns:
T1 T2 T3 T4 T1234
3 2 1 0 6
Instead create a view, you can use a CTE (Common Table Expression). It works like a view, but not persists on database. Please try it:
WITH CteDate( Date1 )
AS ( SELECT Date1 FROM t1 UNION ALL
SELECT Date1 FROM t2 UNION ALL
SELECT Date1 FROM t3 UNION ALL
SELECT Date1 FROM t4
)
SELECT COUNT(*) FROM CteDate WHERE Date1 <= DATEADD(DAY, -3, GETDATE())
It works for all SQL Server greater or equal then 2005.
I'm trying to combine range.
WITH a AS (
select '2017-09-16 07:12:57' as begat,'2017-09-16 11:30:22' as endat
union
select '2017-09-18 17:05:21' ,'2017-09-19 13:18:01'
union
select '2017-09-19 15:34:40' ,'2017-09-22 13:29:37'
union
select '2017-09-22 12:24:16' ,'2017-09-22 13:18:29'
union
select '2017-09-28 09:48:54' ,'2017-09-28 13:39:13'
union
select '2017-09-20 13:52:43' ,'2017-09-20 14:14:43'
), b AS (
SELECT *, lag(endat) OVER (ORDER BY begat) < begat OR NULL AS step
FROM a
)
, c AS (
SELECT *, count(step) OVER (ORDER BY begat) AS grp
FROM b
)
SELECT min(begat), coalesce( max(endat), 'infinity' ) AS range
FROM c
GROUP BY grp
ORDER BY 1
Result
1 "2017-09-16 07:12:57";"2017-09-16 11:30:22"
2 "2017-09-18 17:05:21";"2017-09-19 13:18:01"
3 "2017-09-19 15:34:40";"2017-09-22 13:29:37"
4 "2017-09-22 12:24:16";"2017-09-22 13:18:29"
5 "2017-09-28 09:48:54";"2017-09-28 13:39:13"
positions 3,4 intersect (endata> next begat)
How do I make the union of all the intersections into one large interval
I need result
1 "2017-09-16 07:12:57";"2017-09-16 11:30:22"
2 "2017-09-18 17:05:21";"2017-09-19 13:18:01"
3 "2017-09-19 15:34:40";"2017-09-22 13:29:37"
4 "2017-09-28 09:48:54";"2017-09-28 13:39:13"
Hey I would suggest using the following process :
1- Identify when a row is new, so you give a value of 1 to values that do not overlap (CTE b)
2- Sequence together the rows that have overlaps with others. This way you can see have a common identifier that will allow you to MAX and MIN begat and endat (CTE c)
3- For each sequence, give the MIN of begat and the MAX of endat so you will have your final values
WITH a AS (
select '2017-09-16 07:12:57' as begat,'2017-09-16 11:30:22' as endat
union
select '2017-09-18 17:05:21' ,'2017-09-19 13:18:01'
union
select '2017-09-19 15:34:40' ,'2017-09-22 13:29:37'
union
select '2017-09-22 12:24:16' ,'2017-09-22 13:18:29'
union
select '2017-09-28 09:48:54' ,'2017-09-28 13:39:13'
union
select '2017-09-20 13:52:43' ,'2017-09-20 14:14:43'
)
, b AS (
SELECT
begat
, endat
, (begat > MAX(endat) OVER w IS TRUE)::INT is_new
FROM a
WINDOW w AS (ORDER BY begat ROWS BETWEEN UNBOUNDED PRECEDING AND 1 PRECEDING)
)
, c AS (
SELECT
begat
, endat
, SUM((is_new)) OVER (ORDER BY begat) seq
FROM b
)
SELECT
MIN(begat) beg_at
, MAX(endat) end_at
FROM c
GROUP BY seq
EDITED
If you need speed you can use a psql function:
create or replace function append_ranges_in_a() returns setof a
language plpgsql
as
$BODY$
declare
v_current a%rowtype;
v_new a%rowtype;
v_first boolean:=true;
begin
for v_current in select begat, endat from a order by begat, endat
loop
if v_first then
v_first := false;
v_new.begat := v_current.begat;
elsif v_new.endat < v_current.begat then
return next v_new;
v_new.begat := v_current.begat;
end if;
v_new.endat := greatest(v_current.endat,v_new.endat);
end loop;
return next v_new;
return;
end;
$BODY$;
select * from append_ranges_in_a()
I test it with ~ 400000 rows:
delete from a;
insert into a (begat, endat)
select time::text, (time+'1 day'::interval)::text
from (select t+(round(random()*23.0)||' hours')::interval as time
from generate_series('1401-01-01'::timestamp,'2018-08-21'::timestamp,'1 day'::interval) t
) t;
select count(*) from a;
select * from append_ranges_in_a() offset 100000 limit 10
and it is twice fast as O(n^2) pure SQL version.
OLD slow solution:
You can use a recursive WITH query https://www.postgresql.org/docs/current/static/queries-with.html to construct the result row by row.
I create the table
The first row is the candidate first row (ending where ending), but the row is not "ready"
Then I look at the next row (step) and if it is not intersecting I add a ready row,
Also I add a not ready row with the current (last) observed range
When I do not have more rows I calculate the last row
I retain ready rows and the last row
Here is the code
CREATE TABLE a as
select '2017-09-16 07:12:57' as begat,'2017-09-16 11:30:22' as endat
union
select '2017-09-18 17:05:21' ,'2017-09-19 13:18:01'
union
select '2017-09-19 15:34:40' ,'2017-09-22 13:29:37'
union
select '2017-09-22 12:24:16' ,'2017-09-22 13:18:29'
union
select '2017-09-28 09:48:54' ,'2017-09-28 13:39:13'
union
select '2017-09-20 13:52:43' ,'2017-09-20 14:14:43';
WITH RECURSIVE t(begat, endat, ready, step) AS (
select * from (
select *,false,1 from a order by begat, endat limit 1) a
UNION ALL
SELECT new_rows.*
FROM (SELECT * FROM t ORDER BY begat DESC limit 1) t,
lateral (SELECT * FROM a ORDER BY begat, endat OFFSET step LIMIT 1) a,
lateral (
SELECT t.begat, t.endat, true as ready, step WHERE t.endat < a.begat
UNION SELECT CASE WHEN t.endat < a.begat THEN a.begat ELSE t.begat END, greatest(a.endat, t.endat), false, step+1
) new_rows
)
select begat, endat
from (
select begat, endat, ready, row_number() over (order by begat desc, endat desc)=1 is_last
from t
order by begat, endat) t
where ready or is_last;
i using range type
https://www.postgresql.org/docs/9.3/static/rangetypes.html
WITH tmp AS (
-- preparation range type
select begat, coalesce( endat, 'infinity' ) as endAt, tsrange( begat, coalesce( endat, 'infinity' ) ) as rg
from (
select '2017-09-11 17:13:03'::timestamp as begat ,'2017-09-12 12:24:09'::timestamp as endat union
select '2017-09-19 15:34:40','2017-09-20 11:04:45' union
select '2017-09-20 08:32:00','2017-09-22 13:28:37' union
select '2017-09-20 13:52:43','2017-09-20 14:14:43' union
select '2017-09-21 12:24:16','2017-09-21 13:28:29' union
select '2017-09-22 12:24:16','2017-09-22 13:28:29' union
select '2017-09-22 12:34:16','2017-09-23 13:28:29' union
select '2017-09-22 12:25:16','2017-09-24 13:28:29' union
select '2017-09-28 09:48:54','2017-09-28 13:39:13' union
select '2017-09-28 14:22:16','2017-09-28 15:52:15' union
select '2017-10-05 12:17:45','2017-10-06 12:35:38' union
select '2017-10-06 16:20:44','2017-10-07 10:11:09' union
select '2017-10-07 20:38:32','2017-10-09 14:42:29' union
select '2017-10-12 18:22:14','2017-10-12 20:52:45'
) a
),a as (
-- group intersecting range
select l.*
from tmp l left join tmp r on l.begAt > r.begAt and r.rg #> l.rg
where r.begAt is null
),
b AS (
SELECT *, lag(endat) OVER (ORDER BY begat) < begat OR NULL AS step
FROM a
)
, c AS (
SELECT *, count(step) OVER (ORDER BY begat) AS grp
FROM b
)
SELECT min(begat), coalesce( max(endat), 'infinity' ) AS range
FROM c
GROUP BY grp
ORDER BY 1
First off: This is a redshift query written in aginity
I need some help with this super long query for a model I am making. Everything works great until my last section, Section 10, where I have to do some unions. I am able to run, but not reference temp-tables Inquiry7 and Inquiry8. If I run the whole query and put a Select * from Inquiry6 at the end, it runs. But if I run the whole query and put a Select * from Inquiry7 at the end, I start getting weird errors like The Connection is broken..
I should also note that I do not have write access for redshift.
Is there anything that stands out to you that I can do to make this more efficient? Due to space limitation, I am only posting the last half of my code.
--------------------------------------------------------------------------------------------------------------------------------------
--Part 6: Appending SQL, SQL Reviewed, and SQO dates
--------------------------------------------------------------------------------------------------------------------------------------
--6.1 SQL Date
sql_date as (
Select a.*, convert_timezone('UTC', 'US/Pacific',b.CreatedDate) as SQL_Date
From part5_final a
Left Outer Join salesforce_opportunity b on a.convertedopportunityid = b.id
),
--6.2 SQL Reviewed Date and Value
sql_reviewed as (
Select a.*,
First_Value (b.CreatedDate) Over (Partition By a.ConvertedContactId, a.MQL_Date ORDER BY b.CreatedDate ASC rows between unbounded preceding and unbounded following) as SQL_reviewed_date,
First_Value (b.NewValue) Over (Partition By a.ConvertedContactId, a.MQL_Date ORDER BY b.CreatedDate ASC rows between unbounded preceding and unbounded following) as SQL_reviewed_value
From sql_date a Left Outer Join (Select a.OpportunityId, convert_timezone('UTC', 'US/Pacific',a.CreatedDate) as CreatedDate, a.NewValue From salesforce_opportunityfieldhistory a Where a.field = 'StageName' and a.OldValue = 'Sales Qualified Lead') b on a.ConvertedOpportunityId = b.OpportunityId
),
--6.3 Discovery Date
Discovery_Date as (
Select a.*,
First_Value (b.CreatedDate) Over (Partition By a.ConvertedContactId, a.MQL_Date ORDER BY b.CreatedDate ASC rows between unbounded preceding and unbounded following) as Discovery_date
From sql_reviewed a Left Outer Join (Select a.OpportunityId, convert_timezone('UTC', 'US/Pacific',a.CreatedDate) as CreatedDate, a.NewValue From salesforce_opportunityfieldhistory a Where a.field = 'StageName' and a.NewValue not in ('Sales Qualified Lead', 'Closed - SQL Rejected')) b on a.ConvertedOpportunityId = b.OpportunityId
),
--6.4 SQO Date
SQO_Date as (
Select a.*,
First_Value (b.CreatedDate) Over (Partition By a.ConvertedContactId, a.MQL_Date ORDER BY b.CreatedDate ASC rows between unbounded preceding and unbounded following) as SQO_date
From Discovery_Date a Left Outer Join (Select a.OpportunityId, convert_timezone('UTC', 'US/Pacific',a.CreatedDate) as CreatedDate, a.NewValue From salesforce_opportunityfieldhistory a Where a.field = 'StageName' and a.NewValue not in ('Sales Qualified Lead', 'Discovery', 'Closed - SQL Rejected', 'Closed - SQL Unqualified')) b on a.ConvertedOpportunityId = b.OpportunityId
),
--6.5 Reorg and dedupe
Part6_Final as (
Select leadid, convertedcontactid, mql_date, mql_reviewed_date, mql_reviewed_value, sal_date, lead_closed_date, lead_closed_value, converteddate, convertedopportunityid, sql_date, sql_reviewed_date, sql_reviewed_value, discovery_date, sqo_date, opportunitycloseddate, opportunitystage
From SQO_Date
Group By leadid, convertedcontactid, mql_date, mql_reviewed_date, mql_reviewed_value, sal_date, lead_closed_date, lead_closed_value, converteddate, convertedopportunityid, sql_date, sql_reviewed_date, sql_reviewed_value, discovery_date, sqo_date, opportunitycloseddate, opportunitystage
),
--102 Seconds
--------------------------------------------------------------------------------------------------------------------------------------
--Part 7: Appending Offer: we are going to build a table of MQL_Status__c changes, then do a date diff to calculate the time from MQL
-- Date, taking the min result as our offer.
--------------------------------------------------------------------------------------------------------------------------------------
--7.1 This grabs the values for the first MQL_Status__c change on the lead history table
offer1 as (
Select a.leadid,
First_Value (convert_timezone('UTC', 'US/Pacific',a.CreatedDate)) Over (Partition By a.LeadId ORDER BY convert_timezone('UTC', 'US/Pacific',a.CreatedDate) ASC rows between unbounded preceding and unbounded following) as CreatedDate,
First_Value (a.OldValue) Over (Partition By a.LeadId ORDER BY convert_timezone('UTC', 'US/Pacific',a.CreatedDate) ASC rows between unbounded preceding and unbounded following) as OldValue,
First_Value (a.NewValue) Over (Partition By a.LeadId ORDER BY convert_timezone('UTC', 'US/Pacific',a.CreatedDate) ASC rows between unbounded preceding and unbounded following) as NewValue
From salesforce_leadhistory a
Where a.Field = 'MQL_Status__c'
),
--7.2 There are instances where a lead was created with MQL_Status__c populated. In order to accurately identify the first
-- change, we will take the Old Value and the lead created date for those
offer2 as (
Select a.leadid, CASE WHEN a.oldvalue <> '' THEN convert_timezone('UTC', 'US/Pacific',b.CreatedDate) ELSE a.CreatedDate END as CreatedDate,
CASE WHEN a.oldvalue <> '' THEN a.oldValue ELSE a.NewValue END as NewValue
From offer1 a
Inner Join salesforce_lead b on a.leadid = b.id
),
--7.3 This grabs the values for the first MQL_Status__c change on the contact history table
offer3 as (
Select a.contactid,
First_Value (convert_timezone('UTC', 'US/Pacific',a.CreatedDate)) Over (Partition By a.ContactId ORDER BY convert_timezone('UTC', 'US/Pacific',a.CreatedDate) ASC rows between unbounded preceding and unbounded following) as CreatedDate,
First_Value (a.OldValue) Over (Partition By a.ContactId ORDER BY convert_timezone('UTC', 'US/Pacific',a.CreatedDate) ASC rows between unbounded preceding and unbounded following) as OldValue,
First_Value (a.NewValue) Over (Partition By a.ContactId ORDER BY convert_timezone('UTC', 'US/Pacific',a.CreatedDate) ASC rows between unbounded preceding and unbounded following) as NewValue
From salesforce_contacthistory a
Where a.Field = 'MQL_Status__c'
),
--7.4 There are instances where a lead was created with MQL_Status__c populated. In order to accurately identify the first
-- change, we will take the Old Value and the lead created date for those
Offer4 as (
Select a.contactid, CASE WHEN a.oldvalue <> '' THEN convert_timezone('UTC', 'US/Pacific',b.CreatedDate) ELSE a.CreatedDate END as CreatedDate,
CASE WHEN a.oldvalue <> '' THEN a.oldValue ELSE a.NewValue END as NewValue
From offer3 a
Inner Join salesforce_contact b on a.contactid = b.id
),
--7.5 combining results from 6.2 and 6.4. Additionally filling in all other contact and lead offer changes. Also, dedupe.
offer5 as (
Select a.PersonId, a.CreatedDate, a.NewValue
From (
Select CASE WHEN b.ConvertedContactId <> '' THEN b.ConvertedContactId else a.LeadId END as PersonId, a.CreatedDate, a.NewValue
From offer2 a
Inner Join salesforce_lead b on a.leadid = b.id
UNION ALL
Select a.ContactId as PersonId, a.CreatedDate, a.NewValue
From offer4 a
UNION ALL
Select a.leadid as PersonId, (convert_timezone('UTC', 'US/Pacific',a.CreatedDate)) as createddate, a.newvalue
From salesforce_leadhistory a
Where a.Field = 'MQL_Status__c'
UNION ALL
Select a.contactid as PersonId, (convert_timezone('UTC', 'US/Pacific',a.CreatedDate)) as createddate, a.newvalue
From salesforce_contacthistory a
Where a.Field = 'MQL_Status__c'
) a
Group By a.PersonId, a.CreatedDate, a.NewValue
),
--7.6 Identification of correct offer per MQL
offer6 as (
Select a.*,
First_Value (b.NewValue) Over (Partition By ISNULL(a.ConvertedContactId, a.LeadId), a.MQL_Date ORDER BY abs(datediff(min,a.mql_date, b.createddate)) ASC rows between unbounded preceding and unbounded following) as Offer
From Part6_Final a
Left Outer Join offer5 b on ISNULL(a.ConvertedContactId, a.LeadId) = b.PersonId
--Group By a.LeadId, a.ConvertedContactId, a.MQL_Date, a.MQL_Reviewed_Date, a.MQL_Reviewed_Value, a.SAL_Date, a.Lead_Closed_Date, a.Lead_Closed_Value, a.ConvertedDate, a.ConvertedOpportunityId, a.SQL_Date, a.SQL_Reviewed_Date, a.SQL_Reviewed_Value, a.Discovery_Date, a.SQO_Date, a.OpportunityClosedDate, a.OpportunityStage
),
--7.7 Dedupe
offer7 as (
Select *
From offer6 a
Group By a.LeadId, a.ConvertedContactId, a.MQL_Date, a.MQL_Reviewed_Date, a.MQL_Reviewed_Value, a.SAL_Date, a.Lead_Closed_Date, a.Lead_Closed_Value, a.ConvertedDate, a.ConvertedOpportunityId, a.SQL_Date, a.SQL_Reviewed_Date, a.SQL_Reviewed_Value, a.Discovery_Date, a.SQO_Date, a.OpportunityClosedDate, a.OpportunityStage, a.Offer
),
--7.8 Cases where a change does not appear in the history table, but the field is populated on the record, this query fills those gaps in.
Part7_Final as (
Select a.LeadId, a.ConvertedContactId, a.MQL_Date, a.MQL_Reviewed_Date, a.MQL_Reviewed_Value, a.SAL_Date, a.Lead_Closed_Date, a.Lead_Closed_Value, a.ConvertedDate, a.ConvertedOpportunityId, a.SQL_Date, a.SQL_Reviewed_Date, a.SQL_Reviewed_Value, a.Discovery_Date, a.SQO_Date, a.OpportunityClosedDate, a.OpportunityStage,
CASE WHEN a.Offer IS NULL AND a.ConvertedContactId is NULL THEN c.MQL_Status__c
WHEN a.Offer IS NULL and a.ConvertedContactId is NOT NULL THEN b.MQL_Status__c
ELSE a.Offer
END as Offer
From Offer7 a
Left Outer Join salesforce_contact b on a.convertedcontactid = b.id
Left Outer Join salesforce_lead c on a.leadid = c.id
),
--160 seconds
--21,506 rows
--------------------------------------------------------------------------------------------------------------------------------------
--Part 8: Lead Source / Marketing Channel
--------------------------------------------------------------------------------------------------------------------------------------
--8.1 Determine previous MQL date so we have a start end end date for determining correct last touch attribution.
mktgchannel1 as (
Select a.*,
ISNULL(Last_Value (b.mql_date) Over (Partition By ISNULL(a.ConvertedContactId, a.LeadId), a.MQL_Date ORDER BY b.mql_date ASC rows between unbounded preceding and unbounded following), '10/1/2016') as PrevMQLDate
From Part7_Final a
Left Outer Join (
Select a.leadid, a.convertedcontactid, a.mql_date
From Part7_Final a
Group By a.leadid, a.convertedcontactid, a.mql_date
) b on ISNULL(a.ConvertedContactId, a.LeadId) = ISNULL(b.ConvertedContactId, b.LeadId) AND a.mql_date > b.mql_date
),
--209 seconds
--8.2 Building marketing channel source table from Bizible data
mktgchannel2 as (
Select CASE WHEN b.bizible2__Contact__c <> '' THEN b.bizible2__Contact__c ELSE b.bizible2__Lead__c END as PersonId, convert_timezone('UTC', 'US/Pacific',a.bizible2__Touchpoint_Date__c) as TouchpointDate, a.Id
From salesforce_bizible2__bizible_touchpoint__c a
Inner Join salesforce_bizible2__bizible_person__c b on a.bizible2__Bizible_Person__c = b.Id
Where a.bizible2__Touchpoint_Position__c like '%Form%'
),
--8.3 Dedupe
mktgchannel3 as (
Select *
From (
Select a.*,
Last_Value (b.id) Over (Partition By ISNULL(a.ConvertedContactId, a.LeadId), a.MQL_Date ORDER BY b.touchpointdate ASC rows between unbounded preceding and unbounded following) as TouchpointId,
Last_Value (b.TouchpointDate) Over (Partition By ISNULL(a.ConvertedContactId, a.LeadId), a.MQL_Date ORDER BY b.touchpointdate ASC rows between unbounded preceding and unbounded following) as TouchpointDate
From mktgchannel1 a
Left Outer Join mktgchannel2 b on ISNULL(a.ConvertedContactId, a.LeadId) = b.PersonId and b.TouchpointDate between a.PrevMQLDate and a.MQL_Date
)a
Group By a.leadid, a.convertedcontactid, a.mql_date, a.mql_reviewed_date, a.mql_reviewed_value, a.sal_date, a.lead_closed_date, a.lead_closed_value, a.converteddate, a.convertedopportunityid, a.sql_date, a.sql_reviewed_date, a.sql_reviewed_value, a.discovery_date, a.sqo_date, a.opportunitycloseddate, a.opportunitystage, a.offer, a.prevmqldate, a.touchpointid, a.touchpointdate
),
--226 seconds
--8.4 Building attribution marketing channel source table from Bizible data
mktgchannel4 as (
Select a.bizible2__opportunity__c as opportunityid, id as attributionid, convert_timezone('UTC', 'US/Pacific',a.bizible2__Touchpoint_Date__c) as attributiondate
From salesforce_bizible2__bizible_attribution_touchpoint__c a
Where bizible2__form_url__c <> ''
),
--8.5 Appending attribution id / date
mktgchannel5 as (
Select a.*,
Last_Value (b.attributionid) Over (Partition By ISNULL(a.ConvertedContactId, a.LeadId), a.MQL_Date ORDER BY b.attributiondate ASC rows between unbounded preceding and unbounded following) as AttributionId,
Last_Value (b.attributionDate) Over (Partition By ISNULL(a.ConvertedContactId, a.LeadId), a.MQL_Date ORDER BY b.attributiondate ASC rows between unbounded preceding and unbounded following) as AttributionDate
From mktgchannel3 a
Left Outer Join mktgchannel4 b on a.convertedopportunityid = b.opportunityid AND attributiondate between a.prevmqldate and mql_date
),
--8.6 Dedupe
mktgchannel6 as (
Select *
From mktgchannel5 a
Group by a.leadid, a.convertedcontactid, a.mql_date, a.mql_reviewed_date, a.mql_reviewed_value, a.sal_date, a.lead_closed_date, a.lead_closed_value, a.converteddate, a.convertedopportunityid, a.sql_date, a.sql_reviewed_date, a.sql_reviewed_value, a.discovery_date, a.sqo_date, a.opportunitycloseddate, a.opportunitystage, a.offer, a.prevmqldate, a.touchpointid, a.touchpointdate, a.attributionid, a.attributiondate
),
--8.7 Determining whether the touchpoint or attribution touchpoint is more relevent to the last touch model
mktgchannel7 as (
Select a.leadid, a.convertedcontactid, a.mql_date, a.mql_reviewed_date, a.mql_reviewed_value, a.sal_date, a.lead_closed_date, a.lead_closed_value, a.converteddate, a.convertedopportunityid, a.sql_date, a.sql_reviewed_date, a.sql_reviewed_value, a.discovery_date, a.sqo_date, a.opportunitycloseddate, a.opportunitystage, a.offer, a.prevmqldate,
CASE WHEN a.touchpointid IS NULL and a.attributionid IS NOT NULL THEN a.attributionid
WHEN a.touchpointid IS NOT NULL and a.attributionid IS NULL THEN a.touchpointid
WHEN a.touchpointid IS NOT NULL and a.attributionid IS NOT NULL AND a.touchpointdate > a.attributiondate THEN a.touchpointid
WHEN a.touchpointid IS NOT NULL and a.attributionid IS NOT NULL and a.touchpointdate < a.attributiondate THEN a.attributionid
WHEN a.touchpointid IS NOT NULL and a.attributionid IS NOT NULL and a.touchpointdate = a.attributiondate THEN a.touchpointid
WHEN a.touchpointid IS NULL and a.attributionid IS NULL THEN NULL
END as TouchpointId,
CASE WHEN a.touchpointid IS NULL and a.attributionid IS NOT NULL THEN a.attributiondate
WHEN a.touchpointid IS NOT NULL and a.attributionid IS NULL THEN a.touchpointdate
WHEN a.touchpointid IS NOT NULL and a.attributionid IS NOT NULL AND a.touchpointdate > a.attributiondate THEN a.touchpointdate
WHEN a.touchpointid IS NOT NULL and a.attributionid IS NOT NULL and a.touchpointdate < a.attributiondate THEN a.attributiondate
WHEN a.touchpointid IS NOT NULL and a.attributionid IS NOT NULL and a.touchpointdate = a.attributiondate THEN a.touchpointdate
WHEN a.touchpointid IS NULL and a.attributionid IS NULL THEN NULL
END as TouchpointDate
From mktgchannel6 a
),
--8.8 Bringing in the marketing channel and form url from bizible data
Part8_Final as (
Select a.leadid, a.convertedcontactid as contactid, ISNULL(a.touchpointdate, a.mql_date) as inquiry_date, a.mql_date, a.mql_reviewed_date, a.mql_reviewed_value, a.sal_date, a.lead_closed_date, a.lead_closed_value, a.converteddate as converted_date, a.convertedopportunityid as opportunityid, a.sql_date, a.sql_reviewed_date, a.sql_reviewed_value, a.discovery_date, a.sqo_date, a.opportunitycloseddate as opportunity_closed_date, a.opportunitystage as opportunity_stage, a.offer, a.prevmqldate as prev_mql_date, a.touchpointid, a.touchpointdate as touchpoint_date,
ISNULL(b.bizible2__marketing_channel__c, c.bizible2__marketing_channel__c) as Lead_Source,
ISNULL(b.bizible2__form_url__c, c.bizible2__form_url__c) as Lead_Source_URL
From mktgchannel7 a
Left Outer Join salesforce_bizible2__bizible_touchpoint__c b on a.touchpointid = b.id
Left Outer Join salesforce_bizible2__bizible_attribution_touchpoint__c c on a.touchpointid = c.id
),
--282 secs
--------------------------------------------------------------------------------------------------------------------------------------
--Part 9: New leads that haven't had a status change yet
--------------------------------------------------------------------------------------------------------------------------------------
--9.1 identification of MQLs that are waiting to be evaluated and do not appear in the model thus far
newlead1 as (
Select a.Id as LeadId, convert_timezone('UTC', 'US/Pacific',a.CreatedDate) as LeadCreatedDate
From salesforce_lead a
Left Outer Join (
Select Distinct LeadId
From Part8_Final a
) b on a.Id = b.LeadId
Where a.Status = 'New'
AND a.MQL_Status__c <> ''
AND a.MQL_Status__c IS NOT NULL
AND b.LeadId IS NULL
),
--9.2 This takes the leads identified in 9.1 and looks for the first offer change
newlead2 as (
Select b.LeadId, b.LeadCreatedDate,
First_Value (convert_timezone('UTC', 'US/Pacific',a.CreatedDate)) Over (Partition By b.LeadId ORDER BY convert_timezone('UTC', 'US/Pacific',a.CreatedDate) ASC rows between unbounded preceding and unbounded following) as CreatedDate,
First_Value (a.OldValue) Over (Partition By b.LeadId ORDER BY convert_timezone('UTC', 'US/Pacific',a.CreatedDate) ASC rows between unbounded preceding and unbounded following) as OldValue,
First_Value (a.NewValue) Over (Partition By b.LeadId ORDER BY convert_timezone('UTC', 'US/Pacific',a.CreatedDate) ASC rows between unbounded preceding and unbounded following) as NewValue
From (Select * From salesforce_leadhistory Where field = 'MQL_Status__c') a
Right Outer Join newlead1 b on a.leadid = b.leadid
),
--9.3 dedupe
newlead3 as (
Select *
From newlead2
Group By leadid, leadcreateddate, createddate, oldvalue, newvalue
),
--9.4 Identifying the appropriate offer and offer date to use
newlead4 as (
Select a.leadid,
CASE WHEN a.oldvalue = '' THEN a.createddate
WHEN a.oldvalue <> '' THEN a.leadcreateddate
WHEN a.oldvalue IS NULL THEN a.leadcreateddate
END as MQL_Date,
CASE WHEN a.oldvalue = '' THEN a.newvalue
WHEN a.oldvalue <> '' THEN a.oldvalue
WHEN a.oldvalue IS NULL THEN b.MQL_Status__c
END as Offer
From newlead3 a
Inner Join salesforce_lead b on a.leadid = b.id
),
--9.5 Indentifying the bizible touchpoint to use
newlead5 as (
Select a.*,
Last_Value (c.id) Over (Partition By a.LeadId, a.MQL_Date ORDER BY convert_timezone('UTC', 'US/Pacific',c.bizible2__touchpoint_date__c) ASC rows between unbounded preceding and unbounded following) as TouchpointId,
Last_Value (convert_timezone('UTC', 'US/Pacific',c.bizible2__touchpoint_date__c)) Over (Partition By a.LeadId, a.MQL_Date ORDER BY convert_timezone('UTC', 'US/Pacific',c.bizible2__touchpoint_date__c) ASC rows between unbounded preceding and unbounded following) as Touchpoint_Date,
Last_Value (bizible2__marketing_channel__c) Over (Partition By a.LeadId, a.MQL_Date ORDER BY convert_timezone('UTC', 'US/Pacific',c.bizible2__touchpoint_date__c) ASC rows between unbounded preceding and unbounded following) as Lead_Source,
Last_Value (bizible2__form_url__c) Over (Partition By a.LeadId, a.MQL_Date ORDER BY convert_timezone('UTC', 'US/Pacific',c.bizible2__touchpoint_date__c) ASC rows between unbounded preceding and unbounded following) as Lead_Source_URL
From newlead4 a
Left Outer Join salesforce_bizible2__bizible_person__c b on a.leadid = b.bizible2__lead__c
Left Outer Join salesforce_bizible2__bizible_touchpoint__c c on b.id = c.bizible2__bizible_person__c AND a.mql_date > convert_timezone('UTC', 'US/Pacific',c.bizible2__touchpoint_date__c)
Where c.bizible2__Touchpoint_Position__c like '%Form%'
),
--9.6 Dedupe
newlead6 as (
Select *
From newlead5
Group By leadid, mql_date, offer, touchpointid, touchpoint_date, lead_source, lead_source_url
),
--9.7 Adding NULL fields to 9.6 so that it can be joined with part8_final
newlead7 as (
Select a.leadid, NULL as contactid, ISNULL(a.touchpoint_date, a.mql_date) as inquiry_date, a.mql_date, NULL::timestamp AT TIME ZONE 'US/Pacific' as mql_reviewed_Date, NULL as mql_reviewed_value, NULL::timestamp AT TIME ZONE 'US/Pacific' as sal_date, NULL::timestamp AT TIME ZONE 'US/Pacific' as lead_closed_date, NULL as lead_closed_value, NULL::timestamp AT TIME ZONE 'US/Pacific' as converted_date, NULL as opportunityid, NULL::timestamp AT TIME ZONE 'US/Pacific' as sql_date, NULL::timestamp AT TIME ZONE 'US/Pacific' as sql_reviewed_Date, NULL as sql_reviewed_value, NULL::timestamp AT TIME ZONE 'US/Pacific' as discovery_date, NULL::timestamp AT TIME ZONE 'US/Pacific' as sqo_date, NULL::timestamp AT TIME ZONE 'US/Pacific' as opportunity_closed_date, NULL as opportunity_stage, a.offer, NULL::timestamp AT TIME ZONE 'US/Pacific' as prev_mql_date, a.touchpointid, a.touchpoint_date, a.lead_source, a.lead_source_url
From newlead6 a
),
--241 secs
--9.8 The Union. This brings together all mqls being worked (part8_final) and all leads that have mql'd but haven't been worked
Part9_Final as (
Select * from newlead7
UNION ALL
Select * from part8_final
),
--790 secs
--------------------------------------------------------------------------------------------------------------------------------------
--Part 10: Inquiries
--------------------------------------------------------------------------------------------------------------------------------------
--10.1 Identification of inquiries already in the model
Inquiry1 as (
Select distinct touchpointid from part9_final where touchpointid IS NOT NULL
),
Inquiry2 as (
Select a.*
From salesforce_bizible2__bizible_touchpoint__c a
Left Outer Join Inquiry1 b on a.Id = b.TouchpointId
Where b.TouchpointId IS NULL
AND a.bizible2__Touchpoint_Position__c like '%Form%'
),
Inquiry3 as (
Select b.Bizible2__Lead__c,
b.Bizible2__Contact__c,
convert_timezone('UTC', 'US/Pacific',a.bizible2__touchpoint_date__c) as Inquiry_Date,
NULL::timestamp AT TIME ZONE 'US/Pacific' as MQL_Date, NULL::timestamp AT TIME ZONE 'US/Pacific' as mql_reviewed_Date, NULL as mql_reviewed_value, NULL::timestamp AT TIME ZONE 'US/Pacific' as sal_date, NULL::timestamp AT TIME ZONE 'US/Pacific' as lead_closed_date, NULL as lead_closed_value, NULL::timestamp AT TIME ZONE 'US/Pacific' as converted_date, NULL as opportunityid, NULL::timestamp AT TIME ZONE 'US/Pacific' as sql_date, NULL::timestamp AT TIME ZONE 'US/Pacific' as sql_reviewed_Date, NULL as sql_reviewed_value, NULL::timestamp AT TIME ZONE 'US/Pacific' as discovery_date, NULL::timestamp AT TIME ZONE 'US/Pacific' as sqo_date, NULL::timestamp AT TIME ZONE 'US/Pacific' as opportunity_closed_date, NULL as opportunity_stage, NULL as offer, NULL::timestamp AT TIME ZONE 'US/Pacific' as prev_mql_date,
a.id as touchpointid,
convert_timezone('UTC', 'US/Pacific',a.bizible2__touchpoint_date__c) as touchpoint_date,
a.bizible2__marketing_channel__c as lead_source,
a.bizible2__form_url__c as lead_source_url
From Inquiry2 a
Inner Join salesforce_bizible2__bizible_person__c b on a.bizible2__bizible_person__c = b.id
),
Inquiry4 as (
Select *
From Inquiry3 a
Where a.Bizible2__Contact__c <> ''
),
Inquiry5 as (
Select b.Id as LeadId,
a.bizible2__contact__c as contactid, a.inquiry_date, a.mql_Date, a.mql_reviewed_date, a.mql_reviewed_value, a.sal_date, a.lead_closed_date, a.lead_closed_value, a.converted_date, a.opportunityid, a.sql_date, a.sql_reviewed_date, a.sql_reviewed_value, a.discovery_date, a.sqo_date, a.opportunity_closed_date, a.opportunity_stage, a.offer, a.prev_mql_date, a.touchpointid, a.touchpoint_date, a.lead_source, a.lead_source_url
From Inquiry4 a
Left Outer Join salesforce_lead b on a.bizible2__contact__c = b.ConvertedContactId
),
Inquiry6 as (
Select *
From Inquiry3 a
Where a.Bizible2__Contact__c = ''
),
--663 secs
Inquiry7 as (
Select * From Inquiry6
UNION ALL
Select * From Inquiry5
),
Inquiry8 as (
Select * from Inquiry7
UNION ALL
Select * from Part9_Final
),
Inquiry9 as (
Select a.*, CASE WHEN b.bizible2__touchpoint_position__c like '%LC%' THEN 1 ELSE NULL END as Net_New_Name
From Inquiry8 a
Left Outer Join (Select * from salesforce_bizible2__bizible_touchpoint__c Where bizible2__Touchpoint_Position__c like '%Form%')b on a.touchpointid = b.id
)
Select * From Inquiry9
Redshift can optimize CTEs but this is an exceptionally large number.
I'd recommend converting your CTE subqueries to temp tables.
Did this query cause cluster restart? Please check with the following query,
SELECT sysdate current_ts, endtime AS restart_ts
FROM stl_utilitytext
WHERE text LIKE '%xen_is_up.sql%'
ORDER BY endtime DESC;
Is your cluster dc1.large? For long query, Redshift needs more memory to rewrite it, it could cause out of memory thus trigger a cluster restart.
As Joe mentioned, please try to use temp table instead CTE.