Redshift: Need efficiency assistance with super long query - amazon-redshift

Question

First off: This is a redshift query written in aginity
I need some help with this super long query for a model I am making. Everything works great until my last section, Section 10, where I have to do some unions. I am able to run, but not reference temp-tables Inquiry7 and Inquiry8. If I run the whole query and put a Select * from Inquiry6 at the end, it runs. But if I run the whole query and put a Select * from Inquiry7 at the end, I start getting weird errors like The Connection is broken..
I should also note that I do not have write access for redshift.
Is there anything that stands out to you that I can do to make this more efficient? Due to space limitation, I am only posting the last half of my code.
--------------------------------------------------------------------------------------------------------------------------------------
--Part 6: Appending SQL, SQL Reviewed, and SQO dates
--------------------------------------------------------------------------------------------------------------------------------------
--6.1 SQL Date
sql_date as (
Select a.*, convert_timezone('UTC', 'US/Pacific',b.CreatedDate) as SQL_Date
From part5_final a
Left Outer Join salesforce_opportunity b on a.convertedopportunityid = b.id
),
--6.2 SQL Reviewed Date and Value
sql_reviewed as (
Select a.*,
First_Value (b.CreatedDate) Over (Partition By a.ConvertedContactId, a.MQL_Date ORDER BY b.CreatedDate ASC rows between unbounded preceding and unbounded following) as SQL_reviewed_date,
First_Value (b.NewValue) Over (Partition By a.ConvertedContactId, a.MQL_Date ORDER BY b.CreatedDate ASC rows between unbounded preceding and unbounded following) as SQL_reviewed_value
From sql_date a Left Outer Join (Select a.OpportunityId, convert_timezone('UTC', 'US/Pacific',a.CreatedDate) as CreatedDate, a.NewValue From salesforce_opportunityfieldhistory a Where a.field = 'StageName' and a.OldValue = 'Sales Qualified Lead') b on a.ConvertedOpportunityId = b.OpportunityId
),
--6.3 Discovery Date
Discovery_Date as (
Select a.*,
First_Value (b.CreatedDate) Over (Partition By a.ConvertedContactId, a.MQL_Date ORDER BY b.CreatedDate ASC rows between unbounded preceding and unbounded following) as Discovery_date
From sql_reviewed a Left Outer Join (Select a.OpportunityId, convert_timezone('UTC', 'US/Pacific',a.CreatedDate) as CreatedDate, a.NewValue From salesforce_opportunityfieldhistory a Where a.field = 'StageName' and a.NewValue not in ('Sales Qualified Lead', 'Closed - SQL Rejected')) b on a.ConvertedOpportunityId = b.OpportunityId
),
--6.4 SQO Date
SQO_Date as (
Select a.*,
First_Value (b.CreatedDate) Over (Partition By a.ConvertedContactId, a.MQL_Date ORDER BY b.CreatedDate ASC rows between unbounded preceding and unbounded following) as SQO_date
From Discovery_Date a Left Outer Join (Select a.OpportunityId, convert_timezone('UTC', 'US/Pacific',a.CreatedDate) as CreatedDate, a.NewValue From salesforce_opportunityfieldhistory a Where a.field = 'StageName' and a.NewValue not in ('Sales Qualified Lead', 'Discovery', 'Closed - SQL Rejected', 'Closed - SQL Unqualified')) b on a.ConvertedOpportunityId = b.OpportunityId
),
--6.5 Reorg and dedupe
Part6_Final as (
Select leadid, convertedcontactid, mql_date, mql_reviewed_date, mql_reviewed_value, sal_date, lead_closed_date, lead_closed_value, converteddate, convertedopportunityid, sql_date, sql_reviewed_date, sql_reviewed_value, discovery_date, sqo_date, opportunitycloseddate, opportunitystage
From SQO_Date
Group By leadid, convertedcontactid, mql_date, mql_reviewed_date, mql_reviewed_value, sal_date, lead_closed_date, lead_closed_value, converteddate, convertedopportunityid, sql_date, sql_reviewed_date, sql_reviewed_value, discovery_date, sqo_date, opportunitycloseddate, opportunitystage
),
--102 Seconds
--------------------------------------------------------------------------------------------------------------------------------------
--Part 7: Appending Offer: we are going to build a table of MQL_Status__c changes, then do a date diff to calculate the time from MQL
-- Date, taking the min result as our offer.
--------------------------------------------------------------------------------------------------------------------------------------
--7.1 This grabs the values for the first MQL_Status__c change on the lead history table
offer1 as (
Select a.leadid,
First_Value (convert_timezone('UTC', 'US/Pacific',a.CreatedDate)) Over (Partition By a.LeadId ORDER BY convert_timezone('UTC', 'US/Pacific',a.CreatedDate) ASC rows between unbounded preceding and unbounded following) as CreatedDate,
First_Value (a.OldValue) Over (Partition By a.LeadId ORDER BY convert_timezone('UTC', 'US/Pacific',a.CreatedDate) ASC rows between unbounded preceding and unbounded following) as OldValue,
First_Value (a.NewValue) Over (Partition By a.LeadId ORDER BY convert_timezone('UTC', 'US/Pacific',a.CreatedDate) ASC rows between unbounded preceding and unbounded following) as NewValue
From salesforce_leadhistory a
Where a.Field = 'MQL_Status__c'
),
--7.2 There are instances where a lead was created with MQL_Status__c populated. In order to accurately identify the first
-- change, we will take the Old Value and the lead created date for those
offer2 as (
Select a.leadid, CASE WHEN a.oldvalue <> '' THEN convert_timezone('UTC', 'US/Pacific',b.CreatedDate) ELSE a.CreatedDate END as CreatedDate,
CASE WHEN a.oldvalue <> '' THEN a.oldValue ELSE a.NewValue END as NewValue
From offer1 a
Inner Join salesforce_lead b on a.leadid = b.id
),
--7.3 This grabs the values for the first MQL_Status__c change on the contact history table
offer3 as (
Select a.contactid,
First_Value (convert_timezone('UTC', 'US/Pacific',a.CreatedDate)) Over (Partition By a.ContactId ORDER BY convert_timezone('UTC', 'US/Pacific',a.CreatedDate) ASC rows between unbounded preceding and unbounded following) as CreatedDate,
First_Value (a.OldValue) Over (Partition By a.ContactId ORDER BY convert_timezone('UTC', 'US/Pacific',a.CreatedDate) ASC rows between unbounded preceding and unbounded following) as OldValue,
First_Value (a.NewValue) Over (Partition By a.ContactId ORDER BY convert_timezone('UTC', 'US/Pacific',a.CreatedDate) ASC rows between unbounded preceding and unbounded following) as NewValue
From salesforce_contacthistory a
Where a.Field = 'MQL_Status__c'
),
--7.4 There are instances where a lead was created with MQL_Status__c populated. In order to accurately identify the first
-- change, we will take the Old Value and the lead created date for those
Offer4 as (
Select a.contactid, CASE WHEN a.oldvalue <> '' THEN convert_timezone('UTC', 'US/Pacific',b.CreatedDate) ELSE a.CreatedDate END as CreatedDate,
CASE WHEN a.oldvalue <> '' THEN a.oldValue ELSE a.NewValue END as NewValue
From offer3 a
Inner Join salesforce_contact b on a.contactid = b.id
),
--7.5 combining results from 6.2 and 6.4. Additionally filling in all other contact and lead offer changes. Also, dedupe.
offer5 as (
Select a.PersonId, a.CreatedDate, a.NewValue
From (
Select CASE WHEN b.ConvertedContactId <> '' THEN b.ConvertedContactId else a.LeadId END as PersonId, a.CreatedDate, a.NewValue
From offer2 a
Inner Join salesforce_lead b on a.leadid = b.id
UNION ALL
Select a.ContactId as PersonId, a.CreatedDate, a.NewValue
From offer4 a
UNION ALL
Select a.leadid as PersonId, (convert_timezone('UTC', 'US/Pacific',a.CreatedDate)) as createddate, a.newvalue
From salesforce_leadhistory a
Where a.Field = 'MQL_Status__c'
UNION ALL
Select a.contactid as PersonId, (convert_timezone('UTC', 'US/Pacific',a.CreatedDate)) as createddate, a.newvalue
From salesforce_contacthistory a
Where a.Field = 'MQL_Status__c'
) a
Group By a.PersonId, a.CreatedDate, a.NewValue
),
--7.6 Identification of correct offer per MQL
offer6 as (
Select a.*,
First_Value (b.NewValue) Over (Partition By ISNULL(a.ConvertedContactId, a.LeadId), a.MQL_Date ORDER BY abs(datediff(min,a.mql_date, b.createddate)) ASC rows between unbounded preceding and unbounded following) as Offer
From Part6_Final a
Left Outer Join offer5 b on ISNULL(a.ConvertedContactId, a.LeadId) = b.PersonId
--Group By a.LeadId, a.ConvertedContactId, a.MQL_Date, a.MQL_Reviewed_Date, a.MQL_Reviewed_Value, a.SAL_Date, a.Lead_Closed_Date, a.Lead_Closed_Value, a.ConvertedDate, a.ConvertedOpportunityId, a.SQL_Date, a.SQL_Reviewed_Date, a.SQL_Reviewed_Value, a.Discovery_Date, a.SQO_Date, a.OpportunityClosedDate, a.OpportunityStage
),
--7.7 Dedupe
offer7 as (
Select *
From offer6 a
Group By a.LeadId, a.ConvertedContactId, a.MQL_Date, a.MQL_Reviewed_Date, a.MQL_Reviewed_Value, a.SAL_Date, a.Lead_Closed_Date, a.Lead_Closed_Value, a.ConvertedDate, a.ConvertedOpportunityId, a.SQL_Date, a.SQL_Reviewed_Date, a.SQL_Reviewed_Value, a.Discovery_Date, a.SQO_Date, a.OpportunityClosedDate, a.OpportunityStage, a.Offer
),
--7.8 Cases where a change does not appear in the history table, but the field is populated on the record, this query fills those gaps in.
Part7_Final as (
Select a.LeadId, a.ConvertedContactId, a.MQL_Date, a.MQL_Reviewed_Date, a.MQL_Reviewed_Value, a.SAL_Date, a.Lead_Closed_Date, a.Lead_Closed_Value, a.ConvertedDate, a.ConvertedOpportunityId, a.SQL_Date, a.SQL_Reviewed_Date, a.SQL_Reviewed_Value, a.Discovery_Date, a.SQO_Date, a.OpportunityClosedDate, a.OpportunityStage,
CASE WHEN a.Offer IS NULL AND a.ConvertedContactId is NULL THEN c.MQL_Status__c
WHEN a.Offer IS NULL and a.ConvertedContactId is NOT NULL THEN b.MQL_Status__c
ELSE a.Offer
END as Offer
From Offer7 a
Left Outer Join salesforce_contact b on a.convertedcontactid = b.id
Left Outer Join salesforce_lead c on a.leadid = c.id
),
--160 seconds
--21,506 rows
--------------------------------------------------------------------------------------------------------------------------------------
--Part 8: Lead Source / Marketing Channel
--------------------------------------------------------------------------------------------------------------------------------------
--8.1 Determine previous MQL date so we have a start end end date for determining correct last touch attribution.
mktgchannel1 as (
Select a.*,
ISNULL(Last_Value (b.mql_date) Over (Partition By ISNULL(a.ConvertedContactId, a.LeadId), a.MQL_Date ORDER BY b.mql_date ASC rows between unbounded preceding and unbounded following), '10/1/2016') as PrevMQLDate
From Part7_Final a
Left Outer Join (
Select a.leadid, a.convertedcontactid, a.mql_date
From Part7_Final a
Group By a.leadid, a.convertedcontactid, a.mql_date
) b on ISNULL(a.ConvertedContactId, a.LeadId) = ISNULL(b.ConvertedContactId, b.LeadId) AND a.mql_date > b.mql_date
),
--209 seconds
--8.2 Building marketing channel source table from Bizible data
mktgchannel2 as (
Select CASE WHEN b.bizible2__Contact__c <> '' THEN b.bizible2__Contact__c ELSE b.bizible2__Lead__c END as PersonId, convert_timezone('UTC', 'US/Pacific',a.bizible2__Touchpoint_Date__c) as TouchpointDate, a.Id
From salesforce_bizible2__bizible_touchpoint__c a
Inner Join salesforce_bizible2__bizible_person__c b on a.bizible2__Bizible_Person__c = b.Id
Where a.bizible2__Touchpoint_Position__c like '%Form%'
),
--8.3 Dedupe
mktgchannel3 as (
Select *
From (
Select a.*,
Last_Value (b.id) Over (Partition By ISNULL(a.ConvertedContactId, a.LeadId), a.MQL_Date ORDER BY b.touchpointdate ASC rows between unbounded preceding and unbounded following) as TouchpointId,
Last_Value (b.TouchpointDate) Over (Partition By ISNULL(a.ConvertedContactId, a.LeadId), a.MQL_Date ORDER BY b.touchpointdate ASC rows between unbounded preceding and unbounded following) as TouchpointDate
From mktgchannel1 a
Left Outer Join mktgchannel2 b on ISNULL(a.ConvertedContactId, a.LeadId) = b.PersonId and b.TouchpointDate between a.PrevMQLDate and a.MQL_Date
)a
Group By a.leadid, a.convertedcontactid, a.mql_date, a.mql_reviewed_date, a.mql_reviewed_value, a.sal_date, a.lead_closed_date, a.lead_closed_value, a.converteddate, a.convertedopportunityid, a.sql_date, a.sql_reviewed_date, a.sql_reviewed_value, a.discovery_date, a.sqo_date, a.opportunitycloseddate, a.opportunitystage, a.offer, a.prevmqldate, a.touchpointid, a.touchpointdate
),
--226 seconds
--8.4 Building attribution marketing channel source table from Bizible data
mktgchannel4 as (
Select a.bizible2__opportunity__c as opportunityid, id as attributionid, convert_timezone('UTC', 'US/Pacific',a.bizible2__Touchpoint_Date__c) as attributiondate
From salesforce_bizible2__bizible_attribution_touchpoint__c a
Where bizible2__form_url__c <> ''
),
--8.5 Appending attribution id / date
mktgchannel5 as (
Select a.*,
Last_Value (b.attributionid) Over (Partition By ISNULL(a.ConvertedContactId, a.LeadId), a.MQL_Date ORDER BY b.attributiondate ASC rows between unbounded preceding and unbounded following) as AttributionId,
Last_Value (b.attributionDate) Over (Partition By ISNULL(a.ConvertedContactId, a.LeadId), a.MQL_Date ORDER BY b.attributiondate ASC rows between unbounded preceding and unbounded following) as AttributionDate
From mktgchannel3 a
Left Outer Join mktgchannel4 b on a.convertedopportunityid = b.opportunityid AND attributiondate between a.prevmqldate and mql_date
),
--8.6 Dedupe
mktgchannel6 as (
Select *
From mktgchannel5 a
Group by a.leadid, a.convertedcontactid, a.mql_date, a.mql_reviewed_date, a.mql_reviewed_value, a.sal_date, a.lead_closed_date, a.lead_closed_value, a.converteddate, a.convertedopportunityid, a.sql_date, a.sql_reviewed_date, a.sql_reviewed_value, a.discovery_date, a.sqo_date, a.opportunitycloseddate, a.opportunitystage, a.offer, a.prevmqldate, a.touchpointid, a.touchpointdate, a.attributionid, a.attributiondate
),
--8.7 Determining whether the touchpoint or attribution touchpoint is more relevent to the last touch model
mktgchannel7 as (
Select a.leadid, a.convertedcontactid, a.mql_date, a.mql_reviewed_date, a.mql_reviewed_value, a.sal_date, a.lead_closed_date, a.lead_closed_value, a.converteddate, a.convertedopportunityid, a.sql_date, a.sql_reviewed_date, a.sql_reviewed_value, a.discovery_date, a.sqo_date, a.opportunitycloseddate, a.opportunitystage, a.offer, a.prevmqldate,
CASE WHEN a.touchpointid IS NULL and a.attributionid IS NOT NULL THEN a.attributionid
WHEN a.touchpointid IS NOT NULL and a.attributionid IS NULL THEN a.touchpointid
WHEN a.touchpointid IS NOT NULL and a.attributionid IS NOT NULL AND a.touchpointdate > a.attributiondate THEN a.touchpointid
WHEN a.touchpointid IS NOT NULL and a.attributionid IS NOT NULL and a.touchpointdate < a.attributiondate THEN a.attributionid
WHEN a.touchpointid IS NOT NULL and a.attributionid IS NOT NULL and a.touchpointdate = a.attributiondate THEN a.touchpointid
WHEN a.touchpointid IS NULL and a.attributionid IS NULL THEN NULL
END as TouchpointId,
CASE WHEN a.touchpointid IS NULL and a.attributionid IS NOT NULL THEN a.attributiondate
WHEN a.touchpointid IS NOT NULL and a.attributionid IS NULL THEN a.touchpointdate
WHEN a.touchpointid IS NOT NULL and a.attributionid IS NOT NULL AND a.touchpointdate > a.attributiondate THEN a.touchpointdate
WHEN a.touchpointid IS NOT NULL and a.attributionid IS NOT NULL and a.touchpointdate < a.attributiondate THEN a.attributiondate
WHEN a.touchpointid IS NOT NULL and a.attributionid IS NOT NULL and a.touchpointdate = a.attributiondate THEN a.touchpointdate
WHEN a.touchpointid IS NULL and a.attributionid IS NULL THEN NULL
END as TouchpointDate
From mktgchannel6 a
),
--8.8 Bringing in the marketing channel and form url from bizible data
Part8_Final as (
Select a.leadid, a.convertedcontactid as contactid, ISNULL(a.touchpointdate, a.mql_date) as inquiry_date, a.mql_date, a.mql_reviewed_date, a.mql_reviewed_value, a.sal_date, a.lead_closed_date, a.lead_closed_value, a.converteddate as converted_date, a.convertedopportunityid as opportunityid, a.sql_date, a.sql_reviewed_date, a.sql_reviewed_value, a.discovery_date, a.sqo_date, a.opportunitycloseddate as opportunity_closed_date, a.opportunitystage as opportunity_stage, a.offer, a.prevmqldate as prev_mql_date, a.touchpointid, a.touchpointdate as touchpoint_date,
ISNULL(b.bizible2__marketing_channel__c, c.bizible2__marketing_channel__c) as Lead_Source,
ISNULL(b.bizible2__form_url__c, c.bizible2__form_url__c) as Lead_Source_URL
From mktgchannel7 a
Left Outer Join salesforce_bizible2__bizible_touchpoint__c b on a.touchpointid = b.id
Left Outer Join salesforce_bizible2__bizible_attribution_touchpoint__c c on a.touchpointid = c.id
),
--282 secs
--------------------------------------------------------------------------------------------------------------------------------------
--Part 9: New leads that haven't had a status change yet
--------------------------------------------------------------------------------------------------------------------------------------
--9.1 identification of MQLs that are waiting to be evaluated and do not appear in the model thus far
newlead1 as (
Select a.Id as LeadId, convert_timezone('UTC', 'US/Pacific',a.CreatedDate) as LeadCreatedDate
From salesforce_lead a
Left Outer Join (
Select Distinct LeadId
From Part8_Final a
) b on a.Id = b.LeadId
Where a.Status = 'New'
AND a.MQL_Status__c <> ''
AND a.MQL_Status__c IS NOT NULL
AND b.LeadId IS NULL
),
--9.2 This takes the leads identified in 9.1 and looks for the first offer change
newlead2 as (
Select b.LeadId, b.LeadCreatedDate,
First_Value (convert_timezone('UTC', 'US/Pacific',a.CreatedDate)) Over (Partition By b.LeadId ORDER BY convert_timezone('UTC', 'US/Pacific',a.CreatedDate) ASC rows between unbounded preceding and unbounded following) as CreatedDate,
First_Value (a.OldValue) Over (Partition By b.LeadId ORDER BY convert_timezone('UTC', 'US/Pacific',a.CreatedDate) ASC rows between unbounded preceding and unbounded following) as OldValue,
First_Value (a.NewValue) Over (Partition By b.LeadId ORDER BY convert_timezone('UTC', 'US/Pacific',a.CreatedDate) ASC rows between unbounded preceding and unbounded following) as NewValue
From (Select * From salesforce_leadhistory Where field = 'MQL_Status__c') a
Right Outer Join newlead1 b on a.leadid = b.leadid
),
--9.3 dedupe
newlead3 as (
Select *
From newlead2
Group By leadid, leadcreateddate, createddate, oldvalue, newvalue
),
--9.4 Identifying the appropriate offer and offer date to use
newlead4 as (
Select a.leadid,
CASE WHEN a.oldvalue = '' THEN a.createddate
WHEN a.oldvalue <> '' THEN a.leadcreateddate
WHEN a.oldvalue IS NULL THEN a.leadcreateddate
END as MQL_Date,
CASE WHEN a.oldvalue = '' THEN a.newvalue
WHEN a.oldvalue <> '' THEN a.oldvalue
WHEN a.oldvalue IS NULL THEN b.MQL_Status__c
END as Offer
From newlead3 a
Inner Join salesforce_lead b on a.leadid = b.id
),
--9.5 Indentifying the bizible touchpoint to use
newlead5 as (
Select a.*,
Last_Value (c.id) Over (Partition By a.LeadId, a.MQL_Date ORDER BY convert_timezone('UTC', 'US/Pacific',c.bizible2__touchpoint_date__c) ASC rows between unbounded preceding and unbounded following) as TouchpointId,
Last_Value (convert_timezone('UTC', 'US/Pacific',c.bizible2__touchpoint_date__c)) Over (Partition By a.LeadId, a.MQL_Date ORDER BY convert_timezone('UTC', 'US/Pacific',c.bizible2__touchpoint_date__c) ASC rows between unbounded preceding and unbounded following) as Touchpoint_Date,
Last_Value (bizible2__marketing_channel__c) Over (Partition By a.LeadId, a.MQL_Date ORDER BY convert_timezone('UTC', 'US/Pacific',c.bizible2__touchpoint_date__c) ASC rows between unbounded preceding and unbounded following) as Lead_Source,
Last_Value (bizible2__form_url__c) Over (Partition By a.LeadId, a.MQL_Date ORDER BY convert_timezone('UTC', 'US/Pacific',c.bizible2__touchpoint_date__c) ASC rows between unbounded preceding and unbounded following) as Lead_Source_URL
From newlead4 a
Left Outer Join salesforce_bizible2__bizible_person__c b on a.leadid = b.bizible2__lead__c
Left Outer Join salesforce_bizible2__bizible_touchpoint__c c on b.id = c.bizible2__bizible_person__c AND a.mql_date > convert_timezone('UTC', 'US/Pacific',c.bizible2__touchpoint_date__c)
Where c.bizible2__Touchpoint_Position__c like '%Form%'
),
--9.6 Dedupe
newlead6 as (
Select *
From newlead5
Group By leadid, mql_date, offer, touchpointid, touchpoint_date, lead_source, lead_source_url
),
--9.7 Adding NULL fields to 9.6 so that it can be joined with part8_final
newlead7 as (
Select a.leadid, NULL as contactid, ISNULL(a.touchpoint_date, a.mql_date) as inquiry_date, a.mql_date, NULL::timestamp AT TIME ZONE 'US/Pacific' as mql_reviewed_Date, NULL as mql_reviewed_value, NULL::timestamp AT TIME ZONE 'US/Pacific' as sal_date, NULL::timestamp AT TIME ZONE 'US/Pacific' as lead_closed_date, NULL as lead_closed_value, NULL::timestamp AT TIME ZONE 'US/Pacific' as converted_date, NULL as opportunityid, NULL::timestamp AT TIME ZONE 'US/Pacific' as sql_date, NULL::timestamp AT TIME ZONE 'US/Pacific' as sql_reviewed_Date, NULL as sql_reviewed_value, NULL::timestamp AT TIME ZONE 'US/Pacific' as discovery_date, NULL::timestamp AT TIME ZONE 'US/Pacific' as sqo_date, NULL::timestamp AT TIME ZONE 'US/Pacific' as opportunity_closed_date, NULL as opportunity_stage, a.offer, NULL::timestamp AT TIME ZONE 'US/Pacific' as prev_mql_date, a.touchpointid, a.touchpoint_date, a.lead_source, a.lead_source_url
From newlead6 a
),
--241 secs
--9.8 The Union. This brings together all mqls being worked (part8_final) and all leads that have mql'd but haven't been worked
Part9_Final as (
Select * from newlead7
UNION ALL
Select * from part8_final
),
--790 secs
--------------------------------------------------------------------------------------------------------------------------------------
--Part 10: Inquiries
--------------------------------------------------------------------------------------------------------------------------------------
--10.1 Identification of inquiries already in the model
Inquiry1 as (
Select distinct touchpointid from part9_final where touchpointid IS NOT NULL
),
Inquiry2 as (
Select a.*
From salesforce_bizible2__bizible_touchpoint__c a
Left Outer Join Inquiry1 b on a.Id = b.TouchpointId
Where b.TouchpointId IS NULL
AND a.bizible2__Touchpoint_Position__c like '%Form%'
),
Inquiry3 as (
Select b.Bizible2__Lead__c,
b.Bizible2__Contact__c,
convert_timezone('UTC', 'US/Pacific',a.bizible2__touchpoint_date__c) as Inquiry_Date,
NULL::timestamp AT TIME ZONE 'US/Pacific' as MQL_Date, NULL::timestamp AT TIME ZONE 'US/Pacific' as mql_reviewed_Date, NULL as mql_reviewed_value, NULL::timestamp AT TIME ZONE 'US/Pacific' as sal_date, NULL::timestamp AT TIME ZONE 'US/Pacific' as lead_closed_date, NULL as lead_closed_value, NULL::timestamp AT TIME ZONE 'US/Pacific' as converted_date, NULL as opportunityid, NULL::timestamp AT TIME ZONE 'US/Pacific' as sql_date, NULL::timestamp AT TIME ZONE 'US/Pacific' as sql_reviewed_Date, NULL as sql_reviewed_value, NULL::timestamp AT TIME ZONE 'US/Pacific' as discovery_date, NULL::timestamp AT TIME ZONE 'US/Pacific' as sqo_date, NULL::timestamp AT TIME ZONE 'US/Pacific' as opportunity_closed_date, NULL as opportunity_stage, NULL as offer, NULL::timestamp AT TIME ZONE 'US/Pacific' as prev_mql_date,
a.id as touchpointid,
convert_timezone('UTC', 'US/Pacific',a.bizible2__touchpoint_date__c) as touchpoint_date,
a.bizible2__marketing_channel__c as lead_source,
a.bizible2__form_url__c as lead_source_url
From Inquiry2 a
Inner Join salesforce_bizible2__bizible_person__c b on a.bizible2__bizible_person__c = b.id
),
Inquiry4 as (
Select *
From Inquiry3 a
Where a.Bizible2__Contact__c <> ''
),
Inquiry5 as (
Select b.Id as LeadId,
a.bizible2__contact__c as contactid, a.inquiry_date, a.mql_Date, a.mql_reviewed_date, a.mql_reviewed_value, a.sal_date, a.lead_closed_date, a.lead_closed_value, a.converted_date, a.opportunityid, a.sql_date, a.sql_reviewed_date, a.sql_reviewed_value, a.discovery_date, a.sqo_date, a.opportunity_closed_date, a.opportunity_stage, a.offer, a.prev_mql_date, a.touchpointid, a.touchpoint_date, a.lead_source, a.lead_source_url
From Inquiry4 a
Left Outer Join salesforce_lead b on a.bizible2__contact__c = b.ConvertedContactId
),
Inquiry6 as (
Select *
From Inquiry3 a
Where a.Bizible2__Contact__c = ''
),
--663 secs
Inquiry7 as (
Select * From Inquiry6
UNION ALL
Select * From Inquiry5
),
Inquiry8 as (
Select * from Inquiry7
UNION ALL
Select * from Part9_Final
),
Inquiry9 as (
Select a.*, CASE WHEN b.bizible2__touchpoint_position__c like '%LC%' THEN 1 ELSE NULL END as Net_New_Name
From Inquiry8 a
Left Outer Join (Select * from salesforce_bizible2__bizible_touchpoint__c Where bizible2__Touchpoint_Position__c like '%Form%')b on a.touchpointid = b.id
)
Select * From Inquiry9

Answer 1

Redshift can optimize CTEs but this is an exceptionally large number.
I'd recommend converting your CTE subqueries to temp tables.

Answer 2

Did this query cause cluster restart? Please check with the following query,
SELECT sysdate current_ts, endtime AS restart_ts
FROM stl_utilitytext
WHERE text LIKE '%xen_is_up.sql%'
ORDER BY endtime DESC;
Is your cluster dc1.large? For long query, Redshift needs more memory to rewrite it, it could cause out of memory thus trigger a cluster restart.
As Joe mentioned, please try to use temp table instead CTE.