Using dbt_utils.union_relations wrong but don't know how/why? - macros

Question

So I am a new DBT user, super cool stuff, but I am running into an issue with the union_relations macro. I am feeding relations to this function, but the compiled/run query is not finding any columns from the relations.
Here is the code I'm running:
{{dbt_utils.union_relations(relations=[ref('transform_hist_ca_map_stu_obj_assessment'), ref('transform_hist_sc_map_stu_obj_assessment')])}}
)
select *
from conformed_obj_assessment
where student_assessment_identifier is not null
and assessment_identifier is not null
and identification_code is not null
and student_unique_id is not null
and performance_level is not null
And this is the error I receive:
syntax error at or near "from" LINE 1706: from __dbt__CTE__transform_hist_ca_map_stu_obj_a... ^ compiled SQL at target/run/rally_dw/conformed/conformed_student_objective_assessment.sql
Basically the first column is a DBT generated column, and there are supposed to be columns from the relations after that, but for some reason these columns are not being pulled in. I'm wondering if this is because the relations I want to pull from are currently ephemeral, so not materialized, So I'm wondering if that's causing an issue. Here is the compiled SQL, the CTEs return data, but for some reason it's not getting pulled into the last CTE.
create table "dashboarding"."dev_em_conformed"."conformed_student_objective_assessment__dbt_tmp"
as (
with __dbt__CTE__historical_ca_map_stu_obj_assessment as (
with hist_ca_map_stu_obj_assess as (
select * from "dashboarding"."raw_ea"."historical_ca_map_student_obj_assessment"
),
cleaned as (
select distinct
source_org,
assessment_id as assessment_identifier,
student_assessment_identifier,
student_unique_id,
performance_levels as performance_level,
scale_score as score,
assessment_id,
to_date(test_date, 'YYYY-MM-DD') as test_date,
identification_code,
null as parent_objective_assessment_name
from hist_ca_map_stu_obj_assess
)
select * from cleaned
), __dbt__CTE__transform_hist_ca_map_stu_obj_assessment as (
with hist_ca_stu_obj_assess as (
select * from __dbt__CTE__historical_ca_map_stu_obj_assessment
),
final as(
select
null as source_org,
student_assessment_identifier,
assessment_id as assessment_identifier,
identification_code as identification_code,
null as school_year,
student_unique_id,
null as student_grade_level,
null as assessment_grade_level,
NULL as administration_date,
null as administration_end_date,
null as objective_assessment_name,
score,
performance_level,
parent_objective_assessment_name,
null as parent_objective_assessment_id
from hist_ca_stu_obj_assess
)
select * from final
), __dbt__CTE__historical_sc_map_stu_obj_assessment as (
with hist_sc_map_soa as (
select * from "dashboarding"."raw_ea"."historical_sc_map_student_obj_assessment"
),
cleaned as (
select distinct
source_org,
assessment_id as assessment_identifier,
student_assessment_identifier,
student_unique_id,
performance_levels as performance_level,
scale_score as score,
assessment_id,
to_date(test_date, 'YYYY-MM-DD') as test_date,
identification_code,
null as parent_objective_assessment_name
from hist_sc_map_soa
)
select * from cleaned
), __dbt__CTE__transform_hist_sc_map_stu_obj_assessment as (
with hist_sc_stu_obj_assess as (
select * from __dbt__CTE__historical_sc_map_stu_obj_assessment
),
final as(
select
null as source_org,
student_assessment_identifier,
assessment_id as assessment_identifier,
identification_code as identification_code,
null as school_year,
student_unique_id,
null as student_grade_level,
null as assessment_grade_level,
NULL as administration_date,
null as administration_end_date,
null as objective_assessment_name,
score,
performance_level,
parent_objective_assessment_name,
null as parent_objective_assessment_id
from hist_sc_stu_obj_assess
)
select * from final
), conformed_obj_assessment as(
(
select
cast('__dbt__CTE__transform_hist_ca_map_stu_obj_assessment' as
varchar
) as _dbt_source_relation,
---NO MORE COLUMNS???
from __dbt__CTE__transform_hist_ca_map_stu_obj_assessment
)
union all
(
select
cast('__dbt__CTE__transform_hist_sc_map_stu_obj_assessment' as
varchar
) as _dbt_source_relation,
---NO MORE COLUMNS??
from __dbt__CTE__transform_hist_sc_map_stu_obj_assessment
)
)
select *
from conformed_obj_assessment
where student_assessment_identifier is not null
and assessment_identifier is not null
and identification_code is not null
and student_unique_id is not null
and performance_level is not null
);
Any thoughts would be super appreciated thank you!

Answer 1

The union_relations macro relies on knowing what columns are in your relations (tables/views), as stored in the information schema. Since this model is ephemeral, there aren't any records in the information schema, which is why there's SQL like this:
select
cast('__dbt__CTE__transform_hist_ca_map_stu_obj_assessment' as
varchar
) as _dbt_source_relation,
from __dbt__CTE__transform_hist_ca_map_stu_obj_assessment
I noticed that you're using a slightly older version of dbt-utils — while we haven't fixed this issue, we have improved the way this issue is handled (released in v0.5.0).
A newer version of dbt-utils will helpfully tell you something like:
Compilation Error in model test_ephemeral (models/test_ephemeral.sql)
The `union_relations` macro cannot be used with ephemeral models, as it relies on the information schema.
`__dbt__CTE__my_ephemeral` is an ephemeral model. Consider making is a view or table instead.` is an ephemeral model. Consider making is a view or table instead.
So, as the (new) error message suggests — the only way around this is to make your upstream models a view or table.