copy tables from one database to another using macros in dbt - macros

Question

What I want to achieve is to copy tables from one database to another using macros in dbt
I created a macro which list my table names from a schema as below
{%- macro get_dimension_names(database = target.database, schema = 'dbo', dry_run = True) -%}
{% set get_tables_query %}
select distinct table_name
--- '{{ database | upper }}.' || table_schema ||'.'|| table_name
from {{database}}.information_schema.tables
where table_schema = upper('{{schema}}')
and table_schema ='DBO'
and table_type='BASE TABLE'
{% endset %}
{{ log('\nGenerating dimensions list...\n', info=True) }}
{% set get_dimension_names = run_query(get_tables_query).columns[0].values() %}
-- iterates through the statments generated and executes them if dry_run is false else it just logs the statements
{% for query in get_dimension_names %}
{% if dry_run %}
{{ log(query, info=True) }}
{% else %}
{{ log('Generating tables: ' ~ query, info=True) }}
{% do run_query(query) %}
{% endif %}
{% endfor %}
{%- endmacro -%}
I have another macro which calls the above macro to generate sql select * from the above list to generate sql statements
{%- macro statement_list() -%} ------database = target.database, schema = 'dbo', dry_run = True) -%}
{% set tables = get_dimension_names() %} --- this is from macro which lists table names
{{ log('\nGenerating sql statements list...\n', info=True) }}
{% set query %}
select *
from tables
-- {% if tables -%}
-- where lower(table_name) in {% for t in tables -%} {{ t }} {%- endfor -%}
--- {%- endif -%}
{% endset %}
{{ log('\nGenerating query...\n', info=True) }}
{% set results = run_query(query).columns[0].values() %} -------.columns[0].values() %}
-- iterates through the statments generated and executes them if dry_run is false else it just logs the statements
{% for query in results %}
{% if dry_run %}
{{ log(query, info=True) }}
{% else %}
{{ log('Generating tables: ' ~ query, info=True) }}
{% do run_query(query) %}
{% endif %}
{% endfor %}
{%- endmacro -%}
I am getting the below error
17:58:25 Encountered an error while running operation: Database Error
002003 (42S02): SQL compilation error:
Object 'TABLES' does not exist or not authorized.
I did change my second macro as below
{%- macro generate_dimension_names(database = target.database, schema = 'dbo') -%}
{% set get_tables_list %}
-- query to select table names with column __is_row_current as not all tables in dbo have __is_row_current column
select distinct table_name
from {{database}}.information_schema.columns
where
column_name = '__IS_ROW_CURRENT'
AND table_schema ='DBO'
-- and table_name in ( 'POSTCODE')
{% endset %}
{% set get_tables = run_query(get_tables_list).columns[0].values() %}
{{return(get_tables)}}
{%- endmacro -%}
Even though the macro generates create statements, it is throwing an error as below
Encountered an error while running operation: Database Error
001003 (42000): SQL compilation error:

Answer 1

There are quite a few issues with your code.
The one generating the error:
Your statement_list macro contains the following:
{% set query %}
select *
from tables
...
{% endset %}
Normally you don't nest curlies in jinja, but in a set block, if you want to template a variable, you need to include the curlies:
{% set query %}
select *
from {{ table }}
...
{% endset %}
In the statement_list macro, you're not iterating over the list of tables properly. You need something like this, which will compile to a single string that will have all of your select statements separated by a semicolon:
{% set query %}
{% for table in tables %}
create or replace table destination_schema.{{table}} as (
select *
from {{ table }}
...
)
;
{% endfor %}
{% endset %}
Then once you have query defined, you just run that single statement (gated by dry_run if you want)
You don't return anything from your get_dimension_names macro. You need to add:
{% set dimension_names = run_query(get_tables_query).columns[0].values() %}
{{ return(dimension_names) }}
You need to remove this final block from your get_dimension_names, which seems like it belongs in your other macro:
-- iterates through the statments generated and executes them if dry_run is false else it just logs the statements
{% for query in get_dimension_names %}
{% if dry_run %}
{{ log(query, info=True) }}
{% else %}
{{ log('Generating tables: ' ~ query, info=True) }}
{% do run_query(query) %}
{% endif %}
{% endfor %}
{%- endmacro -%}
Finally, I don't think this is a great approach. If you want to persist a model into a new schema, you can use dbt's custom schema model config. If you want them in two places, you should probably use a view (which you can manage as an additional dbt model with a custom schema config) or if on snowflake, a zero-copy clone.