I want to use a query in a copy job for my source in an Azure Data Factory pipeline together with a date function - here is the dummy query:
SELECT * FROM public.report_campaign_leaflet WHERE day="{today - 1d}"
I´ve found some documentation about dynamic content and some other stuff but no information on how to use date functions directly in a sql query.
Maybe someone has a hint for me?
Thanks & best,
Michael
Here is the possible solution for your problem.
In your copy activity, at the source side, you choose query in Use Query option, and then in the query box you write an expression
Here is the expression #concat('SELECT * FROM public.report_campaign_leaflet WHERE day=','"',formatDateTime(adddays(utcnow(),-1), 'yyyy-MM-dd'),'"')
formatDateTime function will just format the output of addDays(utcnow(),-1) into yyyy-MM-dd format
Again, you can have a parameter in your pipeline processDate for example, and to set this value from expression in trigger definition, and then just to call that parameter in the query. (suggestion)
You need to replace the double quote (") with two single quotes (''):
#concat('SELECT * FROM public.report_campaign_leaflet WHERE day=','''',formatDateTime(adddays(utcnow(),-1), 'yyyy-MM-dd'),'''')
Related
Forum Newbie...
I want to utilise the ADF Copy function, to carry out incremental table extracts from one Azure DB to another. Every table in the database that I need all have the same 2 relevant fields i.e. date1, date2. For Watermark comparison purposes, I need to use isnull(date1,date2), but unsure how to do this, i.e. I am not sure how I can add this consistent derived value to the Source as an additional field that can perhaps be added via the Query or Stored Procedure Option on the source, to utilise the #item().source.schema and #item().source.table values that have already been generated as parameters..?
You can use the query option in the Copy data activity source and add a new column in the query itself to get the results of isnull(date1,date2) and include the parameter values to get the table name instead of hardcoding them as shown below.
In source, select Query option under Use query and add dynamic content to concat() select statement with parameter values.
#concat('select *, isnull(date1,date2) as final_dt from ',pipeline().parameters.schema,'.',pipeline().parameters.table)
Sink table data output:
I've created a dataflow task in azure data factory and used derived column transformation. One of the source derived column value is '678396' which is extracted through Substring function and datatype "String" by default. I want to convert it into "Integer" because my target column datatype is "Integer".
I've to converted the column in this expression:
ToInteger(Substring(Column_1,1,8))
Please help me with correct expression.
Kind regards,
Rakesh
You don't need to build the expression. If you column data are all like int string "678396", or the output of Substring(Column_1,1,8) are int String
Data Factory can convert the int string to integer data type directly from source to sink. We don't need convert again.
Make sure you set column mapping correctly in sink settings. All things would works well.
Update:
This my csv dataset:
You can choose the Quote character to singe quote, then could solve the problem. See the source data preview in Copy active and Data Flow:
Copy active source:
Data Flow overview:
In data flow, we will get the alert like you said comment, we could ignore it and debug the data flow directly:
HTH.
you don't even need to substruct quotes '', as ToInteger function can convert numbers as string type
I have a Mapping Data Flow, where I want to use a custom SQL query for the Source, but I cannot break it on multiple lines, I get an error stating:
token recognition error at: ''
If I remove the newline and put the whole query on a single line it works, but it looks bearly readable. I would like to preserve the query formatting.
Does anyone have an idea how to do this?
LE the same happens with a simple statement like
select
1
This is how it looks in ADF:
You can enter a query directly into the SQL Query box as multi-lines (see image). You only need to use the expression builder or dynamic content if you are going to use expressions or parameters.
You cannot use CTEs in ADF data flow source queries, that's the issue.
How do I parameterize the where condition in a lookup activity query in azure data factory? I have created a pipeline parameter and tried to pass it to the lookup activity query as given below.
select max(dt) as dt from tab1 where col='#pipeline.parameters.parama1'
I have tried with quotes, without quotes, curly brackets, but still not firing. Any help would be appreciated.
Regards,
Sandeep
Official doc here: https://learn.microsoft.com/en-us/azure/data-factory/control-flow-expression-language-functions
Expressions can also appear inside strings, using a feature called string interpolation where expressions are wrapped in #{ ... }.
Taking this into consideration, this may work for you:
select max(dt) as dt from tab1 where col=#{pipeline().parameters.param}
Hope this helped!
Is there a way in the V2 Copy Activity to operate upon one of the input columns (of type string) with an expression? Before I load rows to the destination, I need to limit the number of characters in the column.
My hope was to simply switch from something like this:
"ColumnMappings": "inColumn: outColumn"
to something like this:
"ColumnMappings": "#substring(inColumn, 1, 300): outColumn"
If anyone can point me to where I can read-up on where & when string expressions can be used, I could use the guidance.
This is the official documentation on expressions and functions: https://learn.microsoft.com/en-us/azure/data-factory/control-flow-expression-language-functions
And this is the documentation on mappings: https://learn.microsoft.com/en-us/azure/data-factory/copy-activity-schema-and-type-mapping
Also remember that if you are using a defined query in the copy activity, you can use sql functions like CAST([fieldName] as varchar(300)) to limit the amount of characters on a particular field.
Hope this helped!
When you don't have a SQL Source, but your destination is a SQL sink, you can use a Stored Procedure to insert your data into the final table. That way, you can define these kinds of transformations in the stored procedure. I don't think the Data Factory can handle these kinds of activities, it is more intended as an orchestrator.
Have a look here:
https://learn.microsoft.com/en-us/azure/data-factory/connector-sql-server#invoke-stored-procedure-from-sql-sink