I'm new to work with data factory. I want to know how can I fetch ForEach activity's inner activity result from outer Until activity. I have my pipeline as below.
My ForEach1 activity looks like this,
My until activity looks like this.
I want to fetch Azure Function activity's output from Web activity, but pipeline is failing with the error "ErrorCode=InvalidTemplate, ErrorMessage=cannot reference action 'Log_extraction_func_copy1'. The action 'Log_extraction_func_copy1' is nested in a foreach scope of multiple levels. Referencing repetition actions from outside the scope is supported only when there are no multiple levels of nesting". Is there any way out for this? Sorry, I'm not good at English.Hope my question is clear.
You can assign the output of function in a variable using set variable and access the variable value from web activity.
Related
I wanted the result of a query on the databricks notebook to be the success or failure condition of the pipeline to reprocess for example the "copy data" in the azure data factory.
For example:
If x = 1, terminate the pipeline, if not, reprocess (with a limit of 3 attempts).
What's the best way to do this?
You can do this with the help of if and until activities in ADF.
Please go through the sample demonstration below:
This is the sample Notebook code from databricks.
#your code
x=1
dbutils.notebook.exit(x)
In ADF, first create an array variable which will be used in the until activity.
This array length is used for n number of times re-process.
Next give your databricks notebook.
Now use an if activity and give the below expression in that.
#equals(activity('Notebook1').output.runOutput,1)
If this is true, our pipeline has to be terminated. So, add a fail activity in the True activities of if.
Here you can give any message that you want.
Leave the Fail activities of if as it is.
Now, use an until activity and give the success of if to it.
Inside Until activities we can give any activity. if you want to reprocess another pipeline then you can give execute pipeline also. Here I have given a copy activity.
After copy activity use an append variable activity and give the array variable that we defined in the first and append with any single value that you want.
Now in the until expression give the below.
#equals(length(variables('iter')),4)
So, the activities inside until will reprocess 3 times if x!=1.
If x=1 in notebook, pipeline failed and terminated at if.
if x!=1 in Notebook, until reprocessed copy activity 3 times.
I have 20 file formats and 1 Data Flow activity that maps to each one of them. Based on the file name, I know which data flow activity to execute. Is the only way to handle this through a "Switch" activity? Is there another way? e.g. can I parameterize the data flow to execute by a variable name?:
Unfortunately , there is no option to run one out of list of dataflows based on input condition.
To perform data migration and transformation for multiple tables, you can use same dataflow and parameterize the dataflow by providing the table names either during the runtime or use a control table to hold all the tablenames and inside foreach , call the dataflow activity. In the sink settings, use merge schema option.
I have a switch statement that looks at a variable value and based on that determines which data flow to execute. The problem with this is, I need to update the switch statement every time I add a new ID/dataflow.
Is there an alternative design to this? What if my dataflows had the same name as the variable value - would it be possible to parameterize the name of the dataflow to execute?
e.g. variable value = "1" execute data flow with name "1_dataflow", 2 execute "2_dataflow" etc. How would I accomplish this?
Yes, you can parameterize the values in any activity in Azure Data Factory and make the pipeline dynamic instead of giving hard-coded values.
You can use parameters to pass external values into pipelines,
datasets, linked services, and data flows. Once the parameter has been
passed into the resource, it cannot be changed. By parameterizing
resources, you can reuse them with different values each time.
You can also use Set Variable activity in to define and variable and set value to it which you can use in Switch activity and also can change it later.
Refer: How to use parameters, expressions and functions in Azure Data Factory, Set Variable Activity in Azure Data Factory and Azure Synapse Analytics
I'm doing a simple data flow pipeline between 2 cosmos dbs. The pipeline starts with the dataflow, which grabs the pipeline variable "LastPipelineStartTime" and passes that parameter to the dataflow for the query to use in order to get all new data where c._ts >= "LastPipelineStartTime". Then, on data flow success, updates the variable via Set Variable to the pipeline.TriggerTime(). Essentially so I'm always grabbing new data between pipeline runs.
My question is: it looks like the variable during each debug run reverts back to its Default Value of 0, and instead grabs everything each time. Am I misunderstanding or using pipeline variables wrong? Thanks!
As i know,the variable which is set in the Set Variable Activity has it's own life cycle: during current execution of pipeline.Any change of variable can't persist until next execution stage.
To implement your needs,pls refer to my workarounds as below:
1.If you execute ADF pipeline in the schedule,you could just pass the schedule time as parameter into it to make sure you grab new data.
2.If the frequency is random,persist the trigger time into other residence(e.g. simple file in the blob storage),before data flow activity,use LookUp Activity to grab that time from blob storage file.
I have a simple custom activity with a private member variable (integer).
When i put it inside a sequence activity which is inside a while activity and start iterating i have a problem:
My member variable is zeroed in each iteration even though i increment it by one every time the activity is executed.
What am i doing wrong?
Thanks,
Adi Barda
Without seeing the code it is hard to say, but when you are working inside of a While activity you have to be careful how you modify state on your child activities. The While activity spawns multiple execution execution contexts and will clone your activity from a template (in other words - you aren't executing the same activity multiple times, the workflow creates multiple instances of your custom activity). See: http://blogs.msdn.com/advancedworkflow/archive/2006/03/21/557121.aspx and http://msdn.microsoft.com/en-us/magazine/cc163414.aspx