I have a pipeline in Synapse that reloads data from a number of REST-endpoints. It looks as follows:
What I want to achieve is that the 'CheckAccessTokenValidity' activity checks whether my API-token is still valid. If yes, all is fine, proceed to ForEachDivision activity. If no, refresh tokens, Proceed to ForEachDivision activity.
I thought I had implemented this logic in the screenshot provided. However, this is not the case. My pipeline is now deadlocked because the ForEachDivision apparently expects all three preceding activities to be succesful before running (Which is impossible by design).
How do I implement the logic described above in Azure Synapse?
You can modify your pipeline and accommodate following changes:
Create a variable 'var1' , by default set it to 'False'
After 'CheckAccessTokenValidity' , add set variable activity 'set variable1' pointing to 'var1' and set its value as 'True'
After SetRefreshToken , add another set variable activity 'set variable2' pointing to same 'var1' and set its value as 'True' , so that either of the step 2 or 3 can turn the value of variable to 'True'.
Remove all activities starting ForEach block from this pipeline and cut paste those activities and keep in another new pipeline 'pipeline2'
After 'set variable2' activity, attach an If activity with 'success' and 'Skipped' conditional path . If step 2 gets successful, 'set variable2' would be skipped and it will call the next activity that is 'Step6' . If step 2 fails, and step 3 is called, it will call 'set variable2' and after its success, Step6 will be completed.
In this if activity, write the condition to check the value of var1 . If var1=='True' , then execute the 'pipeline2'
Related
I'm trying to execute Azure Durable Function in ADF.
I have "Get Current Function Status" Activity inside Until activity. Possible value are pending, completed, running and exception. I have variable until_flag which do get value "true" if runTimeStatus get value Completed.
My problem is that even runTimeStatus is completed Until loop never stop. It keeps going. What is wrong?
I'm following tutorial https://www.youtube.com/watch?v=aD3k8k5sdao
I have tried:
#equals(bool(variables('until_flag')), 'true')
and
#bool(variables('until_flag'))
If you look into the official MS docs on UntilActivity
Enter an expression that will be evaluated after all child activities
defined in the Until activity are executed. If the expression
evaluates to false, the Until activity will execute all its child
activities again. When it evaluates to true, the Until activity will
complete. The expression can be a literal string expression, or any
combination of dynamic expressions, functions, system variables, or
outputs from other activities.
So try with this expression to evaluate condition
#equals(variables('until_flag'), 'false')
I have many files in a blob container. However I wanted to run a Stored procedure only IF a certain file (e.g. SRManifest.csv) exists on the blob Container. I used Get metadata and IF Condition on Data Factory. Can you please help me with the dynamic script for this. I tried this #bool(startswith(
activity('Get Metadata1').output.childitems.ItemName,
'SRManifest.csv')). It doesnt work.
Then I thought, what if i used #greaterOREquals(activity('Get Metadata1').output.LastModified,adddays(utcnow(),-2))But this checks the last modified within 2 days of the Bloob not the file exist. Thank you.
Please see below my diagram
I have understood your requirement differently I think.
I wanted to run a Stored procedure only IF a certain file (e.g. SRManifest.csv) exists on the blob Container
1 Change your metadata activity to look for existence of sentinel file (SRManifest.csv)
2 Follow with an IF activity, use this condition:
3 Put your sp in the True part of the IF activity
If you also needed the file list passed to the sp then you'll need the GetMetadata with childitems option inside the IF-True activity
Based on your diagram, since you are looping over all the blob names already, you can add a Boolean variable to the pipeline and set its default value to false:
Inside the ForEach activity, you only want to attempt to set the variable if the value is still false, and if the blob name is found, set it to true. Since Set Variable cannot be self-referential, do this inside the False branch of an If activity:
This will only attempt to process if the value is false (so the file name has not been found yet), and will do nothing if the value is true. Now set the variable based on your file name:
[NOTE: This value can be hard coded, parameterized, or based on a variable]
When you execute the pipeline, you'll see the Set Variable stops attempting once the value is set to true:
In the main pipeline, after the ForEach activity has completed, you can use the variable to set the condition of your final If activity. If the blob is never found, it will still be false, so put the Stored Procedure activity inside the True branch.
How can I spy into my values when I'm on ADF debug mode ?
I want to build a simple pipeline that digs into a storage account table. For each row, enter the value of the second column, use it to create a URL and call a web service.
I saw the output of the Lookup command but how can I saw, for example, the content of each input() into the foreach activity. Can I used the user properties for debugging reason ?
When debugging, I frequently make use of the 'Set Variable' activity. Viewing the output of a 'Set Variable' activity is spying on the value.
You want to see the input to each iteration of your ForEach. Prepend the inner activity with a Set Variable activity. Dynamic content #string(item()) should be enough.
I have initialized a pipeline parameter, lets say a flag, with a value ('true'). Now inside the pipeline I want to update the flag value based on some condition.
So lets say I have a web activity inside the pipeline, I want to update the value of that flag to 'false' if i dont get the response from the web activity.
Any idea how to do that?
You could try #empty(body('web1'))
So if I understand correctly you want to call a Web activity from your pipeline and if there is no response then set a variable?
In this case, could you not just set a low timeout and use a "Set Variable" activity on the Fail condition? So if there was no response it would fail and set the variable
I have a scenario where I would like to skip a component to execute based on the condition and run its consecutive components in Talend.
Is it at all possible?
You have two options available to you for conditionally executing parts of your job.
Where the component that follows your conditional check can be a starting component (if you drop it to the canvas then it should have a green background) then you can use the Run if connector to link it to the previous part of your job like so:
In this example we simply call another tJava component conditionally but this could be any component that is startable.
Where the first tJava component (Set condition boolean) is configured with the following code:
Boolean condition = false;
globalMap.put("condition",condition);
And the two Run if connectors are set as ((Boolean)globalMap.get("condition")) == true and ((Boolean)globalMap.get("condition")) == false respectively.
A better option may be to use the filtering in a tMap or tFilterRow component and this also allows you to link to components that aren't starting components. To do this you would set your job up as below:
In this job I have hard coded some tabular data in a tFixedFlowInput component:
We then use a tMap to filter the flows of data to any following components:
In which we test the value of the boolean condition column of our data. As an illustration I have also applied some simple, conditional transformation to the data where "true" rows have 1000 added to their value and "false" rows have 100 subtracted from their value.
From here you can then carry on the flow of your job as normal, in this case we link to a tSystem component to execute system commands as per your comment.
I've mocked up a job for you:
I have a context variable called: startFrom
It can be accessed with context.startFrom
I've placed a tJava with a few tWarns:
I use 4 context settings:
Default
Normal
Opt
two
So my Job:
Does nothing
Start from "Start"
Start from "Optional_Start"
Start from "RecoverFromHere"
If settings are the following:
context.startFrom.equals("opt")
Recovery and Recovery1 prints out their names using System.out
If I start my job I can select where I want to start it. If I don't select anything: context value is null, it won't do anything.
you can not use prejob as it does not have runif trigger, but you can do like this
prejob -->oncomponentok-->tJava (in here you poupulate you evaluate your condition say as given below)---->RUN IF Trigger - you put your condition here..((String)globalMap.get("var_myCondition")).equals("true") --->component to run in true condition
--->RUN IF Trigger on (tJava) ---((String)globalMap.get("var_myCondition")).equals("false")--->component to run in false condition
in short your job would be like
prejob-->tJava---(RUNIF TRIGGER)------>component/flow to run in true condition
---(RUNIF TRIGGER)------>component/flow to run in false condition
tJava code
String myCondition="false";
globalMap.put("var_myCondition",myCondition);