I do not see lineage for this pipeline copy activity in Purview.
In ADF in regards of the lineage, I'm getting the following errors in all of the for-each-loop activities having dynamic parameter:
Below is the pipeline task configuration passing dynamic parameter for each executed item:
Please advise.
Thanks,
Stan
Related
I am building a pipeline in ADF and I must save in the database the name of the pipeline and the activities that are being used, how can I save this information in the database?
You would get a better answer if you could be more specific on when/where you want to do that, i.e the usage scenario. Without that, my best-guess answer is that you can use PowerShell to obtain that information.
Specifically, you can use the cmdlet Get-AzDataFactoryV2Pipeline, as specified here: https://learn.microsoft.com/en-us/powershell/module/az.datafactory/get-azdatafactoryv2pipeline?view=azps-5.8.0
You can use a python script to parse these details and then load it into the database this can all be done using Azure DevOps pipelines.
I have a pipeline built on Azure data Factory. It has:
a "LookUp" task that has an SQL query that returns a column [CountRecs]. This columns holds a value 0 or more.
an "if" task to check this returned value. I want to fail the pipeline when the value of [CountRecs]>0
Is this possible?
You could probably achieve this by having a Web Activity when your IF Condition is true ([CountRecs]>0) in which the web activity should call the below REST API to cancel the pipeline run by using the pipelinerunID (you can get this value by using dynamic expression - #pipeline().RunId)
Sample Dynamic Expression for Condition: #greater(activity('LookupTableRecordCount').output.firstRow.COUNTRECS, 0)
REST API to Cancel the Pipeline Run: POST https://management.azure.com/subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.DataFactory/factories/{factoryName}/pipelineruns/{runId}/cancel?api-version=2018-06-01
MS Doc related to Rest API: ADF Pipeline Runs - Cancel
One other possible way is to have an invalid URL in your web activity which will fail the Web activity in-turn it will fail the IfCondition activity, which inturn will result in your pipeline to fail.
There is an existing feature request related to the same requirement in ADF user voice forum suggested by other ADF users. I would recommend you please up-vote and/or comment on this feedback which will help to increase the priority of the feature request implementation.
ADF User voice feedback related to this requirement: https://feedback.azure.com/forums/270578-data-factory/suggestions/38143873-a-new-activity-for-cancelling-the-pipeline-executi
Hope this helps.
As a sort-of hack-solution you can create a "Set variable" activity which incurs division by zero if a certain condition is met. I don't like it but it works.
#string(
div(
1
, if(
greater( int(variables('date_diff')), 100 )
, 0
, 1
)
)
)
I have created a pipeline (LogPipeline) that logs other pipelines' status to a database. The idea is that every pipeline will call the LogPipeline at the start and at the end by passing pipeline name and pipeline ID along with other parameters like started/ended/failed.
The last parameter is "Reason" where I want to capture the error message of why a pipeline may have failed.
However, in a given pipeline there are multiple activities that could fail. So I want to direct any and all failed activities to my Execute Pipeline activity and pass the error message.
But on the Execute Pipeline when filling out the parameters, I can only reference an activity by its name, e.g. Reason = #activity['Caller Activity'].Error.Message.
However, since multiple activities are calling this Execute Pipeline, is there a way to say
Reason = #activity[activityThatCalledExecutePipeline].Error.Message?
If my understanding is correct,there are multiple activities call the LogPipeline and you want to get those failed activities' names so that you could know the names inside LogPipeline. Per my knowledge,your requirement is not supported in ADF.
I'm not sure why you have to construct such complex scenario,even though you just want to log the specific fail activities and error messages anyway which is common requirement.There are many monitor ways supported by ADF,please follow below links:
1.https://learn.microsoft.com/en-us/azure/data-factory/monitor-using-azure-monitor#alerts
2.https://learn.microsoft.com/en-us/azure/data-factory/monitor-programmatically
I would suggest you getting an idea of Alerts and Monitor in ADF portal:
And you could set the Target Criteria
It includes:
I have a data factory pipeline that has an "If Condition" Activity and I want the pipeline to fail on a certain condition. What is the best way to achieve this? There is no fail activity..
They've just added a "Fail Activity" in Data Factory and Synapse Analytics:
https://learn.microsoft.com/en-us/azure/data-factory/control-flow-fail-activity
It's found in the "General" category in "Activities" in the Data Factory Studio.
Update September 2021: There is now a Fail activity in ADF.
First of all, please vote for this feedback. This is a common need.
There are several workarounds such as a Web Activity to throw an error by trying to connect to https://ThrowAnError or a Stored Procedure activity which executes a raiserror function in Azure SQL Database. But that’s as close as we can get to solving your problem currently.
Is there a way by which we can get the list of skipped tasks from the build?
For example, I have 2 tasks that run conditionally only based on external factors. So how can I see, whether the tasks were skipped or actually ran from Azure DevOps REST API?
I need to trigger another build conditionally based on the above factor.
Any help will be appreciated!
You should look into the Build Timeline REST API. If you issue the following GET request:
GET https://dev.azure.com/{organization}/{project}/_apis/build/builds/{buildId}/timeline
where buildId is the ID of the build you're examining, it returns the Timeline object. It is essentially a collection of TimelineRecord objects, each representing an entry in a build's timeline. You should filter out this collection to leave only those, where "type": "Task" and "result": "skipped".