Is there any method to explicitly fail an azure data factory pipeline?
If you would like to fail your pipeline explicitly, one possible way is to have an invalid URL in your web activity which will fail the Web activity, which inturn will result in your pipeline to fail.
There is an existing feature request related to the same requirement in ADF user voice forum suggested by other ADF users. I would recommend you please up-vote and/or comment on this feedback which will help to increase the priority of the feature request implementation.
ADF User voice feedback related to this requirement: https://feedback.azure.com/forums/270578-data-factory/suggestions/38143873-a-new-activity-for-cancelling-the-pipeline-executi
Additional info:
In case if you just want to cancel your pipeline run then you can have a Web activity which calls the below REST API to cancel the pipeline run by using the pipelinerunID (you can get this value by using dynamic expression - #pipeline().RunId)
REST API to Cancel the Pipeline Run: POST https://management.azure.com/subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.DataFactory/factories/{factoryName}/pipelineruns/{runId}/cancel?api-version=2018-06-01
MS Doc related to Rest API: ADF Pipeline Runs - Cancel
Hope this helps.
Just set a variable with #string(div(1, 0))
It depends at which level you would like to stop the operation. For example at MSSQL DB operation, from SP you may RAISERROR above level 18 to raise an exception. As already stated incase of web Activity, request a non existent URL. That would also stop the process
Related
I want to perform some validation checks in ADF on my input data and any validation failures want to capture into Azure log analytics.
Can someone guide me how to capture the custom logs into log analytics through Azure Data Factory please.
Any example dataflow/pipeline would be very helpful.
Thanks,
Kumar
If I understand correctly you want to be able to get the Azure Monitor logs for ADF and query/store these logs?
Well the good news is most of the information you would want to see is already collected through Azure Monitor.
One of the simple methods to pull the information is to use the Azure Monitor REST API. You can then store the response into a file or table, or you can just query the api for specific pipelines or triggers etc.
Here is a link with example of Authorization and using the Azure Monitor API:
https://learn.microsoft.com/en-us/azure/azure-monitor/essentials/rest-api-walkthrough#authenticating-azure-monitor-requests
This is an example of the HTTP URL using the Azure Monitor REST API to get Activity Run data (Dynamic content syntax):
#{concat('https://management.azure.com/subscriptions/', linkedService().SubscriptionID, '/resourceGroups/', linkedService().ResourceGroupName, '/providers/Microsoft.DataFactory/factories/', linkedService().DataFactoryName, '/pipelineruns/', linkedService().RunID, '/queryActivityruns?api-version=2018-06-01')}
Here are all the different ADF Metrics that can be pulled from Azure Monitor:
https://learn.microsoft.com/en-us/azure/data-factory/monitor-using-azure-monitor#data-factory-metrics
Here is an example of a REST Linked service in ADF that can be used to call the rest api:
You could then create a dataset that will pass all the values to the linked service so that you can call the API, and copy the response to a DB, like here:
This particular example is getting the status of a specific Pipeline RunID, but it can be much more broad than that.
Here is what the request body looks like so that I could filter to just failed pipelines in the last day:
I'm looking into this myself, as far as I can tell you would have to use a REST or HTTP connector to send a POST request to the HTTP Data Collector API in Log Analytics. More details here: https://learn.microsoft.com/en-gb/azure/azure-monitor/logs/data-collector-api
I'm trying to query the Azure Devops REST API to see when a project first came into existence and, perhaps, who created it. I'm using the link below and my results do match the documentation which, unfortunately, doesn't include create date. Is there any other ways to query AzDo REST API to get this data?
https://learn.microsoft.com/en-us/rest/api/azure/devops/core/projects/get%20project%20properties?view=azure-devops-rest-5.1
I am afraid there is no rest api to get the start date of project. The start date property is not included in the returned results of Get Project Api.
This issue has been reported to Microsoft develop team. You can vote up this thread or submit a new feature(Click Suggest a feature and choose Azure Devops).
In Azure Devops under the "Summary" tab associated with a build, there are a list of changes (Git hashes, etc.) associated with the build. I'm trying to figure out how to get these via the rest API.
I found this REST API to get the changes between the two builds.
It works if I have the previous build id in the pipeline, however in the situation I'm interested in, I have the current build id, I need to somehow use the REST API to find the previous id.
Is there a way to do this using the REST API, or a better way to accomplish what I'm trying to do here?
As far as I can see, there's a special REST API call for that. You pass the buildId as a part of the URL and get the collection of changes (Change[]) in response.
Using only the REST API, I am able to upload a file to Azure Media Services from my local machine and start an encoding job. Then I need to poll the job for status to see when it is done. But, what I really want is for Azure Media Services to send a request to my callback URL when it is done. Is there way to do this?
Take a look at our Notifications features which supports WebHooks.
https://learn.microsoft.com/en-us/azure/media-services/media-services-dotnet-check-job-progress-with-webhooks
It integrates well with Azure Functions also - if you want to host your callback in Azure Functions and just leverage the WebHook trigger in there.
We have some examples of doing that up here:
https://github.com/Azure-Samples/media-services-dotnet-functions-integration/tree/master/101-notify-webhooks
I'm trying to Enable/Disable a logic app on Azure using the management APIs. I always get a 403 saying the client: with object id does not have authorization to perform 'Microsoft.Logic/workflows/disable/action' ...
I do use the authentication token in my request and so far have been able to use the API to list all workflows, get trigger histories and in/out messages using the same method.
Any suggestion?
I've seen this issue a lot before if you are calling the http:// instead of https:// - we are looking into automatically redirecting, but for now you will need to make sure you are calling the https:// endpoint with the correct method (in this case a PUT)
EDIT: We discovered the issue was the account being used to perform the enable/disable didn't have contribute permissions.