Using only the REST API, I am able to upload a file to Azure Media Services from my local machine and start an encoding job. Then I need to poll the job for status to see when it is done. But, what I really want is for Azure Media Services to send a request to my callback URL when it is done. Is there way to do this?
Take a look at our Notifications features which supports WebHooks.
https://learn.microsoft.com/en-us/azure/media-services/media-services-dotnet-check-job-progress-with-webhooks
It integrates well with Azure Functions also - if you want to host your callback in Azure Functions and just leverage the WebHook trigger in there.
We have some examples of doing that up here:
https://github.com/Azure-Samples/media-services-dotnet-functions-integration/tree/master/101-notify-webhooks
Related
I want to perform some validation checks in ADF on my input data and any validation failures want to capture into Azure log analytics.
Can someone guide me how to capture the custom logs into log analytics through Azure Data Factory please.
Any example dataflow/pipeline would be very helpful.
Thanks,
Kumar
If I understand correctly you want to be able to get the Azure Monitor logs for ADF and query/store these logs?
Well the good news is most of the information you would want to see is already collected through Azure Monitor.
One of the simple methods to pull the information is to use the Azure Monitor REST API. You can then store the response into a file or table, or you can just query the api for specific pipelines or triggers etc.
Here is a link with example of Authorization and using the Azure Monitor API:
https://learn.microsoft.com/en-us/azure/azure-monitor/essentials/rest-api-walkthrough#authenticating-azure-monitor-requests
This is an example of the HTTP URL using the Azure Monitor REST API to get Activity Run data (Dynamic content syntax):
#{concat('https://management.azure.com/subscriptions/', linkedService().SubscriptionID, '/resourceGroups/', linkedService().ResourceGroupName, '/providers/Microsoft.DataFactory/factories/', linkedService().DataFactoryName, '/pipelineruns/', linkedService().RunID, '/queryActivityruns?api-version=2018-06-01')}
Here are all the different ADF Metrics that can be pulled from Azure Monitor:
https://learn.microsoft.com/en-us/azure/data-factory/monitor-using-azure-monitor#data-factory-metrics
Here is an example of a REST Linked service in ADF that can be used to call the rest api:
You could then create a dataset that will pass all the values to the linked service so that you can call the API, and copy the response to a DB, like here:
This particular example is getting the status of a specific Pipeline RunID, but it can be much more broad than that.
Here is what the request body looks like so that I could filter to just failed pipelines in the last day:
I'm looking into this myself, as far as I can tell you would have to use a REST or HTTP connector to send a POST request to the HTTP Data Collector API in Log Analytics. More details here: https://learn.microsoft.com/en-gb/azure/azure-monitor/logs/data-collector-api
Can the namespace preferences and program preferences be set via REST API calls? If yes, what is the syntax for it?
Generally in Cloud Data Fusion, when we intend to perform the action on GCP side, like create/delete/restart etc. instance, it's feasible to use domestic Google Cloud API, giving the opportunity to interact with a service endpoint via JSON/HTTP calls interface as described in Google Cloud API design document.
Dedicated to Data Fusion you can follow the Cloud Data Fusion REST API reference document, nicely explaining the methods for composing REST API HTTP calls to manage Data Fusion instances, moreover every method description from the documentation contains Google API Explorer sub-panel, to get handy experience building JSON request on a live data.
Said above, I assume your initial question is related more to CDAP REST API, as it includes the methods for pure CDAP instance metadata/namespaces/application configuration.
From the user perspective your workflow might be the following:
Identify the CDAP API endpoint as explained in this guideline;
Compose an HTTP PUT/GET request relevant to Data Fusion
Namespace/Metadata/Preferences/Configuration
object via CDAP RESTful API.
Yes of course! You have two methods.
The first method is creating it from the platform. Follow the steps below:
Open your data fusion instance
Go to System Admin => Configuration => Make HTTP calls
To create a namespace, submit an HTTP PUT request:
PUT /v3/namespaces/<namespace-id>
Link of CDAP: CDAP
The second method is using terraform.
Is there any method to explicitly fail an azure data factory pipeline?
If you would like to fail your pipeline explicitly, one possible way is to have an invalid URL in your web activity which will fail the Web activity, which inturn will result in your pipeline to fail.
There is an existing feature request related to the same requirement in ADF user voice forum suggested by other ADF users. I would recommend you please up-vote and/or comment on this feedback which will help to increase the priority of the feature request implementation.
ADF User voice feedback related to this requirement: https://feedback.azure.com/forums/270578-data-factory/suggestions/38143873-a-new-activity-for-cancelling-the-pipeline-executi
Additional info:
In case if you just want to cancel your pipeline run then you can have a Web activity which calls the below REST API to cancel the pipeline run by using the pipelinerunID (you can get this value by using dynamic expression - #pipeline().RunId)
REST API to Cancel the Pipeline Run: POST https://management.azure.com/subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.DataFactory/factories/{factoryName}/pipelineruns/{runId}/cancel?api-version=2018-06-01
MS Doc related to Rest API: ADF Pipeline Runs - Cancel
Hope this helps.
Just set a variable with #string(div(1, 0))
It depends at which level you would like to stop the operation. For example at MSSQL DB operation, from SP you may RAISERROR above level 18 to raise an exception. As already stated incase of web Activity, request a non existent URL. That would also stop the process
We want to show our developers a user feedback modal dialog (in browser) upon completion of Pull Request in Azure DevOps. I have tried to Google it and have gone through Pull Request Workflow Extensibility documentation but couldn't find anything suggesting it is possible. Is it even possible to do? If yes, can someone point us to resources\code which can help us implement this functionality?
We want to show our developers a user feedback modal dialog (in
browser) upon completion of Pull Request in Azure DevOps.
We usually use extensibility points to extend the UI in Azure Devops Web portal. But if you want to get the user feedback modal dialog (in browser), the extensibility points doesn't support this behavior.
The Web Hooks in Service Hooks is the best thing I can imagine to trigger the completion of Pull Request.
Any time when we complete the PR, it will send a json representation to specified service (The Url which the Http Post will be sent). Then we can combine WebHooks with other service like Azure Functions to do our customization, like Use Azure Functions to create custom branch policies. It seems you need to create own Azure Function or website which shows up a dialog if receiving the Http post from Web Hooks.
I need to make an API call from post trigger of Azure DocumentDB.
I tried calling external APIs - but found that such calls are blocked in DocumentDB. If I host an API on Azure's same account will that API is allowed to be called from the post trigger ? If not what can be the alternatives.
Server-side code runs in a sandboxed environment and cannot make external calls. You can, instead, make the same call from your client after your request is acknowledged by the service (indicating your post-trigger succeeded/failed).
DocumentDB now supports a private preview of server side change feed. Using this model you can pull down all the changes from a DocumentDB collection and execute custom logic to perform desired post processing like invoking web services.
Please email askdocdb at Microsoft dot com if you want to learn more about this pattern.
In server-side scripts you can use all JavaScript functionality and CRUD/Query API for DocumentDB. Other "hosting" API is not exposed. For instance, JavaScript by itself doesn't have built-in support for file system or web API, functionality like needs to be provided by host (IE, cscript, etc). DocumentDB doesn't provide additional API other than CRUD/Query API.