We are building a data extraction DAG. The source we are extracting from will email us upon new data being available. It will also send the date for which there is new data for. How do we:
(a) - Trigger a DAG upon receiving such email?
(b) - Access email content inside the DAG (e.g. the date listed in email content)
I don't think that there is something in Airflow to do that, but you can use an external service to receive the email, process it and send an HTTP request to Airflow API to trigger a run. In this case you can provide the content of the email (subject, body, sender, ...) as dag run configuration, and your dag tasks can access them from {{ dag_run.conf }}.
To do that, you need an email server with hooks (https://www.cloudmailin.com/, https://mailnuggets.com/), you can also deploy your own server (here is an example for GCP App Engine). Then you need to activate Airflow API (doc). Finally in the hook, you configure your hook to send an POST HTTP request as explained in this page.
Related
We have a few pipelines that are taking backups on schedule. We would like to receive notifications on Discord in case if specific Pipelines were to fail.
There is a way to receive an email notification and an official application to integrate Slack into Azure DevOps.
But is there a good way to do that for Discord?
You can use the Discord Webhook extension that give you a task to send messages to Discord during your pipeline.
When creating a webhook for a channel in a Discord server, you will be given a url which contains the Channel ID and the Secret Key for that webhook in this format: https://discordapp.com/api/webhooks/{channelId}/{webhookKey}.
You will not need to provide the url to the task, but rather just the Channel ID and Secret Key. (The first section in the link above, Making a Webhook, is all that needs to be followed to be able to use this task).
More info you can find here.
After you know which values put in the task you can dd it to your pipeline and configure it to run only if the pipeline failed with the condition failed().
We have on-prem machines, and a bunch of them when offline, and we didn't know for several days because the pipelines just kept waiting for the machines. How can we set up an email when they go offline?
You could leverage something like the Azure Logic App, call the Azure REST API - Agents - List via the Http Action to check the TaskAgentStatus, and send some notification like send an email when the status is offline.
Overview: I try to trigger an Azure DevOps 2020 Pipeline (YAML) in the event that a WorkItem of a specific type is created. So my idea I had was to create a connection between "WorkItem" -> Service Hook of Type "Web Hook" -> Service Connection of Type "Incoming Webhook" -> resource:webhook triggers the Pipeline in YAML.
My problem now is, that I could create the serviceendpoint by GUI or by API, but the documentation and the created endpoint does not make sense for me. I am running my server on-premise and the endpoint I got looks like this:
{"count":1,"value":[{"data":{},"id":"1babbef7-1edb-4b01-bf18-b6e3c309caae","name":"TestIncomingWebhook","type":"incomingwebhook","url":"https://dev.azure.com", ...
The url does not make sense to me.
I can alter the url if I use the REST API to create the endpoint by using this format
POST https://{instance}/{collection}/{project}/_apis/serviceendpoint/endpoints
as described in Microsoft Azure API Documentation.
Questions:
Why is the url that the service connection gets pointed to https://dev.azure.com while I am running on-premise from a server on a totally different url? Would that be the url I have to POST to trigger the ServiceConnection?
What must be the correct URL-Format for my local Server? Where should I point that URL to?
How can I trigger the Service Connection from curl? Do I have to use GET with a lot of parameters or POST with a JSON payload?
I would appreciate if someone could help me out on this. Perhaps someone even got a full solution on how to trigger pipelines on workitem events. I searched a lot but could not find a solution, yet.
Thanks, Peter.
I created the WebHook and ServiceConnection from the GUI like this:
WebHook Configuration Screenshot
ServiceHook Configuration Screenshot
My real problem is, that I got no feedback what the correct ServiceHook URL is. I have no idea where to point my WebHook.
Addition:
The correct URL for accessing the Incoming Webhook Service Endpoint in Azure DevOps is https://{instance}/{Collection name}/_apis/public/distributedtask/webhooks/{Service Connection name}?api-version=6.0-preview
This should be related to your machine settings, you could share the detail steps here
Since you are using Azure DevOps Server 2020, the url format for your local Server should be https://{instance}. You could open Azure DevOps Server Administration Console to check it.
It will trigger the webhook via the event you selected.
You could refer to this ticket for more details.
In addition, you could try this Logic Apps then create flow to trigger build when specific type is created
Update1
Create Service connection Incoming WebHook, eg. Web Hooks name is HookTests and Service connection name is TestHook
Create Web Hooks and enter the URL https://{instance}/{Collection name}/_apis/public/distributedtask/webhooks/{Web Hook name eg. HookTests}?api-version=6.0-preview
Create YAML build and add service connection resources in the yaml pipeline see below:
resources:
webhooks:
- webhook: MyWebhookTrigger
connection: {Service connection name, eg.TestHook}
Above steps actually do below things:
New workitem is created-->automatically Trigger Azure pipeline.
In addition, we could also add task power shell and call REST API to trigger another pipeline.
Completely new to Webhook concept and Rundeck. I have a job in rundeck where it checkes health of some servers, code being in python.
Fetch 200 Ok status after running Curl Command and using that status write a condition using python in RUNDECK
i want to use webhook to provide update via email/slack channel to 5-6 users
Created a webhook, i selected a job which it should invoke, but i didnt understand what options to be entered options section[Job Option arguments, in the form -opt1 value -opt2 "other value"]
when i click on the webhook URL it gives 404 error found.
this might be the very basic questions. sorry kindly help
i want to use webhook to provide update via email/slack channel to 5-6 users
Webhooks are used to enable third-party applications to trigger jobs. If you just want to send notifications of job status when a job is run, you don't need to use a webhook.
When you configure the job there's a "Notifications" tab. You can select when and how to send notifications. For slack specifically, there is a notification plugin you can install.
To pass options to webhook you need to pass it in this way. Here you have a good example to integrate it with python code.
UPDATE: Anyway, this answer is the way to achieve your goal.
Using the Azure portal, I can click the Run Once button on a Scheduler job to execute it. Is that functionality available via a REST call from my app? The scenario is that periodically a job is run to check for changes in data. The first time the app starts up, I would like to execute that job once as not to have to wait for the scheduled time.
What is the call to kick off a scheduled job? Would I have to duplicate code in both the scheduler job and custom API and then call the custom API from my code?
You can execute a scheduler job by sending a POST request to /jobs/<jobName> (no request body required). But you need to pass the master key of your mobile service (in the x-zumo-master HTTP header), so you should only do that from a location that is not visible to your users - you don't want anyone getting a hold of your master key, as this would open up your service for all kinds of attacks.