So I am REALLY new to Airflow, DAGs, the whole nine.
So I have a task where I need to email some data to a list of recipients.
Right now, I have a python function which will send emails, and I was gonna assign this function to a task in my DAG.
However, in the task I was given, I was told that an admin should be able to trigger the DAG, such that additional recipients can receive the data by passing their emails through the airflow configuration json. I am assuming this means, airflow.cfg.
I have been googling for advice, but it seems like most people set up the email section in their airflow.cfg file to just send them an email if there's an error in the dag, not to a list of recipients.
I want to know:
-If I pass a list of email recipients through my airflow.cfg file, can I still assign this email sending process to a task in the DAG?
You should use params, which is a dictionary that can be defined at DAG level parameters and remains accesible in every task. This way you can pass the list of recipients, and then use the EmailOperator to send the emails.
From the UI:
from airflow.operators.email_operator import EmailOperator
email_task = EmailOperator(
to='{{ params.mail_recipients }}',
task_id='email_task',
subject='My subject',
html_content=" Templated HTML content - Today is {{ ds }}"
)
Remember to previously configure SMTP block in Airflow settings. Check this answer for details and examples using params.
Hope that works for you!
Related
Need to send email with Attachment from Matillion Tool
I have checked SNS Message,Send Email component from Matillion but it does not have attachment option.
I have Error log table into Amazon Redshift and I want to retrieve those records and load into one file on daily basis( can be put on S3/SFTP ) this file I want to add as attachment to email and send it to vendor for further analysis.
Found this : https://metlcommunity.matillion.com/s/question/0D54G00007lwu2DSAQ/send-email-with-attachment-of-errors but couldn't help.
https://metlcommunity.matillion.com/s/question/0D54G00007lwu2DSAQ/send-email-with-attachment-of-errors
Based on this message on the Matillion site, it looks like currently there is no built-in component for sending an email with an attachment, but it is on the product roadmap to add one.
There are some hosted downloadable jobs that can send emails - e.g. this and this. It is not clear if they can handle attachments, but at least the Python code inside them might be helpful..
You can do this with the python component in Matillion.
If the target file is on S3, you can use the python boto3 package to download the file to the matillion server.
Then you can use the smtp package in python to create the email and attach the file.
this is example code to attach a file where the filename is in the variable filename_out
filename_out = "/tmp/" + "myfile.logs"
part = MIMEBase('application', "octet-stream")
part.set_payload(open(filename_out, "rb").read())
encoders.encode_base64(part)
part.add_header('Content-Disposition', 'attachment; filename=' + filename_out)
message.attach(part)
I've been trying to test something out, basically looking to do one of the following things:
Change name of recipient in a sent file. I've tried using Outlook Spy (great tool) but every time I changed the recipient in PR_DISPLAY_TO_W it returned the following error:
Could not edit the property: HrSetOneProp returned MAPL_E_COMPUTED
Attaching a file to a sent email file. (I don't know if this one is possible, but would be useful if it was.)
I appreciate any responses.
PR_DISPLAY_TO / CC / BCC are computed properties. The store provider updates them whenever recipients list is modified.
Use the MailItem.Recipients collection to modify the recipients.
MailItem.Attachments.Add.
Background
We're building an application that will process & send emails via Mailgun. These are sometimes one-off messages, initiated by a transaction. Some emails, though, will be sent to 30k+ at once.
Eg, a newsletter to all members.
Considerations
Mailgun offers a Batch Sending option with their API. Using "Recipient Variables", you can include dynamic values that are paired with a particular user.
This Batch Sending functionality is limited, however. You cannot send more than 1,000 recipients per request, which means we have to iterate through a recipient list (on our database) for each set of 1,000. Mailgun provides an example of how this might work, using Python (scroll about 2/3 down).
Question
Are there any advantages to batch sending (ie, sending an email to a group of recipients through a single API call, using recipient variables) as opposed to making our own loop, variable substitutions and individual API calls?
I assume this is more taxing on our server, as it would be processing each message itself, instead of just offloading all that data to Mailgun's server for heavy-lifting on their end. But I also like the flexibility & simplicity of handling that on our end and sending a "fully-rendered" message to Mailgun, one at a time, without having to iterate 1k at a time.
Any thoughts on best practices, or considerations we should take into account?
Stumbled onto this today, and felt it provided a pretty good summary/answer for my original question. I wanted to post this as an answer, in case anybody else has this question and hasn't found this Mailgun post. Straight from the horse's mouth, too. The nutshell version:
For PHP, at least, the SDK has a Mailgun class, with a BatchMessage() method. This actually handles the counting of recipients for you, so you can just queue up as many email addresses as you want (ie, more than 1k) and Mailgun will fire off to the API endpoint as needed. Pretty slick!
Here's their original wording, plus a link to the page.
Sending a message with Mailgun PHP SDK + Batch Message:
Batch Message
In addition to Message Builder, we have Batch Message. This class
allows you to build a message and submit a template message in
batches, up to 1,000 recipients per post. The benefit of using this
class is that the recipients tally is monitored and will automatically
submit the message to the endpoint when you've added the 1,000th
recipient. This means you can build your message and begin iterating
through your database. Forget about sending the message, the SDK will
keep track of posting to the API when necessary.
// First, instantiate the SDK with your API credentials and define your domain.
$mgClient = new Mailgun("key-example");
$domain = "example.com";
// Next, instantiate a Message Builder object from the SDK, pass in your sending domain.
$batchMsg = $mgClient->BatchMessage($domain);
// Define the from address.
$batchMsg->setFromAddress("dwight#example.com",
array("first"=>"Dwight", "last" => "Schrute"));
// Define the subject.
$batchMsg->setSubject("Help!");
// Define the body of the message.
$batchMsg->setTextBody("The printer is on fire!");
// Next, let's add a few recipients to the batch job.
$batchMsg->addToRecipient("pam#example.com",
array("first" => "pam", "last" => "Beesly"));
$batchMsg->addToRecipient("jim#example.com",
array("first" => "Jim", "last" => "Halpert"));
$batchMsg->addToRecipient("andy#example.com",
array("first" => "Andy", "last" => "Bernard"));
// ...etc...etc...
// After 1,000 recipeints,
// Batch Message will automatically post your message to the messages endpoint.
// Call finalize() to send any remaining recipients still in the buffer.
$batchMsg->finalize();
The answer of #cdwyer and #nikoshr is very helpful, but bit legacy. Used methods in the example are deprecated. Here is current usage of lib:
$batchMessage = $this->mailgun->messages()->getBatchMessage('mydomain.com');
$batchMessage->setFromAddress('user#domain.com');
$batchMessage->setReplyToAddress('user2#domain.com');
$batchMessage->setSubject('Contact form | Company');
$batchMessage->setHtmlBody('<html>...</html>');
foreach ($recipients as $recipient) {
$batchMessage->addToRecipient($recipient);
}
$batchMessage->finalize();
More info at documentation.
I have incoming email set up for SharePoint 2010, and am sending forms from InfoPath using that method, with SMTP server set up on the SharePoint Server machine. It seems easier than elevating the permissions of an anonymous user with the InfoPath code-behind option. I also tried a custom method of sending the emails but was unable to find a way to send the form data along with the email message using the InfoPath 2010 code-behind option. So, I found another way, using the advice from SharePoint UK's website.
My problem is that while removing the x-mailer header actually solved my problem regarding whole email messages, it did not allow the attached form data to be sent to the library. The message would be imported and then just disappear if I did not have the "Save original e-mail" setting enabled. Inspecting messages in the drop folder I found they had a content type of "multipart/related" and if I edited them with NotePad++ I could change that to "multipart/mixed" and the attached form data would then be imported as desired.
So I went about trying to modify the VB Script to make this happen BUT now nothing happens with the script at all. Clearly I'm doing something wrong with it and VBScript really isn't my area of expertise. Maybe it's just not possible to edit those fields or maybe I need to attack it from a different angle. How can I get my VBScript to edit the content type for these emails?
CODE:
Set shell = CreateObject("WScript.Shell")
shell.LogEvent 4, "starting mail filter"
if instr(iMsg.Fields("urn:schemas:mailheader:content-type"), "related") > 1 then
Dim logMsg = "Mail Type: " & iMsg.Fields("urn:schemas:mailheader:content-type")
Dim tempType = Replace(iMsg.Fields("urn:schemas:mailheader:content-type"), "related", "mixed")
shell.LogEvent 4, iMsg.Fields("urn:schemas:mailheader:content-type")
iMsg.Fields.Delete("urn:schemas:mailheader:content-type")
iMsg.Fields.Item("urn:schemas:mailheader:content-type") = tempType
iMsg.Fields.Update
shell.LogEvent 4, iMsg.Fields("urn:schemas:mailheader:content-type")
end if
shell.LogEvent 4, "end mail filter"
iMsg.DataSource.Save
EventStatus = 0
I looked into this in greater detail and found that the content-type property of an email message is one of the many that is read-only using this particular vbscript library. So, I would have to find some other way of editing the file, and the work involved is more than what it would take to find an alternate solution that skipped email altogether. Therefore, I am abandoning the plan of using email in this case. If you take something away from this, let it be that there are read-only properties for the mail message object in this script.
I'm using SO_NEW_DOCUMENT_ATT_SEND_API1 function to send an email with an attachment both for a sap user and for an external email. So far, so good. I discovered that the sender of these emails is the sap user that fired the report and the function doesn't have the sender exportation parameter like the old SO_DOCUMENT_SEND_API1 has. The problem is that I need the sender to be an external email. I've been searching and couldn't make this happen. Can anyone help or give me an idea? Thanks
As I have already stated in an answer to another question of yours, stop using the old API and use the new BCS API. The documentation contains an example on how to change the sender address:
DATA: lr_send_request TYPE REF TO cl_bcs,
lr_sender TYPE REF TO cl_cam_address_bcs.
lr_sender = cl_cam_address_bcs=>create_internet_address( 'foo.bar#baz.com' ).
lr_send_request->set_sender( lr_sender ).
Just try to copy the example report BCS_EXAMPLE_1 to your local namespace and exchange the line that sets the sender (line 50 in my release, might be a different one on your system).