I want to fail the task if task duration exceeds a certain time for which I configured execution_timeout in every task of the dag. However, the task is failing, but I am not getting email notification of it. Does anyone know the reason? Any help would be appreciated.
default_args = {
'owner': 'airflow',
'depends_on_past': False,
'start_date': datetime.today() - timedelta(days = 2),
'email':['email#company.com'],
'email_on_failure': True
}
[2022-10-27 03:06:42,587] {logging_mixin.py:112} INFO - [2022-10-27 03:06:42,585] {timeout.py:42} ERROR - Process timed out, PID: 15847
[2022-10-27 03:06:42,588] {bash_operator.py:140} INFO - Sending SIGTERM signal to bash process group
[2022-10-27 03:06:42,602] {taskinstance.py:1145} ERROR - Timeout, PID: 15847
Traceback (most recent call last):
File "/home/airflow_user/.local/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 978, in _run_raw_task
result = task_copy.execute(context=context)
File "/home/airflow_user/.local/lib/python3.7/site-packages/airflow/operators/bash_operator.py", line 124, in execute
for line in iter(self.sub_process.stdout.readline, b''):
File "/home/airflow_user/.local/lib/python3.7/site-packages/airflow/utils/timeout.py", line 43, in handle_timeout
raise AirflowTaskTimeout(self.error_message)
airflow.exceptions.AirflowTaskTimeout: Timeout, PID: 15847
I can see the task turning to red. But no email is sent even with the email_on_failure config.
You have not configured your smtp in airflow.cfg, you need configure it:
[smtp]
smtp_host = #your smtp host
smtp_starttls = True
smtp_ssl = False
# Example: smtp_user = airflow
smtp_user = # your email to send
# Example: smtp_password = airflow
smtp_password = # App password
smtp_port = # your smtp port
smtp_mail_from = # your email to send
smtp_timeout = 30
smtp_retry_limit = 5
Example, if you are using gmail:
[smtp]
smtp_host = smtp.gmail.com
smtp_starttls = True
smtp_ssl = False
# Example: smtp_user = airflow
smtp_user = example#gmail.com
# Example: smtp_password = airflow
smtp_password = #app password
smtp_port = 25
smtp_mail_from = example#gmail.com
smtp_timeout = 30
smtp_retry_limit = 5
Note: you can take your app password in here: https://support.google.com/mail/answer/185833?hl=en
Gmail port can be 25, 465 or 587
Related
Im trying to use python to build an email sender.
Ive been getting an SSL certificate error, though I downloaded it.
Im also getting an SMTP error, which i think shouldn't be happening.
I've tried to fix this error but the solution I found was that of Mac and not windows.
The error:
with smtplib.SMTP_SSL( 'smtp.gmail.com' , 465, context=context) as smtp:
File "C:\Program Files\Inkscape\lib\python3.10\smtplib.py", line 1050, in __init__
SMTP.__init__(self, host, port, local_hostname, timeout,
File "C:\Program Files\Inkscape\lib\python3.10\smtplib.py", line 255, in __init__
(code, msg) = self.connect(host, port)
File "C:\Program Files\Inkscape\lib\python3.10\smtplib.py", line 341, in connect
self.sock = self._get_socket(host, port, self.timeout)
File "C:\Program Files\Inkscape\lib\python3.10\smtplib.py", line 1057, in _get_socket
new_socket = self.context.wrap_socket(new_socket,
File "C:\Program Files\Inkscape\lib\python3.10\ssl.py", line 513, in wrap_socket
return self.sslsocket_class._create(
File "C:\Program Files\Inkscape\lib\python3.10\ssl.py", line 1071, in _create
self.do_handshake()
File "C:\Program Files\Inkscape\lib\python3.10\ssl.py", line 1342, in do_handshake
self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:997)
The code:
from email.message import EmailMessage
from emailApp import password
import ssl
try:
_create_unverified_https_context = ssl._create_unverified_context
except AttributeError:
# Legacy Python that doesn't verify HTTPS certificates by default
pass
else:
# Handle target environment that doesn't support HTTPS verification
ssl._create_default_https_context = _create_unverified_https_context
import smtplib
email_sender = 'example2#mail.com'
email_password = password
email_receiver = 'example#gmail.com'
subject = "msg"
body = """
Hello
"""
em = EmailMessage()
em['From'] = email_sender
em['To'] = email_receiver
em['Subject'] = subject
em.set_content(body)
context = ssl.create_default_context()
context = ssl.create_default_context()
with smtplib.SMTP_SSL( 'smtp.gmail.com' , 465, context=context) as smtp:
smtp.login(email_sender, email_password)
smtp.sendmail(email_sender, email_receiver, em.as_string())
I want to use SMTP email in GCP cloud composer. I have followed the GCP documentation and did the following -
Created a secret with my SMTP password (tested with secret name as airflow-variables-smtp-password & airflow-config-smtp-password)
Provided secret accessor role to my svc account
Modified email backend and all configs -
[email]
email_backend = airflow.utils.email.send_email_smtp
[smtp]
smtp_port = 587
smtp_user = ****
smtp_mail_from = ****
smtp_password_secret = smtp-password
smtp_host = smtp.office365.com
Changed secret backend -
[secrets]
backend = airflow.providers.google.cloud.secrets.secret_manager.CloudSecretManagerBackend
Below is my email dag which is subject to fail (tested with email operator as well) -
from airflow import DAG
from airflow.operators.python_operator import PythonOperator
def print_hello():
pppp
return 'Hello Wolrd'
args = {
'owner': '***',
'depends_on_past': False,
'start_date': "2021-09-14",
'email_on_failure': True
}
with DAG(
dag_id='mail_testing',
default_args=args,
) as dag:
python_task = PythonOperator(
task_id='email_task',
python_callable=print_hello,
email_on_failure=True,
email= ['***'],
dag=dag
)
When running the dag, I am seeing it is looking for sendgrid to send the email.
[2021-09-29 17:42:20,958] {logging_mixin.py:104} WARNING - /opt/python3.8/lib/python3.8/site-packages/airflow/providers/sendgrid/utils/emailer.py:122 PendingDeprecationWarning: Fetching Sendgrid credentials from environment variables will be deprecated in a future release. Please set credentials using a connection instead.
[2021-09-29 17:42:21,081] {taskinstance.py:1503} ERROR - Task failed with exception
Traceback (most recent call last):
File "/opt/python3.8/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1158, in _run_raw_task
self._prepare_and_execute_task_with_callbacks(context, task)
File "/opt/python3.8/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1333, in _prepare_and_execute_task_with_callbacks
result = self._execute_task(context, task_copy)
File "/opt/python3.8/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1363, in _execute_task
result = task_copy.execute(context=context)
File "/opt/python3.8/lib/python3.8/site-packages/airflow/operators/email.py", line 79, in execute
send_email(
File "/opt/python3.8/lib/python3.8/site-packages/airflow/utils/email.py", line 55, in send_email
return backend(
File "/opt/python3.8/lib/python3.8/site-packages/airflow/providers/sendgrid/utils/emailer.py", line 122, in send_email
_post_sendgrid_mail(mail.get(), conn_id)
File "/opt/python3.8/lib/python3.8/site-packages/airflow/providers/sendgrid/utils/emailer.py", line 141, in _post_sendgrid_mail
response = sendgrid_client.client.mail.send.post(request_body=mail_data)
File "/opt/python3.8/lib/python3.8/site-packages/python_http_client/client.py", line 277, in http_request
self._make_request(opener, request, timeout=timeout)
File "/opt/python3.8/lib/python3.8/site-packages/python_http_client/client.py", line 184, in _make_request
raise exc
python_http_client.exceptions.UnauthorizedError: HTTP Error 401: Unauthorized
[2021-09-29 17:42:21,084] {taskinstance.py:1546} INFO - Marking task as UP_FOR_RETRY.
What am I doing wrong? Is there anything I am missing!
PendingDeprecationWarning: Fetching Sendgrid credentials from environment variables will be deprecated in a future release. Please set credentials using a connection instead.
These docs describe configuring Sendgrid credentials in a connection. I would follow them, as they include steps for troubleshooting.
The other possibility is that it's an issue with your Sendgrid credentials. You can test your sendGrid credentials are valid by sending a test-email via a curl request.
I am searching for days to find out how can I set Office365 SMTP server in my VB6 application. My code is working properly with port 465 and other mail servers.
BUT it is not working with port 587 and smtp.office365.com
Is there any way I could have TLS via 587 in VB6?
Thanks
With this code, I get to send mail using CDO to Office365.
Private Message As CDO.Message
Private Attachment, Expression, Matches, FilenameMatch, i
Sub enviar_mail()
Set Message = New CDO.Message
Message.Subject = "Test Mail"
Message.From = "YourEmail#yourdomain.com"
Message.To = ""
Message.CC = ""
Message.TextBody = "my text body here"
Dim Configuration
Set Configuration = CreateObject("CDO.Configuration")
Configuration.Load -1 ' CDO Source Defaults
Configuration.Fields.Item("http://schemas.microsoft.com/cdo/configuration/sendusing") = 2
Configuration.Fields.Item("http://schemas.microsoft.com/cdo/configuration/smtpserver") = "smtp.office365.com"
Configuration.Fields.Item("http://schemas.microsoft.com/cdo/configuration/smtpserverport") = 25
Configuration.Fields.Item("http://schemas.microsoft.com/cdo/configuration/smtpauthenticate") = 1
Configuration.Fields.Item("http://schemas.microsoft.com/cdo/configuration/sendusername") = "YourEmail#yourdomain.com"
Configuration.Fields.Item("http://schemas.microsoft.com/cdo/configuration/sendpassword") = "YourPass"
Configuration.Fields.Item("http://schemas.microsoft.com/cdo/configuration/smtpusessl") = True
Configuration.Fields.Update
Set Message.Configuration = Configuration
Message.Send
End Sub
This code worked for me until a few days ago when we switched ISP's (or maybe something changed coincidentally on the server side). If I specify port 587, I get a transport error and have not found the solution for that with VBA. Please let me know if this works for you and if you find a way to use 587. (Port 25 doesn't work for me either, same error.)
Public Function SMTPSend(vSendTo, vsubject As Variant, vmessage As Variant)
'This works
Set emailObj = CreateObject("CDO.Message")
emailObj.From = "name#myemail.com"
emailObj.To = vSendTo
emailObj.Subject = vsubject
emailObj.TextBody = vmessage
'emailObj.AddAttachment "c:\windows\win.ini"
Set emailConfig = emailObj.configuration
emailConfig.Fields("http://schemas.microsoft.com/cdo/configuration/sendusing") = 2
emailConfig.Fields("http://schemas.microsoft.com/cdo/configuration/smtpauthenticate") = 1
emailConfig.Fields("http://schemas.microsoft.com/cdo/configuration/smtpserver") = "smtp.office365.com"
'Must exclude port if specifying SSL
'emailConfig.Fields("http://schemas.microsoft.com/cdo/configuration/smtpserverport") = 587
emailConfig.Fields("http://schemas.microsoft.com/cdo/configuration/smtpusessl") = True
emailConfig.Fields("http://schemas.microsoft.com/cdo/configuration/sendusername") = "name#myemail.com"
emailConfig.Fields("http://schemas.microsoft.com/cdo/configuration/sendpassword") = "mypassword"
emailConfig.Fields.Update
emailObj.Send
If Err.Number = 0 Then SMTPSend = True Else SMTPSend = False
End Function
There seem to be a number of posts on various forums suggesting that the CDO library doesn't work with port 587, but that's not true. I think the real issue is that SMTP services using port 587 are doing so because they require STARTTLS authentication and the http://schemas.microsoft.com/cdo/configuration/smtpusessl config option is specific to SSL, not TLS (I know, I know - not the most accurate description but it'll suffice for now).
Instead, there is another setting - http://schemas.microsoft.com/cdo/configuration/sendtls - that does support TLS, so using this with port 587 works fine:
config.Fields("http://schemas.microsoft.com/cdo/configuration/sendusing").Value = 2 '2 = cdoSendUsingPort
config.Fields("http://schemas.microsoft.com/cdo/configuration/smtpserver").Value = "smtp.example.com"
config.Fields("http://schemas.microsoft.com/cdo/configuration/smtpserverport").Value = 587
config.Fields("http://schemas.microsoft.com/cdo/configuration/smtpauthenticate").Value = 1 '1 = cdoBasic
config.Fields("http://schemas.microsoft.com/cdo/configuration/sendusername").Value = "my_username"
config.Fields("http://schemas.microsoft.com/cdo/configuration/sendpassword").Value = "myy_very_secret_password"
config.Fields("http://schemas.microsoft.com/cdo/configuration/sendtls").Value = True
config.Fields("http://schemas.microsoft.com/cdo/configuration/smtpconnectiontimeout").Value = 60
config.Fields.Update()
This is what worked for me:
flds("http://schemas.microsoft.com/cdo/configuration/sendusing") = 2
flds("http://schemas.microsoft.com/cdo/configuration/smtpserver")= "smtp.office365.com"
flds("http://schemas.microsoft.com/cdo/configuration/smtpserverport")= 25
flds("http://schemas.microsoft.com/cdo/configuration/smtpauthenticate")= 1
flds("http://schemas.microsoft.com/cdo/configuration/sendusername")= "name#myemail.com"
flds("http://schemas.microsoft.com/cdo/configuration/sendpassword") = "mypasword"
flds("http://schemas.microsoft.com/cdo/configuration/smtpusessl")= True
flds("http://schemas.microsoft.com/cdo/configuration/smtpconnectiontimeout")= 60
I don't know if you could fix it, but I got it with this:
Private Sub Command1_Click()
Dim oSmtp As New EASendMailObjLib.Mail
oSmtp.LicenseCode = "TryIt"
' Set your Hotmail email address
oSmtp.FromAddr = "liveid#hotmail.com"
' Add recipient email address
oSmtp.AddRecipientEx "support#emailarchitect.net", 0
' Set email subject
oSmtp.Subject = "test email from hotmail account"
' Set email body
oSmtp.BodyText = "this is a test email sent from VB 6.0 project with hotmail"
' Hotmail SMTP server address
oSmtp.ServerAddr = "smtp.live.com"
' Hotmail user authentication should use your
' Hotmail email address as the user name.
oSmtp.UserName = "liveid#hotmail.com"
oSmtp.Password = "yourpassword"
' Set port to 25, if you want to use 587 port, please change 25 to 587
oSmtp.ServerPort = 25
' detect SSL/TLS connection automatically
oSmtp.SSL_init
MsgBox "start to send email ..."
If oSmtp.SendMail() = 0 Then
MsgBox "email was sent successfully!"
Else
MsgBox "failed to send email with the following error:" & oSmtp.GetLastErrDescription()
End If
End Sub
Font: https://www.emailarchitect.net/easendmail/kb/vb.aspx?cat=4
Remember download the library:
http://easendmail-smtp-component-net-edition.soft112.com/
Just Replace Parameters!
Trying to configure Exim mail server using this article. I can use this server inside my local network but when I try to use it from internet I taking some errors. Ports 10000-20000 translate to server machine. Everything I doing inside the docker image of CentOS 7. Host machine with CentOS 7 too.
Abbreviations:
test_domain.tk - my test domain
test1, test2 - test users
test#external.com - test external email
123.456.789.876 - my external ip (I have router with NAT)
10.0.7.30 - docker's tunnel
Starting docker with command: docker run -d --name mail -h test_domain.tk -p 10025:25 -p 10587:587 -p 10465:465 -p 10143:143 -p 10993:993 mail/server:localwork start_server
start_server:
#!/bin/bash -e
/usr/sbin/dovecot && /usr/sbin/exim -v -bdf -q30m
/etc/exim/exim.conf:
primary_hostname = test_domain.tk
domainlist local_domains = # : localhost : test_domain.tk
domainlist relay_to_domains =
hostlist relay_from_hosts =
acl_smtp_mail = acl_check_mail
acl_smtp_rcpt = acl_check_rcpt
acl_smtp_data = acl_check_data
acl_smtp_mime = acl_check_mime
av_scanner = clamd:/var/run/clamd.exim/clamd.sock
tls_advertise_hosts = *
tls_certificate = /etc/ssl/default.crt
tls_privatekey = /etc/ssl/default.key
daemon_smtp_ports = 25 : 465 : 587
tls_on_connect_ports = 465
allow_domain_literals
never_users = root
auth_advertise_hosts = *
rfc1413_hosts = *
rfc1413_query_timeout = 5s
ignore_bounce_errors_after = 2d
timeout_frozen_after = 7d
begin acl
acl_check_mail:
deny condition = ${if eq{$sender_helo_name}{} {1}}
message = Nice boys say HELO first
warn condition = ${if eq{$sender_host_name}{} {1}}
set acl_m_greylistreasons = Host $sender_host_address lacks reverse DNS\n$acl_m_greylistreasons
accept
acl_check_rcpt:
accept hosts = :
control = dkim_disable_verify
deny message = Restricted characters in address
domains = +local_domains
local_parts = ^[.] : ^.*[#%!/|]
deny message = Restricted characters in address
domains = !+local_domains
local_parts = ^[./|] : ^.*[#%!] : ^.*/\\.\\./
accept local_parts = postmaster
domains = +local_domains
require verify = sender
accept hosts = +relay_from_hosts
control = submission
control = dkim_disable_verify
accept authenticated = *
control = submission
control = dkim_disable_verify
require message = relay not permitted
domains = +local_domains : +relay_to_domains
require verify = recipient
accept
acl_check_data:
warn condition = ${if !def:h_Message-ID: {1}}
set acl_m_greylistreasons = Message lacks Message-Id: header. Consult RFC2822.\n$acl_m_greylistreasons
accept
acl_check_mime:
deny message = Blacklisted file extension detected
condition = ${if match \
{${lc:$mime_filename}} \
{\N(\.exe|\.pif|\.bat|\.scr|\.lnk|\.com)$\N} \
{1}{0}}
accept
begin routers
dnslookup:
driver = dnslookup
domains = ! +local_domains
transport = remote_smtp
ignore_target_hosts = 0.0.0.0 : 127.0.0.0/8
# if ipv6-enabled then instead use:
# ignore_target_hosts = <; 0.0.0.0 ; 127.0.0.0/8 ; ::1
no_more
system_aliases:
driver = redirect
allow_fail
allow_defer
data = ${lookup{$local_part}lsearch{/etc/aliases}}
# user = exim
file_transport = address_file
pipe_transport = address_pipe
userforward:
driver = redirect
check_local_user
# local_part_suffix = +* : -*
# local_part_suffix_optional
file = $home/.forward
allow_filter
no_verify
no_expn
check_ancestor
file_transport = address_file
pipe_transport = address_pipe
reply_transport = address_reply
procmail:
driver = accept
check_local_user
require_files = ${local_part}:+${home}/.procmailrc:/usr/bin/procmail
transport = procmail
no_verify
localuser:
driver = accept
check_local_user
# local_part_suffix = +* : -*
# local_part_suffix_optional
transport = local_delivery
cannot_route_message = Unknown user
begin transports
remote_smtp:
driver = smtp
remote_msa:
driver = smtp
port = 587
hosts_require_auth = *
procmail:
driver = pipe
command = "/usr/bin/procmail -d $local_part"
return_path_add
delivery_date_add
envelope_to_add
user = $local_part
initgroups
return_output
local_delivery:
driver = appendfile
directory = $home/Maildir
maildir_format
maildir_use_size_file
delivery_date_add
envelope_to_add
return_path_add
address_pipe:
driver = pipe
return_output
address_file:
driver = appendfile
delivery_date_add
envelope_to_add
return_path_add
address_reply:
driver = autoreply
begin retry
* * F,2h,15m; G,16h,1h,1.5; F,4d,6h
begin rewrite
begin authenticators
dovecot_login:
driver = dovecot
public_name = LOGIN
server_socket = /var/run/dovecot/auth-client
server_set_id = $auth1
dovecot_plain:
driver = dovecot
public_name = PLAIN
server_socket = /var/run/dovecot/auth-client
server_set_id = $auth1
exim log:
8 LOG: MAIN
8 exim 4.84 daemon started: pid=8, -q30m, listening for SMTP on port 25 (IPv6 and IPv4) port 587 (IPv6 and IPv4) and for SMTPS on port 465 (IPv6 and IPv4)
16 LOG: host_lookup_failed MAIN
16 no host name found for IP address 123.456.789.876
16 LOG: MAIN REJECT
16 H=([10.0.7.30]) [123.456.789.876] X=SSLv3:DHE-RSA-AES128-SHA:128 F=<test1#test_domain.tk> rejected RCPT <test#external.com>: relay not permitted
16 LOG: lost_incoming_connection MAIN
16 unexpected disconnection while reading SMTP command from ([10.0.7.30]) [123.456.789.876]
When I try to connect from internet I got timeout error in mail client and empty logs in Exim. It's probably problem of work with router. How to make it works?
Ask me if you need more data. Thanks in advance.
You have your docker internal ports (for example 10025) mapped to standard smtp ports (for example 25), but you have exim listening on the standard ports instead of the mapped internal ports. Configure exim to listen on 10025, 10465, 10587 and see if the behavior changes.
It seems there's an issue with reverse DNS lookup according to this ancient post. Try disabling host_lookup and see if that works:
host_lookup = 0.0.0.0/0
If it does you'll have to fix your DNS settings to map the right domain to your host.
Trac does not appear to be triggering an email event when submitting a change to a ticket with a cc'ed email address. I currently have logging enabled, and I'm not finding anything pertaining to email in the trac.log file. I'm also monitoring /var/log/maillog - nothing appears there either.
I am not using SMTP. I'm attempting to configure Trac to use sendmail.
My current [notifications] section:
admit_domains =
always_notify_owner = false
always_notify_reporter = false
always_notify_updater = true
email_sender = SmtpEmailSender
ignore_domains =
mime_encoding = none
sendmail_path = /usr/sbin/sendmail
smtp_always_bcc = known_good_address#domain.tld
smtp_always_cc =
smtp_default_domain =
smtp_enabled = false
smtp_from = trac#localhost
smtp_from_name =
smtp_password =
smtp_port = 25
smtp_replyto = trac#localhost
smtp_server = localhost
smtp_subject_prefix = __default__
smtp_user =
ticket_subject_template = $prefix #$ticket.id: $summary
use_public_cc = false
use_short_addr = false
use_tls = false
Path to sendmail is correct:
[box]# which sendmail
/usr/sbin/sendmail
Should all of the smtp references be removed from the trac.ini [notification] section in order for Trac's sendmail hook to work?
It should be as easy as changing
smtp_enabled = false
to
smtp_enabled = true