I'm trying to send email from the command line in ubuntu so that I can incorporate it into my script.
When I run this command:
echo "This is a test email again"| sendmail tdunphy#mycompany.com
I see this in the mail logs:
Jul 9 13:41:21 tdunphy-VirtualBox postfix/local[31388]: B8BF17F: to=<tdunphy#mycompany.com>, relay=local, delay=0.17, delays=0.1/0.04/0/0.03, dsn=2.0.0, status=sent (delivered to command: procmail -a "$EXTENSION")
Jul 9 13:41:21 tdunphy-VirtualBox postfix/qmgr[31307]: B8BF17F: removed
But for some reason the email never arrives. Any tips on how to troubleshoot this would be appreciated.
Related
I am attempting to run perl CGI script on RHEL 7, i've used yum installer and it's successfully installed 5.26, i can run perl programs from terminal without problem, but when i run a simple CGI script it writes error to httpd/error.log. I've posted the exact error below.
enabled software collection
$scl enable rh-perl526 bash
perl -v
This is perl 5, version 26, subversion 3 (v5.26.3) built for x86_64-linux-thread-multi
(with 27 registered patches, see perl -V for more detail)
$which perl
/opt/rh/rh-perl526/root/usr/bin/perl
$perl -e 'print "Hello Perl\n"'
Hello Perl
When i run a simple CGI script from browser or curl -XGET, i got 500 internal server for
http://serverip/cgi-bin/test-cgi.pl
apache error
[Wed Mar 24 21:24:38.308033 2021] [cgi:error] [pid 9437] [client x.x.x.x:35220] AH01215: /opt/rh/rh-perl526/root/usr/bin/perl: error while loading shared libraries: libperl.so.rh-perl526-5.26: cannot open shared object file: No such file or directory
[Wed Mar 24 21:24:38.308245 2021] [cgi:error] [pid 9437] [client x.x.x.x:35220] End of script output before headers: test.cgi.pl
I've added libperl.so path to LD_LIBRARY_PATH variable in .bashrc and i've rh-perl526-perl-libs.x86_64 package installed.
While installing rh-perl526, i noticed it also installed default perl on system as well due to YUM dependency
any idea how to fix this issue ?
LD_LIBRARY_PATH needs to set in apache http.conf file
i have added below line in httpd.conf and it worked
SetEnv LD_LIBRARY_PATH /path/to/libperl
I also removed perl 5.16 from system using yum.
We are seeing suck sendmail processes when we are attempting to send email from a Perl FCGI process. These processes are taking too long, hours to a day, since it should just be doing a relay to a server configured in sendmail as the smart host. Most of the mail from the FCGI processes takes less than 5 seconds. The slow sendmail processes are easy to find on the our servers with $ ps -ef | grep sendmail
Almost all of the email works normally from these web nodes. I'd guess thousands of mails go through with no problem. Sending test email from the command line goes smoothly. The sendmail command gets stuck rarely and we don't have a way to reproduce it.
It seems that most of this stuck email gets through sooner or later. These seem to be sending mail hours later, sometimes over a day later.
All of the sendmail that we've seen stuck has been a command that was run by a Perl process, which is a child process of a FCGI process.
Looking at the logs of the smart host we see that most of this mail does get through sooner or later but we have found some that don't seem to have ever been sent.
This is running in FCGI for Catalyst and then added to a IO::Async::Loop which does some processing, and in the IO::Async::Loop, Email::Sender::Transport::Sendmail is used which does a open($fh, '|-', #args) and pipes the mail header+body and does a close($fh).
I've seen this http://perldoc.perl.org/perlipc.html#Avoiding-Pipe-Deadlocks but don't know how to apply it in this situation. The child sendmail has only STDIN open.
When we have one of these stuck sendmails the sendmail is waiting on STDIN:
[<ffffffff8119ce8b>] pipe_wait+0x5b/0x80
[<ffffffff8119d8ad>] pipe_read+0x34d/0x4d0
[<ffffffff8119204a>] do_sync_read+0xfa/0x140
[<ffffffff81192945>] vfs_read+0xb5/0x1a0
[<ffffffff81192c91>] sys_read+0x51/0xb0
[<ffffffff8100b0d2>] system_call_fastpath+0x16/0x1b
[<ffffffffffffffff>] 0xffffffffffffffff
and the async perl process is waiting on the child to die:
#0 0x00007f8849e6065e in waitpid () from /lib64/libc.so.6
#1 0x000000000046dc2d in Perl_wait4pid ()
#2 0x000000000046de2d in Perl_my_pclose ()
#3 0x00000000004cec4e in Perl_io_close ()
#4 0x00000000004ceda8 in Perl_do_close ()
#5 0x00000000004c2629 in Perl_pp_close ()
#6 0x00000000004804de in Perl_runops_standard ()
#7 0x000000000042e7ad in perl_run ()
#8 0x000000000041bbc5 in main ()
An example of one that didn't get through:
Job #1653576 (that's just our internal job number) has a sendmail process that started on Aug 19 13:04.
Process on webnode2:
fcgi-user 13621 13466 0 13:04 ? 00:00:00 /usr/sbin/sendmail -i -f admin#ourServer.org -- proffunnyhat#mit.edu
I don't see the record I expect to see on our smart host for this in /var/log/maillog that would indicate that it was relayed to nexus and then to MIT.
I do see successful email for proffunnyhat#mit.edu on Aug 21 (from web2 /var/log/maillog):
Aug 21 00:00:02 node-008 sendmail[13621]: u7JH4tbr013621: to=proffunnyhat#mit.edu, ctladdr=admin#ourServer.org (10520/10520), delay=1+10:55:07, xdelay=00:00:01, mailer=relay, pri=32292, relay=[127.0.0.1] [127.0.0.1], dsn=2.0.0, stat=Sent (u7L401Z1026237 Message accepted for delivery)
Aug 21 00:00:02 node-008 sendmail[26247]: u7L401Z1026237: to=<proffunnyhat#mit.edu>, delay=00:00:01, xdelay=00:00:00, mailer=relay, pri=122657, relay=mail.ourServer.org. [128.84.4.11], dsn=2.0.0, stat=Sent (u7L402jx001185 Message accepted for delivery)
and then on mail.ourServer.org:
bdc34 #mail.ourServer.org: log$ sudo grep u7L402jx001185 maillog*
maillog-20160821:Aug 21 00:00:02 web2 sendmail[1185]: u7L402jx001185: from=<admin#ourServer.org>, size=2874, class=0, nrcpts=1, msgid=<201608191704.u7JH4tbr013621#mail.ourServer.org>, proto=ESMTP, daemon=MTA, relay=mail.ourServer.org [128.84.4.13]
maillog-20160821:Aug 21 00:00:03 mail.ourServer.org[1200]: u7L402jx001185: to=<proffunnyhat#mit.edu>, ctladdr=<e-admin#ourServer.org> (10519/10519), delay=00:00:01, xdelay=00:00:01, mailer=esmtp, pri=122874, relay=dmz-mailsec-scanner-8.mit.edu. [18.7.68.37], dsn=2.0.0, stat=Sent (OK 5E/2D-20045-34729B75)
An example of one that was stuck but seems to have been sent:
mail.ourServer.org:/var/log/sendmail:
Aug 19 02:19:51 mail.ourServer.org sendmail[20792]: u7J6JlP6020790: to=<jxjx#connect.ust.hk>, ctladdr=<admin#ourServer.org> (10519/10519), delay=00:00:04, xdelay=00:00:04, mailer=esmtp, pri=122504, relay=connect-ust-hk\
.mai...ction.outlook.com. [213.199.154.87], dsn=2.0.0, stat=Sent (<201608190619.u7J6Jlda000738#web2.ourServer.org> [InternalId=15526306777069,...1MB1197.apcprd01.prod.exchangelabs.com] 9137 bytes in 0.189, 47.082 KB/sec\
Queued mail for delivery)
Things we have tried
I've modified Email::Sender::Transport::Sendmail to send a '\x00' to the pipe, that didn't work.
I've replaced IO::Async::Loop::Poll with IO::Async::Loop::Select. That didn't change anything.
I've tried sending signals to the sendmail and its parent. That killed them but the mail was aborted.
Added our fcgi user to sendmail's trusted users file. Didn't change anything.
I wrote a wrapper script that read from STDIN and writes to sendmail. If nothing comes in on STDIN for 5 seconds it exits. This feels really hacky to me but it does seem to work. Since mail is a critical part of our system I'd rather have a real solution.
ikegami comment lead us to the answer of doing a double fork. Looking at the signal handlers and file handles set up under FCGI made it clear that excessively clever things were happening. So I moved to cut all ties with the parent process using a double fork like when starting a daemon. That worked.
# FCGI does some clever signal handeling and file handers to do
# its work. This causes problems with forked processes that then
# fork and wait for other processes. So use exec to get a process
# that is not a copy of the parent. Double fork to avoid zombies.
# Check -e of submit script because $! has a cryptic messsage if it doens't exist
my $script = "$SAFEBINDIR/submit.pl";
unless( -e -r -x $script ){
$submission->submit_log->error($submission->submission_id . ": $script doesn't exist");
$c->flash( message => "There was a problem");
$c->res->redirect( $c->uri_for('/user') );
$c->detach;
}
# Do the double fork + exec to have an independent process
my $pid;
unless( $pid = fork() ) { #this is the child
unless( fork() ){ #this is the grandchild
exec( $script, $submission->submission_id ) #should never return
or $submission->submit_log->error($submission->submission_id
. ": Error when trying to exec() $script '$!'");
exit(0);
}
}
waitpid($pid,0); #wait for child, grandchild will get ppid 1
}
So I followed [the guide][1] on how to set up a simple mail filter with Postfix, so that I can do a find-replace in the body of outgoing emails. I created a script at /tmp/mailfilter.sh, and changed the /etc/postfix/master.cf file as instructed
# ==========================================================================
# service type private unpriv chroot wakeup maxproc command + args
# (yes) (yes) (yes) (never) (100)
# ==========================================================================
smtp inet n - n - - smtpd
-o content_filter=filter:dummy
filter unix - n n - 10 pipe
flags=Rq user=filter null_sender=
argv=/tmp/mailfilter.sh -f ${sender} -- ${recipient}
I created a user called filter and made it the owner of the script. But when I tried sending an email, I get the following error:
Jun 7 03:01:53 localhost postfix/qmgr[31288]: 134D944A0673: from=<sender#gmail.com>, size=894, nrcpt=1 (queue active)
Jun 7 03:01:53 localhost pipe[31603]: fatal: pipe_command: execvp /tmp/mailfilter.sh: Permission denied
Jun 7 03:01:53 localhost postfix/pipe[31562]: 134D944A0673: to=<receiver#gmail.com>, relay=filter, delay=8974, delays=8974/0/0/0.01, dsn=4.3.0, status=deferred (temporary failure. Command output: pipe: fatal: pipe_command: execvp /tmp/mailfilter.sh: Permission denied )
Specifically what I'm assuming is relevant is
(temporary failure. Command output: pipe: fatal: pipe_command: execvp /tmp/mailfilter.sh: Permission denied )
/tmp/mailfilter.sh has chmod a+x and is owned by filter. I tried removing everything in it so it's just an empty file, and I still get the permission denied error.
I can't figure out what I'm missing. I've set every permission I can find, but Postfix is doing something arcane that I don't understand.
CentOS uses SELinux as a MAC framework, so maybe you need to set properly the type of your executable. You can check in /var/log/audit/audit.log for any security violation. If SELinux is denying you, you can try this command as root:
chcon -t postfix_pipe_exec_t /tmp/mailfilter.sh
That manual is a good reference: http://linux.die.net/man/8/postfix_selinux
I am trying to learn Selinux. With a sandbox and using VSFTPD to experiment with, I have a vsfptd server running in Centos. I have annonmous users to place files in /var/ftp/incoming. On a remote machine I can have the user successfully log in but could not place the file on the remove vsftpd server:
$ftp mysql_server
Connected to mysql_server (192.168.1.31).
220 Welcome to blah FTP service.
Name (mysql_server:root): anonymous
331 Please specify the password.
Password:
230 Login successful.
Remote system type is UNIX.
Using binary mode to transfer ftp> put atd
local: atd remote: atd
227 Entering Passive Mode (192,168,1,31,19,161).
553 Could not create file.
ftp>
On the VSFTPD server, aureport -a report shows:
[root#mysql_server ftp]# aureport -a
AVC Report
========================================================
# date time comm subj syscall class permission obj event
========================================================
4. 04/08/2013 13:30:36 vsftpd unconfined_u:system_r:ftpd_t:s0-s0:c0.c1023 21 dir write system_u:object_r:public_content_t:s0 denied 28
5. 04/08/2013 13:34:57 vsftpd unconfined_u:system_r:ftpd_t:s0-s0:c0.c1023 2 dir write system_u:object_r:public_content_t:s0 denied 47
I checked the directory and the file contexts look good, so I don't understand why Selinux won't allow vsftpd to write to the incoming directory:
[root#mysql_server ftp]# ls -Z
drwx-wx---. root ftp system_u:object_r:public_content_t:s0 incoming
drwxr-xr-x. root root system_u:object_r:public_content_t:s0 pub
[root#mysql_server ftp]#
You need to run the following commands to allow in SELinux upload and edit files:
setsebool -P allow_ftpd_full_access on
setsebool -P ftp_home_dir on
Your SELinux type is not correct. Use 'public_content_rw_t' instead of 'public_content_t'. Read more on http://beginlinux.com/blog/2008/11/vsftpd-and-selinux-on-centos/
I'm running a JMeter test plan from command line and it's currently outputting something along the lines of:
Created the tree successfully using C:\*****\TestPlan.jmx
Starting the test # Thu Oct 11 10:20:43 EDT 2012 (1349965243947)
Waiting for possible shutdown message on port 4445
Tidying up ... # Thu Oct 11 10:20:46 EDT 2012 (1349965246384)
... end of run
Is there any way to turn off this output and have the plan execute 'silently'?
Found a way to do this, by following this article http://www.robvanderwoude.com/battech_redirection.php
and appending > NUL to the command
jmeter -n -t C:\***\TestPlan.jmx -Jhostname=%1 > NUL