we receive bursts of mails from our IDS that we are postprocessing to create incident reports, forward them to request tracker, etc. Right now, we have one script that does all the postprocessing that is called by procmail according to rules in /etc/aliases. Something like:
ids-report: "|/opt/ids/process.pl"
Now, the problem is that running many instances of this script because of mail bursts can lead to memory depletion and some synchronization nastyness.
In my script, I tried using Sys::RunAlone, but found it suboptimal.
I would like procmail to feed my script one mail from queue at a time, but I have not found a way to do it.
Do you guys have any idea, what could I do? Out-of-the-box solutions are welcomed too...
Thanks.
Why not save every mail to a file or database as soon as you receive them and process the saved records one at a time from another script ?
If you post some code you may receive more (and possibly better) answers :)
I second Georgi Rangelov's answer but if you are seriously asking how to run only one message at a time in Procmail, the answer is to use a lock file.
:0:only1.lock
| /opt/ids/process.pl
This means, if the file only1.lock exists, wait. Once it's gone, create it, run the pipe, and then remove the lock file when done.
See also http://porkmail.org/era/procmail/mini-faq.html#locking
"IMAP with idle" mailbox as buffer/pipeline of email messages
You may deliver messages to IMAP mailbox. You can use fetchmail with --idle (to get real time processing) and --mda (to execute your script).
You can make procmail deliver messages to maildir as use dovecot even without daemon running to allow fetchmail access maildir using IMAP protocol.
P.S. I may provide a few more details if you are interested. AFAIR I had used it to handle spams catched by my spamtrap server.
Related
I know I should probably ask MS about this, but I dont know if I trust their communication tools to get me a satisfying answer.
I have recently taken the task to test an application of mine.
Part of this test is to send an e-mail to it, then log in and see if I get the result I want (this is automated)
For this the process is:
Open Outlook -> Send Mail
Close Outlook
Open App -> Check result
Open Outlook -> Send Mail
Close Outlook
But for some reason, every second time the outlook app is opened, (including manually opening the app to check on configurations etc.) Mails will go to the outbox but will not be sent, unless I manually trigger them to.
Now, there are possible solutions like keeping the app running continously, or telling my testing-suite to press F9 after every sent mail, but I want to tackle the root cause, and fix the underlying problem.
The Outlook Version used is the latest Version of Office 365 Outlook.
Has anyone else had this experience and figured out a fix?
Thank you in advance.
Keep in mind that message submission is an asynchronous process, so if you close Outlook while it is still sending, the message might end up stuck in the Outbox.
You can call Namespace.SendAndReceive to force submission, but it is still asynchronous. You can hook into the SyncObject.SyncEnd event on the first (All Accounts) SyncObject from the Namespace.SyncObjects collection and quite only after that event fires.
Since this problem occured on a machine I do not own, I tried avoiding certain easy troubleshooting steps. After I couldnt find an answer on my own, and I didnt get responses here that would've helped me solve this without a wooden hammer method, I asked for permission to issue a repair to the local office 365 suite.
After the repair it had redownloaded and installed the entirety of office 365.
After connecting my company MS account to outlook, the issue no longer appeared.
I guess this counts as solving this issue.
I need to write a windows service to send emails. The emails will likely be stored in a database table and they should be sent as early as convenient. It would be advantages to have multiple threads sending messages as there will be hurts at certain times of the day however it is not good to send the same message multiple times.
So I'm having a little bit of trouble understanding in this kind of scenario how I can best leverage quartz.net to alleviate some of queueing and concurrency issues. So my architecture questions are:
1. For this kind of scenario, is it best for a Job to check if there are emails to send or should a job be to actually send one email?
2. If the answer to 1) is to check for emails to be sent then that would leave me with a concurrency issue and I would need to use DisallowConcurrentExecution which would result in only 1 email being sent at a time?
3. If there answer to 1) is send a single email then I take it the job details would need to reflect the specific ID of the email to be sent?
4. In either case - two web users could trigger the creation of the same email job (concurrently). So it doesn't seem that Quartz really helps solve my problem - it might provide a nice architecture for a unit of work and controlling polling frequency but not really the core of my problem? Or am I mssing / over thinking something?
Finally, just to be clear, each email relates to a specific Order so there is ID and state potential. So because two web users can send the same email at the same instant in time should not result in two emails being sent.
Look forward to any advice.
Thanks
Josh
Quartz.Net would meet your scheduling needs.
However, you have conflicting needs. You want "more than one thread" to send the emails, but you also want "Do not want duplicate emails".
The DisallowConcurrentExecution will prevent multiple instances of the same job running at the same time. However, if you have only one instance of the job running, you don't know which individuals emails have been sent or not sent.
If you only keep "these emails have been sent, and these haven't" in memory.....you're always at risk of sending duplicates.
You can solve this, but you're gonna have to have a "pessimistic" flag on which emails have already been sent. Like at the database level.
So if you want multiple threads to send emails...that's ok. But your "get some emails to send" code is going to have to 'mark' the emails it is working on. (So the next thread doesn't get them). Then you have to mark them again right after they are sent.
Quartz is good for scheduling the "when" your jobs run. But it doesn't have the ability to "track" which emails you need to send and which ones have already been sent. That's gonna be your responsibility.
I had this similar problem....where I had many many users trying to "get at" a bunch of to-do items. Thus why I wrote this blog entry for Sql Server. I needed to "mark" the rows, but also had to order them before I marked them.
http://granadacoder.wordpress.com/2009/07/06/update-top-n-order-by-example/
I also added some "hints".......
WITH ( UPDLOCK, READPAST , ROWLOCK ) –<<Optional Hints
because so many different users were trying to "get-at" the items.
(Think about how T1cket M#ster has to work.......there has to be some pessimistic locking on the tickets....and they have a timer that releases the locks if you don't buy the tickets in time).
Hope that helps.
Due to high network traffic during the day, many of our Jenkins builds must run in evenings and during the night. Emails are sent containing reports, notifications of broken builds, etc. However, I don't want the emails to be sent to developers in evenings and during the night. Is it possible to queue all the emails and send them e.g. between 8-17 office hours? So if a build breaks during the night, an email is sent at 8am.
Unfortunately, as far as I know, there is no plugin that allows to delay email sending. However, maybe you can give a try to the script capability of the email-ext plugin. It allows you to use JS or Groovy scripts in the template. In such script, you may write a loop that "waits" 8am to send the email.
But personnally, I don't like that idea, it's not really a good way to achieve that, and in addition it will certainly make the final result of the build wait until 8am (the build will only finish once the mail is effectively sent). This will also have the drawback that the job will take one place in the Jenkins job queue, potentially blocking another job...
Maybe developing your own plugin (by forking mail-ext plugin for example) would be a better idea...
Let me spread my ideas.
I'm also not aware of any existing functionality to achieve that via Jenkins.
Plugin would be probably the best way (possibly beneficial for others is published to public).
The alternative solution coming to my mind is in case you are in a situation, where you have control of the e-mail server, that might also be place to achieve your goal.
As for the SMTP (based on the sever you use) there might be a solution.
Possibly solution provided here (sendmail in queue-only mode) could help you:
How can I delay mail delivery through an SMTP relay, possibly sendmail
I have roughly 9,000 undelivered messages in my mail spool in Coldfusion 9. As far as I can tell the only way to manage these messages is to manipulate them 10 at a time through the CF Admin GUI.
I'm looking for a way to expedite this process. I'd like to just clear the queue, or batch send them all.
Does anyone know how to do this?
Thank you,
-Dave
Go into the filesystem and move the files from cfusion/mail/undelivr to cfusion/mail/spool. Simple!
Take a look at Ray Camden's SpoolMail (http://spoolmail.riaforge.org/). This is a very handy plug-in that you can add to all your servers and at bulk move your emails to spool and resend them.
Someone sent me a snippet at some point that would try to reprocess the queue periodically at some point, but for the life of me, I can;t find it or see it on google. - sorry.
HOWEVER: the undeliverable queue is just a bunch of files, you can write yourself a little application that will try to reprocess the queue periodically and prune out the ones that have been in the undeliverable too long.
I am pretty sure that the spool only tries to deliver mail once
just move the files back into the spool directory to have the spooler "retry"
you would have to keep a file or DB to track what has been tried & how many times.
I would also send a notification on what has been deleted - or at least log it.
-sean
PS> 9000? what is your traffic like? I would suspect there is a problem if you have that many undeliverables....
You should write simple CF program for tracking and deleting for undelivered mails.
Use cfdirectory tag.
If you moved your mail server and the spool doesn't seem to ever empty out, you need to open up each .cfmail file and change the IP number. I moved my mail server as well, and when I brought it back up I had forgotten to change the IP setting for mail in CF Admin, and wound up with 21,000 emails in my spool by the next day. Ugh. I could have run a cf script on it to open up each one, change the IP number, and then move the file into the spool dir, but opted instead of downloading a free search and replace utility from download dot com. Worked like a charm. It took about twenty minutes to do the full s&r and then a few seconds to move them all over.
The file system is the simplest way.
To attempt to resend the emails, move the files in ~\ColdFusion2016\cfusion\Mail\Undelivr to ~\ColdFusion2016\cfusion\Mail\Spool.
If you just don't care about those email files then simply delete them.
I point my development mail server to point to nowhere (smtp.gmail.com.dontSend) so no mail goes out and they all just stack up in undelivered. That way the rest of my development team does not get all the errors I generate and catch through email. More importantly, no test emails accidentally go out to real accounts.
I go into the CF-Administrator to look at and manage my undelivered mail when testing but if/when it gets huge I just delete them from the file system.
I am writing a script to handle an automated email inbox. I've never really dealt with mail servers before, but have gotten to a point where every email sent is executing my script. I am using a .forward file in the home directory of the recipient user that looks like this:
"|exec /home/[MY USER]/magic || exit 75 #[MY USER]"
The script simply takes it's input and writes it to a file at this point. The problem is, I don't know how to find the email being sent, and I can't seem to find any documentation on how the .forward-file works in detail. How do I get to the email?
Any help is greatly appeciated :)
Some general information about the ~/.forward file and the interaction with the mail server is described here, and the Filtering Mail FAQ also contains some information.
While just using a dot-forward file might be sufficient you very typically want to use procmail for more powerful mail processing.