---+ BRIEF:
using Outlook 2016 as my email client, connecting to Gmail account
Q: what features allow orthogonal properties to be specified independently for messages?
AFAICT the only orthogonal features are read/unread, and only a single type of Outlook/flag mapping to single type of Gmail star.
Which is leading me to consider
Flagged/starred=not-handled-yet
have a (Gmail) rule set the Gmail star as messages are received
clear the flag/star when you have handled it
which is slightly safer than
Flagged/starred=deferred
new messages in inbox will be unflagged/unstarred and hence not deferred yet)
must take care to set the flag/star whenever deferring
must take care to set the flag/star when moving to a topic folder/label(and out of Inbox) or else unhandled messages may be lost
and must take care to clear the flag/star whenever archiving
and of course both of these policies conflict with historical/legacy uses of flags and stars
---+ LESS BRIEF:
I am (perhaps unfortunately) impelled[*] to use Microsoft Outlook as my email client on my PC.
More precisely Microsoft Outlook for Microsoft 365 MSO … which as far as I can tell is essentially Outlook 2016 with slight updates. NOT the web version.
I am connecting it to my Gmail account of many years.
This connection is IMAP/SMTP. I am individual, personal, no exchange servers in the mix etc.
My email workflow and organization is nothing very special. But I'll describe briefly, for motivation. My email workflow for many years has wanted to be something like the following [**]:
• Scan my Inbox
• if I can handle an email right away, do so
• if not, mark it as "Deferred"
I typically have several subfolders or labels under "Deferred", such as work/personal, urgent/soon/later. Sometimes I use GTD like classifications or lists. Sometimes I want to move or mark or label or tag emails as related to particular projects or topics. Sometimes I have rules that apply such marks or labels or tags or folders as email is received. Plus I tend to use Gmail labels or Outlook folders for topics, projects, etc. I really like using saved searches or search folders. I really liked being able to sort email, e.g. by subject, sender, thread, etc..
Basically, I want to be able to be able to mark or classify my email with several ORTHOGONAL or INDEPENDENT properties. and I want to be able to query combinations of these orthogonal/independent properties.
For example:
• email that is associated with a topic
• read/unread
• deferred/not
• handled
○ I'm finished with the email, etc.
And I want to be able to make queries such as
• "show me all email that I have not completely handled yet"
○ whether or not it is in my Inbox, or in a Deferred folder, etc.
Email systems have various ways of indicating such properties, orthogonal or not
• e.g. Outlook folders and subfolders
○ strictly speaking not truly orthogonal
• e.g. Gmail labels - which are truly orthogonal since you can have multiple labels per email
• Gmail stars
• Outlook flags
• Outlook categories
The kluge that maps Gmail labels to IMAP folders Give the Outlook users some control over Gmail's truly orthogonal labels. But AFAIK the Outlook user has no way to detect whether an email has more than one Gmail label, i.e. no way to look for the same message being in more than one Outlook folder. So I can't have Outlook do the equivalent of a Gmail search "in:Inbox AND label:Deferred AND label:Someday". Not by mapping Outlook folders to Gmail labels.
(If there were some way of having Outlook detect that a message has been copied to more than one folder, I think I would be done. Some cheap way. Email deduplication tools accomplish this in some sense, but they really want to what they think are duplicates, which are really Gmail messages with multiple labels.)
Outlook in its natural state has limited orthogonal properties in the form of flags and categories. Good enough for stuff like
• show me all the email in folder Topic-X that has category "Needs-Handling"
• show me all the email in folder Topic-X that is flagged …
other you quickly run out of categories and flags in a way that you do not with Gmail labels.
But when I'm attaching my Outlook email client to the Gmail message store (across IMAP, if it matters)
Outlook categories are not supported by Gmail. if you set a category in Outlook it does not appear on the Gmail message. it appears that the Outlook category may survive for a while, perhaps as long as it's in the Outlook cache of Gmail, but if you uninstall and then reinstall Outlook on a different machine, the categories are lost. AFAICT.
As far as I can tell the only orthogonal properties that are common to the intersection of Outlook on Gmail are
• the klugey Gmail label / Outlook folder mapping
• Gmail star mapped to Outlook flag
and only one flavor of star/flag. Gmail supports multiple colors of star, but Outlook only sees them as a single flag. and vice versa.
Q: are there any other orthogonal message properties that can be set and queried by both Outlook and Gmail?
I highly suspect not, but I just wanted to check.
---+ WORKFLOW given these limitations
messages received by Gmail
○ optionally labeled automatically by Gmail rules
○ and/or left in:Inbox
I scan Inbox (using either Outlook or Gmail clients)
○ handle immediately, and archive (remove the Inbox label)
○ or defer
○ or leave in Inbox as read/unread.
while I might defer by moving out of Inbox into the Defer folder-label or a subfolder, doing this does not allow me to query "Topic1 && deferred/…" in Outlook.
If Outlook-flag/Gmail-star is really the only orthogonal property, I think that I might have to use it to indicate deferred.
Two choices:
Flagged/starred=deferred
• new messages in inbox will be un-flagged unstarred and hence not deferred yet)
• must take care to set the flag/star whenever deferring
• must take care to set the flag/star when moving to a topic folder/label (and out of Inbox)
○ or else unhandled messages may be lost
• and must take care to clear the flag/star whenever archiving
or
Flagged/starred=not-handled-yet
• have a (Gmail) rule set the Gmail star as messages are received
• clear the flag/star when you have handled it
I think the latter choice, Flagged/starred=not-handled-yet, is "safer", in the sense that the most common user error would be to not clear the flag/star when it is handled or archived. this could be annoying, clutter up your searches for " work I haven't finished yet", but it doesn't lose anything. whereas the other polarity can quite easily result in work falling through the cracks.
Note: deferred is a subset of not-handled-yet.
The biggest problem with flagged/starred=not-handled-yet is that I, and probably many others, have historically used flags/stars to indicate things like " this is interesting", or "this is important". i.e. there is a semantic conflict between these historic uses and using flag/start to indicate deferred or not handled yet
Q: has anybody else used the flag/star=not-handled-yet approach? problems?
---+ MORE BACKGROUND
Note**: "My email workflow for many years has wanted to be something like the following [**]:…"
I say "wanted to be" because I've tried to get as close to this is possible, not always successfully.
I was most successful back in the the 20th century when using GNUS email on emacs. Procmail sorting into folders. I would only mark email as "read" when I finished processing it.
Unfortunately, more and more people need to know the distinction between the following three states:
"I have not seen/read your email yet"
"I have seen your email, read it but not yet handled it"
"I have both read and handle your email"
Also, way back in these GNUS/Procmail days, I was free to add my own headers to messages in my saved email tree. But this flexibility is not available nowadays in Gmail/Outlook. I think if I could add my own headers I would be logically happy enough, although the UI might be a bit unsatisfactory.
Note*: "I am (perhaps unfortunately) impelled[] to use Microsoft Outlook as my email client on my PC*."
First, because of what my physical therapist calls computeritis I am trying to use speech recognition to do as much as possible, to avoid typing. Speech recognition just works better for Microsoft Outlook than it does with Gmail.
Second, Gmail does not sort messages in folders or queries. IMHO sorting in other email clients like Outlook and Thunderbird essentially gives you the answer to queries that you do not yet know that you should ask. e.g. you might notice that you have suddenly received 100 messages from a particular sender on different topics. It would be better if you had groups sorts the could return message counts, but eyeballing is often good enough.
... Why use Gmail as the mail storage/server underneath the Outlook user client? why not move my email to a system where all of the Outlook features are supported? First, Gmail free is a good price, but Microsoft has similar. Second, mainly because I have used Gmail for a very long time, and fear the potential work or hassle of moving all of my historic email. and I do need to move all my historic email. Plus, with my luck I would move all of my email from Gmail to Microsoft, and then Microsoft would start charging. But third, I really do like the concept of Gmail labels. I am reluctant to move to a system that does not support such orthogonality. It is unfortunate that Gmail does not support sorting, and I don't know of any other email client that is speech friendly which accesses labels natively.
I spent almost 2 years trying to use Thunderbird rather than Microsoft Outlook as my email client, but I've recently stopped doing so. PRO: Thunderbird sorts like Outlook. CON: Thunderbird is not as speech friendly as Outlook, but I was able to work around much of its problems, so in this respect Thunderbird was slightly more speech friendly than Gmail. CON: Thunderbird interfaces with Gmail labels by this includes that Outlook does. PRO: there is a Thunderbird add-on that can at least show you all of the labels associated with message, although AFAICT no queries. PRO: Thunderbird produces HTML email that more of my correspondence in the open source/hardware community can understand, Whereas Outlook produces HTML email that many people find unreadable on non-Outlook email readers. CON: Thunderbird does not have highlighting (I kluged this, but it was not pretty). PRO: Thunderbird is open source so in theory you can fix things, write your own code, preferably as a add-on. CON: Thunderbird fairly recently underwent a major reorganization of its add-on architecture, and most of the historical add-ons that I want to use no longer work. And it seems that many of them cannot be made to work because of Thunderbird's newish security model. OVERALL, while Thunderbird might be a place to return to, I think I have spent enough time doing so for the moment.
I was and am still most hopeful about being able to return to an emacs based email reader like Gnus. PRO: although not really designed for speech, I have found emacs's command orientation very amenable to speech recognition. In some ways preferable to Outlook. CON: Gnus IMAP Is online only, not acceptable for somebody who needs to read email on planes when disconnected etc. CON: nearly every time I connected Gnus/IMAP/Gmail my emacs would be locked out for 15 minutes, half an hour, etc. CON: I spent some time working with offlineimap (flakey), and mbsync (syncs Gmail-IMAP-Maildir bidirectionally, and then Gnus/Emacs accesses using nnmaildir. but I kept running into Gmail account problems, bandwidth limits, and just plain flakiness. Overall, I get the impression that Google/Gmail are really pushing against non-Oauth authentication. and authentication in open source seems to be a bit of an issue; if an open source app has all of the keys inserts necessary to connect to Gmail Oauth style, AFAICT but might be a violation of the Gmail terms and conditions.
Bottom line: I was and am still hopeful about being able to return to an emacs based mail reader like Gnus accessing Gmail, weather across IMAP or Gmail's native API. off-line. Oauth. and in my dreams with support for Gmail style labels,
If the Gmail labels were exposed as headers, X-GM-LABELS, then I think that my need for orthogonality would be satisfied. I know that I can query such headers in Gnus, and I think they can be queried in Outlook. It remains my hope that I can get headers for Gnus/Emacs. But as far as I know Outlook does not have a way of obtaining such headers. and as far as I know MailDir is hostile the things that modify message content in this way.
All of this last section is just background.
For the moment, all I really want to know is if there are any other orthogonal features when Outlook accesses Gmail. Other than Outlook-flag/Gmail-star.
I will answer my own question, not because I want to do the Q&A, but to clearly state my present understanding, even though it is already mentioned my question.
As far as I can tell
Gmail by itself has the following user settable and query able message properties:
labels: text strings, hierarchical, otherwise fully orthogonal, or close
read/unread
important/not
stars: I believe 12 standard, yellow/Orange/red/purple/blue/green, red-bang!, orange-guillemet, yellow-bang, green-check, blue-info, purple question
categories: Primary, social, promotions, updates, forms, reservation, purchases
I would not be at all surprised if all of these are implemented as labels under the hood, mapped to different icons and operators like labels/category:
Outlook natively, i.e. with exchange, has the following user settable and query able message properties:
folders... not exactly mutually orthogonal, since can only be in a single folder at a time, but folders are orthogonal with the other properties
read/unread
flags - In different flavors ranging from simple flag/unflag, followup flags with a text string, date flags, ...
categories: a limited number of named & colored categories
importance: normal/high/low
sensitivity
And probably some user definable fields.
Although it would seem that Outlook has more orthogonal properties than Gmail, and it does, doesn't mean it's more powerful in terms of querying and setting. it mainly means that some of them have predefined behaviors and GUI features like icons.
However, the intersection of the Gmail and Outlook features is quite narrow:
Gmail does not support Outlook categories, Importance, sensitivity...
AFAICT there is only one orthogonal property apart from the folder/label: Outlook flagged / Gmail starred
There is a simple mapping between Gmail star and Outlook flag.
All of the at least twelve possible Gmail flags maps to a single Flagged value in Outlook - I.e. Outlook cannot tell what the Gmail star is.
If Outlook flagsthe message, it displays as Gmail yellow star. Outlook cannot control what Gmail star is used.
There seem to be synchronization issues between Outlook flagged and Gmail starred. e.g. I have not yet managed to have Outlook clear a flag that Gmail has set, and have that reflected on Gmail's web interface.
I've got a significantly large mailing list - 50K+ subscribers. To avoid stressing my servers, I would like to avoid sending emails through a components embedded on my website. Sending through here sends the CPU usage through the roof - so I'd like to be able to send emails locally. I can easily send emails through mandrillapp, so sending the emails out is not a problem. However, I've hit a bit of a snag.
Phplist seems to assume it is living on a public site, and inserts tracking info which routes the users to a phplist directory on my site (which obviously) does not exist.
Question 1: First of all, I'd like to avoid embedding this tracking - is this possible? Or else is there someway to include it and avoid the 404 error. Would I have to install phplist anyway on the server?
Question 2: I've already got acymailing to handle unsubscribes, so is it actually possible to keep this in place - just to make sure the acymailing is still my point of reference?
Question 3: How do people handle sending out large mailing lists? I know CampaignMonitor, MailChimp etc, but these get a bit expensive for my situation. I'd like to keep sending "internally" so to speak. Is there an elegant solution which will NOT kill my server but is not too expensive? I know I want to have the cake and eat, but it would be nice to hear what people out there are doing.
TIA
David
I need to write a windows service to send emails. The emails will likely be stored in a database table and they should be sent as early as convenient. It would be advantages to have multiple threads sending messages as there will be hurts at certain times of the day however it is not good to send the same message multiple times.
So I'm having a little bit of trouble understanding in this kind of scenario how I can best leverage quartz.net to alleviate some of queueing and concurrency issues. So my architecture questions are:
1. For this kind of scenario, is it best for a Job to check if there are emails to send or should a job be to actually send one email?
2. If the answer to 1) is to check for emails to be sent then that would leave me with a concurrency issue and I would need to use DisallowConcurrentExecution which would result in only 1 email being sent at a time?
3. If there answer to 1) is send a single email then I take it the job details would need to reflect the specific ID of the email to be sent?
4. In either case - two web users could trigger the creation of the same email job (concurrently). So it doesn't seem that Quartz really helps solve my problem - it might provide a nice architecture for a unit of work and controlling polling frequency but not really the core of my problem? Or am I mssing / over thinking something?
Finally, just to be clear, each email relates to a specific Order so there is ID and state potential. So because two web users can send the same email at the same instant in time should not result in two emails being sent.
Look forward to any advice.
Thanks
Josh
Quartz.Net would meet your scheduling needs.
However, you have conflicting needs. You want "more than one thread" to send the emails, but you also want "Do not want duplicate emails".
The DisallowConcurrentExecution will prevent multiple instances of the same job running at the same time. However, if you have only one instance of the job running, you don't know which individuals emails have been sent or not sent.
If you only keep "these emails have been sent, and these haven't" in memory.....you're always at risk of sending duplicates.
You can solve this, but you're gonna have to have a "pessimistic" flag on which emails have already been sent. Like at the database level.
So if you want multiple threads to send emails...that's ok. But your "get some emails to send" code is going to have to 'mark' the emails it is working on. (So the next thread doesn't get them). Then you have to mark them again right after they are sent.
Quartz is good for scheduling the "when" your jobs run. But it doesn't have the ability to "track" which emails you need to send and which ones have already been sent. That's gonna be your responsibility.
I had this similar problem....where I had many many users trying to "get at" a bunch of to-do items. Thus why I wrote this blog entry for Sql Server. I needed to "mark" the rows, but also had to order them before I marked them.
http://granadacoder.wordpress.com/2009/07/06/update-top-n-order-by-example/
I also added some "hints".......
WITH ( UPDLOCK, READPAST , ROWLOCK ) –<<Optional Hints
because so many different users were trying to "get-at" the items.
(Think about how T1cket M#ster has to work.......there has to be some pessimistic locking on the tickets....and they have a timer that releases the locks if you don't buy the tickets in time).
Hope that helps.
for my group at the university I'd like to set up a server-sided email-to-rss service.
It should work like that, that different people can send emails to a certain address (nothing proprietary like gmail but a certain imap or pop server) which will the be translated into an rss feed. One main and important feature has to be that one can see the sender of the email in the feed. Furthermore it would be nice (to take the load off the server) if the emails get translated to a feed only once a day or so.
Does anyone has some input on this subject? Are there any scripts/services which will allow that?
Thanks a bunch.
Instead of "reinventing the wheel", you could use a mailing list that supports RSS. Your people can then write the mails to the mailing list and you can then use the mailing list's RSS feed however you intend to.
This should help you find a solution: https://encrypted.google.com/search?q=mailing%20list%20rss
Pick a programming language you're familiar with, then use either an imap library to fetch the E-mails (through cron, every hour or something like that), or if you have access to procmail on your mail server, launch your script as an email arrives (this shouldn't be too much work to handle for a server, unless you're talking a vast amount of E-mail).
The script would just insert the E-mails into a database, before extracting them and outputting the RSS-feed directly from that (this shouldn't be more than a handful of lines of code).
There's a couple of providers that does this for you, although it seems that the most popular ones have disappeared. Advanced Email2RSS seems to be an option, although I have no idea how good they are or if it'll even solve your issue.
Most services offered online today that claim to "track" e-mails, do so by embedding images in the emails. My questions are:
Is this the only way to do it and if not, what are the other methods?
Are any of the methods actually fool-proof?
Has anybody had any luck with specific software or even an online group?
Yes, this is pretty much the only way to do it. Consider that an email is something that is inherently static. The only way to know if someone has "opened" an email is for the email to send some information back to your server. Most email clients these days support HTML emails, which means that you can get the client to request an image (or anything else) from your server by embedding the proper HTML tags. Other than this, you cannot force an email client to do anything it doesn't want to do. It's a separate program on a remote computer, and you have no control over it.
No, there's no foolproof way. There will always be emails you can't track. If someone downloads their email and disconnects from the internet before reading it, you can't track that email. Most email clients allow you to disable image loading now as well if you want to, so that can block tracking too.
I've usually written my own, so I wouldn't know what to recommend. I imagine most services will be quite similar, so I'd base a product/purchase decision on how easy their front-end is to use.
In addition to pixel tracking, a second way to track open rates is by looking for clickthroughs. If someone clicked through, then they must have opened it. This is infrequent, but it's important not to throw this data away.
More details:
How MailChimp tracks open rates
How CampaignMonitor tracks open rates
Wikipedia on email open rates
Hubspot on open rate issues
Facebook uses a bgsound element in addition to an img element like this:
<img src="http://www.facebook.com/email_open_log_pic.php?mid=999999999999"
style="border:0;width:1px;height:1px;" />
<bgsound src="http://www.facebook.com/email_open_log_pic.php?mid=99999999999&s=a"
volume="-10000" />
This is the best way, and it's hardly ideal - many e-mail clients block images to start with.
No, no methods are foolproof. A foolproof method of detecting if someone had read an e-mail would be a significant privacy issue.
I've used ExactTarget and CampaignMonitor's tracking systems. Both worked pretty well for tracking trends - i.e. twice as many people opened e-mail #1 than #2 - but you never know how many missed opens there are due to images not being shown.
Pixel tracking is the only way to track open rates. Then the links in your emails are also tracked through a redirect service for click rates. Absolutely nothing is going to be foolproof. You will have to use some guess work to figure out your actual open rate since some email clients will only take the text version and not the html and also some clients do not load images by default.
SilverPop is a popular one. They actually use PowerMTA on the back-end. Our company just ended up licensing PowerMTA and writing our own front-end and tracking.
No it's not the only way. Your HTML e-mail can refer to a web server for 'some content' which is then tracked. That could be an image, a stylesheet, some Javascript, etc. Most mail clients hate it and nothing automated is guaranteed to work.
Gain the trust of your recipient and invite them to your website. Track clicks.