Related
---+ BRIEF:
using Outlook 2016 as my email client, connecting to Gmail account
Q: what features allow orthogonal properties to be specified independently for messages?
AFAICT the only orthogonal features are read/unread, and only a single type of Outlook/flag mapping to single type of Gmail star.
Which is leading me to consider
Flagged/starred=not-handled-yet
have a (Gmail) rule set the Gmail star as messages are received
clear the flag/star when you have handled it
which is slightly safer than
Flagged/starred=deferred
new messages in inbox will be unflagged/unstarred and hence not deferred yet)
must take care to set the flag/star whenever deferring
must take care to set the flag/star when moving to a topic folder/label(and out of Inbox) or else unhandled messages may be lost
and must take care to clear the flag/star whenever archiving
and of course both of these policies conflict with historical/legacy uses of flags and stars
---+ LESS BRIEF:
I am (perhaps unfortunately) impelled[*] to use Microsoft Outlook as my email client on my PC.
More precisely Microsoft Outlook for Microsoft 365 MSO … which as far as I can tell is essentially Outlook 2016 with slight updates. NOT the web version.
I am connecting it to my Gmail account of many years.
This connection is IMAP/SMTP. I am individual, personal, no exchange servers in the mix etc.
My email workflow and organization is nothing very special. But I'll describe briefly, for motivation. My email workflow for many years has wanted to be something like the following [**]:
• Scan my Inbox
• if I can handle an email right away, do so
• if not, mark it as "Deferred"
I typically have several subfolders or labels under "Deferred", such as work/personal, urgent/soon/later. Sometimes I use GTD like classifications or lists. Sometimes I want to move or mark or label or tag emails as related to particular projects or topics. Sometimes I have rules that apply such marks or labels or tags or folders as email is received. Plus I tend to use Gmail labels or Outlook folders for topics, projects, etc. I really like using saved searches or search folders. I really liked being able to sort email, e.g. by subject, sender, thread, etc..
Basically, I want to be able to be able to mark or classify my email with several ORTHOGONAL or INDEPENDENT properties. and I want to be able to query combinations of these orthogonal/independent properties.
For example:
• email that is associated with a topic
• read/unread
• deferred/not
• handled
○ I'm finished with the email, etc.
And I want to be able to make queries such as
• "show me all email that I have not completely handled yet"
○ whether or not it is in my Inbox, or in a Deferred folder, etc.
Email systems have various ways of indicating such properties, orthogonal or not
• e.g. Outlook folders and subfolders
○ strictly speaking not truly orthogonal
• e.g. Gmail labels - which are truly orthogonal since you can have multiple labels per email
• Gmail stars
• Outlook flags
• Outlook categories
The kluge that maps Gmail labels to IMAP folders Give the Outlook users some control over Gmail's truly orthogonal labels. But AFAIK the Outlook user has no way to detect whether an email has more than one Gmail label, i.e. no way to look for the same message being in more than one Outlook folder. So I can't have Outlook do the equivalent of a Gmail search "in:Inbox AND label:Deferred AND label:Someday". Not by mapping Outlook folders to Gmail labels.
(If there were some way of having Outlook detect that a message has been copied to more than one folder, I think I would be done. Some cheap way. Email deduplication tools accomplish this in some sense, but they really want to what they think are duplicates, which are really Gmail messages with multiple labels.)
Outlook in its natural state has limited orthogonal properties in the form of flags and categories. Good enough for stuff like
• show me all the email in folder Topic-X that has category "Needs-Handling"
• show me all the email in folder Topic-X that is flagged …
other you quickly run out of categories and flags in a way that you do not with Gmail labels.
But when I'm attaching my Outlook email client to the Gmail message store (across IMAP, if it matters)
Outlook categories are not supported by Gmail. if you set a category in Outlook it does not appear on the Gmail message. it appears that the Outlook category may survive for a while, perhaps as long as it's in the Outlook cache of Gmail, but if you uninstall and then reinstall Outlook on a different machine, the categories are lost. AFAICT.
As far as I can tell the only orthogonal properties that are common to the intersection of Outlook on Gmail are
• the klugey Gmail label / Outlook folder mapping
• Gmail star mapped to Outlook flag
and only one flavor of star/flag. Gmail supports multiple colors of star, but Outlook only sees them as a single flag. and vice versa.
Q: are there any other orthogonal message properties that can be set and queried by both Outlook and Gmail?
I highly suspect not, but I just wanted to check.
---+ WORKFLOW given these limitations
messages received by Gmail
○ optionally labeled automatically by Gmail rules
○ and/or left in:Inbox
I scan Inbox (using either Outlook or Gmail clients)
○ handle immediately, and archive (remove the Inbox label)
○ or defer
○ or leave in Inbox as read/unread.
while I might defer by moving out of Inbox into the Defer folder-label or a subfolder, doing this does not allow me to query "Topic1 && deferred/…" in Outlook.
If Outlook-flag/Gmail-star is really the only orthogonal property, I think that I might have to use it to indicate deferred.
Two choices:
Flagged/starred=deferred
• new messages in inbox will be un-flagged unstarred and hence not deferred yet)
• must take care to set the flag/star whenever deferring
• must take care to set the flag/star when moving to a topic folder/label (and out of Inbox)
○ or else unhandled messages may be lost
• and must take care to clear the flag/star whenever archiving
or
Flagged/starred=not-handled-yet
• have a (Gmail) rule set the Gmail star as messages are received
• clear the flag/star when you have handled it
I think the latter choice, Flagged/starred=not-handled-yet, is "safer", in the sense that the most common user error would be to not clear the flag/star when it is handled or archived. this could be annoying, clutter up your searches for " work I haven't finished yet", but it doesn't lose anything. whereas the other polarity can quite easily result in work falling through the cracks.
Note: deferred is a subset of not-handled-yet.
The biggest problem with flagged/starred=not-handled-yet is that I, and probably many others, have historically used flags/stars to indicate things like " this is interesting", or "this is important". i.e. there is a semantic conflict between these historic uses and using flag/start to indicate deferred or not handled yet
Q: has anybody else used the flag/star=not-handled-yet approach? problems?
---+ MORE BACKGROUND
Note**: "My email workflow for many years has wanted to be something like the following [**]:…"
I say "wanted to be" because I've tried to get as close to this is possible, not always successfully.
I was most successful back in the the 20th century when using GNUS email on emacs. Procmail sorting into folders. I would only mark email as "read" when I finished processing it.
Unfortunately, more and more people need to know the distinction between the following three states:
"I have not seen/read your email yet"
"I have seen your email, read it but not yet handled it"
"I have both read and handle your email"
Also, way back in these GNUS/Procmail days, I was free to add my own headers to messages in my saved email tree. But this flexibility is not available nowadays in Gmail/Outlook. I think if I could add my own headers I would be logically happy enough, although the UI might be a bit unsatisfactory.
Note*: "I am (perhaps unfortunately) impelled[] to use Microsoft Outlook as my email client on my PC*."
First, because of what my physical therapist calls computeritis I am trying to use speech recognition to do as much as possible, to avoid typing. Speech recognition just works better for Microsoft Outlook than it does with Gmail.
Second, Gmail does not sort messages in folders or queries. IMHO sorting in other email clients like Outlook and Thunderbird essentially gives you the answer to queries that you do not yet know that you should ask. e.g. you might notice that you have suddenly received 100 messages from a particular sender on different topics. It would be better if you had groups sorts the could return message counts, but eyeballing is often good enough.
... Why use Gmail as the mail storage/server underneath the Outlook user client? why not move my email to a system where all of the Outlook features are supported? First, Gmail free is a good price, but Microsoft has similar. Second, mainly because I have used Gmail for a very long time, and fear the potential work or hassle of moving all of my historic email. and I do need to move all my historic email. Plus, with my luck I would move all of my email from Gmail to Microsoft, and then Microsoft would start charging. But third, I really do like the concept of Gmail labels. I am reluctant to move to a system that does not support such orthogonality. It is unfortunate that Gmail does not support sorting, and I don't know of any other email client that is speech friendly which accesses labels natively.
I spent almost 2 years trying to use Thunderbird rather than Microsoft Outlook as my email client, but I've recently stopped doing so. PRO: Thunderbird sorts like Outlook. CON: Thunderbird is not as speech friendly as Outlook, but I was able to work around much of its problems, so in this respect Thunderbird was slightly more speech friendly than Gmail. CON: Thunderbird interfaces with Gmail labels by this includes that Outlook does. PRO: there is a Thunderbird add-on that can at least show you all of the labels associated with message, although AFAICT no queries. PRO: Thunderbird produces HTML email that more of my correspondence in the open source/hardware community can understand, Whereas Outlook produces HTML email that many people find unreadable on non-Outlook email readers. CON: Thunderbird does not have highlighting (I kluged this, but it was not pretty). PRO: Thunderbird is open source so in theory you can fix things, write your own code, preferably as a add-on. CON: Thunderbird fairly recently underwent a major reorganization of its add-on architecture, and most of the historical add-ons that I want to use no longer work. And it seems that many of them cannot be made to work because of Thunderbird's newish security model. OVERALL, while Thunderbird might be a place to return to, I think I have spent enough time doing so for the moment.
I was and am still most hopeful about being able to return to an emacs based email reader like Gnus. PRO: although not really designed for speech, I have found emacs's command orientation very amenable to speech recognition. In some ways preferable to Outlook. CON: Gnus IMAP Is online only, not acceptable for somebody who needs to read email on planes when disconnected etc. CON: nearly every time I connected Gnus/IMAP/Gmail my emacs would be locked out for 15 minutes, half an hour, etc. CON: I spent some time working with offlineimap (flakey), and mbsync (syncs Gmail-IMAP-Maildir bidirectionally, and then Gnus/Emacs accesses using nnmaildir. but I kept running into Gmail account problems, bandwidth limits, and just plain flakiness. Overall, I get the impression that Google/Gmail are really pushing against non-Oauth authentication. and authentication in open source seems to be a bit of an issue; if an open source app has all of the keys inserts necessary to connect to Gmail Oauth style, AFAICT but might be a violation of the Gmail terms and conditions.
Bottom line: I was and am still hopeful about being able to return to an emacs based mail reader like Gnus accessing Gmail, weather across IMAP or Gmail's native API. off-line. Oauth. and in my dreams with support for Gmail style labels,
If the Gmail labels were exposed as headers, X-GM-LABELS, then I think that my need for orthogonality would be satisfied. I know that I can query such headers in Gnus, and I think they can be queried in Outlook. It remains my hope that I can get headers for Gnus/Emacs. But as far as I know Outlook does not have a way of obtaining such headers. and as far as I know MailDir is hostile the things that modify message content in this way.
All of this last section is just background.
For the moment, all I really want to know is if there are any other orthogonal features when Outlook accesses Gmail. Other than Outlook-flag/Gmail-star.
I will answer my own question, not because I want to do the Q&A, but to clearly state my present understanding, even though it is already mentioned my question.
As far as I can tell
Gmail by itself has the following user settable and query able message properties:
labels: text strings, hierarchical, otherwise fully orthogonal, or close
read/unread
important/not
stars: I believe 12 standard, yellow/Orange/red/purple/blue/green, red-bang!, orange-guillemet, yellow-bang, green-check, blue-info, purple question
categories: Primary, social, promotions, updates, forms, reservation, purchases
I would not be at all surprised if all of these are implemented as labels under the hood, mapped to different icons and operators like labels/category:
Outlook natively, i.e. with exchange, has the following user settable and query able message properties:
folders... not exactly mutually orthogonal, since can only be in a single folder at a time, but folders are orthogonal with the other properties
read/unread
flags - In different flavors ranging from simple flag/unflag, followup flags with a text string, date flags, ...
categories: a limited number of named & colored categories
importance: normal/high/low
sensitivity
And probably some user definable fields.
Although it would seem that Outlook has more orthogonal properties than Gmail, and it does, doesn't mean it's more powerful in terms of querying and setting. it mainly means that some of them have predefined behaviors and GUI features like icons.
However, the intersection of the Gmail and Outlook features is quite narrow:
Gmail does not support Outlook categories, Importance, sensitivity...
AFAICT there is only one orthogonal property apart from the folder/label: Outlook flagged / Gmail starred
There is a simple mapping between Gmail star and Outlook flag.
All of the at least twelve possible Gmail flags maps to a single Flagged value in Outlook - I.e. Outlook cannot tell what the Gmail star is.
If Outlook flagsthe message, it displays as Gmail yellow star. Outlook cannot control what Gmail star is used.
There seem to be synchronization issues between Outlook flagged and Gmail starred. e.g. I have not yet managed to have Outlook clear a flag that Gmail has set, and have that reflected on Gmail's web interface.
This Odoo company we're working with basically sends a lot of e-mail. One e-mail thread can turn into 100+ e-mails with different people brought into the conversation (CC'd) at different times. Due to the complexity of their e-mail management, they want to use their Gmail interface (Google Hosted) and CC an e-mail into odoo and they want it to get tracked in a thread. I've basically already done that... they have an e-mail like odoo+res.partner-432#domain.com (although it's hashed to not be easily readable) - they CC this and the full body thread gets included in chatter (mail.message) under respective model / id.
The challenge with this: the chatter messages can get huge very fast, due to their e-mail messages (because each e-mail includes main reply, and all previous history on thread). I've looked into some systems that have a "reply above this line" - and it just takes the latest message. And in those systems, eg. ticketing systems such as Zendesk, help scout, I believe the teams are using the ticketing system (not a gmail) and thus there is much more control over the inbox and incoming email (not to mention, those e-mails are usually 1-to-1, not including groups).
My questions:
Is there any other workaround that you see here to have odoo pull in only the last e-mail reply and not the full e-mail thread? I could probably build something like this: https://github.com/zapier/email-reply-parser - and hook it into odoos e-mail parsing, but that works on text format e-mails only (not HTML)... only. So it's not bulletproof, and I'm not sure it's worth it.
Even if this client DID use odoo 100%, I still don't think it would be possible to get it to work the way they want without major customizations (eg. Odoo's default behavior is to include all past e-mail threads)
I'm curious if anyone here see's any other solutions, otherwise – I doubt there is something here I haven't seen. :) (But very open to be proven incorrect!)
I am working on a project, where I need to identify emails sent by real humans as opposed to bulk mails, notifications and newsletters. Is there any definite way of doing that? Is there any information in email header which can help. I am working on top of Gmail IMAP so I already have non-spam emails.
Any help in this regard is appreciated. Thanks!
There isn't a clear way to distinguish bulk mail from personalised mailings. Unlike with spam, most bulk mail is requested/expected, so the sender doesn't do odd things to get round spam filters, which means these emails often blend in fairly well.
However, there are some trends that you can look for. If you want to do it reliably, you will probably need to apply some scoring system, like spam-filters do.
You will also need to accept that you are bound to get a substantial proportion of false positives and false negatives.
Some things that are common to bulk mail that appear less often in personalised correspondence:
"To" and "Cc" addresses do not contain a local recipient. Sometimes the sender will send to "mailList#mydomain.com" instead of "recipientA#recipientAdomain.com", "recipientB#recipientBdomain.com", etc. In these cases, it is also likely that only one address appears in "To" and nothing appears in "Cc"
"From" address is "noreply#", "newsletter#", "do-not-reply#", "mailinglist#", even less common terms like "support#" or "sales#" (but remember, they could cause false positives)
The presence of a "List-Unsubscribe:" header
The message contains an unsubscribe link. Run pattern matching to find common phrases in the final few lines of the email. Look for links, or words such as "unsubscribe", "opt out", etc.
Mailing lists tend to have rich content. Check for heavy use of CSS and lots of images, the entire message being contained within a <table></table> or <ul><li></li></ul> structure. i.e. the stuff that something like Dreamweaver would put in, rather than a mail client.
Headers or bold content at the top of the message. If the first bit of a message resembles a newsletter, it's probably a newsletter.
Lots of links or frequent linking to the same (or same few) websites. Newsletters will try to guide the user to the company's site(s), as much as they can. You may score this even more highly if the linked domain matches (or resembles) the sender domain.
Heavy references to social media. If it's a newsletter containing several articles, each story may have its own "Tweet this", "Like this" link. Personal users are likely to contain (at most) one reference to Twitter, Facebook, etc (in their signature)
Notifications and other auto-generated messages will often follow the same basic format. If you have the capabilities, run some kind of diffing or other comparison against previous messages. A strong match would imply automation.
There is no greeting, or a generic greeting. However, personal emails will often skip the "Dear Fred" bit too, so this isn't a good enough detection by itself; but things like "Dear User" or "Dear Customer" are almost certainly generic.
Unlikely to end in "Regards, Ian" or "Yours Sincerely, John Doe"
The sender has scored highly before. Keep a record. If a sender triggers a high score several times, they are almost certainly bulk mailing.
I'm doing a small newsletter software for my business, and I'm wondering what metrics should I collect. Obviously, bounces and clicks should be tracked, but I'm wondering should I track email opens (via an image or a bgsound element)?
Do popular webmail services and ISPs check for tracking images and possibly increase my spam score? I guess if it increases the chances of ending up in a spam filter, I'd rather not collect that metric.
Thanks.
It's generally bad form to try to track your users in that way. Email opens are a private thing.
If you have an image with a unique URL per message, yes you can track it, but IMO, you really shouldn't. Including unnecessary images in an email is bad for a number of reasons:
Images can increase your spam score. There's a time and place for images. They can improve a message, but used inappropriately, they can look spammy.
It is obvious what you are doing. Sooner or later, one of your customers is likely to get wise to it. Some people won't care; others will feel violated.
It's REALLY unreliable. Most email clients and webmails feature an option to block images by default. You will get massively understated results.
Also remember, some people open an email immediately before they click the "delete" button. You are much better off tracking clicks.
There may be some merit in tracking the images you want to include anyway, but I'd not treat it as anything more than a very basic indicator.
As always it depends on the individual ISP and Webmail services. However, I can share some anecdotal evidence: I periodically use mailchimp to send out mass email notifications, and email opens are tracked in mailchimp using the same approach you mentioned (See following link for reference: http://kb.mailchimp.com/reports/about-open-tracking). I never experienced any issues with ending up in the spam folder, I have only had challenges with bounce backs and ending up in the Gmail promotions tab.
So based on the fact that some companies are already doing this, I have to say it probably won't increase your spam score.
Most services offered online today that claim to "track" e-mails, do so by embedding images in the emails. My questions are:
Is this the only way to do it and if not, what are the other methods?
Are any of the methods actually fool-proof?
Has anybody had any luck with specific software or even an online group?
Yes, this is pretty much the only way to do it. Consider that an email is something that is inherently static. The only way to know if someone has "opened" an email is for the email to send some information back to your server. Most email clients these days support HTML emails, which means that you can get the client to request an image (or anything else) from your server by embedding the proper HTML tags. Other than this, you cannot force an email client to do anything it doesn't want to do. It's a separate program on a remote computer, and you have no control over it.
No, there's no foolproof way. There will always be emails you can't track. If someone downloads their email and disconnects from the internet before reading it, you can't track that email. Most email clients allow you to disable image loading now as well if you want to, so that can block tracking too.
I've usually written my own, so I wouldn't know what to recommend. I imagine most services will be quite similar, so I'd base a product/purchase decision on how easy their front-end is to use.
In addition to pixel tracking, a second way to track open rates is by looking for clickthroughs. If someone clicked through, then they must have opened it. This is infrequent, but it's important not to throw this data away.
More details:
How MailChimp tracks open rates
How CampaignMonitor tracks open rates
Wikipedia on email open rates
Hubspot on open rate issues
Facebook uses a bgsound element in addition to an img element like this:
<img src="http://www.facebook.com/email_open_log_pic.php?mid=999999999999"
style="border:0;width:1px;height:1px;" />
<bgsound src="http://www.facebook.com/email_open_log_pic.php?mid=99999999999&s=a"
volume="-10000" />
This is the best way, and it's hardly ideal - many e-mail clients block images to start with.
No, no methods are foolproof. A foolproof method of detecting if someone had read an e-mail would be a significant privacy issue.
I've used ExactTarget and CampaignMonitor's tracking systems. Both worked pretty well for tracking trends - i.e. twice as many people opened e-mail #1 than #2 - but you never know how many missed opens there are due to images not being shown.
Pixel tracking is the only way to track open rates. Then the links in your emails are also tracked through a redirect service for click rates. Absolutely nothing is going to be foolproof. You will have to use some guess work to figure out your actual open rate since some email clients will only take the text version and not the html and also some clients do not load images by default.
SilverPop is a popular one. They actually use PowerMTA on the back-end. Our company just ended up licensing PowerMTA and writing our own front-end and tracking.
No it's not the only way. Your HTML e-mail can refer to a web server for 'some content' which is then tracked. That could be an image, a stylesheet, some Javascript, etc. Most mail clients hate it and nothing automated is guaranteed to work.
Gain the trust of your recipient and invite them to your website. Track clicks.