I'm trying to troubleshoot some very strange issue with different email clients and IMAP scripts giving me different folders. Folders that have been "deleted" from the account are still visible in Outlook + Thunderbird, but gone from other clients/scripts. This includes when I set up an entirely new profile in Outlook/Thunderbird.
In order to progress at all, I need to get an answer to the fundamental question:
IMAP has 2 steps to delete email messages (not folders):
Delete the message
Purge/expunge it
I want to know: does the IMAP protocol (or certain implementations) also have anything similar for IMAP folders?
I've scoured the web, but can't find anything about this for folders. But I can't understand the inconsistencies I'm seeing with my scripting (in multiple languages) and various email clients.
Note that the folders in question here are not any of the special folders that exist for all users by default (or that MUA clients auto-create). These are just user-created folders with various non-special names.
---+ BRIEF:
using Outlook 2016 as my email client, connecting to Gmail account
Q: what features allow orthogonal properties to be specified independently for messages?
AFAICT the only orthogonal features are read/unread, and only a single type of Outlook/flag mapping to single type of Gmail star.
Which is leading me to consider
Flagged/starred=not-handled-yet
have a (Gmail) rule set the Gmail star as messages are received
clear the flag/star when you have handled it
which is slightly safer than
Flagged/starred=deferred
new messages in inbox will be unflagged/unstarred and hence not deferred yet)
must take care to set the flag/star whenever deferring
must take care to set the flag/star when moving to a topic folder/label(and out of Inbox) or else unhandled messages may be lost
and must take care to clear the flag/star whenever archiving
and of course both of these policies conflict with historical/legacy uses of flags and stars
---+ LESS BRIEF:
I am (perhaps unfortunately) impelled[*] to use Microsoft Outlook as my email client on my PC.
More precisely Microsoft Outlook for Microsoft 365 MSO … which as far as I can tell is essentially Outlook 2016 with slight updates. NOT the web version.
I am connecting it to my Gmail account of many years.
This connection is IMAP/SMTP. I am individual, personal, no exchange servers in the mix etc.
My email workflow and organization is nothing very special. But I'll describe briefly, for motivation. My email workflow for many years has wanted to be something like the following [**]:
• Scan my Inbox
• if I can handle an email right away, do so
• if not, mark it as "Deferred"
I typically have several subfolders or labels under "Deferred", such as work/personal, urgent/soon/later. Sometimes I use GTD like classifications or lists. Sometimes I want to move or mark or label or tag emails as related to particular projects or topics. Sometimes I have rules that apply such marks or labels or tags or folders as email is received. Plus I tend to use Gmail labels or Outlook folders for topics, projects, etc. I really like using saved searches or search folders. I really liked being able to sort email, e.g. by subject, sender, thread, etc..
Basically, I want to be able to be able to mark or classify my email with several ORTHOGONAL or INDEPENDENT properties. and I want to be able to query combinations of these orthogonal/independent properties.
For example:
• email that is associated with a topic
• read/unread
• deferred/not
• handled
○ I'm finished with the email, etc.
And I want to be able to make queries such as
• "show me all email that I have not completely handled yet"
○ whether or not it is in my Inbox, or in a Deferred folder, etc.
Email systems have various ways of indicating such properties, orthogonal or not
• e.g. Outlook folders and subfolders
○ strictly speaking not truly orthogonal
• e.g. Gmail labels - which are truly orthogonal since you can have multiple labels per email
• Gmail stars
• Outlook flags
• Outlook categories
The kluge that maps Gmail labels to IMAP folders Give the Outlook users some control over Gmail's truly orthogonal labels. But AFAIK the Outlook user has no way to detect whether an email has more than one Gmail label, i.e. no way to look for the same message being in more than one Outlook folder. So I can't have Outlook do the equivalent of a Gmail search "in:Inbox AND label:Deferred AND label:Someday". Not by mapping Outlook folders to Gmail labels.
(If there were some way of having Outlook detect that a message has been copied to more than one folder, I think I would be done. Some cheap way. Email deduplication tools accomplish this in some sense, but they really want to what they think are duplicates, which are really Gmail messages with multiple labels.)
Outlook in its natural state has limited orthogonal properties in the form of flags and categories. Good enough for stuff like
• show me all the email in folder Topic-X that has category "Needs-Handling"
• show me all the email in folder Topic-X that is flagged …
other you quickly run out of categories and flags in a way that you do not with Gmail labels.
But when I'm attaching my Outlook email client to the Gmail message store (across IMAP, if it matters)
Outlook categories are not supported by Gmail. if you set a category in Outlook it does not appear on the Gmail message. it appears that the Outlook category may survive for a while, perhaps as long as it's in the Outlook cache of Gmail, but if you uninstall and then reinstall Outlook on a different machine, the categories are lost. AFAICT.
As far as I can tell the only orthogonal properties that are common to the intersection of Outlook on Gmail are
• the klugey Gmail label / Outlook folder mapping
• Gmail star mapped to Outlook flag
and only one flavor of star/flag. Gmail supports multiple colors of star, but Outlook only sees them as a single flag. and vice versa.
Q: are there any other orthogonal message properties that can be set and queried by both Outlook and Gmail?
I highly suspect not, but I just wanted to check.
---+ WORKFLOW given these limitations
messages received by Gmail
○ optionally labeled automatically by Gmail rules
○ and/or left in:Inbox
I scan Inbox (using either Outlook or Gmail clients)
○ handle immediately, and archive (remove the Inbox label)
○ or defer
○ or leave in Inbox as read/unread.
while I might defer by moving out of Inbox into the Defer folder-label or a subfolder, doing this does not allow me to query "Topic1 && deferred/…" in Outlook.
If Outlook-flag/Gmail-star is really the only orthogonal property, I think that I might have to use it to indicate deferred.
Two choices:
Flagged/starred=deferred
• new messages in inbox will be un-flagged unstarred and hence not deferred yet)
• must take care to set the flag/star whenever deferring
• must take care to set the flag/star when moving to a topic folder/label (and out of Inbox)
○ or else unhandled messages may be lost
• and must take care to clear the flag/star whenever archiving
or
Flagged/starred=not-handled-yet
• have a (Gmail) rule set the Gmail star as messages are received
• clear the flag/star when you have handled it
I think the latter choice, Flagged/starred=not-handled-yet, is "safer", in the sense that the most common user error would be to not clear the flag/star when it is handled or archived. this could be annoying, clutter up your searches for " work I haven't finished yet", but it doesn't lose anything. whereas the other polarity can quite easily result in work falling through the cracks.
Note: deferred is a subset of not-handled-yet.
The biggest problem with flagged/starred=not-handled-yet is that I, and probably many others, have historically used flags/stars to indicate things like " this is interesting", or "this is important". i.e. there is a semantic conflict between these historic uses and using flag/start to indicate deferred or not handled yet
Q: has anybody else used the flag/star=not-handled-yet approach? problems?
---+ MORE BACKGROUND
Note**: "My email workflow for many years has wanted to be something like the following [**]:…"
I say "wanted to be" because I've tried to get as close to this is possible, not always successfully.
I was most successful back in the the 20th century when using GNUS email on emacs. Procmail sorting into folders. I would only mark email as "read" when I finished processing it.
Unfortunately, more and more people need to know the distinction between the following three states:
"I have not seen/read your email yet"
"I have seen your email, read it but not yet handled it"
"I have both read and handle your email"
Also, way back in these GNUS/Procmail days, I was free to add my own headers to messages in my saved email tree. But this flexibility is not available nowadays in Gmail/Outlook. I think if I could add my own headers I would be logically happy enough, although the UI might be a bit unsatisfactory.
Note*: "I am (perhaps unfortunately) impelled[] to use Microsoft Outlook as my email client on my PC*."
First, because of what my physical therapist calls computeritis I am trying to use speech recognition to do as much as possible, to avoid typing. Speech recognition just works better for Microsoft Outlook than it does with Gmail.
Second, Gmail does not sort messages in folders or queries. IMHO sorting in other email clients like Outlook and Thunderbird essentially gives you the answer to queries that you do not yet know that you should ask. e.g. you might notice that you have suddenly received 100 messages from a particular sender on different topics. It would be better if you had groups sorts the could return message counts, but eyeballing is often good enough.
... Why use Gmail as the mail storage/server underneath the Outlook user client? why not move my email to a system where all of the Outlook features are supported? First, Gmail free is a good price, but Microsoft has similar. Second, mainly because I have used Gmail for a very long time, and fear the potential work or hassle of moving all of my historic email. and I do need to move all my historic email. Plus, with my luck I would move all of my email from Gmail to Microsoft, and then Microsoft would start charging. But third, I really do like the concept of Gmail labels. I am reluctant to move to a system that does not support such orthogonality. It is unfortunate that Gmail does not support sorting, and I don't know of any other email client that is speech friendly which accesses labels natively.
I spent almost 2 years trying to use Thunderbird rather than Microsoft Outlook as my email client, but I've recently stopped doing so. PRO: Thunderbird sorts like Outlook. CON: Thunderbird is not as speech friendly as Outlook, but I was able to work around much of its problems, so in this respect Thunderbird was slightly more speech friendly than Gmail. CON: Thunderbird interfaces with Gmail labels by this includes that Outlook does. PRO: there is a Thunderbird add-on that can at least show you all of the labels associated with message, although AFAICT no queries. PRO: Thunderbird produces HTML email that more of my correspondence in the open source/hardware community can understand, Whereas Outlook produces HTML email that many people find unreadable on non-Outlook email readers. CON: Thunderbird does not have highlighting (I kluged this, but it was not pretty). PRO: Thunderbird is open source so in theory you can fix things, write your own code, preferably as a add-on. CON: Thunderbird fairly recently underwent a major reorganization of its add-on architecture, and most of the historical add-ons that I want to use no longer work. And it seems that many of them cannot be made to work because of Thunderbird's newish security model. OVERALL, while Thunderbird might be a place to return to, I think I have spent enough time doing so for the moment.
I was and am still most hopeful about being able to return to an emacs based email reader like Gnus. PRO: although not really designed for speech, I have found emacs's command orientation very amenable to speech recognition. In some ways preferable to Outlook. CON: Gnus IMAP Is online only, not acceptable for somebody who needs to read email on planes when disconnected etc. CON: nearly every time I connected Gnus/IMAP/Gmail my emacs would be locked out for 15 minutes, half an hour, etc. CON: I spent some time working with offlineimap (flakey), and mbsync (syncs Gmail-IMAP-Maildir bidirectionally, and then Gnus/Emacs accesses using nnmaildir. but I kept running into Gmail account problems, bandwidth limits, and just plain flakiness. Overall, I get the impression that Google/Gmail are really pushing against non-Oauth authentication. and authentication in open source seems to be a bit of an issue; if an open source app has all of the keys inserts necessary to connect to Gmail Oauth style, AFAICT but might be a violation of the Gmail terms and conditions.
Bottom line: I was and am still hopeful about being able to return to an emacs based mail reader like Gnus accessing Gmail, weather across IMAP or Gmail's native API. off-line. Oauth. and in my dreams with support for Gmail style labels,
If the Gmail labels were exposed as headers, X-GM-LABELS, then I think that my need for orthogonality would be satisfied. I know that I can query such headers in Gnus, and I think they can be queried in Outlook. It remains my hope that I can get headers for Gnus/Emacs. But as far as I know Outlook does not have a way of obtaining such headers. and as far as I know MailDir is hostile the things that modify message content in this way.
All of this last section is just background.
For the moment, all I really want to know is if there are any other orthogonal features when Outlook accesses Gmail. Other than Outlook-flag/Gmail-star.
I will answer my own question, not because I want to do the Q&A, but to clearly state my present understanding, even though it is already mentioned my question.
As far as I can tell
Gmail by itself has the following user settable and query able message properties:
labels: text strings, hierarchical, otherwise fully orthogonal, or close
read/unread
important/not
stars: I believe 12 standard, yellow/Orange/red/purple/blue/green, red-bang!, orange-guillemet, yellow-bang, green-check, blue-info, purple question
categories: Primary, social, promotions, updates, forms, reservation, purchases
I would not be at all surprised if all of these are implemented as labels under the hood, mapped to different icons and operators like labels/category:
Outlook natively, i.e. with exchange, has the following user settable and query able message properties:
folders... not exactly mutually orthogonal, since can only be in a single folder at a time, but folders are orthogonal with the other properties
read/unread
flags - In different flavors ranging from simple flag/unflag, followup flags with a text string, date flags, ...
categories: a limited number of named & colored categories
importance: normal/high/low
sensitivity
And probably some user definable fields.
Although it would seem that Outlook has more orthogonal properties than Gmail, and it does, doesn't mean it's more powerful in terms of querying and setting. it mainly means that some of them have predefined behaviors and GUI features like icons.
However, the intersection of the Gmail and Outlook features is quite narrow:
Gmail does not support Outlook categories, Importance, sensitivity...
AFAICT there is only one orthogonal property apart from the folder/label: Outlook flagged / Gmail starred
There is a simple mapping between Gmail star and Outlook flag.
All of the at least twelve possible Gmail flags maps to a single Flagged value in Outlook - I.e. Outlook cannot tell what the Gmail star is.
If Outlook flagsthe message, it displays as Gmail yellow star. Outlook cannot control what Gmail star is used.
There seem to be synchronization issues between Outlook flagged and Gmail starred. e.g. I have not yet managed to have Outlook clear a flag that Gmail has set, and have that reflected on Gmail's web interface.
I am using MailKit as SMTP client to send emails.
I see there are two properties XMessagePriority and MessagePriority
what is the difference between these two and does one override the other?
One of the things you'll discover in the world of email is that there are legacy bits and pieces here and there.
This is one of those.
The original "email" specification did not specify a header for designating the priority of a message, so some mail clients began using a non-standard header called X-Priority (non-standard headers always start with X-). (Note: Microsoft products began using X-MSMail-Priority instead. There's some info about it here: https://learn.microsoft.com/en-us/openspecs/exchange_server_protocols/ms-oxcmail/2bb19f1b-b35e-4966-b1cb-1afd044e83ab)
Later, many X.400 message properties were mapped to message headers, including a Priority header but the values were not the same as the ones used in the X-Priority header.
Once a piece of software begins doing something and users like/demand the feature, other software begins to adopt that way of doing something in order to compete. Then, once a real standard is defined, if it isn't identical the the way it was already being done, there are now 2 ways of doing the same thing and software must do both.
You might be thinking, "but doesn't that mean software has to implement both? And if they implement both, why not drop the old way?"
Because old software still exists out there that can only handle the old way of doing things, so in order to be compatible with that older software that may exist, the legacy way of doing things persists.
(And by "mail software", that includes more than mail clients like Outlook or official mail server software like Exchange - it also includes automated shell scripts that admins wrote decades ago that are probably still running because no one has ever bothered to update them if they even know that anything should be updated).
does one override the other?
Yes. No. It all depends on the implementation of the receiving mail software.
Is it possible to create a new file (simple text file) and store it within an IMAP folder?
So e.g. my program analyzes the mails and wants to store a file with statistics in an IMAP folder, so it's stored on the server and not locally so that you can get the information through all platforms. Of course this may be a misuse of these folders (normally they are supposed to store emails), but exists a functionality in IMAP that lets me upload whatever I want to specific folders on the server?
What you want is the METADATA extension. Not very widely supported, but perhaps your goal is a server that happens to support it.
The people who voted your question down should do penance by reading RFC5464.
my last question
I write collector for imap (perl). There was a question, if the user marks a message as deleted, what should happen with the message from which I gathered all? Remove it completely, or move in a trash, or create a new archive folder for these messages? If I flagged /Deleted gmail and yandex, for example, removes it completely, my work mail service move letter to trash folder. What is the correct implementation?
It is unclear what you are doing. Presumably, you're working on a program which serves some purpose. What is that purpose?
If you are writing some tool which accesses an IMAP account as a client, then you should not handle any sorts of policies such as "what to do with deleted messages". Some other IMAP client marked that message as \Deleted, and the server decided that it's good to implement IMAP's usual two-phase deleted with \Deleted and EXPUNGE. There are, as you already mentioned, other possible policies where the server might auto-expunge upon seeing the \\Deleted flag getting set, perhaps by moving the e-mail to a virtual Trash folder, etc. As an observer of an IMAP account, though, you are most likely not suppossed to care.