I'm looking for a way to know definitively if an email I receive is in response to a specific email I sent. I manually set the Message-Id of the outgoing message using make_msgid, store this value, and then check the In-Reply-To of an incoming email to determine if it is equal to the original Message-Id I sent.
This approach is basically what is suggested here in this very helpful answer by Mohammad Eghlima.
But I wonder if this approach is "foolproof" and if there is a better way to accomplish this? For example if there are some clients other than outlook, gmail etc. that do not follow this convention of setting In-Reply-To to the Message-Id of the original mail for replies, or if they set their own Message-Id for some reason (ex. Gmail does this if it determines the existing message id doesn't follow RFC standards)?
I've seen some other answers mention other potential methods to accomplish what I'm trying to do - for example, here but most of these questions/answers are from 10 years ago so I'm wondering if there is a better way to accomplish this now.
No, nothing is entirely foolproof. Junior PHP programmers write new email-sending code every day and none of it conforms to any particular set of conventions or RFCs. And then there's Microsoft and Google ... Oh, you are already familiar with them.
There have been no significant developments in email standardization on this particular front in the last decade, so advice from 10 years ago is by and large still relevant.
If anything, the field has been polarized by Microsoft and Google plunging ahead to "innovate" in various aspects of what may charitably be characterized as usability improvements over traditional email, but the motivation has often been to silo in users to prefer or be forced to use their solutions, not standardize anything.
(The efforts to improve e.g. email security through DMARC etc has been better coordinated and standardized.)
The post you link to basically summarizes the information from D. J. Bernstein's excellent email reference resource https://cr.yp.to/mail.html; see in particular the threading conventions at https://cr.yp.to/immhf/thread.html
---+ BRIEF:
using Outlook 2016 as my email client, connecting to Gmail account
Q: what features allow orthogonal properties to be specified independently for messages?
AFAICT the only orthogonal features are read/unread, and only a single type of Outlook/flag mapping to single type of Gmail star.
Which is leading me to consider
Flagged/starred=not-handled-yet
have a (Gmail) rule set the Gmail star as messages are received
clear the flag/star when you have handled it
which is slightly safer than
Flagged/starred=deferred
new messages in inbox will be unflagged/unstarred and hence not deferred yet)
must take care to set the flag/star whenever deferring
must take care to set the flag/star when moving to a topic folder/label(and out of Inbox) or else unhandled messages may be lost
and must take care to clear the flag/star whenever archiving
and of course both of these policies conflict with historical/legacy uses of flags and stars
---+ LESS BRIEF:
I am (perhaps unfortunately) impelled[*] to use Microsoft Outlook as my email client on my PC.
More precisely Microsoft Outlook for Microsoft 365 MSO … which as far as I can tell is essentially Outlook 2016 with slight updates. NOT the web version.
I am connecting it to my Gmail account of many years.
This connection is IMAP/SMTP. I am individual, personal, no exchange servers in the mix etc.
My email workflow and organization is nothing very special. But I'll describe briefly, for motivation. My email workflow for many years has wanted to be something like the following [**]:
• Scan my Inbox
• if I can handle an email right away, do so
• if not, mark it as "Deferred"
I typically have several subfolders or labels under "Deferred", such as work/personal, urgent/soon/later. Sometimes I use GTD like classifications or lists. Sometimes I want to move or mark or label or tag emails as related to particular projects or topics. Sometimes I have rules that apply such marks or labels or tags or folders as email is received. Plus I tend to use Gmail labels or Outlook folders for topics, projects, etc. I really like using saved searches or search folders. I really liked being able to sort email, e.g. by subject, sender, thread, etc..
Basically, I want to be able to be able to mark or classify my email with several ORTHOGONAL or INDEPENDENT properties. and I want to be able to query combinations of these orthogonal/independent properties.
For example:
• email that is associated with a topic
• read/unread
• deferred/not
• handled
○ I'm finished with the email, etc.
And I want to be able to make queries such as
• "show me all email that I have not completely handled yet"
○ whether or not it is in my Inbox, or in a Deferred folder, etc.
Email systems have various ways of indicating such properties, orthogonal or not
• e.g. Outlook folders and subfolders
○ strictly speaking not truly orthogonal
• e.g. Gmail labels - which are truly orthogonal since you can have multiple labels per email
• Gmail stars
• Outlook flags
• Outlook categories
The kluge that maps Gmail labels to IMAP folders Give the Outlook users some control over Gmail's truly orthogonal labels. But AFAIK the Outlook user has no way to detect whether an email has more than one Gmail label, i.e. no way to look for the same message being in more than one Outlook folder. So I can't have Outlook do the equivalent of a Gmail search "in:Inbox AND label:Deferred AND label:Someday". Not by mapping Outlook folders to Gmail labels.
(If there were some way of having Outlook detect that a message has been copied to more than one folder, I think I would be done. Some cheap way. Email deduplication tools accomplish this in some sense, but they really want to what they think are duplicates, which are really Gmail messages with multiple labels.)
Outlook in its natural state has limited orthogonal properties in the form of flags and categories. Good enough for stuff like
• show me all the email in folder Topic-X that has category "Needs-Handling"
• show me all the email in folder Topic-X that is flagged …
other you quickly run out of categories and flags in a way that you do not with Gmail labels.
But when I'm attaching my Outlook email client to the Gmail message store (across IMAP, if it matters)
Outlook categories are not supported by Gmail. if you set a category in Outlook it does not appear on the Gmail message. it appears that the Outlook category may survive for a while, perhaps as long as it's in the Outlook cache of Gmail, but if you uninstall and then reinstall Outlook on a different machine, the categories are lost. AFAICT.
As far as I can tell the only orthogonal properties that are common to the intersection of Outlook on Gmail are
• the klugey Gmail label / Outlook folder mapping
• Gmail star mapped to Outlook flag
and only one flavor of star/flag. Gmail supports multiple colors of star, but Outlook only sees them as a single flag. and vice versa.
Q: are there any other orthogonal message properties that can be set and queried by both Outlook and Gmail?
I highly suspect not, but I just wanted to check.
---+ WORKFLOW given these limitations
messages received by Gmail
○ optionally labeled automatically by Gmail rules
○ and/or left in:Inbox
I scan Inbox (using either Outlook or Gmail clients)
○ handle immediately, and archive (remove the Inbox label)
○ or defer
○ or leave in Inbox as read/unread.
while I might defer by moving out of Inbox into the Defer folder-label or a subfolder, doing this does not allow me to query "Topic1 && deferred/…" in Outlook.
If Outlook-flag/Gmail-star is really the only orthogonal property, I think that I might have to use it to indicate deferred.
Two choices:
Flagged/starred=deferred
• new messages in inbox will be un-flagged unstarred and hence not deferred yet)
• must take care to set the flag/star whenever deferring
• must take care to set the flag/star when moving to a topic folder/label (and out of Inbox)
○ or else unhandled messages may be lost
• and must take care to clear the flag/star whenever archiving
or
Flagged/starred=not-handled-yet
• have a (Gmail) rule set the Gmail star as messages are received
• clear the flag/star when you have handled it
I think the latter choice, Flagged/starred=not-handled-yet, is "safer", in the sense that the most common user error would be to not clear the flag/star when it is handled or archived. this could be annoying, clutter up your searches for " work I haven't finished yet", but it doesn't lose anything. whereas the other polarity can quite easily result in work falling through the cracks.
Note: deferred is a subset of not-handled-yet.
The biggest problem with flagged/starred=not-handled-yet is that I, and probably many others, have historically used flags/stars to indicate things like " this is interesting", or "this is important". i.e. there is a semantic conflict between these historic uses and using flag/start to indicate deferred or not handled yet
Q: has anybody else used the flag/star=not-handled-yet approach? problems?
---+ MORE BACKGROUND
Note**: "My email workflow for many years has wanted to be something like the following [**]:…"
I say "wanted to be" because I've tried to get as close to this is possible, not always successfully.
I was most successful back in the the 20th century when using GNUS email on emacs. Procmail sorting into folders. I would only mark email as "read" when I finished processing it.
Unfortunately, more and more people need to know the distinction between the following three states:
"I have not seen/read your email yet"
"I have seen your email, read it but not yet handled it"
"I have both read and handle your email"
Also, way back in these GNUS/Procmail days, I was free to add my own headers to messages in my saved email tree. But this flexibility is not available nowadays in Gmail/Outlook. I think if I could add my own headers I would be logically happy enough, although the UI might be a bit unsatisfactory.
Note*: "I am (perhaps unfortunately) impelled[] to use Microsoft Outlook as my email client on my PC*."
First, because of what my physical therapist calls computeritis I am trying to use speech recognition to do as much as possible, to avoid typing. Speech recognition just works better for Microsoft Outlook than it does with Gmail.
Second, Gmail does not sort messages in folders or queries. IMHO sorting in other email clients like Outlook and Thunderbird essentially gives you the answer to queries that you do not yet know that you should ask. e.g. you might notice that you have suddenly received 100 messages from a particular sender on different topics. It would be better if you had groups sorts the could return message counts, but eyeballing is often good enough.
... Why use Gmail as the mail storage/server underneath the Outlook user client? why not move my email to a system where all of the Outlook features are supported? First, Gmail free is a good price, but Microsoft has similar. Second, mainly because I have used Gmail for a very long time, and fear the potential work or hassle of moving all of my historic email. and I do need to move all my historic email. Plus, with my luck I would move all of my email from Gmail to Microsoft, and then Microsoft would start charging. But third, I really do like the concept of Gmail labels. I am reluctant to move to a system that does not support such orthogonality. It is unfortunate that Gmail does not support sorting, and I don't know of any other email client that is speech friendly which accesses labels natively.
I spent almost 2 years trying to use Thunderbird rather than Microsoft Outlook as my email client, but I've recently stopped doing so. PRO: Thunderbird sorts like Outlook. CON: Thunderbird is not as speech friendly as Outlook, but I was able to work around much of its problems, so in this respect Thunderbird was slightly more speech friendly than Gmail. CON: Thunderbird interfaces with Gmail labels by this includes that Outlook does. PRO: there is a Thunderbird add-on that can at least show you all of the labels associated with message, although AFAICT no queries. PRO: Thunderbird produces HTML email that more of my correspondence in the open source/hardware community can understand, Whereas Outlook produces HTML email that many people find unreadable on non-Outlook email readers. CON: Thunderbird does not have highlighting (I kluged this, but it was not pretty). PRO: Thunderbird is open source so in theory you can fix things, write your own code, preferably as a add-on. CON: Thunderbird fairly recently underwent a major reorganization of its add-on architecture, and most of the historical add-ons that I want to use no longer work. And it seems that many of them cannot be made to work because of Thunderbird's newish security model. OVERALL, while Thunderbird might be a place to return to, I think I have spent enough time doing so for the moment.
I was and am still most hopeful about being able to return to an emacs based email reader like Gnus. PRO: although not really designed for speech, I have found emacs's command orientation very amenable to speech recognition. In some ways preferable to Outlook. CON: Gnus IMAP Is online only, not acceptable for somebody who needs to read email on planes when disconnected etc. CON: nearly every time I connected Gnus/IMAP/Gmail my emacs would be locked out for 15 minutes, half an hour, etc. CON: I spent some time working with offlineimap (flakey), and mbsync (syncs Gmail-IMAP-Maildir bidirectionally, and then Gnus/Emacs accesses using nnmaildir. but I kept running into Gmail account problems, bandwidth limits, and just plain flakiness. Overall, I get the impression that Google/Gmail are really pushing against non-Oauth authentication. and authentication in open source seems to be a bit of an issue; if an open source app has all of the keys inserts necessary to connect to Gmail Oauth style, AFAICT but might be a violation of the Gmail terms and conditions.
Bottom line: I was and am still hopeful about being able to return to an emacs based mail reader like Gnus accessing Gmail, weather across IMAP or Gmail's native API. off-line. Oauth. and in my dreams with support for Gmail style labels,
If the Gmail labels were exposed as headers, X-GM-LABELS, then I think that my need for orthogonality would be satisfied. I know that I can query such headers in Gnus, and I think they can be queried in Outlook. It remains my hope that I can get headers for Gnus/Emacs. But as far as I know Outlook does not have a way of obtaining such headers. and as far as I know MailDir is hostile the things that modify message content in this way.
All of this last section is just background.
For the moment, all I really want to know is if there are any other orthogonal features when Outlook accesses Gmail. Other than Outlook-flag/Gmail-star.
I will answer my own question, not because I want to do the Q&A, but to clearly state my present understanding, even though it is already mentioned my question.
As far as I can tell
Gmail by itself has the following user settable and query able message properties:
labels: text strings, hierarchical, otherwise fully orthogonal, or close
read/unread
important/not
stars: I believe 12 standard, yellow/Orange/red/purple/blue/green, red-bang!, orange-guillemet, yellow-bang, green-check, blue-info, purple question
categories: Primary, social, promotions, updates, forms, reservation, purchases
I would not be at all surprised if all of these are implemented as labels under the hood, mapped to different icons and operators like labels/category:
Outlook natively, i.e. with exchange, has the following user settable and query able message properties:
folders... not exactly mutually orthogonal, since can only be in a single folder at a time, but folders are orthogonal with the other properties
read/unread
flags - In different flavors ranging from simple flag/unflag, followup flags with a text string, date flags, ...
categories: a limited number of named & colored categories
importance: normal/high/low
sensitivity
And probably some user definable fields.
Although it would seem that Outlook has more orthogonal properties than Gmail, and it does, doesn't mean it's more powerful in terms of querying and setting. it mainly means that some of them have predefined behaviors and GUI features like icons.
However, the intersection of the Gmail and Outlook features is quite narrow:
Gmail does not support Outlook categories, Importance, sensitivity...
AFAICT there is only one orthogonal property apart from the folder/label: Outlook flagged / Gmail starred
There is a simple mapping between Gmail star and Outlook flag.
All of the at least twelve possible Gmail flags maps to a single Flagged value in Outlook - I.e. Outlook cannot tell what the Gmail star is.
If Outlook flagsthe message, it displays as Gmail yellow star. Outlook cannot control what Gmail star is used.
There seem to be synchronization issues between Outlook flagged and Gmail starred. e.g. I have not yet managed to have Outlook clear a flag that Gmail has set, and have that reflected on Gmail's web interface.
This Odoo company we're working with basically sends a lot of e-mail. One e-mail thread can turn into 100+ e-mails with different people brought into the conversation (CC'd) at different times. Due to the complexity of their e-mail management, they want to use their Gmail interface (Google Hosted) and CC an e-mail into odoo and they want it to get tracked in a thread. I've basically already done that... they have an e-mail like odoo+res.partner-432#domain.com (although it's hashed to not be easily readable) - they CC this and the full body thread gets included in chatter (mail.message) under respective model / id.
The challenge with this: the chatter messages can get huge very fast, due to their e-mail messages (because each e-mail includes main reply, and all previous history on thread). I've looked into some systems that have a "reply above this line" - and it just takes the latest message. And in those systems, eg. ticketing systems such as Zendesk, help scout, I believe the teams are using the ticketing system (not a gmail) and thus there is much more control over the inbox and incoming email (not to mention, those e-mails are usually 1-to-1, not including groups).
My questions:
Is there any other workaround that you see here to have odoo pull in only the last e-mail reply and not the full e-mail thread? I could probably build something like this: https://github.com/zapier/email-reply-parser - and hook it into odoos e-mail parsing, but that works on text format e-mails only (not HTML)... only. So it's not bulletproof, and I'm not sure it's worth it.
Even if this client DID use odoo 100%, I still don't think it would be possible to get it to work the way they want without major customizations (eg. Odoo's default behavior is to include all past e-mail threads)
I'm curious if anyone here see's any other solutions, otherwise – I doubt there is something here I haven't seen. :) (But very open to be proven incorrect!)
I'm writing a piece of software that will run on computers as well as phones.
The service uses an HTTP API for communication and will be published over the local network using Zeroconf.
Initially I published my service using _http._tcp. as the service type but I quickly discovered that both my NAS and my music receiver(!) also broadcasts themselves with that exact service type.
So the question now arises how to differentiate between my service and other services that are using HTTP.
Alternatives
Using a different service type
The is certainly the most certainly the easiest way and (almost) guarantees no other services will be picked up.
However, according to Apple1 new services should be registered with IANA. This is obviously not required but seeing as they recommend it it feels like it would be the wrong way to do it
Using the TXT record
Apple2 describes the TXT record like this:
When a service is registered, three related DNS records are created: a service (SRV) record, a pointer (PTR) record, and a text (TXT) record. The TXT record contains additional data needed to resolve or use the service, although it is also often empty.
The certainly feels like it could be the right way to do it, but I'm still not sure and it's hard to find a description of what the field should contain.
My first though would be to put something like <service_name>-<version> which will then be parsed to see which service it actually is.
My NAS seems to use this for identifying model and version numbers.
Try talking to the service
After finding a service one could always perform a HEAD request on a known endpoint and look for a known header set by the service.
This feels like a fairly slow approach and who knows what making a HEAD request to my receiver will do.
And just to be clear, this question has nothing to do with a specific language or framework, it's about the concepts of zeroconf.
I could show some code but I don't see how that would help.
First, does the service you're advertising actually meet the qualifications for _http as defined by RFC 2782. Specifically- is it not just using HTTP for a transport but is also:
can be displayed by "typical" web browser client software, and
is intended primarily to be viewed by a human user.
If no, register your own service type (there are a couple other services that use HTTP as a transport but don't meet those qualifications so they have -http as a suffix to the service name, see pgpkey-http, senteo-http, xul-http).
If yes, there are a couple ways to go depending on how strict one's interpretation of the RFC is. The least strict being just adding a TXT record as you've already noted in your question. iTunes registers itself with a TXT record in the format iTSh Version=196618.
If you're feeling a little more strict, the RFC only explicitly states that the u=, p= and path= TXT records exist for HTTP. Perhaps someone can chime in on this, but I haven't seen much discussion on whether adding TXT records to already existing entries is frowned upon or not. So with that, the other way is to just an algorithmic instance name. For example, adding the suffix "-NicklasAService" to the device name. Hopefully giving it a unique name to the local network but still making it so that the service can be easily picked out by the PTR record by just looking for the suffix.
I'm building a REST service where I want to implement a way to deprecate certain URIs when they shouldn't be supported anymore for one reason or another. As functions are deprecated, they will be replaced by new ones that work in similar (but not identical) ways. This means that at some point, I will have to start responding with 410 Gone.
The idea is that all client software should be updated, and after say six months all users should have had the chance to upgrade. At this time, the deprecated URIs will start to inform the client that it's out of date, so that the client can display a message to the user. This time is not known in advance, though, and can't explicitly be written in the documentation.
The problem I want to solve is:
Is there an HTTP header field I should use to indicate that a certain URI will cease to work at a certain time and, if so, which?
This can't be the first time someone wants to solve this problem. Is there an unofficial header field already in use, or should I design my own? Note that I don't want to add this information to the content itself, as that would mean that every resource was changed and needs to be refreshed by the client, which is of course not what happened.
Strictly speaking, no. The resources should be driving your applications state, so if there is a change, the uri linking would provide the nessessary changes to your application.
For a HTTP header, you are free to add custom headers. Normally starting with X- but its important to know changes to uri's is only interesting to developers not users.