I'm building a service which scans peoples email for specific PDF attachments and indexes them. I implemented this with OAuth2 for Gmail using their extensive Gmail API which works perfectly fine.
I now want to implement the same for outlook/live/hotmail. So I searched around, and I mainly read that you can "Connect to Outlook.com IMAP using OAuth 2.0" (tutorial here). The thing is that this implements a full IMAP connection. As far as I know this is more meant for aftermarket applications with which the user can view and send his email, not for applications which need to download some email in the background (like mine).
I haven't worked with IMAP from within code, but the main problems I see with this now is that:
If I read emails they will be set to "read" in the inbox of the user, which I obviously don't want (I don't want to interfere with the normal email usage of the user).
I will need to either stay connected with all email inboxes, or constantly loop through all email inboxes to get new emails.
My questions are actually;
Is there no other way than IMAP to get users outlook.com email?
Or are my problems actually not problems and should I just create an IMAP "receiver" for all the outlook email accounts?
In answer to point #1, according to Max, you can use body.peak to avoid this.
In answer to point #2, according to triplee, you can connect to all of the accounts and then use an IMAP extension called IDLE.
In answer to point #3, I found the Outlook REST API which includes options to read the email and you can call GET https://outlook.office.com/api/beta/me/messages on a regular basis to get all messages in the user's account or you can call GET https://outlook.office.com/api/v1.0/me/messages on a regular basis to get all messages in the user's inbox. You can find a full list of the REST APIs operations here.
In answer to point #4, I believe that some of your problems are problems and others aren't. I believe that you don't have any technical problems. However, I believe that you are correct that IMAP is not designed for applications like yours as is shown in the Wikipedia article on IMAP which reads (bold is mine),
In computing, the Internet Message Access Protocol (IMAP) is an Internet standard protocol used by e-mail clients to retrieve e-mail messages from a mail server over a TCP/IP connection. IMAP is defined by RFC 3501.
Also, in RFC 3501 which defines the original IMAP protocol it refers to email clients which from the absract reads (bold is mine),
The Internet Message Access Protocol, Version 4rev1 (IMAP4rev1) allows a client to access and manipulate electronic mail messages on a server.
Related
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 4 years ago.
Improve this question
In Cuba, web access is extremely censored, so I created a tool that allows more than 50,000 people to browse the Internet through email. Cubans send me an email with an URL in the subject line, and I email them back with the response. Read more at https://apretaste.com.
It was working like a charm, till the communist government of Cuba started blocking my emails. My solution was rotation.
I started with Amazon SES, and I was changing the domain each time it was blocked, but Amazon adds a header to all emails, and once they blocked the header no email from SES was able to reach Cuba any more. The same happened with Mailgun and others, they all add headers.
Currently I am creating Gmail accounts and sending via SMTP, but Google blocks me for no reason and only allows to send 100 emails a day per account. Also I can only create few emails using the same IP address/phone, so I was forced to use anonymous proxies and fake Chinese phones. Now I am fighting a war on two fronts.
An email can be blocked by three parameters: IP address, domain, and email address.
It will be terrific if I can set up my own Postfix server at a VPS that auto-rotates the IP address. Even better if I can simulate "gmail.com", to avoid purchasing a new domain every day.
All the intents to create what I call "the ultimate sender" just either reach the spam folder or add unwanted headers making it too easy to block. I feel exhausted. I hit a knowledge barrier here.
I know I am crossing to the dark side, but this is for a very good cause. Thousands count on this service as their only source of unbiased news, social network and to feel part of the 21st century.
Can you please help me implementing "the ultimate sender", or pointing to another solution that I may be missing?
I have a few suggestions for you.
The first one relies on The Onion Router also known as Tor.
Since you are crossing to the dark side, why not also take a look into the darknet?
Take a look at this list of Tor email providers. If you have your own email server that can be accessed through Tor, it becomes much harder for anybody to stop people from using this service. After all, Tor was developed to offer people uncensored access to the web.
You can read about Tor in detail here, it uses Onion Routing and this is how you would set up your server to use Tor.
Here is an example how you could use it:
The steps that involve the setup, receiving an URL request and sending back the reply are as follows:
Set up an email server.
Configure your email server to use Tor.
Publish the public service name. (e.g. "duskgytldkxiuqc6.onion")
Deploy a client that takes the service name and a URL, and let it send an email with a request to your server.
The client now waits for a reply.
You send a reply and the client receives it.
You can change your service name on a regular basis, but you need to make it accessible to those who will use this service.
Having an own email server means being able to control the email header.
Here is one example how you could make use of it:
Configure your email server so that it receives and recognizes
emails which contain the requested URLs.
Before you send a reply modify the email header so that it shows a random IP address and a random sender email address including a random domain name.
Send your reply.
Sending an email that way means that you cannot be replied back to. But since your reply already contains the requested information there is no need to.
I hope this helps.
Crowd source it.
Find a way that volunteers can send some emails for you. This is the only long term approach that I can think of. A simple web interface with mail to links would be be enough to get started although there are other potential problems with this approach too.
Because you are talking about low numbers of users, you could also use crowdsourcing to create the single email address per person approach. They can create an account on a specific set of email providers and give you the credentials. This would allow the single email per user approach or could be used to rotate through a large set of email accounts to send emails.
The simplest solution is perhaps to set up a local SMTP server on your own computer. You don't even need a server per se.
https://sourceforge.net/projects/winsmtpserver/
There are many other such applications. They are usually used to test SMTP functions during local development, but there is nothing against actually sending spam through them.
I know this would be quite a large task, but how about pairing the users with one or just a few emails so they always receive an email from that email.
I'd assume people wouldn't have more than 100 queries per day, if so they could start receiving them from a backup email
I'd imagine it would look less suspicious for them to appear to be in constant contact with one unique email rather than 50,000 being in contact with one
I know this would be a huge undertaking, but I feel like it solves your issue.
Since the users are willing to receive emails form you then your shouldn't be blocked.
When you mentioned you are getting block does it mean your mail is going in spam or is getting lost in between sending and receiving or it is getting bounced back??
My suggestion would be to setup your own mail server and follow as below:
-Get approx 25 or more ip to rotate. (IP is the most imp part which is tracked and is accountable for the reputation of your mail server)
Don't start sending emails in bulk from the word go it is better to gradullay increase the email volume so that mail server reputation nicely built
keep changing the format of the email often
encourage user to add yourself to there contact list
your best part is user are willing to receive emails from you and you would reply to revived email is the USP of yours but still i will recommend you to register for FBL so that you would know which user is reporting you as spam and you can remove him from your list and never send him email again.
using best practice to send emails like dkim, SPF, dmarc are also vital.
Hope my answer was of some help to you. If you need step by step guide to step up mail server let me know.
My friend, do you remember what made Hillary Clinton lose the last elections to Trump?
It was the "mail" affair. And what was it? People discovered she shared confidential information through a non-official, non-governmental email account (i.e., she used some Gmail, Yahoo or another of a kind). Until here, nothing new with direct relation to your matters. But there is an small particularity on this history, and this can put, maybe not a solution, but maybe a light on a new path you could follow: Clinton actually never sent those emails; the email account she used had the password shared and the communication between people (Clinton-someone) occurred only using the drafts of the account.
How? One side logs in and accesses the drafts folder. There he/she reads the last message and edits it, cutting and writing new data - then save the draft message. On the next turn, the other side of the communication line logs in and do the same. And so forth, so never really sending those messages, but instead just updating the drafts (this "Hillary" method does schooled people... Dilma Rousseff, impeached ex-president of Brazil, actually did this method down there in Brazil too).
So, maybe if you could establish a pact with your user that he/she doesn't delete the account's password, you could pass those information by this method - without "really" exchanging emails. Maybe a "parent" email account (some that could reset a lost password) could be useful too.
Alternative: aren't you able to contract a regular HTTP webserver? You could rely on FTP to publish data to your user, he/she asks for it and you publish a page with that content.
Salvi, have you tried something with Telnet? OK, we are talking here about a text-only environment, but if nothing more would rest in the future, this could be better than nothing. Maybe you could implement a podcast-like, or push-like service based on it. Look what people do with it with references to your walk on the dark side...
If in Windows, open your command prompt.
Type telnet and press Enter.
Type "o" without quotes and press Enter.
Type "towel.blinkenlights.nl" without the quotes and press Enter.
i have an idea about system where users respond to website's messages using the emails they received.
Is it even possible to have multiple imaginary emails with hash in the e-mail's name field(f.e., 1h2149g0as1gasd9123#mysite.com ) and those all imaginary e-mails just forward content to messaging#mysite.com e-mail. And afterwards we are able to parse the contents and know for which conversation the message was sent.
The emails are hosted in Google service. Could not find any information about this in internet.
Such option would be excellent, because there would be no need of including the identificator in e-mail's content which might be deleted if no quoted text is in replied e-mail.
I'm open to any other suggestions that would fit in this situation.
When using Google's service, you can add tags after the e-mail address using a +. For example...
test#domain.com
test+12345#domain.com
test+wuishw78#domain.com
will all go through to test#domain.com, and you can then look at the address.
Note this is not supported by all e-mail systems, so while this works on Google's hosted e-mail, it may not work on others.
I have a project to send some email to end clients. My client need to know what exactly "Email Client" they use to read the mail. I know a hidden can get the open event and even the user agent they use, so by parsing user agent i can get most email clients info.
But it's hard to detect some popular web mail clients like "Gmail", "Hotmail" and "Yahoo mail". Because user agent return is only the browser user agent string.
Edit: i think i need a result more like this:
here
You will not be able to perfectly detect the e-mail client your users are using.
In E-Mail headers some programs choose to include the X-Mailer tag, which tells you exactly with what program and version your user is sending the e-mail - of course that can be faked. Not all programs use the X-Mailer tag, I e.g. couldn't find it in a mail sent with Microsoft Outlook 2010.
Besides that you could do some guesswork by the Received from tag in the e-mail headers, but in the end you can use SMTP and POP3 with most webmailers like GMail or Yahoo. That means even though your e-mail is sent via servers from google.com, the originating client could still be Outlook or Thunderbird and not GMail itself.
Maybe we can help you if you better if you could tell us why exactly your client wants to know the programs the users use to read their mail? Probably to tune the appearance of newsletters?
I know this is a really old topic, but the most reliable way to detect webmail client for gmail, hotmail, outlook.com and office365 is to use a tracking pixel. What you will want to do is geo-locate the IP address and you'll find gmail all comes from mountain-view and microsoft based products from redmond.
I haven't validated this with non-US users, but I'd imagine the caching services they use will all be in the same place.
Cheers
Given multi-tenant application, How are vendors implementing email notifications from an email account setup and programming perspective:
Sending emails could come from a generic account: eg notifications#VendorName.com or noreply#VendorName.com, this seems reasonable considering reply addresses and lilnks can be contained within the email contents.
Receiving Emails: How would an application receive email, for instance; to generate support tickets or assign comments in an email to a project/task. I have seen ID's within the subject and some reply to addresses containing the account name eg: notifications#AccountName.VendorName.com
I realise one can programatically connect to a pop3 server and receive emails and look for the IDs with the subject, but is there a way of setting up and receiving email to a single pop3 account from multiple sub-host name email addresses (not sure on terminology there) eg: noreply#AccountName1.VendorName.com or noreply#AccountName2.VendorName.com and check the Account Name from the address? (similar to checking subdomains on a URL)
Any practices, experience, comments or sughestions?
(not sure its relevant, but using C# asp.net-mvc and services etc)
For sending notification emails, we have a notification send to address associated with each account and simply send from our domain to that address. Our from address is monitored and replies end up in the CSR work queue.
For inbound emails, we use FogBugz (from the makers of Stack Overflow) for case tracking. That accepts new cases via email (e.g. cases#mycompany.com). Tickets are auto-created from the email. My only complaint there is that the customer needs to check an obscure link for case updates (no "my cases" web portal, but maybe that will come out in an upcoming version of FogBugz).
We have a custom field in FogBugz to indicate the customer the ticket is from. We could theoretically write a plugin to FogBugz that auto-assigns that using the senders domain, but I guess the CSR's haven't complained loudly enough yet :-)
We (at muHive) are an inbound email/social conversations management product. If you are looking at a handling inbound email or social media conversations from customers, we have an impressive toolset.
For our own outbound needs, the simplest way is to use an Email sending API. Don't bother with SMTP sending by yourself. We use Amazon SES and have also tried Sendgrid which gave us additional benefits like delivery status and email parsing.
There are two ways in which you can handle multiple accounts to a catch all email address. If your target system can differentiate between different customers and assign tasks to the correct representatives based on either the content/sender, ask all your customers to send an email to support#company.com.
As you rightly said, you could also create *accountName_support#company.com* email addresses and use different accounts on whatever CRM/Support solution use to manage these emails.
Another approach is to have your customers send you an email to support#company.com and you use a rule based system (like muHive) to forward these mails to the appropriate account executives based on the customer/account who sent the mail.
A number of applications have the handy feature of allowing users to respond to notification emails from the application. The responses are slurped back into the application.
For example, if you were building a customer support system the email would likely contain some token to link the response back to the correct service ticket.
What are some guidelines, hints and tips for implementing this type of system? What are some potential pitfalls to be aware of? Hopefully those who have implemented systems like this can share their wisdom.
Some guidelines and considerations:
The address question: The best thing to do is to use the "+" extension part of an email (myaddr**+custom**#gmail.com) address. This makes it easier to route, but most of all, easier to keep track of the address routing to your system. Other techniques might use a token in the subject
Spam: Do spam processing outside the app, and have the app filter based on a header.
Queuing failed messages: Don't, for the most part. The standard email behavior is to try for up to 3 days to deliver a message. For an application email server, all this does is create giant spool files of mail you'll most likely never process. Only queue messages if the failure reasons are out of your control (e.g., server is down).
Invalid message handling: There are a multiple of ways a message can be invalid. Some are limitations of the library (it can't parse the address, even though its an RFC valid one). Others are because of broken clients (e.g., omitting quotes around certain headers). Other's might be too large, or use an unknown encoding, be missing critical headers, have multiple values where there should only be one, violate some semantic specific to your application, etc, etc, etc. Basically, where ever the Java mail API could throw an exception is an error handling case you must determine how to appropriately handle.
Error responses: Not every error deserves a response. Some are generated because of spam, and you should avoid sending messages back to those addresses. Others are from automated systems (yourself, a vacation responder, another application mail system, etc), and if you reply, it'll send you another message, repeating the cycle.
Client-specific hacks: like above, each client has little differences that'll complicate your code. Keep this in mind anytime you traverse the structure of a message.
Senders, replies, and loops: Depending on your situation, you might receive mail from some of the following sources:
Real people, maybe from external sources
Mailing lists
Yourself, or one of your own recipient addresses
Other mail servers (bounces, failures, etc)
Entity in another system (my-ldap-group#company.com, system-monitor#localhost)
An automated system
An alias to one of the above
An alias to an alias
Now, your first instinct is probably "Only accept mail from correct sources!", but that'll cause you lots of headaches down the line because people will send the damnedest things to an application mail server. I find its better to accept everything and explicitly deny the exceptions.
Debugging: Save a copy of the headers of any message you receive. This will help out tremendously anytime you have a problem.
--Edit--
I bought the book, Building Scalable Web Sites, mentioned by rossfabricant. It -does- have a good email section. A couple of important points it has are about handling email from wireless carriers and authentication of emails.
You can set the address that the email is sent from, what will be put into the To: address if someone just presses 'Reply-to'. Make that unique, and you'll be able to tell where it came from, and to where it must be directed back to.
When it comes to putting a name beside it though '"something here" ' - put something inviting to have them just reply to the mail. I've seen one major web-app, with Email capturing that has 'do not reply', which turns people off from actually sending anything to it though.
Building Scalable Web sites has a nice section on handling email. It's written by a Flickr developer.
(source: lsl.com.au)
EDIT: I misunderstood your question.
You could configure your email server to catch-all, and generate a unique reply-to address. E.g. CST-2343434#example.com.
A polling process on the server could read the inbox and parse out the relevant part from the received email, CS-2343434 could mean Customer Support ticket ID no. 2343434.
I implemented something like this using JavaMail API.
Just a thought.
The best way to achieve this will be to write a window service that acts like a mail client [pop3 or imap]. This windows service should execute a timed action triggered by a timer, which connects to the mail server and polls the server for any unread message(s) available in the email inbox. The email ID to check for is the email ID on which the users will give their input on/to. If the windows service client finds that there exists any new mail(s) then it should download and filter the email body and push further for processing based on the user input in the email. You can host the input processing in the same windows service but it is not advisable to do so. The windows service can put the inputs in a special application directory or database from where your main appication can read the user inputs received in email and process them as needed.
You will be required to develop a high performance TCP/IP client for doing so. I advise you not to use the default .Net library due to performance issues, instead use one of the best availabel open source TCP/IP implementations for .Net like XF.Server from kodart. we have used this in our applications and achieved remarkably grear results.
Hope this helps..
Bose has a pretty great system where they embed a Queue and Ticket ID into the email itself.
My company has the traditional Case # on the subject line, but when CREATING a case, require a specific character string "New Case" "Tech Support Issue" on the subject line to get through the spam filters.
If the email doesn't match the create or update semantics, the autoresponder sends an email back to the recipient demonstrating how to properly send an email, or directs them to our forums or web support site.
It helps eliminate the spam issue, and yet is still accessible to a wide technical audience that is still heavily email dependent.
Spam is going to be a bit of a concern. However since you are initiating the conversation you can use the presence of your unique identifier (I prefer to use the subject line - "Trouble ticket: Unable to log into web...[artf123456]") to filter out spam. Be sure to check the filter on occasion since some folks mangle the subject when replying.
Email is a cesspool of bad standards and broken clients. You need to be prepared to accept almost anything as input. You will need to be very forgiving about what kinds of input are tolerated. Anything easy for you to program will likely be difficult for your users to use correctly. Consider the old mailing list programs that require you to issue commands in the subject line. Only hardcore nerds can use those effectively. And some of those trouble-ticket CRM things you mentioned have bizarre requirements, such as forcing the user to reply between two specific text markers in the text. That sort of thing is confusing to people.
You'll need to deal with email clients that send you formatted text instead of plain text. Some email clients still don't handle HTML properly (cough GMail) so your replies will also need to be designed appropriately. There are various ways in which photos might be "uploaded" via email as well, especially when mobile phones are involved. You will need to implement various hacks and heuristics to deal with these situations.
It's also entirely possible that you will get email that is valid but unusable by the email parsing library you are using. Whether or not this is important enough to roll your own will be a judgement call.
Finally, others have mentioned using specific email addresses to uniquely identify a "conversation". This is probably the easiest way to do this, as the content of the mail will often not survive a round trip to a client. Be prepared, however, to get mail to old IDs from old customers who, instead of opening a new ticket somehow, reply to an old ticket. Your application will probably need some way to push emails with an old ID into a new case, either manually or automatically. For a CRM system it's very likely that a user would reply to an old email even if you already sent him a new email with a new ID in it. As for whether you should use some.email.address+some.id#yourdomain.com or just some.id#yourdomain.com, I'd go with the latter because the plus-sign confuses some email clients. Make your IDs guids or something and have some way to validate them (such as a CRC or something) and you'll get less junk. Humans should never have to type in the GUIDs, just reply to them. The downside is spam filtering: a user's computer might view such email addresses as spam, and there wouldn't be an easy way to whitelist the addresses.
Which reminds me: sending email these days is full of pitfalls. There are many anti-spam technologies which make it extremely hard for you to send email to your customers. You will need to research all of these and you need to be careful, and do some testing, to ensure that you can reach the major email providers. A website like Campaign Monitor
can help you if you are sending email.