Just signed up a third party email marketing provider, when I provide the template they give me a small tag to place which they subsitute with a user specific unsubscribe link.
My concern is that the link is single click, there is no subsequent confirmation, etc.. and whilst I am all for easy removal, I worry that any combination of malware scanners, AV engines, spam scanners will follow the link and thus unsubscribe many legitmate users.
Is this the norm to have a single HTTP GET request unsubscribe a user?
How are other developers handling this issue?
Note: The provider in question is critsend
Interesting question. It’s not the norm. But it’s common with cautious email service providers. For example, MailChimp also has a 1-click unsubscribe for his freemium users. I’m not a big fan of that, too. (I’d prefer a prefilled form field, where the user confirms his wish to unsubscribe by clicking "submit".) However, I didn’t witness any problems using 1-click-unsub until now.
FYI, here’s a discussion addressing a similar topic (false positive double opt-in confirmations). You might also want to check out this article and this discussion (forum registration required).
The norm is once clicked, it goes to a form which you click a button to confirm removal. That's strange there are even single clicks avaliable
Any side effect changing HTTP GET request is non-conforming as far as HTTP is concerned. In particular, see this from RFC 2616, section 9.1.1:
In particular, the convention has been established that the GET and HEAD methods SHOULD NOT have the significance of taking an action other than retrieval. These methods ought to be considered "safe". This allows user agents to represent other methods, such as POST, PUT and DELETE, in a special way, so that the user is made aware of the fact that a possibly unsafe action is being requested.
It would be more standard to put the actual unsubscribe behind a form submission to cause a POST.
I know Campaign Monitor has built in procedures to catch non-user unsubscribes. Not sure about critsend.
Related
I have read a few questions on here about e-mail clients prefetching URLs in e-mails. An answer to this seems to be to add a new confirmation page, where the user has to click a button to confirm the desired action.
But, this answer states the following:
As of Feb 2017 Outlook (https://outlook.live.com/) scans emails
arriving in your inbox and it sends all found URLs to Bing, to be
indexed by Bing crawler.
This effectively makes all one-time use links like
login/pass-reset/etc useless.
(Users of my service were complaining that one-time login links don't
work for some of them and it appeared that BingPreview/1.0b is hitting
the URL before the user even opens the inbox)
Drupal seems to be experiencing the same problem:
https://www.drupal.org/node/2828034
My major concern is with this statement:
As of Feb 2017 Outlook (https://outlook.live.com/) scans emails
arriving in your inbox and it sends all found URLs to Bing, to be
indexed by Bing crawler.
If this is the case, any URL in an e-mail meant to confirm an action, e.g. confirming a login, subscription, or unsubscription, can end up searchable in a search engine, if that's whats meant by indexed in the quote above. In this case, it's Bing. Not even a dedicated confirmation page where the user confirms the desired action truly mitigates this.
Scenario #1
If I email the user a login link with a one-time token in the URL, that URL will end up in Bing. This token will have a short lifetime, lets say 5 minutes, so I doubt anyone will manage to search on Bing and find the URL before the user clicks it or it expires.
Scenario #2
The user gets an e-mail with a link to confirm a subscription. This link is perhaps valid for 24 hours. This might(?) be long enough for someone else to stumble over the link on a search engine and accidentally (or on purpose) confirm the subscription on behalf of the user.
Scenario #2 is not uncommon, it's even best practice to use double opt-in as far as I am aware.
Scenario #3
Unsubscribe URLs in the bottom of newsletters. Maybe valid for forever? You don't want this publicly searchable in an search engine.
Assume all the one-time confirmation links land on a confirmation page where the user confirms the desired action.
Is it truly the issue that URLs in e-mails are indexed by search engines, at least Bing? And will they actually end up publicly searchable? If not, what is meant by indexed in the quote above?
I'll add for the sake of completion that I don't think I've had much of a problem with this in my own use of the web, so my gut feeling is that this is unlikely the case.
Is it truly the issue that URLs in e-mails are indexed by search engines, at least Bing?
I can't definitely say if they are being indexed or not, only Bing could answer this question, but they are surely being visited, at least with a simple GET request. I just tested this sending myself a link to a page on my website that logs the requests that are made against it, and indeed I'm seeing a GET coming from 207.46.13.181 (reverse DNS says msnbot-207-46-13-181.search.msn.com), which suggests that an automated program from search.msn.com is crawling the link. This leads me to believe that yes, they are trying to index the link's content somehow, but it's only my opinion really.
And will they actually end up publicly searchable? If not, what is meant by "indexed" in the quote above?
Well, again, impossible to say unless you work for Bing. In any case, "indexing" means exactly what you think it does: parsing the content of a page to potentially include it in search results.
The real question here is: does this somehow represent a security problem or will it compromise my website's functionality?
It surely has the potential to: if your confirmation/reset/subscription/whatever process only relies on a single GET request with the appropriate GET parameter, then you should definitely revisit the strategy, as it obviously allows anyone to perform the action (even maliciously for example enumerating possible IDs for your GET parameters).
If the link you are trying to send contains sensible information or can be used to alter important data for an user of your website, then you should at least put it behind a login page only giving access to the interested user. This way, anyone who wants to access it (including search engines) will be redirected to a login page if not already logged in.
If the link you are trying to send is just some kind of harmless confirmation link (e.g. subscribe/unsubscribe from a newsletter), then at least use a form inside the web page to do the actual confirmation through a POST request (possibly also using a CSRF token), otherwise you will unequivocally end up with false positives.
I'd like to provide a single-click "Unsubscribe" links in the footer of the emails my service sends.
Obviously, many spam scanners will scan emails, and will follow any links found in the emails to scan their contents for malware. A workaround I have used so far:
If the "Unsubscribe" page is requested via HTTP GET, it renders a simple confirmation form and a bit of JS that submits the form on page load
If the "Unsubscribe" page is requested via HTTP POST then we unsubscribe the user
This way, the user will usually only need a single click on the form and they will get a "You have been unsubscribed" message. If they have JS disabled, they can still manually submit the confirmation form.
Now the problem is, some scanners like Office365's ATP will open the pages, and execute JS inside them. By executing JS they submit the form and cause user to be auto-unsubscribed.
I've considered adding checks to the auto-submit JS logic:
don't auto-submit for specific user agents
don't auto-submit for specific client IP ranges
trigger the auto-submit on mouse move event
But these all seem like brittle methods, hacks at best, that are bound to break as email scanners change their tactics.
I'm sure this problem has bit many people before me. Are there known reasonable workarounds, aside from just giving up the single-click functionality?
PS. I have added support for RFC 8058 but users are still going to click links in the footer.
This is a topic of ongoing debate at M³AAWG (The Messaging, Malware, and Mobile Anti-Abuse Working Group). It's a mess and there are no easy solutions. It sounds like you're doing everything right, but some anti-spam systems are a little too aggressive.
The big issue is that anything you can do can also be done by an abusive marketer or spammer.
The best proposal I've heard is just to put a timer on the action. Add a captcha for users that unsubscribe within 5 minutes of delivery and remove the captcha afterwards. (Do not implement this for your RFC 8058 List-Unsubscribe-Post link.)
My next favorite proposal is to add a canary link to the message. This should be invisible to human readers. If it is followed, it reverts recent click activity from that IP and bans the IP from action triggers for a time.
I like your ideas too, just make sure that if Javascript is disabled, the user can still unsubscribe after a confirmation button click.
There's a part of me (warning, I'm an anti-spam researcher) that wants these false positives. Hopefully that will teach my peers that they're doing such a bad job and that these escalations will keep coming to them. From your perspective, you get to pass the buck (though you will lose a few subscribers in the process).
Spam detection systems must be careful to avoid subscription management links (at least until the bad guys start disguising their payloads as unusbscribe links).
Does SendGrid support double opt-in to Lists as a feature or is that something we will have to implement for ourselves?
https://sendgrid.api-docs.io/v3.0/contacts-api-recipients/add-recipients
It doesn't appear to me to be anywhere in the docs, but I thought I'd ask in case I missed it.
Not as of the current date; I asked their support staff and received the following answer:
Double opt-in needs to be implemented by you in the form/page you're subscribing your recipients. The confirmation email can be sent through SendGrid.
For Marketing Campaigns we have the SendGrid’s WordPress Subscription Widget that makes it easy for people visiting your WordPress site to subscribe to your marketing emails;
or Building a SendGrid Subscription Widget.
I got this answer from their support. It turns out we have to implement it by ourselves.
The double opt-in functionality is not something SendGrid provides as
we expect our customers to handle any opt-in practices on their side.
We apologize for any inconvenience.
SendGrid will be GDPR compliant by May, 25, 2018. Please note that
SendGrid does not – and does not currently have plans to – use servers
or data centers in the European Union to process email. Thus, SendGrid
cannot restrict data to the EU. However, neither current EU law nor
the GDPR require this. Instead, what is required is that SendGrid must
provide "appropriate safeguards" for data that it hosts and processes
on its US servers (see Art 46 of the GDPR here). SendGrid offers a
Data Processing Addendum (DPA) to provide such adequate safeguards,
which includes provisions for when GDPR goes into effect.
More info on GDPR can be found here. Our DPA can be reviewed and
signed by filling out the information here.
They do not support it. I asked support many times, which is a strange as it would seem a company of that size could spare the dev resources to build a feature that literally all of their customers need.
However, https://sgwidget.com is a third party product that provides double opt in functionality for Sendgrid accounts.
Full Disclosure: I am a developer at SG Widget.
No, indeed still today, they do not. Not in their forms, nor in their API is there simple, flip-switchable support for double opt-in. But, with email automation fairly recently implemented in their marketing services ("free" and "advanced" plans, not "essential") you can send an automated email directly upon sign-up.
My solution is to have 2 lists for new contacts, where one is a "pre-confirmation" list and the other being the "real" list. Here´s a way to use automation:
Create initial signup form, either via their sparse Web forms or via your own, using HTML/JS/PHP and API endpoint:
Create 2 separate lists, one for "pre-confirmation" emails and the other for people who confirm their addresses.
Make the form sign up new contacts to the first list, "pre-confirmation".
Create a marketing automation flow that triggers upon new signups to the "pre-confirmation" list. Make the automation trigger an email that contains a button or a link with the following link structure:
https://yoursite.com?email=user#email.com&passphrase=[phrase-you-set-manually]
where ?email= is your user´s email, substitute this in the email template/design by {{ Sender_Email }}
where &passphrase= is a phrase long enough to not be guessed. Since you only have one single email design here, and you can only enter one single phrase, unless you make a script or a hash, you make it difficult enough for people to think it was generated by a server :).
On your server/application, yoursite.com, use $_POST['email'] and $_POST['passphrase'], or whatever you name them, to validate the email clicks from your list and then enter all validated emails to the correct list using the PUT
/marketing/contacts endpoint.
you may also have to delete the user from the previous list, using DELETE
/marketing/lists/{id}/contacts, but I do think that the PUT /marketing/contacts takes care of placing the contact in only the lists specified in the list_ids field.
once the contact has been entered into the correct list, you can also have a marketing automation set up for that list, which sends him/her a welcome message.
This method takes care of double opt-in for SendGrid without using one single email credit from the Email API (transactional plan). The only catch is that we utilize one initial and one second/final list to achieve it.
Note: the initial sign-up message that here acts as the "confirm your email" message, will be tied to the first list and will require a marketing unsubscribe link in the footer. Make it clear in the bottom of the email that it is a temporary list, to not get any spam complaints. But it will not be an issue, as we wont be sending to anyone in that list except for this initial time. Unless you have a user who enters his/her email twice, after some time of inactivity when they forgot they already signed up. That could happen. But it´s a separate issue.
I think this is possible by switching the flow of a typical email subscriber. When the user clicks your subscribe button, instead of calling the sendgrid members/contact PUT api to add to your list, send an email with a link to a URL of yours that will then trigger the members/contact PUT api call.
Not sure what stack you are using but I was able to build something like this with next.js utilizing their api routes
I'm sorry if this is duplicating another post. I have a possible answer to a question in another post but I'm not sure if its a good solution and I wanted to ask people for their views.
The problem is the one raised in this post, how to protect emails from spam bots.
Rather than have the addresses on the page, split into different vars and then assembled by JavaScript, I send an ajax request to the server (just a GET to the welcome_controller) with a key ie 'address_id_42' in the params and it returns a mailto link which is then inserted into the page.
Is there any gain by not having any address data on the page initially? Is any advantage immediately lost by the fact the server will just hand out mailto links if you send it the right address id?
I could easily extend it so the server replies with some custom structure which gets unraveled by the js, but I agree that really this is not the right place to focus and that better spam filtering is the way forward, but I'm interesting in what people think to using ajax as a level of obfuscation?
Cheers :)
It depends on the kind of website it is.
Is the page only accessible after authentication (login)?
Is there another (simpler) way of doing it rather than getting it using AJAX?
The answer to your question really depends on these things.
But in a general way, yes, it might help. But such AJAX requests should only be triggered by some "humanly" action like clicking on "show email" button or something like that.
Also you could convert the email text to an image (which I believe is pretty easy to do with PHP).
Also other solutions could involve separating the two parts of the email address (part before and after '#' symbol) by putting them in different 'spans' etc.
I think obfuscating content through AJAX is a great idea. However, you can also try ready to use third party implementations like Mailhide instead of building all of this yourself. You get an additional layer of security by making the user fill up a CAPTCHA before the email address is revealed.
Jeff Lindsay, who coined the term 'webhook', said that the difference between webhook and http callback is that webhooks are user-defined. I think I understand what he meant, but I was thinking about it and I asked myself, can webhooks be effectively used by regular users (I mean: non-developers)?
Usually people don't have a clue how the internet works, they don't know what http is, terms like URL, callback, or request-response don't say anything to them. I've heard that many people do not know the difference between a web browser and a web site, they think that internet really starts at google.com and they type in all urls in the google search box... I mean, what's the use of webhooks when you're not a developer?
Do you think services like AlertGrid make sense? It's a webhook consumer that you can configure to dispatch alerts (SMS, phone, email) either when the callback is NOT received in x amount of time, or when the received data meets user-defined condition, plus it does some data visualization. We wanted it to make webhooks usable for non-developers. But still it requires an initial integration by someone who at least knows how to configure the source to send the webhook events. In many cases it only takes pasting an url to a textbox, but it seems to be beyond the skills of a typical user.
So, are the webhook doomed to be used by software developers only, or is there a chance that millions of Facebook or Twitter users will start making use of them somehow?
I think that something implemented using Webhooks can be made very user friendly.
Suppose Stack Exchange allowed users to define a webhook that would be notified whenever you earned a badge. You could supply a custom URL, or there could be simple buttons to click that would set it up for your Facebook or Twitter account. It could be as simple as the Facebook Like button.
YES I think this is a great idea. It's actually something I designed in my head a couple months ago and didn't think the product existed.
Webhooks are extremely powerful and having a 'service bus' aggregate/manage/dispatch these callbacks is extremely compelling to me.
I think that we are a long way from the general public consuming webhooks in any sort of meaningful way but I don't see why not. I remember when RSS was a 'developer' only technology.
Thanks for the link. I'll be digging in more this weekend.