SPF=Neutral sending through gmail - email

I'm using phpMailer on a Linode server. The DNS records are set to allow sending through the gmail mail server which is hosing my mail account membership#oiyc.org. I just added DKIM to try to get the SPF rating up. The DKIM apparently is fine but I still get SPF Neutral.
The sender set up is:
$mail = new PHPMailer();
$mail->isSMTP();
$mail->SMTPKeepAlive = true;
$mail->SMTPAuth = true;
$mail->SMTPSecure = "tls"; // sets the prefix to the server
$mail->Host = "smtp.gmail.com"; // sets GMAIL as the SMTP server
$mail->Port = 587; // set the SMTP port
$mail->Username = "membership#oiyc.org"; // GMAIL username
$mail->Password = "*******************"; // GMAIL password
$mail->isHTML(true); // send as HTML
$mail->WordWrap = 100; // set word wrap
$mail->Sender = "membership#oiyc.org";
$mail->addReplyTo($_SESSION['se-reply'],$_SESSION['se-from']);
$mail->setFrom($_SESSION['se-reply'],$_SESSION['se-from']);
$mail->DKIM_domain = "oiyc.org";
$mail->DKIM_private = "*********/rsa.private"; //path to file on the disk.
$mail->DKIM_selector = "mainkey";// change this to whatever you set during step 2
$mail->DKIM_passphrase = "";
$mail->DKIM_identity = $mail->Sender;
Here is the source received from an email sent through my linode server.
Delivered-To: ********#gmail.com
Received: by 10.46.25.85 with SMTP id p82csp1388830lje;
Sun, 4 Feb 2018 11:11:56 -0800 (PST)
X-Received: by 10.98.196.204 with SMTP id h73mr11556131pfk.143.1517771515865;
Sun, 04 Feb 2018 11:11:55 -0800 (PST)
ARC-Seal: i=1; a=rsa-sha256; t=1517771515; cv=none;
d=google.com; s=arc-20160816;
b=*****
qrIA==
ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816;
h=content-transfer-encoding:mime-version:list-unsubscribe:message-id
:subject:reply-to:to:date:from:dkim-signature
:arc-authentication-results;
bh=ptVvqh2PiSco0+Kb7wjBXHUijnbEm43LU4E+zStVvb0=;
b=********
iuTg==
ARC-Authentication-Results: i=1; mx.google.com;
dkim=pass header.i=#oiyc-org.20150623.gappssmtp.com header.s=20150623 header.b=ytsz7YWm;
spf=neutral (google.com: 209.85.220.41 is neither permitted nor denied by best guess record for domain of membership#oiyc.org) smtp.mailfrom=membership#oiyc.org
Return-Path: <membership#oiyc.org>
Received: from mail-sor-f41.google.com (mail-sor-f41.google.com. [209.85.220.41])
by mx.google.com with SMTPS id i3sor1037208pgs.91.2018.02.04.11.11.55
for <********#gmail.com>
(Google Transport Security);
Sun, 04 Feb 2018 11:11:55 -0800 (PST)
Received-SPF: neutral (google.com: 209.85.220.41 is neither permitted nor denied by best guess record for domain of membership#oiyc.org) client-ip=209.85.220.41;
Authentication-Results: mx.google.com;
dkim=pass header.i=#oiyc-org.20150623.gappssmtp.com header.s=20150623 header.b=ytsz7YWm;
spf=neutral (google.com: 209.85.220.41 is neither permitted nor denied by best guess record for domain of membership#oiyc.org) smtp.mailfrom=membership#oiyc.org
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
d=oiyc-org.20150623.gappssmtp.com; s=20150623;
h=from:date:to:reply-to:subject:message-id:list-unsubscribe
:mime-version:content-transfer-encoding;
bh=ptVvqh2PiSco0+Kb7wjBXHUijnbEm43LU4E+zStVvb0=;
b=*********
SsBA==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
d=1e100.net; s=20161025;
h=x-gm-message-state:from:date:to:reply-to:subject:message-id
:list-unsubscribe:mime-version:content-transfer-encoding;
bh=ptVvqh2PiSco0+Kb7wjBXHUijnbEm43LU4E+zStVvb0=;
b=*************
r+zA==
X-Gm-Message-State: AKwxytcQCxD/95gmJfS/DyCC4XOh8K3K+Jj9QONeHmVyCH5ebJDtxvIl tQwyBjpS9etVQopYODbtnZZ2Kw0k1Pc=
X-Google-Smtp-Source: AH8x227kdTn+9Ee7QoJFUYDPq/ax7LmKHzsDAtCNr/5cL0MidmAB3GWuEw4RU28Zb3jl8Kx0uAnegw==
X-Received: by 10.99.96.80 with SMTP id u77mr6305435pgb.401.1517771515191;
Sun, 04 Feb 2018 11:11:55 -0800 (PST)
Return-Path: <membership#oiyc.org>
Received: from oiyc.org ([2600:3c01::f03c:91ff:fe56:5129])
by smtp.gmail.com with ESMTPSA id m65sm14046167pfc.150.2018.02.04.11.11.54
for <********#gmail.com>
(version=TLS1 cipher=ECDHE-RSA-AES128-SHA bits=128/128);
Sun, 04 Feb 2018 11:11:54 -0800 (PST)
From: Bob Brunius <membership#oiyc.org>
X-Google-Original-From: Bob Brunius <********#gmail.com>
Date: Sun, 4 Feb 2018 11:11:53 -0800
To: ********#gmail.com
Reply-To: Bob Brunius <********#gmail.com>
Subject: A different sort of test 123d
Message-ID: <MR3sDgtyN4siuc2vYCZLxL34VuLFlexvK0WbbcEH7FA#oiyc.org>
X-Mailer: PHPMailer 6.0.3 (https://github.com/PHPMailer/PHPMailer)
List-Unsubscribe: <information#oiyc.org>, <https://oiyc.org/membershipDatabaseForms/unsubscribe.php?email=********#gmail.com&member=242>
MIME-Version: 1.0
Content-Type: multipart/alternative; boundary="b1_MR3sDgtyN4siuc2vYCZLxL34VuLFlexvK0WbbcEH7FA"
Content-Transfer-Encoding: 8bit
--b1_MR3sDgtyN4siuc2vYCZLxL34VuLFlexvK0WbbcEH7FA
Content-Type: text/plain; charset=us-ascii
Hello 12345678-abcd
--b1_MR3sDgtyN4siuc2vYCZLxL34VuLFlexvK0WbbcEH7FA
Content-Type: text/html; charset=us-ascii
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<body>
Hello 12345678-abcd
</body>
</html>
--b1_MR3sDgtyN4siuc2vYCZLxL34VuLFlexvK0WbbcEH7FA--

Currently your SPF record is published in your DNS as...
"v=spf1 include:_spf.google.com include:oiyc.org ~all"
which has an include:oiyc.org recursively referencing itself. This is technically incorrect, but strangely may work if the preceding mechanism include:_spf.google.com is satisfied and returns a result to the SPF query. Therafter the include:oiyc.org mechanism will cause the lookup to fail, so the ~all mechanism would never be processed.
The include: mechanism is intended to reference an externally published set of SPF details, usually at a different domain.
I suspect that you intended to reference the IP address of your own domain, presumably defined in DNS as an A record for the bare domain name, in which case you would us the mechanism a:oiyc.org which can itself be shortened to just a
So your resulting TXT record might be something like...
"v=spf1 include:_spf.google.com a ~all"

There is no connection between using DKIM and SPF.
From the log:
Received-SPF: neutral (google.com: 209.85.220.41 is neither permitted nor denied by best guess record for domain of membership#oiyc.org) client-ip=209.85.220.41;
When checking for the SPF record, this happens:
$ host -t TXT oiyc.org
oiyc.org has no TXT record
This means, you haven't set up any SPF rules. Therefore, there cannot be a SPF check. As the message states, there is no reason to have anything other than neutral.
This also applies to ARC-Authentication-Results and Authentication-Results headers.

The SPF that was supposed to add for the current IP listed is
# TXT "v=spf1 a ip4:209.85.220.41 ~all"
visit this link to create and the SPF : https://mxtoolbox.com/SPFRecordGenerator.aspx

It has been a while since this post. But in case this helps anyone. If you already have your MX record set as Google you don't need to repeat it in your SPF. So this will be sufficient: v=spf1 mx ~all.
But then I received a softfail error: softfail (google.com: domain of transitioning a#d.c does not designate xxx.xx.xx.xx as permitted sender)
Adding the ip4 addresses to the TXT record fixed that error too and I received a pass. These were the 2 Google IPs that appeared on my errors, so I added them to the permitted list.
v=spf1 mx ip4:209.85.220.41 ip4:209.85.220.65 ~all
P.S. If you already have a TXT record, the spf value can be added in a new line within the same TXT record.

Related

SPF + DKIM + DMARC = Passed yet message ends in spam

Trying to send out emails yet they end up in spam folders for some reason. So we tweaked and adjusted Dkim and Dmarc, now all records show passed within Gmail, yet still flags as spam.
Also tested via dkimvalidator , this results in all records pass and showing
SpamAssassin Score: -0.1
Message is NOT marked as spam
Now the breakdown of an email is shown below.
Sending IP has been changed for this post to 123.456.789.12
Sending domain has been changed to somedomain.net
sending email address has been changed to support#somedomain.net
Receiver email address has been changed to someuser#gmail.com
Original Message
Message ID <3164f55daebbc258d0a4846eda47142b#somedomain.net>
Created at: Fri, Mar 15, 2019 at 3:06 PM (Delivered after 1 second)
From: support#somedomain.net
To: someuser#gmail.com
Subject: Are you getting our emails?
SPF: PASS with IP 123.456.789.12Learn more
DKIM: 'PASS' with domain somedomain.netLearn more
DMARC: 'PASS' Learn more
Delivered-To: someuser#gmail.com
Received: by 2002:a05:6504:1158:0:0:0:0 with SMTP id r24csp952466ltn;
Fri, 15 Mar 2019 12:06:09 -0700 (PDT)
X-Google-Smtp-Source: APXvYqwbxqRfiq2UKjVTB57wP4g1MW2NvFWKdyYY9P4PITERpXVsMmcdriTP6Fp9rsf+DU2Ky1nQ
X-Received: by 2002:a1c:230e:: with SMTP id j14mr3316801wmj.9.1552676769338;
Fri, 15 Mar 2019 12:06:09 -0700 (PDT)
ARC-Seal: i=1; a=rsa-sha256; t=1552676769; cv=none;
d=google.com; s=arc-20160816;
b=kI1D5Zd50f7UIVNtLYscVA3w9dhrAzg3YfuSMJWIIag6Le+YNqzYUpTxlQ11pK0Jvd
9bQ8KztlKnEwTWvC90bUrX7FvNTdKTEeatTQao5I5z77QxBG8q96cYYnbmzHVzsKxcS6
eBJYwO7gabkTsJzdGHYeYR9pYHPANr5vhtkpGYn4OSEAvrTokXHnI7Lma5MlI+QcJFhO
eGb6omkWY05TGfZZYMJ9ny/8WBhG6W3mpuT+x3Z0mv9IWnuxhhnDWgYGb+qEK/cUkCjl
CLHaxq+uRh8MXnjzoChWHXc4elm8yKo9CS3qqAppMThxQ3X/2kp9SVg+EPKut9R3OtG0
D+ng==
ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816;
h=message-id:user-agent:subject:to:from:date
:content-transfer-encoding:mime-version:dkim-signature;
bh=GrxIPJ9zPlbqzf38J/MOr7Lt7eP6CUv0wQxJqM2McmE=;
b=QgXB7YGir4NdgYnbt+2pORQCK5bMsOT+mHtSghj+CrTIORIqW04ie/3v8iWpMop2uc
ZYby4O/YD5TohP7VnYT1/lHGcQO8BCK+BCqXJaJb1JyzoRFyAM2aVMfJonRphm3VSfzH
JCGAAQjW3hTRsANmiU4Toh5IgXVgLvz2ZFSbOjY+xEopHJar1XXg9kN9N+A8kZMJWIIk
dVXrhwSDV/1Z0/8ObepuXI+KZNRpyKItMjMMDDyh1Py7cQl1MJXnoTzr53aCeDw3nWWg
yUDRjq3kDUK2Rm0X/O43DGHQvAHvk7gge4UaE5rDFjtBphLVYO0Zadto3bQAFwAMouR9
H9Qg==
ARC-Authentication-Results: i=1; mx.google.com;
dkim=pass header.i=#esomedomain.net header.s=mail header.b="cLQsKmq/";
spf=pass (google.com: domain of support#somedomain.net designates 123.456.789.12 as permitted sender) smtp.mailfrom=support#somedomain.net;
dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=somedomain.net
Return-Path: <support#somedomain.net>
Received: from ms101.somedomain.net (vmi247326.someotherotherdomain.net. [123.456.789.12])
by mx.google.com with ESMTPS id k4si945224wmi.131.2019.03.15.12.06.09
for <someuser#gmail.com>
(version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128);
Fri, 15 Mar 2019 12:06:09 -0700 (PDT)
Received-SPF: pass (google.com: domain of support#somedomain.net designates 123.456.789.12 as permitted sender) client-ip=123.456.789.12;
Authentication-Results: mx.google.com;
dkim=pass header.i=#somedomain.net header.s=mail header.b="cLQsKmq/";
spf=pass (google.com: domain of support#somedomain.net designates 123.456.789.12 as permitted sender) smtp.mailfrom=support#somedomain.net;
dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=somedomain.net
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=somedomain.net; s=mail; h=Message-ID:Subject:To:From:Date: Content-Transfer-Encoding:Content-Type:MIME-Version:Sender:Reply-To:Cc: Content-ID:Content-Description:Resent-Date:Resent-From:Resent-Sender: Resent-To:Resent-Cc:Resent-Message-ID:In-Reply-To:References:List-Id: List-Help:List-Unsubscribe:List-Subscribe:List-Post:List-Owner:List-Archive;
bh=GrxIPJ9zPlbqzf38J/MOr7Lt7eP6CUv0wQxJqM2McmE=; b=cLQsKmq/CfXyv0nzG9Fwi6cs4 Ei3zkeoZTgWBvLZfF303+EVPnwUdqgKKlXfLntlfW+8lp54rm3S/dI9p640dC6IIoGJrdkYwWUw+0 J0K6U9rSTs1yUJ8mM6kWdYURmUQ7eAMvbqFtCIXhcXPBBlKDgvFGCpwz5GnSark8zkU7c=;
Received: from localhost ([127.0.0.1] helo=ms101.somedomain.net) by vmi247326.someotherdomain.net with esmtp (Exim 4.91) (envelope-from <support#somedomain.net>) id 1h4s9g-0001pt-Nb for someuser#gmail.com; Fri, 15 Mar 2019 15:06:08 -0400
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
Date: Fri, 15 Mar 2019 15:06:08 -0400
From: suppor#somedomain.net
To: someuser#gmail.com
Subject: Are you getting our emails?
User-Agent: Roundcube Webmail/1.4-rc1
Message-ID: <3164f55daebbc258d0a4846eda47142b#somedomain.net>
X-Sender: support#somedomain.net
Currently running centos 7 --> VestaCP
Roundcube using postfix, yet don't mind switching to horde or 3rd party ESP if needed.
The only thing that I can think is the sending domain is a sub domain of the root yet we've added MX record with priority 1 within DNS on cloudflare to accept mail for root domain, yet the sub domain is hosted on same server and same IP.
The Message ID is being generated from somewhere, I am not sure. This may be the reason for spam flag as well.
Any help is greatly appreciated.
I worked in G Suite support a while back, spent a lot of time working with Gmail.
Some things to keep in mind,
1) Try to have your SPF with ~all instead of ?all (I can still see your domain in the return path, you may want to edit that out).
2) Your DKIM needs to be added into the domain listed in the return path. If the return-path is domaina.com, the DKIM needs to be from domaina.com
3) DMARC doesn't really help that much when sending email, is mostly used to prevent inbound spoofing against your domain, it's still good to have it nonetheless.
4) Try sending emails from a different server, sometimes specific servers have pretty bad rep, causing emails to be marked as spam (mail-tester.com tool helps get an idea of how spammy your emails are).
5) Try not to send marketing emails to people who have never heard from you. In case you're actually doing that, use a dedicated service (like Mailchimp) that maay help the delivery (also, avoid bccs).
Other than that, the tough answer, you can do everything good, but there is 0 guarantee that your email will never be marked as spam, sadly, it's half in your control, the other half is what others have reported from your domain, your server, your content, your actual username, links included, etc. Keep in mind that your reputaion varies per server, so if emails land in spam in Gmail, it doesn't necesarily mean that it'll land in spam in hotmail. Sometimes using a completely different domain/server/service helps, but if the reputation is already tainted, only marking emails as 'not spam' would eventually help the delivery (I know, it sucks when that happens).
Cheers!

Increasing understanding of return-path and SPF records

I have a website with a web form that is hosted with a third party. I'm having trouble with their server config/architecture and wanted to confirm a few things and show the mail headers I am receiving.
I know that I must explicitly set my mail to send through an outgoing mail server and port which I have specified (this is in order for the mail to send, it will not be sent if not sent through the SMTP server).
Upon setting the SMTP server and port here is the mail header:
x-store-info:sbevkl2QZR7OXo7WID5ZcdV2tiiWGqTnhQzu7BHe69dd2ZvcRr0xBttv16txT0x/MHyyxbQQOWxD0k3WKrQDVl56gwTtl9T9YlQDKWZad3R7ZbliBL6BSfw52gpz37cwL/qGTahKP+U=
Authentication-Results: hotmail.com; spf=softfail (sender IP is 213.171.216.60) smtp.mailfrom=test#hotmail.co.uk; dkim=none header.d=hotmail.co.uk; x-hmca=fail header.id=test#hotmail.co.uk
X-SID-PRA: test#hotmail.co.uk
X-AUTH-Result: FAIL
X-SID-Result: FAIL
X-Message-Status: n:n
X-Message-Delivery: Vj0xLjE7dXM9MDtsPTA7YT0xO0Q9MTtHRD0xO1NDTD0y
X-Message-Info: NhFq/7gR1vSyCjVJ7Q2iIHhF9oW5eW+g+jrtzv+TwMhHX16XKDSEsIIxc1qXk1NO+AwFoToprpXBtEMxyoOvMnJSYUEEN4JngTWWsUg0/J3120nOI8GDZ8sF8m5iNGKuZt7Ds7svv3bOfUNBVLmpGlsbHUCNwXFNgry/rw0sWtsI0nyKa01KIdnxHDoLHp7GPP/klJGbBhJE2FLEW70tX5XZujwdcC9+R5m/pk5uo4uPxfmnwQK9yQ==
Received: from cust-smtp-auth1.fasthosts.net.uk ([213.171.216.60]) by BLU004-MC1F22.hotmail.com with Microsoft SMTPSVC(7.5.7601.23143);
Mon, 28 Dec 2015 05:59:03 -0800
Received: from fun-booths.co.uk (unknown [88.208.252.229])
by cust-smtp-auth1.fasthosts.net.uk (Postfix) with ESMTP id 923AF74021D
for <c_qatest#hotmail.co.uk>; Mon, 28 Dec 2015 13:59:02 +0000 (GMT)
Date: Mon, 28 Dec 2015 13:59:02 +0000
To: c_qatest#hotmail.co.uk
From: "test#hotmail.co.uk" <test#hotmail.co.uk>
Subject: Fun Booths - Booking form
Message-ID: <918f36d31bf876f19ea6d9563c1ad348#fun-booths.co.uk>
X-Priority: 3
X-Mailer: PHPMailer 5.2.10 (https://github.com/PHPMailer/PHPMailer/)
Reply-To: test#hotmail.co.uk
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
Return-Path: test#hotmail.co.uk
X-OriginalArrivalTime: 28 Dec 2015 13:59:03.0161 (UTC) FILETIME=[E8B5FE90:01D14177]
Focusing on the Authentication-Results the (sender IP is 213.171.216.60) which is definitely the correct outgoing SMTP server. However the the spf result is spf=softfail.
After doing some research it was suggested that
The Return-Path header is empty. This means that during the SMTP
conversation, the MAIL FROM command did not specify an email address
where bounces are sent. It is the domain from this email address that
SPF tests. If there is no email address, a softfail will result.
Is this referring to the smtp.mailfrom field in the Authentication-Results which is set to a value of test#hotmail.co.uk in the example above?
I believe I have done what is required to implement SPF:
(1) I have set the SPF record in the domain's DNS zone
v=spf1 a ip4:213.171.216.0/24 mx -all
Now the confusion arises here. When someone else's mail server receives a message claiming to come from that domain, then
(2) the receiving server can check whether the message complies with the domain's stated policy
How does hotmail's receiving mail server for example do this check? because if I set a Return-Path of hello#fun-booths.co.uk then this results in spf=pass. Just to be clear www.fun-booths.co.uk is the domain being used.
x-store-info:J++/JTCzmObr++wNraA4Pa4f5Xd6uens6FBov4shFUrwGsQPla5CZKHNFpj4XdT2wfaqUtXggI++7RpfQIpooWW0Sp2ynYP894LLfhswpqbr+Di/ao+0Ofc9Btl/xdHLsTQXTk39KBE=
Authentication-Results: hotmail.com; spf=pass (sender IP is 213.171.216.60) smtp.mailfrom=hello#fun-booths.co.uk; dkim=none header.d=fun-booths.co.uk; x-hmca=pass header.id=hello#fun-booths.co.uk
X-SID-PRA: hello#fun-booths.co.uk
X-AUTH-Result: PASS
X-SID-Result: PASS
X-Message-Status: n:n
X-Message-Delivery: Vj0xLjE7dXM9MDtsPTE7YT0xO0Q9MTtHRD0xO1NDTD0w
X-Message-Info: NhFq/7gR1vTQzco4wDfDIuNexRCLt7KFLQW7EkmNLn/2YehuSC93bNZTp87n+KmseY8TwxSqCjOondyBGOJR9CRbKyT/FU2B2nhMw3SU8HjmnNyAmDcRFqxvARiDy1lMz5O7U5B61WNdLZsDb1vLPQ93l4XO90mQcjMfCI4SWr50rtHEJwK9Y/c2zDWf8jdVXEgQOyBm4pQwu9z7isJFvrHl9HRMGMcWeNHQVVCsFOoqJ8mhQItxPg==
Received: from cust-smtp-auth2.fasthosts.net.uk ([213.171.216.60]) by SNT004-MC2F7.hotmail.com with Microsoft SMTPSVC(7.5.7601.23143);
Mon, 28 Dec 2015 06:05:27 -0800
Received: from fun-booths.co.uk (unknown [88.208.252.229])
by cust-smtp-auth2.fasthosts.net.uk (Postfix) with ESMTP id 66414740221
for <c_qatest#hotmail.co.uk>; Mon, 28 Dec 2015 14:05:26 +0000 (GMT)
Date: Mon, 28 Dec 2015 14:05:26 +0000
To: c_qatest#hotmail.co.uk
From: "hello#fun-booths.co.uk" <hello#fun-booths.co.uk>
Subject: Fun Booths - Booking form
Message-ID: <85119bedb602f9865290c2ea218315b4#fun-booths.co.uk>
X-Priority: 3
X-Mailer: PHPMailer 5.2.10 (https://github.com/PHPMailer/PHPMailer/)
Reply-To: test#hotmail.co.uk
X-Sender: hello#fun-booths.co.uk
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
Return-Path: hello#fun-booths.co.uk
X-OriginalArrivalTime: 28 Dec 2015 14:05:27.0582 (UTC) FILETIME=[CDD7FBE0:01D14178]
This leads me on to my final questions - Is it true that the Return-Path must be set to a real email address on the site domain (www.fun-booths.co.uk), or mail will not send?
Because I have set up the mailbox hello#fun-booths.co.uk, however it seems even when I delete the mailbox that this still results in an spf=pass. It seems that in other words that not having a real email address on the domain, and instead just setting the Return-Path to hello#fun-booths.co.uk without the mailbox existing will result in an spf=pass. Should this be the case?
Is the email #fun-booths.co.uk set in the Return-Path used to determine the domain for the SPF checks?
Finally in the quotation above that mentions the SMTP conversation, is the MAIL FROM command referencing the smtp.mailfrom field in the Authentication-Results?
SPF Checks are always done with the envelope sender (MAIL FROM:) - smtp.mailfrom in your case. In the first mail you have test#hotmail.co.uk as envelope sender, but your server is not listed in the SPF record for hotmail.co.uk, and since that record has ~all as the last mechanism you get a Softfail.
In the second mail the envelope sender is hello#fun-booths.co.uk, so it's the SPF record for fun-booths.co.uk that is being used in that check.
So you should use a fun-booths.co.uk address as your envelope sender.
Depending on how you send mail from the web form there are different ways to control the envelope sender, but it may be that value of Return-Path is used as the envelope sender.
There don't have to be a mail-box for the envelope sender for the SPF to work, since SPF normally only look at the domain part of the address, but if the mail bounces somewhere in the process, the bounce message will normally be sent the the envelope sender, so it's a good idea to keep the mailbox for that address.

DMARC/SPF/DKIM not authenticating with third-party mail

We recently implemented a DMARC record for our domain:
"v=DMARC1; p=quarantine; pct=100; rua=mailto:me#mydomain.com"
(quarantine 100% of non-authenticated emails and send aggregate report to "me")
We use a third-party vendor to issue invites. The vendor sends email from invites#invites.vendordomain.com which is then sent through a mail relay "smtp3.mailrelaydomain.it". I also know that the mail relay uses a single ip address.
That address is included in our SPF record:
"v=spf1 ...[SNIP reference for other mail servers SNIP]... ip4:[ip address for the mail relay] ~all"
When I send an invite using the vendor's service, the message is quarantined.
When I view the aggregate DMARC report I see that the invite:
is recognized as being from an SPF-Authorized Server
passes raw SPF authentication for the sender's domain (invites#invites.vendordomain.com")
passes raw DKIM authentication for the mail relay domain (smtp3.mailrelaydomain.it)
Fails DMARC authentication for both DKIM and SPF for mydomain
Here is a sample headers from an invite.
BEGIN SAMPLE EMAIL HEADER
Delivered-To: someone#mydomain.com
Received: by 10.64.252.9 with SMTP id zo9csp100581iec;
Wed, 21 Oct 2015 11:40:13 -0700 (PDT)
X-Received: by 10.55.195.147 with SMTP id r19mr12995508qkl.12.1445452813709;
Wed, 21 Oct 2015 11:40:13 -0700 (PDT)
Return-Path: <invites#invites.vendordomain.com>
Received: from smtp3.mailrelaydomain.it (smtp3.mailrelaydomain.it. [ip for mail relay])
by mx.google.com with ESMTP id w15si9297939qha.131.2015.10.21.11.40.13
for <someone#mydomain.com>;
Wed, 21 Oct 2015 11:40:13 -0700 (PDT)
Received-SPF: pass (google.com: domain of invites#invites.vendordomain.com designates [mail relay ip] as permitted sender) client-ip=[mail relay ip];
Authentication-Results: mx.google.com;
spf=pass (google.com: domain of invites#invites.vendordomain.com designates [mail relay ip] as permitted sender) smtp.mailfrom=invites#invites.vendordomain.com;
dkim=pass header.i=#mailrelaydomain.it;
dmarc=fail (p=QUARANTINE dis=QUARANTINE) header.from=mydomain.com
Received: from FS-S05.vendorparentdomain.com (unknown [vendor parent ip])
(using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits))
(No client certificate requested)
by smtp3.mailrelaydomain.it (Postfix) with ESMTPSA id 23387A0CBC
for <someone#mydomain.com>; Wed, 21 Oct 2015 15:07:35 -0400 (EDT)
DKIM-Signature: [DKIM Content]
Content-Type: multipart/alternative;
boundary="===============2166944298367943586=="
MIME-Version: 1.0
Subject: Please take our survey
From: Me <me#mydomain.com>
To: Someone Else <someone#mydomain.com>
Cc:
Date: Wed, 21 Oct 2015 18:39:48 -0000
Message-ID: <20151021183948.27448.90706#FS-S05.vendorparentdomain.com>
List-Unsubscribe: [unsubscribe link],
<mailto:invites#invites.vendordomain.com>
Reply-To: Me <me#mydomain.com>
X-Sender: invites#invites.vendordomain.com
I believe the issue is related to the from domain in the message not matching the domain for the message envelope; however, the vendor is unable to change their settings (i.e., envelope will always be from the vendor domain) so any chance of this working with DMARC will have to come from my end.
Knowing that the SPF record can (and does) identify the invite as being from an SPF-Authorized Server, are there any other settings or records I can add to also ensure DMARC authentication for invites from the vendor?
Having read several online articles and "DMARC -spf and DKIM record queries" I suspect I am out of luck, but need to ask the question plainly/specific to my situation just to be sure.
Thanks
You are correct, you are out of luck unless the vendor can change something. What is failing is Identifier Alignment - https://www.rfc-editor.org/rfc/rfc7489#section-3.1 - because what is being authenticated (invites.vendordomain.com via SPF) does not align to the domain the user sees (me#mydomain.com) and the message then, correctly, fails DMARC.
There are three options:
Stop sending with a From: header of your domain at the vendor; you can still use a Reply-To: header with your own address.
Have the vendor align the mail from to your domain. If they don't do this they can't pass DMARC, and at some point they will want to pass DMARC or people will find other solutions. You can have them send with an envelope from of vendorname.mydomain.com and you can set up an MX for that subdomain that points to them to support bounce processing. This has been BCP for a while.
Have the vendor sign with DKIM and us an aligned DKIM signature. This is also best common practice. You only need SPF or DKIM to pass, and DKIM passes are more valuable (because they survive forwarding in many cases) than SPF, so this is the option I would personally prioritize if I were you.
Back in like 2012 and 2013 a lot of vendors pushed back against both of these options, but I honestly haven't seen a vendor in a long time (I spend 100% of my day job on DMARC) that won't support at least aligned DKIM.

Identify spam indicators based on emails headers

Using PHPMailer with SMTP, the following email was sent from my VPS (mydomain.com hosted by phpwebhosting) as if it came from my Comcast account (jane.doe#comcast.net) to my Gmail email (john.doe#gmail.com), and Gmail flagged it as spam.
Looking at the email headers, how can I determine what might make a email client flag an email as spam?
Delivered-To: jane.doe
Received: by 10.28.7.197 with SMTP id 188csp518471wmh;
Fri, 8 May 2015 06:51:39 -0700 (PDT)
X-Received: by 10.43.17.135 with SMTP id qc7mr4244827icb.14.1431093098853;
Fri, 08 May 2015 06:51:38 -0700 (PDT)
Return-Path: <john.doe#comcast.net>
Received: from smtp1.phpwebhosting.com (smtp1.phpwebhosting.com. [145.242.148.75])
by mx.google.com with SMTP id ag10si4096698icc.25.2015.05.08.06.51.38
for <jane.doe>;
Fri, 08 May 2015 06:51:38 -0700 (PDT)
Received-SPF: neutral (google.com: 145.242.148.75 is neither permitted nor denied by domain of john.doe#comcast.net) client-ip=145.242.148.75;
Authentication-Results: mx.google.com;
spf=neutral (google.com: 145.242.148.75 is neither permitted nor denied by domain of john.doe#comcast.net) smtp.mail=john.doe#comcast.net;
dmarc=fail (p=NONE dis=NONE) header.from=comcast.net
Received: (qmail 29774 invoked from network); 8 May 2015 13:51:37 -0000
Received: from unknown (HELO test.sites.mydomain.com) (outgoing#mydomain.com#145.242.134.91)
by smtp1.phpwebhosting.com with (DHE-RSA-AES256-SHA encrypted) SMTP; Fri, 08 May 2015 09:51:37 -0400
Date: Fri, 8 May 2015 06:51:36 -0700
To: Jane Doe <jane.doe>
From: John Doe <john.doe#comcast.net>
Reply-To: john.doe#comcast.net
Subject: Mydomain Password for Test Site
Message-ID: <fa1b444df47091d2ca100f40d93b14cc#test.sites.mydomain.com>
X-Priority: 3
X-Mailer: PHPMailer 5.2.9 (https://github.com/PHPMailer/PHPMailer/)
MIME-Version: 1.0
Content-Type: multipart/alternative;
boundary="b1_fa1b444df47091d2ca100f40d93b14cc"
Content-Transfer-Encoding: 8bit
--b1_fa1b444df47091d2ca100f40d93b14cc
Content-Type: text/plain; charset=us-ascii
Hello Jane,
I have added you to our Mydomain sales tracking and bid solicitation tool.
Your username is: jane.doe.
Click the following link within 24 hours to set your password: https://test.sites.mydomain.com/index.php?cid=25&task=display_p&t=28dba87d5fb8062e40a69f0192660471
Thank you
--b1_fa1b444df47091d2ca100f40d93b14cc
Content-Type: text/html; charset=us-ascii
<p>Hello Alvin,</p>
<p>I have added you to our Mydomain sales tracking and bid solicitation tool.</p>
<p>Your username is: jane.doe.</p>
<p>Click the following link within 24 hours to set your password: https://test.sites.mydomain.com/index.php?cid=25&task=display_p&t=28dba87d5fb8062e40a69f0192660471</p>
<p>Thank you</p>
--b1_fa1b444df47091d2ca100f40d93b14cc--
There is no surefire way to tell why Gmail flags an email as spam. Spam filters in general are black boxes from the perspective of the sender, as only those who know the inner workings and have access to logs can tell for sure what happened to certain email. The reason for flagging can be virtually anything, like violating the sender domain's policies, poor IP reputation, poor reputation of links used, similarity to spam emails, bad standards compliance and so on. Sometimes there is no singular reason either.
It's not that you can't make an educated guess. In this particular case, you are sending an email in the name of a comcast.net user, but you are bypassing Comcast servers entirely. Comcast has SPF and DMARC policies in place and although Comcast's SPF policy evaluation doesn't assert smtp1.phpwebhosting.com either permitted or not (SPF "neutral" result), the DMARC result that Gmail is getting is "fail". The DMARC policy for Comcast is not to flag emails failing email authetication (but report them only), but I'd still guess it's a bad omen. Try sending the email via your authorized Comcast server or use your own domain name for both From: and Return-Path to see if you can avoid getting flagged as spam.

email that my application sends is getting spammed: what's wrong with my headers?

I'm sending out emails via my PHP application. However, they're getting marked as spam by Gmail. Here's how I'm sending the email (PHP):
$headers = "From: test#bookmytakeout.com\r\nReply-To: test#bookmytakeout.com";
$mail_sent = mail( 'munged#gmail.com', 'test mail', $message, $headers, '-ftest#bookmytakeout.com' );
Gmail spams this message. So I went and clicked that handy "show original message" option. Here's what I get:
Delivered-To: munged#gmail.com
Received: by 10.68.71.200 with SMTP id x8cs325812pbu;
Thu, 21 Jul 2011 01:34:52 -0700 (PDT)
Received: by 10.236.114.234 with SMTP id c70mr12483739yhh.163.1311237292052;
Thu, 21 Jul 2011 01:34:52 -0700 (PDT)
Return-Path: <test#bookmytakeout.com>
Received: from vps.bookmytakeout.com ([8.22.200.47])
by mx.google.com with ESMTPS id u61si3662037yhm.119.2011.07.21.01.34.50
(version=TLSv1/SSLv3 cipher=OTHER);
Thu, 21 Jul 2011 01:34:51 -0700 (PDT)
Received-SPF: neutral (google.com: 8.22.200.47 is neither permitted nor denied by best guess record for domain of test#bookmytakeout.com) client-ip=8.22.200.47;
DomainKey-Status: bad format
Authentication-Results: mx.google.com; spf=neutral (google.com: 8.22.200.47 is neither permitted nor denied by best guess record for domain of test#bookmytakeout.com) smtp.mail=test#bookmytakeout.com; domainkeys=neutral (bad format) header.From=test#bookmytakeout.com
DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=default; d=bookmytakeout.com;
h=Received:To:Subject:From:Reply-To:Message-Id:Date;
b=WYWQ+/9+wOAvq+OUSM5KLDAueciIoNiByXaVV29HYa0XbSwm2f+89TCj8pW24G7k1vTGCvR8n64iSwgPQuoEitz6ehbszd0+75Px0WlGsvyeZGrW3PaYEpkRFmkDoaGe;
Received: from munged by vps.bookmytakeout.com with local (Exim 4.69)
(envelope-from <test#bookmytakeout.com>)
id 1QjolW-0001Vn-Us
for munged#gmail.com; Thu, 21 Jul 2011 14:07:31 +0530
To: munged#gmail.com
Subject: test mail
From: test#bookmytakeout.com
Reply-To: test#bookmytakeout.com
Message-Id: <E1QjolW-0001Vn-Us#vps.bookmytakeout.com>
Date: Thu, 21 Jul 2011 14:07:30 +0530
X-AntiAbuse: This header was added to track abuse, please include it with any abuse report
X-AntiAbuse: Primary Hostname - vps.bookmytakeout.com
X-AntiAbuse: Original Domain - gmail.com
X-AntiAbuse: Originator/Caller UID/GID - [1005 1000] / [47 12]
X-AntiAbuse: Sender Address Domain - bookmytakeout.com
test
Now, I have no idea how to diagnose what could be setting off gmail's spam filters. Can someone please point out what part of this email is setting off the spam filter?
If possible, please post a solution as well. I'm more interested in learning what's wrong with the headers than in how to fix this for now.
PS: I have a few suspicions of my own:
the Received header says "vps.bookmytakeout.com" but the From header says "bookmytakeout.com" - but I tried sending it with From as test#vps.bookmytakeout.com - same problem, still spammed.
The headers "Received-SPF", "DomainKey-Status" and "Authentication-Results" seem to indicate some problem. I distinctly remember NOT setting up any MX records for this domain name. Could that be the issue?
I guess you didn't publish any SPF / DKIM authentication record
Received-SPF: neutral (google.com: 8.22.200.47 is neither permitted nor denied by best guess record for domain of test#bookmytakeout.com) client-ip=8.22.200.47;
DomainKey-Status: bad format
Most of the time, Gmail (as well as the other major ISPs) will place the non-authenticated messages in the junk folder.
An SPF record or a DKIM guarantees that you are allowed to use a certain domain as a sender. For example, if you don't own "paypal.com", you can't send an email from "contact#paypal.com". If you've published the right SPF / DKIM, the ISP will consider you as a trusted sender.
SPF & DKIM were first designed to fight against phishing.
DIY SOLUTION: Publish some records by following these instructions:
http://dkim.org/specs/rfc5585.html
http://www.openspf.org/FAQ
EASY SOLUTION: Use a service that will do everything for you. A good ESP will usually sign your emails with DKIM / SPF by default. The problem is that you will sometimes get a "sent via ESP_NAME" mention (in Gmail).
So the best thing to do is to choose an ESP which will provide you personalized DKIM & SPF. This way, it will be 100% transparent.
I work for Mailjet and we offer this service for free. Most of our competitors offer this as an option.
Here's a useful post about this "via-mention" you get when your ESP signs "by default" and how to get rid of it.
http://blog.mailjet.com/post/16922561593/personalized-spf-dkim