I'm trying to build a bash script to check SPF records and so far I have managed to create one to check if there is an SPF record and the type of the SPF record (PASS, SOFT FAIL and FAIL) but I'm trying to figure out how to also check if an SPF record has too many lookups included, let me know if anyone knows a ready-made script for that or perhaps knows a way to do that check. Thanks
Related
I am developing an authentication system using express, So I have a unique email field in the database
should I check the email first and if it exists throw a new custom error Or let the database throw the error?
I want to know what is better
Consumers of your API don't and shouldn't know what kind of database you use.
The error that makes it back to them should encapsulate all of it and specifically tell them what is wrong in some standard format with a good HTTP status code.
Database-specific errors leaking to the user should usually be considered a bug.
Both.
You should write code to check that the email exists before you attempt the insert.
But if that check finds no email, you might still get an error, because of a race condition. For example, in the brief moment between checking for the email and then proceeding to insert the row, some other concurrent session may insert its own row using that email. So your insert will get a duplicate key error in that case, even though you had checked and found the email not present.
Then why bother checking? Because if you use a table with an auto_increment primary key, a failed insert generates and then discards an auto-increment value.
This might seem like a rare and insignificant amount of waste. Also, we don't care that auto-increment id's are consecutive.
But I did help fix an application for a customer where they had a problem that new users were trying 1500 times to create unique accounts before succeeding. So they were "losing" thousands of auto-increment id's for every account. After a couple of months, they exhausted the range of the signed integer.
The fix I recommended was to first check that the email doesn't exist, to avoid attempting the insert if the email is found. But you still have to handle the race condition just in case.
I'm working on a tool to generate SPF TXT records and I am trying to get a loop for scanning the domain for the relevant hosts A records. All I get is just a single entry when I supply the domain name as the domain to check. The other hosts in the domain don't return entries...
Any ideas on how to do this?
Did some research and it seems that you need to do a zone transfer to get the records, as allowing a random IP to access the whole domain has too many security vectors. So no real way to do what I was wanting... Sorry for wasting time!
Like many others, I have navigated the SPF/DKIM/DMARC world with some confusion.
About 4 weeks ago or so I finished setting everything (SPF/DKIM/DMARC) up correctly for a GoDaddy-hosted domain that uses Google's mailservers.
I set the _dmarc TXT record to take zero action with p=none and I used Postmark to monitor the results to see what was passing and failing over a week.
After a week or so I looked at the Postmark results and inserted the include: statements for the domains that I wanted to pass, but weren't. Then I waited another week to see the results. However, the results showed that the domains still weren't passing SPF or DKIM. Below is the SPF record, I've redacted parts of it that are revealing, but two of the domains are legit and still aren't passing.
v=spf1 include:_spf.google.com include:freshemail.io include:cherryroad.com ~all
Do I need to use the actual IP addresses in the include statements instead of the domains? Postmark lists these as well so that would be easy if so.
No, you shouldn't copy their IPs in there because they are subject to change, especially Google's.
If it's failing, presumably you have some results (usually in message headers) that tell you exactly which IP is failing, and you can track it down manually though those includes, do a reverse lookup on it, etc.
However, you're also using GoDaddy, which is mostly guaranteed not to work as they either block outbound SMTP or route it through their own servers, so you're very unlikely to get an SPF pass.
The issue was with SPF DNS lookup limits. I had no idea this was a thing and I'm amazed that this isn't mentioned anywhere on the documentation (whether that's Google's official documentation or otherwise) on setting up SPF/DKIM/DMARC, and didn't come up in Googling of this issue. This limit is designed to prevent denial of service attacks and infinite DNS loops.
For anyone else who sees this post
v=spf1 include:_spf.google.com include:freshemail.io include:cherryroad.com ~all
This SPF record actually has almost 15 DNS lookups, and the limit is 10 per domain. You can find out how many SPF DNS lookups your domain has with a service like AutoSPF or Easy DMARC
The solution, once you see your total DNS lookups, comes in four options:
Create subdomains and use those to diversify the records. For example using "email#business.mydomain.com" as the email for freshemail.io. Then on the SPF record for that subdomain, you would only have v=spf1 include:freshemail.io resulting in less than 10 DNS lookups for that domain.
As #Synchro mentioned, you don't want to use IPs because those can very well change, but the concept of using IPs instead of the domain names does essentially work because an IP address doesn't cost a DNS lookup. Check with the support/engineering of whatever service you're using, it's possible that they have an IP (or an IP range) that doesn't change often. You might be able to bring your DNS lookups under ten using this.
Note that Google takes up about 3 DNS Lookups, and you'll probably want to leave that one as the _spf.google.com value
Note that every SPF record also has a 255 character limit, so if you're using only IPs you'll need to break that up into a lot of SPF records probably
Use an SPF flattening or compressing service like AutoSPF. Essentially, these services employ method #2, but do some backend work every few hours to check and update the IP addresses associated with the domains. Then they provide you with a "compressed" record like v=spf1 include:_6359384.autospf.com ~all that references all of your records and results in far fewer DNS lookups.
Create your own method that acts kind of like #2 and #3, using GoDaddy's API and brew up something that performs updated lookups on a schedule/job and updates separate SPF records including all of the IPs.
My DNS provider works perfectly for A records.
I am having great difficulty understanding the syntax of SPF records. I have no prior experience.
The DNS provider supports SPF records and it has two control boxes for information: 'Name' and 'SPF data'.
The A record which functions fine looks like this:
Name: potsandpins.info
IPV4 Address: 45.61.228.207
The SPF record which is giving me no joy looks like this:
Name: potsandpins.info
SPF Data: "v=spf1 a -all" (including the quotation marks)
My emails are received with a red flag in Gmail which says 'Gmail couldn't verify that potsandpins.info actually sent this message'.
Can anyone suggest anything as I've tried all sensible permutations?
You don't seem to currently have an SPF record for potsandpins.info maybe you deleted it because you ran into trouble. Anyway, think of the SPF as a whitelist of any IP addresses or hosts you've given permission to send email on your behalf.
The name would be either the root domain, sometimes designated by the #, or a hostname, foo, which you'd use if you were sending email out as example#foo.example.com.
The SPF data would be the version number (v=spf1), then mechanisms (e.g., a), and then the ip addresses or hosts you'd like to authorize, then the qualifier such as -ALL, which intends a hard fail. You may want to back off from that using ~ALL for now, which intends a softfail. I think it's better to be specific in SPF records as then they're easier to follow exactly what they're authorizing.
Here's an example SPF record. Let's say you wanted to authorize 192.0.2.10 and Google.
v=spf1 ip4:192.0.2.10 include:_spf.google.com ~all
Let's say you wanted to authorize a range of IP addresses and MailChimp:
v=spf1 ip4:192.0.2.0/24 include:servers.mcsv.net ~all
Here's a good article on common mistakes in SPF records.
Then it's important to validate your SPF record using a tool such as the SPF Survey. I like this tool because it gives more detailed, actionable error messages when there's a problem.
if you post the full headers of an example email and indicate any other services you use to send email, then it would be possible to provide more specific advice. For future reference, it's best to provide more details when you post to Stack Overflow as that makes it easier to help. I tried in this post but the information you provided limited how specific the answer could be.
Also, for future reference, it's best to post using example.com rather than a real domain name and use IP addresses from an IPv4 block reserved for documentation.
The blocks 192.0.2.0/24 (TEST-NET-1), 198.51.100.0/24 (TEST-NET-2),
and 203.0.113.0/24 (TEST-NET-3) are provided for use in documentation.
Anyway, I hope this helps.
I have read through the documentation but the difference is still not very clear to me.
My understanding is:
1) include: will allow the IPs listed for the specified domain, and also any additional domains listed in that domain's own SPF records
2) a: simply allows IPs listed for the specified domain
Is that correct?
Another try at an answer for anyone else trying to put this together, and based on the syntax doc that OP is probably referring to, the open-spf.org "SPF Record Syntax" page.
Note: Previously this document lived at http://www.open-spf.org/SPF_Record_Syntax/, but that domain seems to have gone offline permanently in 2009. More details on the history and missing domain can be found here.
My answer is that yes, you seem to have it in mind.
a: Test the IP for a match in the A records for the domain.
include: Fetch the entire SPF record for the domain, evaluate IP against it, and if a PASS is found then that PASS becomes the result of the whole SPF test. If no PASS is found then it's not a fail, but your original/top-level SPF test continues (probably to the -all/~all/?all phase).
Reasons to use "a":
Because it's more predictable and straightforward.
Because you haven't set up SPF on the relevant domains.
Because you don't control those domains and the SPF isn't what you'd want (specifically if it's too lenient in accepting other servers that aren't in it's A records)
Reasons to use "include":
Because you already trust the SPF of the domain.
Because the SPF of the domain is complex, and you want to have a single source of truth for don't-repeat-yourself reasons.
Happy to be corrected!
1) include:other-domain.com just includes SPF records from other-domain.com.
If SPF entries of other-domain.com allows some ip's (for example have an ip4: or a: entry), then those ip will be also be allowed.
2) true