I work for mid-sized email sender, and we are very conservative when it comes to the mailing that we do. We only email users that opt-in to our service and we follow all best practices. Our user reported spam rate as reported in Google Postmaster Tools has been in the 0% to 0.2% for years.
Suddenly, on approximately 10/15/2020 our user-reported spam rate in Postmaster Tools jumped to 1.1%, then 1.9% a few days later, then 2.4% on both 11/01/2020 and 11/02/2020. In the days between and after these peaks it has still been high, averaging close to 1%.
Something seemed to happen on or around 10/15/2020 that we cannot account for.
Despite this, the rest of our metrics from SendGrid, the mailing platform we use, remain completely normal - our volume hasn't changed, our opens/unique open rate is the same it has always been, our click rates remain the same, and even our unsubscribe rate has not changed.
We haven't changed our templates or our content in over a year, we haven't introduced any new mailings, and our other feedback loops for other providers don't show this increase in spam rate.
Our domain and IP reputation in Postmaster Tools remains at "High", but this high spam rate that has continued daily for nearly a month now has us extremely worried and we aren't sure what to do about it.
Has anyone else ever experienced something like this, or have any advice on what we should do?
As others pointed out in the comments, Google later announced that on October 14th of 2020 they had rolled out a change to the way that they calculated spam rates.
Related
I don't know where to ask this, so I'm going to just ask it here and handle the backlash when ever it comes.
First let me state that I'm familiar with the recent issues relating to Pepsi and YouTube how ever what I am currently unsure of is, does this effect website publishers also, or has my traffic just all around stopped converting?
I'm currently getting around 300,000 impressions per day and my revenue has dropped to as low as 0.19cpc when before it was sitting almost around a dollar for every registered click.
Question basically comes down to, are you guys getting effected by this also? What can I do in the mean time if this is in correlation with the advertisers opting out, because it's getting really hard to manage servers with no revenue.
Yes it seems to have had a drastic impact on my earnings as well, although I'm not doing anywhere close to the impressions you're doing.
My cpc and rpm have hit an all time low and its disastrous. I'm seeing the same cpc as you btw, we're not alone other publishers are complaining about the same issue.
I'm considering dropping a few adsense units till i see some better figures.
I created a simple python flask application with 256MB to play around with.
They state that they give up to 2GB free of ram... so I was way below it
After 30 min I see that I'm billed $0.02..
I tried to post it in their "stackoverflow" variant but I got 500 error :).. so my last resort is to ask here, maybe someone can clarify.
The fear will be to put an app there, forget about it, then at the end of the month you wake up with a $999 bill.
I don't really understand your concern, but I'm still going to try to help.
If you are afraid that you will be charged, this is not going to happen while you have the 30-days trial account. You will only have to pay if you decide to keep the account after the 30-days are over.
If you are afraid that you are being over-charged, you can always keep track of what you spent through the "View Full Usage Details", where there are all the pricing details (per region, per app, etc). You can also predict how much an app will cost beforehand using the calculator on the platform (https://new-console.au-syd.bluemix.net/?direct=classic/#/pricing/cloudOEPaneId=pricing&paneId=pricingSheet)
Lastly, if you are afraid that you will spend more than you want, there is a spending notification policy (which applies only to paid accounts, of course). I believe there is also an option to stop applications if they exceed a limit, but personally I haven't used these feature yet.
I hope this helps :)
I just started using owasp rules and got tons of false positives. Example someone in the description field has written:
"we are going to select some users tomorrow for our job platform."
This is detected as sql injection attack (id 950007). Well it is not. It is valid comment. I have tons of this kind false positives.
First I have set up SecRuleEngine DetectionOnly to gather information.
Then I started using "SecRuleUpdateTargetById 950007 !ARGS:desc" or "SecRuleRemoveById 950007" and I already spend a day for this. modsec_audit.log is alreay > 100MB of size.
I am interested from your experience, how long do you fine tune it (roughly). After you turn it on, do you still get false positives and how do you manage to add white lists on time (do you analyze the logs daily) ?
I need this info to tell by boss the estimation for this task. It seems that will be long lasting.
Totally depends on your site, your technology and your test infrastructure. The OWASP CRS is very noisy by default and does require a LOT of tweaking. Incidentally there is some work going on this and next version might have a normal and a paranoid mode, to hopefully reduce false positives.
To give an example I look after a reasonably sized site with a mixture of static pages and a number of apps written in wide variety of technologies (legacy code - urgh!) and a fair amount of visitors.
Luckily I had a nightly regression run in our preproduction environment with good coverage, so that was my first port of call. I released ModSecurity there after some initial testing, in DetectionOnly mode and tweaked it over a month maybe until I'd addressed all of the issues and was comfortable moving to prod. This wasn't a full month of continuous work of course but 30-60 mins on most days to check the previous nights run, tweak the rules appropriately and set it up for next night's run (damn cookies with their random strings!).
Next up I did the same in production, and pretty much immediately ran into issues with free text feedback fields like you have (of course I didn't see most of these in regression runs). That took a lot of tweaking (had to turn off a lot of SQL Injection rules for those fields). I also got a lot of insight how many bots and scripts run against our site! Most were harmless or Wordpress exploit attempts (luckily I don't run Wordpress), so no real risk to my site, but still an eye opener. I monitored the logs hourly initially (paranoid!), then daily, and then weekly.
I would say from memory that it took another 3 months or so until I was fully comfortable turning it on fully and checked it a lot over the next few days. Luckily all hard work paid off and very few false positives.
Since then it's been fairly stable and very few false alerts - mostly dues to bad data (e.g. email##example.com entered as an email address for a field which didn't validate email addresses properly) and I often left those place and fixed the field validation instead.
Some of the common issues and rules I had to tweak are given here: Modsecurity: Excessive false positives (note you may not need or want to turn off all these rules in your site).
We have Splunk installed on our web servers (basically a tool which sucks up log files and can then be searched or automatically alert or report on issues). So set up a few alerts for when the more troublesome, free text fields fields caused a ModSecurity block (have corrected one or two more false positives there), and also on volume (so we get an alert when a threshold passed and could see we were under a sustained attack - happens few times a year) and weekly/monthly reporting.
So a good 4-5 months to implement from scratch end to end with maybe 30-40 man days work over that time. But it was a very complicated site and I had no prior ModSecurity/WAF experience. On plus side learned a lot about web technologies, ModSecurity and got regexpr-blindness from staring at some of the rules! :-)
Does anyone know a good tool to test whether your emails are going into spam folders?
My web app generates emails to users, and I've been getting a lot of reports back from people saying "hey, no one ever responded to my message".
I have SPF rules in place and functioning correctly (email header shows an spf pass). I've also run my message through spam assassin and it scores very low.
Any other ideas?
To know if your email goes in the inbox, you need to get a metric called "Inbox Placement Rate". This indicator can be provided by Return Path, but it's quite expensive. If you're not sending huge volumes it might not worth it. The only way to measure the IPR is actually to have a certain number of test inboxes... In other words: the only way to chech that your email is not in the spam folder is to make the test and see what happen. There is not other magic solution and that's what Return Path is doing.
This means that when you hear about people claiming they have a 99% deliverability / delivery, it might be true be it just means that the email was "accepted" or "delivered" by the ISP. It's a lot, but it's not everything!
What you should do is the following: use an ESP focusing on deliverability. Personally I work for Mailjet. I believe it's the best value you can get: personalized DKIM and SPF are provided for free, you get the antispam scorings, the analytics, Ip reputation monitoring, throttling, etc. It's an all in one tool to avoid the headaches of optimizing yourself. It's more expensive that Amazon SES because you get a lot of added value services, but it has much lower prices than a lot of traditional ESPs!
Bottom line is: optimizing everything yourself is a full time job. Knowing exactly if an email is in the inbox or not will cost you a lot. The best way to proceed is to:
respect the best practices (opt in, not too much images, no red, etc.)
get some metrics such as open rates, click rates, delivery, etc. and watch their evolution over time. Any change from one sending to the other might be a signal for a problem you want to investigate.
Use a tool that takes care all the deliverability optimizations
Mailjet is cool because no matter which plan you pick, you get to use all the options. But if you want a full overview of what is existing, check out this comparison table:
http://socialcompare.com/en/comparison/transactional-emailing-providers-mailjet-sendgrid-critsend
If you're a perfectionist who wants to finetune the layout, how the emails are displayed etc. Check out Litmus, it's also a quite powerful tool!
http://litmus.com/
Simple answer: Use Mailgun!!!!
http://mailgun.net/
They will do all of your email deliverability and setup for you and give you a powerful API to build on! They are amazing. You'll never have to worry about domain keys or SPAM filtering again!
You should also check that your IP is not on any of major blacklists. dnsbl.info
This will at least give you an idea if you actually are getting flagged as spam.
For the past two years, we've used the service DeliveryMonitor.com. However, they've stopped accepting new applications which is a big red flag...
I'm currently evaluating the service from emailreach.com using their free trial
... We are now using DeliveryWatch.com with pertty good results thus far...
I have landed a side project where a company wants me to develop a database / smart client to track participants in their various programs. This app will be written in C# and MS SQL Server.
I was wondering if there are any tips on how to actually bid out a price for the job?
Should I charge a flat rate (if so how do you charge it?)
Should I charge by the hour (if so what is a going rate?)
Should I enter into some sort of contract?
Any comments or tips would be very helpful, as I am completely new to doing "side gigs" outside of my normal programming job.
Thanks in advance
Yes you should enter into a contract, you need to spell out the terms and conditions of the work the expected deliverables and payment. You will also want to indicate who owns the source code.
I prefer Time & Material contracts, that is they pay an hourly rate. This allows them to change the scope without having to renegotiate the terms. If you go with a fixed bid project then as the users starts to learn what they want and change the specification, which they will, you will have to make a choice. Either you do it for free or you negotiate a scope change.
Your goal is to get the gig, and once you have it, make enough money for it to be worth your while, right? With that in mind, you can do flat rate if the client demands it, but otherwise try and get it hourly, because there will always be snags, and the client will always throw changes at you from the first day to the last minute that will suck down your time if you let it. Either way, get a contract that specifies what you are going to deliver and how to handle change requests.
I did a succession of fixed price contracts for a client once, and it worked great. Then they said they were having a little problem getting a signature on this contract from head-office, but if I were to just do the work they'd make sure I got paid for the work eventually. So I spent two months working on it, only to be told that they had been turned down by head office and so I wasn't going to get paid. I'm 90% sure they'd made a copy of my work before I could delete it from their servers, but nothing I could prove in court. So I'll never work without a signed contract again. If they don't sign a contract on day one, get the hell out of there.
If you're a beginner I'd advise against a flat rate ... unless you are prepared to shoulder the risk of more work than is paid for. It is inherently hard to estimate the effort a project takes.
The rate you can take depends on your market and how you value your time. It might be interesting accepting a low rate gig to get started in a certain market when you have free times at your hand or if you're just interested at the subject.
Well there are a few things you need to worry about.
Contract, yes, this is 100% needed, otherwise you might not get paid. The details of the contract are what might by challenging, especially about how to charge, which is what you're asking.
If you charge an hourly rate, you have to worry about being fair when you charge, and I'm sure the client will worry about this. If you're on the clock, it's not fair to check your email, YouTube, or even Stack Overflow (gasp!). Then there is also the question about whether you should charge if you need to learn some new skill. For example, if you have to learn library A, is this charged to the client or should it be done on your own time?
With a flat rate, you have to worry about keeping yourself productive so that you don't end up with a lower hourly rate. On the other hand, if the job turns out to be easy, you can finish quickly, and end up with a good hourly rate. This probably would be a lot more fair to both parties since no one would be monitoring you so that you're not goofing off and you wouldn't be doing something unethical if you decided to surf the internet while you're working.
If this is a side project and not a long-term job, I'd highly recommend a flat rate. Make sure you estimate it properly though. Take a guess of how many hours you think it will take you, add 20%, and then multiply this by some reasonable hourly rate. The other party will most likely be much more open to a flat rate since then it will be like they are paying for a finished product, rather than a developer.
Good luck!
Bidding is a tight rope walking game. Neither too low nor too much. The contract or the hourly rate always drives bidders mad. Be just in your bidding price and do not undercut too much else you might end up working for nothing. Also bid in such sites that hold the project price in escrow once the winning bidder is chosen. One good resource can be found at :- http://www.freelancing-on-net.com/writing_a_successful.php#
Even if you are bidding hourly, customers always want at least an estimate. Don't forget to account for all the activities that you don't think of when you're thinking of creating software.
Specifically, think of project management, meeting and call times with the client, end user meetings, source code management, testing, and bug fixing. In short, everything that you do for this client that prevents you from making money with another client needs to be billed to this client. You can account for that with the number of hours bid or with the hourly rate (usually a combination of the two).
I used to hear, "Prepare your best estimate. Then double it." Then I heard, "Prepare your best estimate, then times it by Pi." Now I hear, "Prepare your best estimate, then times it by Pi, then double it." :)
Unfortunately, the only way you'll get good at accurately bidding projects is to bid some, complete the project, and see where you were over or under.
You're always motivated to bid low to ensure that you get the project. But if you lose money doing a project for half your cost, getting the project might not have been a good thing.