My server is suddenly not receiving IPN requests from Paypal any more, worked for years - paypal

Question

I've had IPN working for years: I'm using a classic Paypal Button to initiate a purchase, and then Paypal sends me an IPN to my web server so that I can record the payment and send the customer an email.
But since Oct 4, 2021, Paypal has not been able to send me IPNs any more.
The "Instant Payment Notification (IPN) history" shows all IPNs since then as either Failed or Pending.
My server, however, responds just fine to the requests if I invoke the IPN URL from a browser. My server (Debian 10 with Apache 2) has not had any configuration changes or package updates in the past week, especially not since the time when it still received IPNs (Oct 3).
The only change I could find around that date (on Oct 3) is an automatic renewal of the Let's Encrypt SSL cert, which happened many times before without causing any such trouble with IPN. I have since recreated the Let's Encrypt cert, even as a dedicated one just for that domain given to Paypal (before the cert was shared between multiple domains on that server), with no improvement.
What works
Invoke the URL (https://paypal-api.tempel.org/) from curl or a web browser. My CGI script is invoked, and the Apache access.log shows the GET request.
What does not work
Use the IPN Simulator, specifying the same URL - then the simulator page shows
IPN was not sent, and the handshake was not verified. Review your information
at the top and I see no request in the Apache access.log
What has changed?
I have checked my firewall (which I hadn't changed in a while), even disabled it, to no avail.
Why would Paypal suddenly fail to even contact my server?
I suspect it has to do with the SSL cert, as its renewal date is the only date relation I could find and since Paypal not accepting the SSL cert would probably also lead to the effect that it won't even issue the GET or POST request to my server, hence no log entry about it.
Am I doing the testing with the Simulator right? How else could I find out why Paypal doesn't perform the "handshake"?
SSL cert oddities
Even though certbot says that the SSL cert is up-to-date, I get this output from openssl for the domain:
% openssl s_client -connect paypal-api.tempel.org:443 -showcerts -CApath /etc/ssl/certs/
CONNECTED(00000005)
depth=1 O = Digital Signature Trust Co., CN = DST Root CA X3
verify error:num=10:certificate has expired
notAfter=Sep 30 14:01:15 2021 GMT
verify return:0
depth=1 O = Digital Signature Trust Co., CN = DST Root CA X3
verify error:num=10:certificate has expired
notAfter=Sep 30 14:01:15 2021 GMT
verify return:0
depth=3 O = Digital Signature Trust Co., CN = DST Root CA X3
verify error:num=10:certificate has expired
notAfter=Sep 30 14:01:15 2021 GMT
verify return:0
---
Certificate chain
0 s:/CN=git.tempel.org
i:/C=US/O=Let's Encrypt/CN=R3
-----BEGIN CERTIFICATE-----
…
-----END CERTIFICATE-----
---
Server certificate
subject=/CN=git.tempel.org
issuer=/C=US/O=Let's Encrypt/CN=R3
---
No client certificate CA names sent
Server Temp Key: ECDH, X25519, 253 bits
---
SSL handshake has read 4574 bytes and written 281 bytes
---
New, TLSv1/SSLv3, Cipher is ECDHE-RSA-CHACHA20-POLY1305
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
Protocol : TLSv1.2
Cipher : ECDHE-RSA-CHACHA20-POLY1305
Session-ID: 523AEDD29014010191C7B3DD16F4DC7179B8D91372C3063CEF8D6AD2ED7B800E
Session-ID-ctx:
Master-Key: 36D105B8EAD319DA6B8D2EA30484FC6676F2A2D64CA0EE8889B16396ECA71178FE5154638DCE1E9AF051D8D2FA44472E
Start Time: 1633776598
Timeout : 7200 (sec)
Verify return code: 0 (ok)
---
closed
Why does it say "certificate has expired" at the top, even though it's ok at the end? Could this be the issue?
More strangely, when I give the IPN Simulator another domain of mine, which is also using Let's Encrypt, e.g. www.tempel.org, the Simulator responded yesterday with a positive result, yet today, it gives the same "handshake" error. Also, openssl already gave the same "certificate has expired" error yesterday when the IPN Simulator still accepted it.
So, this all seems to have to do with Let's Encrypt's certificates, somehow. But how do I fix this, apart from buying a different SSL cert?
Or is this just some odd caching issue? When I open https://paypal-api.tempel.org/ in Safari, and check its cert info, it does not indicate any expired certs. What's going on here?
Related to recent expiry of X3 root cert?
I have a hunch that this is related to the expiry of one of Let's Encrypt's root certificate on Sept 30, 2021. The openssl command I've shown above mentions that X3 certificate, indeed.
I am still not clear on how to fix this, however. I understand the instructions in the SO article about disabling the outdated X3 cert, but don't understand what to do then, yet. Probably force-renew the certs with certbot. But even that doesn't make the Simulator happy, yet, and the openssl cmd keeps mentioning the X3 root regardless.
Updated the certs
After reading here about fixing the X3 expiry issues, I renewed my cert by forcing the X1 root (h1.tempel.org is another domain on the same server):
certbot renew --cert-name h1.tempel.org --preferred-chain "ISRG Root X1" --force-renewal
and then restarted Apache (apachectl graceful).
Oddly, since then, when I give https://h1.tempel.org/ to the IPN Simulator, I get the desired "IPN was sent and the handshake was verified." message. But using https://paypal-api.tempel.org/, which is the same certificate, but just a different Apache site on the same server, I still get the "the handshake was not verified" error msg. I am out of ideas.
Can anyone see a difference in the certs between https://h1.tempel.org/ and https://paypal-api.tempel.org/?

Answer 1

I've finally managed to get it to work again - in parts.
After I had found out that the simulator accepts https://h1.tempel.org/ but not https://paypal-api.tempel.org/, even though both are valid pages that return status 200, and both are on the same server and use the same SSL certificate, I reconfigured the web server so that the h1 subdomain would also serve the cgi handler. Then I updated the handler URL at https://www.paypal.com/cgi-bin/webscr?cmd=_profile-ipn-notify to use the h1 subdomain (instead of paypal-api, and that fixed the issue, at least for NEW Paypal transactions.
However, trying to resend the previously failed notifications still fails, because they keep using the old URL.
Paypal Support responds
After finally having contacted Paypal business support, I learned:
They're using Cisco / Talos Intelligence to rate ANY outside page they contact, apparently.
And for some inexplicable reason, Talos currently rates my IPN URL's domain as "malware"! No other blacklisting site rates my site, because it's virtually invisible, and there's no content delivered by that site, apart from the IPN cgi handler, which delivers nothing but an empty reply with either status 200 or 500, depending on the content.
Which means I now have to get Talos to remove that malware rating (which is still pending 24 hours later!). Interestingly, Talos only rates the domain, not all sites served by that server's IP address. Therefore, even payments.tempel.org, which is literally nothing but a ServerAlias to paypal-api.tempel.org, is rated fine.
Paypal support also explained that the notification URL is stored in the database, and gets re-used when trying to re-send a failed notification, hence it would keep using the old (currently malware-rated) URL despite me having told Paypal to use a different URL now.
But now, a day later, trying resend again on the failed message, suddenly does use the newly set URL, regardless of Paypal support's explanation that this would not happen with re-sends.
So, the moral of the story: Sometimes it's not your fault but it's Paypal who gets this wrong. Or Cisco.
Also worth noting: Paypal won't alert you if they suddenly decide to consider your IPN URL part of a malware site. Nothing in the help or their testing tools will tell you why they're not contacting you any more, leaving you guessing until you contact them (In that regard, the initial answer (now deleted) was right to suggest that I need to contact Paypal directly because Paypal had blacklisted me for some reason). And they can't even whitelist you, but instead leave you fighting with Cisco over getting your rating back to "harmless".