OpenSSL SSL_read: Connection was aborted, errno 10053 (implementing mTLS using istio) - kubernetes

I am trying to implement mTLS between two services. I am using hashicorp vault to manage certs (CA, clients and servers). After deploying the server using istio gateway with secret generated from respective certs. I am trying to access that server using curl. But I am getting the error:
#pemFiles is chain of root and intermediate CA
curl -vvv -HHost:<some-host> --resolve "<some-host>:443:<istio-gateway-ip>" --cacert pemFiles --cert client.crt --key client.key "https://<istio-gateway-ip>:443/" -Lk
* Added <some-host>:443:<istio-gateway-ip> to DNS cache
* Hostname <some-host> was found in DNS cache
* Trying <istio-gateway>:443...
* Connected to <some-host> (<istio-gateway-ip>) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: pemFiles
* CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Request CERT (13):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Certificate (11):
* TLSv1.3 (OUT), TLS handshake, CERT verify (15):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use h2
* Server certificate:
* subject: CN=5402476c-df7d-44bc-9c1b-b0a6afb931d7
* start date: Sep 23 21:28:29 2021 GMT
* expire date: Oct 25 21:28:58 2021 GMT
* issuer: CN=<our-domain>
* SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x293ec1a2fb0)
> GET / HTTP/2
> Host:<some-host>
> user-agent: curl/7.75.0
> accept: */*
>
* OpenSSL SSL_read: Connection was aborted, errno 10053
* Failed receiving HTTP2 data
* OpenSSL SSL_write: Connection was aborted, errno 10053
* Failed sending HTTP2 data
* Connection #0 to host <some-host> left intact
curl: (56) OpenSSL SSL_read: Connection was aborted, errno 10053
When I use the same steps with certs generated manually using OpenSSL, I am not getting any such issue.
When I pass the "--http2" with curl command:
TLSv1.3 (IN), TLS alert, unknown CA (560):
* OpenSSL SSL_read: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca, errno 0
* Failed receiving HTTP2 data
* OpenSSL SSL_write: SSL_ERROR_ZERO_RETURN, errno 0
* Failed sending HTTP2 data
* Connection #0 to host <some-host> left intact
curl: (56) OpenSSL SSL_read: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca, errno 0

A workaround found by OP, in the comments.
The issue was with OP's cert-manager. As a workaround use root CA cert instead of intermediate CA.

Related

Tailscale startup TLS handshake timeout

Several days ago tailscale on my rasberry pi 4 stopped working, I thought it is another wifi issue and rebooted it but after looking at logs I realized wifi isnt issue but in tailscale connectivity:
sudo systemctl status tailscaled.service
tailscaled[154215]: control: LoginInteractive -> regen=true
tailscaled[154215]: control: doLogin(regen=true, hasUrl=false)
tailscaled[154215]: Received error: fetch control key: Get "https://controlplane.tailscale.com/key?v=57": context deadline exceeded
tailscaled[154215]: logtail: upload: log upload of 23460 bytes compressed failed: Post "https://log.tailscale.io/c/tailnode.log.tailscale.io/<>": net/http: TLS handshake timeout
tailscaled[154215]: logtail: [v1] backoff: 27325 msec
tailscale debug ts2021
02:19:14.305767 Fetching keys from https://controlplane.tailscale.com/key?v=57 ...
02:19:24.410086 Do: Get "https://controlplane.tailscale.com/key?v=57": net/http: TLS handshake timeout
Get "https://controlplane.tailscale.com/key?v=57": net/http: TLS handshake timeout
but curl can GET page without any issue
curl -v https://controlplane.tailscale.com/key?v=57
* Trying 18.156.90.224:443...
* Connected to controlplane.tailscale.com (18.156.90.224) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /etc/ssl/certs/ca-certificates.crt
* CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
i reinstalled tailscaled and deleted status file and the problem still persist.

What is the default caching behaviour for Firebase Hosting with Cloud Run?

I have an HTTP service running in Cloud Run that I have mapped to Firebase Hosting using a wildcard to send all routes to the service.
// firebase.json
{
"hosting": [
{
"site": "mysite",
"rewrites": [
{
"source": "/**",
"run": {
"serviceId": "myservice",
"region": "europe-west2"
}
}
]
}
]
}
Update: I've also tried "source": "**" and config with or without the default static hosting using "public": "public" and the result is the same. The CDN appears to cache all content from Cloud Run by default.
The documentation for Firebase Hosting says: (bold my emphasis)
Any requested static content is automatically cached on the CDN. If you redeploy your site's content, Firebase Hosting automatically clears all your cached static content across the CDN until the next request.
However, because Cloud Functions and Cloud Run services generate content dynamically, the content for a given URL can vary based on such things as user input or the user's identity. To account for this, requests that are handled by backend code do not cache on the CDN by default.
However, within the browser developer Network tab I see cache hits (screenshot below):
Note the server: Google Frontend with x-cache: Hit indicating that the CDN served the content. I can confirm there are no Cloud Run logs for subsequent HTTP GET requests.
I've seen other articles online (one on medium) that says the opposite, that by default caching is enabled.
Is the Google documentation currently up to date? If so, what have I missed? I should mention that my HTTP service uses __session cookies for some routes. The example above was before any signin took place, so no cookies were sent from the browser.
I've also tested this using curl and I also get the same result:
> GET / HTTP/2
> Host: myhost
> user-agent: curl/7.68.0
> accept: */*
>
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* Connection state changed (MAX_CONCURRENT_STREAMS == 100)!
< HTTP/2 200
< content-type: text/html; charset=utf-8
< server: Google Frontend
< x-country-code: GB
< accept-ranges: bytes
< date: Mon, 04 Oct 2021 13:17:19 GMT
< x-served-by: cache-lhr7335-LHR
< x-cache: HIT
< x-cache-hits: 1
< x-timer: S1633353439.260200,VS0,VE1
< vary: x-fh-requested-host, accept-encoding
< content-length: 11949
<
Hello world example
package main
import (
"fmt"
"log"
"net/http"
"os"
"time"
)
func main() {
log.Print("starting server...")
http.HandleFunc("/", handler)
port := os.Getenv("PORT")
if port == "" {
port = "8080"
log.Printf("defaulting to port %s", port)
}
// Start HTTP server.
log.Printf("listening on port %s", port)
if err := http.ListenAndServe(":"+port, nil); err != nil {
log.Fatal(err)
}
}
func handler(w http.ResponseWriter, r *http.Request) {
fmt.Fprintf(w, "The time now is %s\n", time.Now())
}
FROM golang:1.17.1-alpine as builder
LABEL stage=builder
WORKDIR /app
COPY . .
RUN go build .
FROM alpine:latest
RUN apk --no-cache add tzdata
RUN mkdir -p /app/bin
WORKDIR /app
COPY --from=builder /app/helloworld ./bin
CMD ["/app/bin/helloworld"]
firebase.json
{
"hosting": {
"public": "public",
"ignore": [
"firebase.json",
"**/.*",
"**/node_modules/**"
],
"rewrites": [{
"source": "**",
"run": {
"serviceId": "helloworld"
}
}]
}
}
Curl output
CALLING CLOUD RUN DIRECTLY
andy#oak:~/projects/test-to-be-removed/public$ curl -v https://helloworld-q2ftkxvbva-uc.a.run.app/
* Trying 216.239.36.53:443...
* TCP_NODELAY set
* Connected to helloworld-q2ftkxvbva-uc.a.run.app (216.239.36.53) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /etc/ssl/certs/ca-certificates.crt
CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use h2
* Server certificate:
* subject: CN=*.a.run.app
* start date: Sep 13 01:08:45 2021 GMT
* expire date: Nov 20 01:08:44 2021 GMT
* subjectAltName: host "helloworld-q2ftkxvbva-uc.a.run.app" matched cert's "*.a.run.app"
* issuer: C=US; O=Google Trust Services LLC; CN=GTS CA 1C3
* SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x55718a90ae10)
> GET / HTTP/2
> Host: helloworld-q2ftkxvbva-uc.a.run.app
> user-agent: curl/7.68.0
> accept: */*
>
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* Connection state changed (MAX_CONCURRENT_STREAMS == 100)!
< HTTP/2 200
< content-type: text/plain; charset=utf-8
< x-cloud-trace-context: 4844f53bed45b95ba5c11f3873217904;o=1
< date: Mon, 04 Oct 2021 15:31:40 GMT
< server: Google Frontend
< content-length: 74
< alt-svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000,h3-T051=":443"; ma=2592000,h3-Q050=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43"
<
The time now is 2021-10-04 15:31:40.017735277 +0000 UTC m=+1034.166310273
* Connection #0 to host helloworld-q2ftkxvbva-uc.a.run.app left intact
CALLING CLOUD RUN VIA FIREBASE HOSTING (1st time Cache MISS)
andy#oak:~/projects/test-to-be-removed/public$ curl -v https://test-to-be-removed-c02e2.web.app/helloworld
* Trying 199.36.158.100:443...
* TCP_NODELAY set
* Connected to test-to-be-removed-c02e2.web.app (199.36.158.100) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /etc/ssl/certs/ca-certificates.crt
CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use h2
* Server certificate:
* subject: CN=web.app
* start date: Sep 20 20:09:34 2021 GMT
* expire date: Dec 19 20:09:33 2021 GMT
* subjectAltName: host "test-to-be-removed-c02e2.web.app" matched cert's "*.web.app"
* issuer: C=US; O=Google Trust Services LLC; CN=GTS CA 1D4
* SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x55ca2e032e10)
> GET /helloworld HTTP/2
> Host: test-to-be-removed-c02e2.web.app
> user-agent: curl/7.68.0
> accept: */*
>
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* Connection state changed (MAX_CONCURRENT_STREAMS == 100)!
< HTTP/2 200
< content-type: text/plain; charset=utf-8
< server: Google Frontend
< x-cloud-trace-context: b62eb4e97a7e8dbd22b1958616455f27;o=1
< x-country-code: GB
< accept-ranges: bytes
< date: Mon, 04 Oct 2021 15:32:12 GMT
< x-served-by: cache-lhr7331-LHR
< x-cache: MISS
< x-cache-hits: 0
< x-timer: S1633361532.898050,VS0,VE234
< vary: x-fh-requested-host, accept-encoding
< content-length: 73
<
The time now is 2021-10-04 15:32:12.06951937 +0000 UTC m=+1066.218094364
* Connection #0 to host test-to-be-removed-c02e2.web.app left intact
CALLING CLOUD RUN VIA FIREBASE HOSTING (2nd time unexpected Cache HIT)
andy#oak:~/projects/test-to-be-removed/public$ curl -v https://test-to-be-removed-c02e2.web.app/helloworld
* Trying 199.36.158.100:443...
* TCP_NODELAY set
* Connected to test-to-be-removed-c02e2.web.app (199.36.158.100) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /etc/ssl/certs/ca-certificates.crt
CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use h2
* Server certificate:
* subject: CN=web.app
* start date: Sep 20 20:09:34 2021 GMT
* expire date: Dec 19 20:09:33 2021 GMT
* subjectAltName: host "test-to-be-removed-c02e2.web.app" matched cert's "*.web.app"
* issuer: C=US; O=Google Trust Services LLC; CN=GTS CA 1D4
* SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x562f5255ae10)
> GET /helloworld HTTP/2
> Host: test-to-be-removed-c02e2.web.app
> user-agent: curl/7.68.0
> accept: */*
>
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* Connection state changed (MAX_CONCURRENT_STREAMS == 100)!
< HTTP/2 200
< content-type: text/plain; charset=utf-8
< server: Google Frontend
< x-cloud-trace-context: b62eb4e97a7e8dbd22b1958616455f27;o=1
< x-country-code: GB
< accept-ranges: bytes
< date: Mon, 04 Oct 2021 15:32:14 GMT
< x-served-by: cache-lhr7371-LHR
< x-cache: HIT
< x-cache-hits: 1
< x-timer: S1633361534.332609,VS0,VE1
< vary: x-fh-requested-host, accept-encoding
< content-length: 73
<
The time now is 2021-10-04 15:32:12.06951937 +0000 UTC m=+1066.218094364
* Connection #0 to host test-to-be-removed-c02e2.web.app left intact

Can't fix Paypal "invalid_client" authentication error

I'm trying to test my Paypal integration from localhost using Paypal Sandbox, but whenever I try to get an access token from Paypal, I get back the following error:
{
"error": "invalid_client",
"error_description": "Client Authentication failed"
}
I make the request with curl:
curl -v https://api.sandbox.paypal.com/v1/oauth2/token \
-H "Accept: application/json" \
-H "Accept-Language: en_US" \
-u "clientId:secret" \
-d "grant_type=client_credentials"
Obviously I am using the actual client id and secret in the request, and I am SURE I am using the correct client id and secret for the sandbox environment (I double checked).
Curl's verbose logs look like this:
curl -v https://api.sandbox.paypal.com/v1/oauth2/token \
> -H "Accept: application/json" \
> -H "Accept-Language: en_US" \
-u "clientId:secret" \
> -d "grant_type=client_credentials"
* Trying 173.0.82.78...
* TCP_NODELAY set
* Connected to api.sandbox.paypal.com (173.0.82.78) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:#STRENGTH
* successfully set certificate verify locations:
* CAfile: /etc/ssl/cert.pem
CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Request CERT (13):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Certificate (11):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server did not agree to a protocol
* Server certificate:
* subject: C=US; ST=California; L=San Jose; O=PayPal, Inc.; OU=PayPal Production; CN=api.sandbox.paypal.com
* start date: Jul 27 00:00:00 2020 GMT
* expire date: Aug 1 12:00:00 2022 GMT
* subjectAltName: host "api.sandbox.paypal.com" matched cert's "api.sandbox.paypal.com"
* issuer: C=US; O=DigiCert Inc; OU=www.digicert.com; CN=DigiCert SHA2 High Assurance Server CA
* SSL certificate verify ok.
* Server auth using Basic with user 'clientId' // The original log has the actual clientId here
> POST /v1/oauth2/token HTTP/1.1
> Host: api.sandbox.paypal.com
> Authorization: Basic QVRQZGl5TTFqeXpQZkdxMW9FQ2xnZG1uUE13VDRQM01yZEJ1MGRGWVZGLWRNVjk1X09xVjZVam1nNU5xRC0wR2ZPT1I3U1ZianYybWQzQXc6RURQSVdoREpPM0dkdndYNVlRaG45QzRCYzBlUTctaEpWZ041ZlVjUlpkbG1Ta0pMelhzZXE0TmpXeEZMV3kwcGdBNE5zTE5MZ1Qyc0pBVW0=
> User-Agent: curl/7.54.0
> Accept: application/json
> Accept-Language: en_US
> Content-Length: 29
> Content-Type: application/x-www-form-urlencoded
>
* upload completely sent off: 29 out of 29 bytes
< HTTP/1.1 401 Unauthorized
< Cache-Control: max-age=0, no-cache, no-store, must-revalidate
< Content-Length: 77
< Content-Type: application/json
< Date: Wed, 07 Apr 2021 08:57:50 GMT
< Paypal-Debug-Id: 891f84363e502
< X-Paypal-Token-Service: IAAS
<
* Connection #0 to host api.sandbox.paypal.com left intact
{"error":"invalid_client","error_description":"Client Authentication failed"}
The problem seems to be here:
ALPN, server did not agree to a protocol
I have NO idea what it means. Could this have something to do with the fact that I'm testing from localhost? How can I fix this?
Decoding the base64 in the verbose log, the client ID passed is ATPdiyM1jyzPfGq1oEClgdmnPMwT4P3MrdBu0dFYVF-dMV95_OqV6Ujmg5NqD-0GfOOR7SVbjv2md3Aw
This does not work when testing at https://developer.paypal.com/demo/checkout/#/pattern/client , so it is not a valid ID.
( Link showing the error, "client-id not recognized for either production or sandbox" )
Create a new REST app, https://www.paypal.com/signin?intent=developer&returnUri=https%3A%2F%2Fdeveloper.paypal.com%2Fdeveloper%2Fapplications

Paypal was working with signature, now requires client certificate

I am integrating PayPal Express Checkout in the Sandbox using SOAP XML. SetExpressCheckout was working properly using a signature for credentials on 9/18/2013. I made no changes to my code or to the web server. I did begin work on Callback using NVP, since no SOAP version of Callback is available. The next day, SetExpressCheckout stopped working. I now get the following error: "80072f0c A certificate is required to complete client authentication" when trying to post to https://api-3t.sandbox.paypal.com/2.0/
Question 1: Could trying to use NVP Callback have caused PayPal's API server to now require a client certificate rather that a signature?
Question 2: Is there some other explanation for this change in behavior?
Question 3: Should I remove the signature from my sandbox account and request an API certificate instead? (Despite PayPal's recommendation that signatures be used rather than certificates.)
Note: I have tried using my own sandbox signature as well as the generic, "always works", sandbox signature. I have also tried posting to both api-3t.sandbox.paypal.com/2.0/ and api.sandbox.paypal.com/2.0/ (without -3t). None of these efforts eliminated the error.
Thanks, Chris H
are you still seeing this issue?
I am unable to reproduce it even using the IP you got back from nslookup.
Here my test with 23.51.43.42. I'm having the same positive result with 23.50.75.42
curl \
-H "Host: api-3t.sandbox.paypal.com" \
-d "USER=guus_1192700083_biz_api1.paypal.com&PWD=XXXXXXXXXX&SIGNATURE=XXXXXXXXXX&VERSION=108&METHOD=SetExpressCheckout&RETURNURL=http://www.paypal.com&CANCELURL=http://www.paypal.com&AMT=0.01&PAYMENTACTION=Authorization" \
https://23.51.43.42/nvp -kv
* About to connect() to 23.51.43.42 port 443 (#0)
* Trying 23.51.43.42...
* 0x8001f188 is at send pipe head!
* STATE: CONNECT => WAITCONNECT handle 0x80057568; line 1032 (connection #0)
* Connected to 23.51.43.42 (23.51.43.42) port 443 (#0)
* successfully set certificate verify locations:
* CAfile: /usr/ssl/certs/ca-bundle.crt
CApath: none
* SSLv3, TLS handshake, Client hello (1):
* STATE: WAITCONNECT => PROTOCONNECT handle 0x80057568; line 1145 (connection #0)
* SSLv3, TLS handshake, Server hello (2):
* SSLv3, TLS handshake, CERT (11):
* SSLv3, TLS handshake, Request CERT (13):
* SSLv3, TLS handshake, Server finished (14):
* SSLv3, TLS handshake, CERT (11):
* SSLv3, TLS handshake, Client key exchange (16):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSL connection using RC4-SHA
* Server certificate:
* subject: C=US; ST=CALIFORNIA; L=San Jose; O=PayPal, Inc.; OU=Partner Support; CN=api-3t.sandbox.paypal.com
* start date: 2013-08-20 00:00:00 GMT
* expire date: 2015-08-21 23:59:59 GMT
* issuer: C=US; O=VeriSign, Inc.; OU=VeriSign Trust Network; OU=Terms of use at https://www.verisign.com/rpa (c)10; CN=VeriSign Class 3 Secure Server CA - G3
* SSL certificate verify ok.
* STATE: PROTOCONNECT => DO handle 0x80057568; line 1164 (connection #0)
> POST /nvp HTTP/1.1
> User-Agent: curl/7.29.0
> Accept: */*
> Host: api-3t.sandbox.paypal.com
> Content-Length: 261
> Content-Type: application/x-www-form-urlencoded
>
* upload completely sent off: 261 out of 261 bytes
* STATE: DO => DO_DONE handle 0x80057568; line 1236 (connection #0)
* STATE: DO_DONE => WAITPERFORM handle 0x80057568; line 1352 (connection #0)
* STATE: WAITPERFORM => PERFORM handle 0x80057568; line 1363 (connection #0)
* HTTP 1.1 or later with persistent connection, pipelining supported
< HTTP/1.1 200 OK
< Server: Apache
< Content-Length: 133
< Content-Type: text/plain; charset=utf-8
< DC: origin2-api-3t.sandbox.paypal.com
< Date: Thu, 03 Oct 2013 20:07:10 GMT
< Connection: keep-alive
< Set-Cookie: DC=origin2-api-3t.sandbox.paypal.com; secure
<
* STATE: PERFORM => DONE handle 0x80057568; line 1533 (connection #0)
* Connection #0 to host 23.51.43.42 left intact
TOKEN=EC%2d03T72513NN7526924&TIMESTAMP=2013%2d10%2d03T20%3a07%3a10Z&CORRELATIONID=4776c1624af4e&ACK=Success&VERSION=108&BUILD=7920936

PayPal REST API: Requesting oauth token returns 500

Two days ago, something broke with our paypal integration. Unfortunately, trying to contact paypal support has proven to be of no help (no reply in 2 days), so I'll try my luck, here.
Requesting the token (curl https://api.paypal.com/v1/oauth2/token -H "Accept: application/json" -H "Accept-Language: en_US" -u "****:****" -d "grant_type=client_credentials") started ALWAYS returning an empty response and well, a 500 error.
Note that the same works just fine with api.sandbox.paypal.com and sandbox credentials.
I double checked our credentials and they are fine. Note as well, that it doesn't matter what credentials I use, it always returns the same - 500.
Here's the output of curl ... -v; looks like you have an internal server error, as the 500 shows:
* About to connect() to api.paypal.com port 443 (#0)
* Trying 173.0.88.98...
* connected
* Connected to api.paypal.com (173.0.88.98) port 443 (#0)
* successfully set certificate verify locations:
* CAfile: /usr/ssl/certs/ca-bundle.crt
CApath: none
* SSLv3, TLS handshake, Client hello (1):
* SSLv3, TLS handshake, Server hello (2):
* SSLv3, TLS handshake, CERT (11):
* SSLv3, TLS handshake, Request CERT (13):
* SSLv3, TLS handshake, Server finished (14):
* SSLv3, TLS handshake, CERT (11):
* SSLv3, TLS handshake, Client key exchange (16):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSL connection using DES-CBC3-SHA
* Server certificate:
* subject: C=US; ST=California; L=San Jose; O=PayPal, Inc.; OU=PayPal Production; CN=api.paypal.com
* start date: 201
* expire date: 201
* subjectAltName: api.paypal.com matched
* issuer: C=U
* SSL certificate verify ok.
* Server auth using Basic with user '****'
> POST /v1/oauth2/token HTTP/1.1
> Authorization: Basic ****
> User-Agent: curl/7.27.0
> Host: api.paypal.com
> Accept: application/json
> Accept-Language: en_US
> Content-Length: 29
> Content-Type: application/x-www-form-urlencoded
>
* upload completely sent off: 29 out of 29 bytes
* additional stuff not fine /usr/src/ports/curl/curl-7.27.0-1/src/curl-7.27.0/lib/transfer.c:1037: 0 0
* HTTP 1.1 or later with persistent connection, pipelining supported
< HTTP/1.1 500 Internal Server Error
< Server: Apache-Coyote/1.1
< Date: Thu, 28 Mar 2013 15:59:53 GMT
< Content-Length: 0
< Connection: close
<
* Closing connection #0
* SSLv3, TLS alert, Client hello (1):
Thanks!
This is something that is currently being looked into. If you have not already opened a ticket with PayPal MTS I would advise opening up a ticket with Technical Support, so that your issue can be added to the examples. You will also then be notified once the issue has been resolved.
edit: This is now resolved as of 04:12 AM GMT (31/03/2013).