upstream connect error or disconnect/reset before headers. reset reason: protocol error - kubernetes

We are using Anthos service mesh on GKE and for one of the API endpoints we are getting the Below error, Any help would be really appreciated, I Tried providing a port name to the service as mentioned in another post. but nothing solved the problem.
< HTTP/2 502
< content-length: 87
< content-type: text/plain
< date: Fri, 23 Sep 2022 15:45:08 GMT
< server: istio-envoy
< x-envoy-upstream-service-time: 52
<
* Connection #0 to host example.com left intact
upstream connect error or disconnect/reset before headers. reset reason: protocol error

Related

Getting HTTP/1.1 404 error on ReadyAPI REST virtual service on specific resource paths

Why is readyAPI responding with a 404 error when the recourse path is set other than /test in a REST Virtual service.
/test
Sun May 29 16:42:53 BST 2022: DEBUG: http-outgoing >>
POST /test HTTP/1.1
Accept-Encoding: gzip,deflate
Content-Type: application/json
Content-Length: 0
Host: localhost:9019
Connection: Keep-Alive
User-Agent: Apache-HttpClient/4.5.2 (Java/16.0.1)
Sun May 29 16:42:53 BST 2022: DEBUG: http-incoming <<
HTTP/1.1 200 OK
Date: Sun, 29 May 2022 15:42:53 GMT
Content-Type: application/xml
Content-Encoding: gzip
Content-Length: 1249
Server: Jetty(9.4.40.v20210413)
[0x1f][0x8b][0x8][0x0][0x0][0x0][0x0][0x0][0x0][0xff][0xcd]W[0xdb]r[0xe2]F[0x10][0xfd][0x15][0xad][0x9e][0xb9]H[0x2]q+[0xca][0x15][0x8c]|[0x8b][0x8d][0xa1][0x0]o[0x92]J[0xb9][0xa8]A[0x1a]`6[0x92]F;3[0xda]][0xd6][0xcb][0xd7][0xe4]G[0xd3][0xad][0x1b][0x17][0xcb]v[0xaa][0xe2][0xad]
O[0xe2]tk[0xa6][0xa7][0xfb][0x9c]#[0xa9]?[0xe7]gZ[0xe0]y[0x82]Jy[0xe6][0x91]/[0xcc][0xab][0x11][0x97]KE~Y[0x7][0x84][0xf9]5[0x97][0x7][0xfd]z[0x1e][0xd7][0xfa][0xb3]x)][0xc1][0x96]T[0xdc][0xd2][0xed][0x19][0x17][0x1e][0x15][0xb][0x97][0x87]+&[0x2][0xa2][0x18][0xf][0x17]4[0xec][0xd7][0x8f][0x93][0xb4][0xfe][0x90][0x87][0x8a][0xb8]j[0xa0][0x14][0x80][0xb1][0xa2][0xc7][0xb][0x1d][0xc1][0x14]7u[0x88]"gO[0xfa][0x9c][0xeb][0xbd]'=[0xdb][[0xef][0xe9][0xe5][0xd5][0xe9][0x15][0xfd]h?H,-[0xeb][0x87][0xd9]h7[0x8c]v[0xab]i[0xfe]xq[0xa1]g[0x85]b[0x1]e[0x95]"^[0x94]
;f[0xff][0xe6][0xdb][0x88][0xbe][0xb4]?,[0xef][0x93]p[0x1d][0x93]5[0xa6]$[0xff][0x3][0xea][0x11][0xdf]g[0xa4]X[0x17][0x2][0x0]K[0xe2]S[0xb9][0xe2][0xc2][0xa5][0xf][0x92][0x8a][0x1b][0xf][0xd0][0x90]$~L1[0x18][0x91]k[0xa8][0x1b]0iXK[0xaa]HMR%[0x98][0xb7][0xa6]2;[0x85][0xa0].[0x8b][0x18][0x15]V[0x19][0xc3][0x12]YYS[0x88]ETH[0x1e][0xce][0x99][0xf2][0x11][0x19]1hlE[0x87]2[0xa5][0xba]'[0x1]B[0xe][0xf6]&[0xa9][0xb5][0x80][0x6]I[0xa3][0x0]K[0xce][0xf8][0xda] [0x2][0xbe]d>[0xbd][0x8f][0x3]h[0x16][0xe4]5[0xec]n[0xd5]0[0x1b][0xd5]N[0xab]e[0xeb][0xbb]J[0xda][0x17],*[0xb9][0xc0][0x83][0x15]3[0xa9][0xe8][0x8a]+[0xe2]O[0x4][0xf7]bWM[0x4]sag[0xab]]3[0xf0]W[0x12][0xbc][0xf8][0xe6][0xfa]Wl[0xa5][0xce][0xf9][0xb7]gy3[0xec][0xdf][0x9c]#[0xc0][0xa8][0xed][0xc1][0x8b]"[0x16][0xae][0x87][0x1b]"p[0x2][0xa6]qt[0xcf][0xc0][0xfb][0x14]K[0x15]$M+"kABo[0x8e]a[0xbd][0xd7]8[0xd9]C[0x91][0xd5][0xca]a[0xd2][0xe5]qrK[0x6];[0x14]+[0xa4][0xde][0xc7][0xc1]<[0x1]e[0xca][0x9d][0x8]90[0x88]"[0x9f]Q8[0xf4][0x8a][0xf8][0x92]Vt7[0x16][0x82][0x86].[0x12][0xf6][0xea]|[0xa2][0xef][0x81][0xd9]6Xr[0xec][0xf3][0xdf]80[0x9f][0xb8][0xd4][0x3][0x9a][0xe1]$,[0xa3]j4[0xab][0x96]aY[0x9a]a[0xf7][0xec].T[0x5])[0xd0]s[0x1f]NV[0x8][0xe5]IW[0xd9]|[0x1d]q2[0xdd][0xb1][0xcf][0xbe]0r<[0xde][0x19][0x9][0xdd][0xfd][0xe] IW[0xb8]c!5!`[0x9a]mm[0xc4]<[0xcf][0xa7]_[0x99][0xbb][0xd1][0xa6]H[0xb][0x97][0xa9]mz[0x8f][0xb7]$[0xee][0x6][0xb][0x4][0x16][0xb8][0xdc][0xc3][0x95][0x86][0xbf][0x99][0xa6]f^^c[0x1e][0xf6]E[0xa4]g[0xc3][0xd1][0xcb][0xac][0xfd][0xff][0xeb]"=
;S[0xb1][0xbd][0x12]<[0x8e][0xa0][0xc6]?_`*[0xa3][0xc3]d-[0xb8][0xfe][0xac][0xb6][0xe3][0x15][0xd6][0x2][0xe9][0xd6][0xc9]
Y[0xe9]{}^[0xc6][0xfe]
fE[0xb1][0xc6][0xbc]![0xe3][0x84][0x1b][0xd8][0xf]DFi[0x8d][0xf][0xb7][0xf7]#[0xe7][0x8f][0xc1][0xe5][0xfc] 1[0xd3][0xc4][0x9e][0xb7]y[0xbe]C[0xb][0x8a][0xc1][0xad][0xf7]<`![0xb0][0xc5][0xd3][0x6]+EE[0xc8]9:M^[0x17]H[0x8][0xa4][0x9b][0x88][0xd3][0x99]8[0xda]5[0xf][0xa8][0xe6]d[0xb1][0xa3][0x1b][0x14][0xb][0xa8][0xf4][0xb9][0x2][0x9e][0xb]4[0x1b][0xd3][0xea][0x19][0xc6][0x1]~[0x11][0xa2]-[0x99][0x9d][0x14]E[0x93][0x99]Q[0x9f]"[0xf5]s[0xaa][0xb6]r[0xaa][0xa2]q[0x10][0x1][0xd4][0x97]*[0xdf][0xaa],[0xc7][0x7][0xec][0xb5][0x8c][0xdc]?[0xb2]f[0x17][0xb3]Q4[0xc0][0xf9][0xb4][[0x96]e5[0x9b][0x15][0x9d]![0x10][0xae][0xf8][0xde]gJ3p[0xa4]F[0xd7]nt[0xd2][0xff][0x13]8[0xe6][0xde][0xb7][0xc][0xbb][0xd5]l[0xb5]P[,[0x0][0xbf]~[0x98][0xde][0x1][0xba]Q*[0x92][0xbd]z=[0x81][0xe4][0x89][0xed][0xd6][0x99]L[0x3][0xf5]=^[0x9f]&[0x93]2L{1[0xbd]p[0x16]#X[0xee]sLB[0x95][0xb0][0xd3][0xac][0xc1][0x8][0xa3][0xcc][0xe7][0xac][0xc2][r[0xd7]:t[0xbd][0xd3][0x9c][0x17][0xfd]J[0xc6][0x89][0xc5][0x88]![0xf7]y,2[0xfa][0xc1][0xd6]([0x0][0x2][0xd6][0xa2]2[0x7][0xc5][0xd6]D[0xe9]e[0xd6][0x7][0xc3][0xe8]Vr[0xe8][0xa8][0x15][0xc5][0x11][0xf4]"[0x9e]+2$[0x91][0xdc]p[0xa5][0xb9][0x82]KY]ro[0xab]-[0xc9][0xfa][0xfd]{[0xb6]D[0x17][0xce]6[0x1d][0xa6]C[0xed][0xd7][0xc1]p|>[0x83][0xc8]W[0x1e][0xc2]|[0xa7][0x9c][0x7][0x97]>[0xec][0x9c][0x1a]+[0xf0][0x84][0xc9][0xd9][0x81][0xef][0x16][0x86][0xcb][0xe4][0xe4]{[0x98][0xfa][0xff][0xec][0xf6][0xe1][0x0][0xce][0x1e]$[0xa5] [0x1a][0xb6][0x12]1[0xcd]h[0x92]<>[0x89][0xcf]$[0xc9]d[0xbb]K[0x3][0x98]=![0xee]_dR[0xc5][0xfe][0xae]Wj %w[0xc7]9[0x5]S[0xce]>[0x83][0xf7][0xcc][0xb4]w[0x8f]h[0x94][0x94][0xa7]/1[0x1][0x11]n[0xf5][0x13]<d[0x97][0xd0]\[0xb2][0xae][0x9a]UA[0xbd][0xc5][0xe1]<[0xe4][0xc9]s[0xee][0xdf]=P[0xe0][0x99][0xad]b|[0x9d][0x1a][0xea][0xbb]][0xa5]T#[0xf6][0x9b]*[0xb2][0xf]T[0xd4]4m[0xc3],QQ[0xdb][0xee]t;[0xcd][0xb6][0xf5]_[0x19][0xd1]5;U[0x13][0xce]\[0x9d][0xdd]]^[0xdd]\[0xce][0xcf][0xc7][0xbf][0xdf][0xdc]_[0xbd]"'[0xfb]m5[0xd9]o[0x8b][0xe9]U[0xc5]4[0xa1] [0xb3]\1[0xe5][0xe5]>[0x97]Oq#[0xb9][0xa3][0xe1][0x8b][0xcc][0xcf]k[0xd4][0xcf]PJ[0xa6][0x8a]C[0xa1][0xe4]y[0xef][0xa1][0x94]|[0xb6][0xfb]|[0xdc][0x1][0x9a][0xfe][0x8a][0x84][0x9a][0xbb][0xc7]$>[0x82][0x17][0x84][0xf4]=[0xfb][0x9a]D[0x11][0xf8][0x12][0x13]j[0xe3][0x91][0xed][0x87][0xf][0xfa][0xa1][0xc4][0xaa][0x8b][0x17][0x87][0xf5][0xe][0xda]z[0xc4]b[0x12]~[0xcd][0x9f][0xf3][0xf0]&[0x88][0xb8]PN[0xc][0xef][0x12][0x90]o[0xec]vp[0xc3]n[0xd7][0xaf][0xef][0xbf][0xb4][0xc3][0xef][0xa8][0xa3]o[0xa4]z[0xd9][0xf7]T[0x1d]>[0xe4][0xfe][0x1][0xe1][0x96]s{[0xcd][0x0][0x0]
/sendEmail
Sun May 29 16:47:12 BST 2022: DEBUG: http-outgoing >>
POST /sendEmail HTTP/1.1
Accept-Encoding: gzip,deflate
Content-Type: application/json
Content-Length: 0
Host: localhost:9019
Connection: Keep-Alive
User-Agent: Apache-HttpClient/4.5.2 (Java/16.0.1)
Sun May 29 16:47:12 BST 2022: DEBUG: http-incoming <<
HTTP/1.1 404 Not Found
Date: Sun, 29 May 2022 15:47:12 GMT
Transfer-Encoding: chunked
Server: Jetty(9.4.40.v20210413)
0
Are there any specific rules to set the resource paths?
I intend to set the resource path as /messaging/v1/messageDefinitionSends/ for the first action(SendEmail) but it's failing.
second action SendToken is behaving the same, if the path is set as /test it response with a 200 ok status but returns a 404 error on setting the path to /v2/token.
Groovy scripts are used for creating the response.
any explanation for this behavior in ReadyAPI?
I think you might not be setting up your operations correctly. It seems like you're reusing the same endpoint but giving them different names and that is likely the reason why you're getting behavior like this.
What I'd recommend is taking a look at ReadyAPI's documentation on how to add in new operations. I think that once you set up the operations correctly you should be able to access them properly.
https://support.smartbear.com/readyapi/docs/virtualization/configure/create-operations.html
The issue was with the resource path, it was missing.
I assumed it was automatically added. after updating it started working.

How to show TLS handshake information and CONNECT request in Invoke-WebRequest

When I'm accessing a site through HTTPS and/or with HTTP proxy, cURL in Linux provides the -v/--verbose flag to show the CONNECT request to the proxy, as well as the SSL/TLS handshake process (including certificate), like
* Rebuilt URL to: https://www.example.com/
* Trying 192.168.2.1...
* Connected to my-proxy.local (192.168.2.1) port 8080 (#0)
* Establish HTTP proxy tunnel to www.example.com:443
> CONNECT www.example.com:443 HTTP/1.1
> Host: www.example.com:443
> User-Agent: curl/7.47.0
> Proxy-Connection: Keep-Alive
>
< HTTP/1.1 200 Connection established
<
* Proxy replied OK to CONNECT request
* found 148 certificates in /etc/ssl/certs/ca-certificates.crt
* found 597 certificates in /etc/ssl/certs
* ALPN, offering http/1.1
* SSL connection using TLS1.2 / ECDHE_RSA_AES_128_GCM_SHA256
* server certificate verification OK
* server certificate status verification SKIPPED
* common name: www.example.org (matched)
* server certificate expiration date OK
* server certificate activation date OK
* certificate public key: RSA
* certificate version: #3
* subject: C=US,ST=California,L=Los Angeles,O=Internet Corporation for Assigned Names and Numbers,OU=Technology,CN=www.example.org
* start date: Tue, 03 Nov 2015 00:00:00 GMT
* expire date: Wed, 28 Nov 2018 12:00:00 GMT
* issuer: C=US,O=DigiCert Inc,OU=www.digicert.com,CN=DigiCert SHA2 High Assurance Server CA
* compression: NULL
* ALPN, server accepted to use http/1.1
> GET / HTTP/1.1
> Host: www.example.com
> User-Agent: curl/7.47.0
> Accept: */*
>
< HTTP/1.1 200 OK
< Cache-Control: max-age=604800
< Content-Type: text/html
< Date: Mon, 27 Nov 2017 23:08:55 GMT
< Etag: "359670651+gzip+ident"
< Expires: Mon, 04 Dec 2017 23:08:55 GMT
< Last-Modified: Fri, 09 Aug 2013 23:54:35 GMT
< Server: ECS (ord/4C84)
< Vary: Accept-Encoding
< X-Cache: HIT
< Content-Length: 1270
<
< (body...)
Is there anyway to get similar information when using Invoke-WebRequest? Or should I use another CmdLet? I tried -Debug and -Verbose, both of which shows neither information. Even the raw content only contains the actual request after proxy, that is, in the above example the GET / HTTP/1.1.
In brief, I want to see something like the line
> CONNECT www.example.com:443 HTTP/1.1
and
* SSL connection using TLS1.2 / ECDHE_RSA_AES_128_GCM_SHA256
You can get some of this information from the .RawContent property of Invoke-WebRequest. Unfortunately if you opt for Invoke-RestMethod, PowerShell basically discards all of the HTTP information which you're interested.
For this example, I'll use https://jsonplaceholder.typicode.com/posts, which is a good test REST Endpoint for fiddling with things like this.
First, I'll make a connection to the site and store it in a Variable, $response.
$response = Invoke-WebRequest -uri https://jsonplaceholder.typicode.com/posts
Now I can interrogate and pull out some of the useful fields to get some of the info you're looking for.
$response.BaseResponse
IsMutuallyAuthenticated : False
Cookies : {__cfduid=d84018de2d621df9d53eb52d97cd33a651511881763}
Headers : {Transfer-Encoding, Connection, Vary, Access-Control-Allow-Credentials...}
SupportsHeaders : True
ContentLength : -1
ContentEncoding :
ContentType : application/json; charset=utf-8
CharacterSet : utf-8
Server : cloudflare-nginx
LastModified : 11/28/2017 10:17:27 AM
StatusCode : OK
StatusDescription : OK
ProtocolVersion : 1.1
ResponseUri : https://jsonplaceholder.typicode.com/posts
Method : GET
IsFromCache : False
We can also get some good info in the first 25 lines or so of the RawContent property, as seen here. RawContent is, well, raw, so I apply a split on new-lines then use array indexing as depicted by [0..20] to select the first 21 lines.
$response.RawContent.Split("`n")[0..20]
HTTP/1.1 200 OK
Transfer-Encoding: chunked
Connection: keep-alive
Vary: Origin, Accept-Encoding
Access-Control-Allow-Credentials: true
Pragma: no-cache
X-Content-Type-Options: nosniff
CF-Cache-Status: HIT
CF-RAY: 3c4e3f804f9d82f7-ATL
Cache-Control: public, max-age=14400
Content-Type: application/json; charset=utf-8
Date: Tue, 28 Nov 2017 15:09:23 GMT
Expires: Tue, 28 Nov 2017 19:09:23 GMT
ETag: W/"6b80-Ybsq/K6GwwqrYkAsFxqDXGC7DoM"
Set-Cookie: __cfduid=d84018de2d621df9d53eb52d97cd33a651511881763; expires=Wed, 28-Nov-18 15:09:23 GMT; path=/; domain=.typicode.com; HttpOnly
Server: cloudflare-nginx
Via: 1.1 vegur
X-Powered-By: Express
I agree that it would be nice to be able to get this information back too. I'll open up an issue on the github.com/PowerShell repo and see if we can get something like this added in the future, and the link will be added to this answer.

Server-Sent Events with Play: response only received when process killed

I'm trying to get the sample webapp play-streaming-scala to run and in some circumstances I get a weird behavior.
I've got the app running directly on port 80 of some host and I'm checking the output with curl -iv --raw http://somehost/scala/eventSource/liveClock.
What I'm expecting is something like this:
* Hostname was NOT found in DNS cache
* Trying 195.176.3.71...
* Connected to somehost (0.0.0.0) port 80 (#0)
> GET /scala/eventSource/liveClock HTTP/1.1
> User-Agent: curl/7.39.0
> Host: somehost
> Accept: */*
>
< HTTP/1.1 200 OK
HTTP/1.1 200 OK
< Transfer-Encoding: chunked
Transfer-Encoding: chunked
< Content-Type: text/event-stream; charset=utf-8
Content-Type: text/event-stream; charset=utf-8
< Date: Wed, 18 Jan 2017 13:24:55 GMT
Date: Wed, 18 Jan 2017 13:24:55 GMT
<
10
data: 14 24 56
10
data: 14 24 56
10
data: 14 24 56
etc., and clearly see the chunks appear one after the other as time goes by.
Now, on some machines, this works well. On some others on campus, this fails. curl only shows this and then stops:
* Trying 195.176.3.71...
* Connected to somehost (0.0.0.0) port 80 (#0)
> GET /scala/eventSource/liveClock HTTP/1.1
> Host: somehost
> User-Agent: curl/7.43.0
> Accept: */*
>
Now the interesting thing is: if I kill the webapp on the host, curl suddenly “catches up” and spits all the chunks together, closing the connection like this:
10
data: 14 35 20
* transfer closed with outstanding read data remaining
* Closing connection 0
curl: (18) transfer closed with outstanding read data remaining
What can be causing the behavior? What on earth is going on and intercepting these events? Is there any way I can “force flush” something from the Play response?
Turns out the local “hidden” proxy set up automatically by OS X's parental controls system is not forwarding chunked responses properly, thus making a system based on Server-Sent Events inoperable. A shame.

curl: (6) could not resolve host ;401 Unauthorized on Openstack Swift (SAIO)

I'm trying to set up a 'Swift All In One' system on a Ubuntu 12.04 VM by the link:http://docs.openstack.org/developer/swift/development_saio.html.
I use VMware WorkStation 12 Pro on Win7 64bit system and use 'Host-only' network mode.The VM ip address is "192.168.137.200".
When I run the command on the VM:
curl -v -H 'X-Storage-User: test:tester' -H 'X-Storage-Pass: testing' http://192.168.137.200/auth/v1.0
It works well.
But when I run the command on the host machine(Win7 platform), It fails and returns:
* Could not resolve host: test:tester'; Host not found
* Closing connection #0
curl: (6) Could not resolve host: test:tester'; Host not found
* Could not resolve host: testing'; Host not found
* Closing connection #0
curl: (6) Could not resolve host: testing'; Host not found
* About to connect() to 192.168.137.200 port 80 (#0)
* Trying 192.168.137.200... connected
* Connected to 192.168.137.200 (192.168.137.200) port 80 (#0)
> GET /auth/v1.0 HTTP/1.1
> User-Agent: curl/7.20.1 (amd64-pc-win32) libcurl/7.20.1 OpenSSL/0.9.8n zlib/1.
2.3
> Host: 192.168.137.200
> Accept: */*
>
< HTTP/1.1 401 Unauthorized
< Date: Fri, 25 Mar 2016 05:57:24 GMT
< Content-Length: 131
< Content-Type: text/html; charset=UTF-8
< Www-Authenticate: Swift realm="unknown"
< X-Trans-Id: tx081d67bec35b457bb4cb8-0056f4d343
< Vary: Accept-Encoding
<
<html><h1>Unauthorized</h1><p>This server could not verify that you are authoriz
ed to access the document you requested.</p></html>* Connection #0 to host 192.1
68.137.200 left intact
* Closing connection #0
Then I make another Ubuntu 12.04 VM and try to run the command above on the second VM, it works well.
Try to use X-Auth-User and X-Auth-Key headers instead.https://swiftstack.com/docs/cookbooks/swift_usage/auth.html

Sporadic 502 error with gwt rpc calls

I have a GWT application that is giving sporadic 502 errors all of a sudden. I have managed to replicate it by opening multiple connections to the application. Eventually I get a 502 error and the response headers for look as follows:
Server: squid/2.6.STABLE5
Date: Fri, 19 Aug 2011 12:08:03 GMT
Content-Type: text/html
Content-Length: 1014
Expires: Fri, 19 Aug 2011 12:08:03 GMT
X-Squid-Error: ERR_ZERO_SIZE_OBJECT 0
X-Cache: MISS from sentinel.bsgza.bsg.co.za
X-Cache-Lookup: MISS from sentinel.bsgza.bsg.co.za:3128
Via: 1.0 sentinel.bsgza.bsg.co.za:3128 (squid/2.6.STABLE5)
Connection: close
The response headers for the successful rpc calls look like this:
Date: Fri, 19 Aug 2011 13:04:37 GMT
Server: Apache/2.2.14 (Ubuntu)
Content-Encoding: gzip
Content-Disposition: attachment
Content-Length: 249
Content-Type: application/json;charset=utf-8
X-Cache: MISS from sentinel.bsgza.bsg.co.za
X-Cache-Lookup: MISS from sentinel.bsgza.bsg.co.za:3128
Via: 1.0 sentinel.bsgza.bsg.co.za:3128 (squid/2.6.STABLE5)
Connection: keep-alive
We have been able to repeat this on a local server too so it is not a network issue
Try not to route your RPC call via a proxy (Squid). Or at least try to configure Squid to not try to cache them, but only to forward.
Update
It's suggested here that such condition might occur with HTTP POST (used by GWT-RPC) by clients behind PPPoA gateways (cable modems) which have wrong MTU set. Do you see this errors from such clients?