Using HAProxy 1.8, I want to slow down certain traffic. This all works when testing over HTTP 1.1. However as soon as http/2 (h2) is enabled in HAProxy, the 10s delay is no longer taking effect. How can I delay h2 requests?
frontend web
bind [...] alpn h2,http/1.1
tcp-request inspect-delay 10s
tcp-request content accept if WAIT_END
[...]
I'm testing using curl:
time curl -I 'https://[url]/' -v
* Trying 10.233.1.97...
* TCP_NODELAY set
* Connected to [url] (10.233.1.97) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
[...]
* ALPN, server accepted to use h2
[...]
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x7fd3f5808200)
> GET / HTTP/2
> Host: [...]
> User-Agent: curl/7.64.1
> Accept: */*
>
* Connection state changed (MAX_CONCURRENT_STREAMS == 100)!
< HTTP/2 411
HTTP/2 411
< content-type: text/html; charset=us-ascii
content-type: text/html; charset=us-ascii
< server: Microsoft-HTTPAPI/2.0
server: Microsoft-HTTPAPI/2.0
< date: Thu, 02 Apr 2020 19:18:22 GMT
date: Thu, 02 Apr 2020 19:18:22 GMT
< content-length: 344
content-length: 344
<
* Excess found in a non pipelined read: excess = 344 url = / (zero-length body)
* Connection #0 to host app.cloudbilling.nl left intact
* Closing connection 0
curl -I 'https://[url]/' -v 0.02s user 0.01s system 28% cpu 0.101 total
Related
I'm trying to retrieve the list of files stored in an Azure File Storage account using the REST API and curl, I correctly computed headers according to the documentation by using the shared key , but curl request neither returns the files list nor any error message.
Here is my request and the response:
curl -v -H "Authorization: SharedKey myaccount:bAJKeY0xyOZLSJOLDoHfXXOqfA4kOGo1DVFP3BejhY8=" -H "x-ms-date:Mon, 13 Aug 2018 15:22:31 GMT" -H "x-ms-version:2017-07-29" --url https://myaccount.file.core.windows.net/myshare/mydir?restype=directory&comp=list
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Trying 52.239.140.8...
* Connected to myaccount.file.core.windows.net (52.239.140.8) port 443 (#0)
* found 148 certificates in /etc/ssl/certs/ca-certificates.crt
* found 597 certificates in /etc/ssl/certs
* ALPN, offering http/1.1
* SSL connection using TLS1.2 / ECDHE_RSA_AES_256_GCM_SHA384
* server certificate verification OK
* server certificate status verification SKIPPED
* common name: *.file.core.windows.net (matched)
* server certificate expiration date OK
* server certificate activation date OK
* certificate public key: RSA
* certificate version: #3
* subject: CN=*.file.core.windows.net
* start date: Thu, 09 Nov 2017 05:42:03 GMT
* expire date: Sat, 09 Nov 2019 05:42:03 GMT
* issuer: C=US,ST=Washington,L=Redmond,O=Microsoft Corporation,OU=Microsoft IT,CN=Microsoft IT TLS CA 5
* compression: NULL
* ALPN, server did not agree to a protocol
GET /myshare/mydir?restype=directory HTTP/1.1
Host: myaccount.file.core.windows.net
User-Agent: curl/7.47.0
Accept: */*
Authorization: SharedKey
myaccount:bAJKeY0xyOZLSJOLDoHfXXOqfA4kOGo1DVFP3BejhY8=
x-ms-date:Mon, 13 Aug 2018 15:22:31 GMT
x-ms-version:2017-07-29
HTTP/1.1 200 OK
Transfer-Encoding: chunked
Last-Modified: Fri, 27 Apr 2018 16:11:14 GMT
ETag: "0x8D5AC597FF96B3D"
Server: Windows-Azure-File/1.0 Microsoft-HTTPAPI/2.0
x-ms-request-id: 75d6d7c8-f01a-0011-5b19-33104d000000
x-ms-version: 2017-07-29
x-ms-server-encrypted: true
Date: Mon, 13 Aug 2018 15:22:29 GMT
{ [5 bytes data]
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
* Connection #0 to host myaccount.file.core.windows.net left intact
No XML with file list is returned.
I tried to retrieve the share list under myaccount and it works, as well as downloading a single file, but I cannot receive the list of files under a directory.
Two points:
See url in curl command
--url https://myaccount.file.core.windows.net/myshare/mydir?restype=directory&comp=list
You forget to put the url in "" so the parameter &comp=list is cut because & is a reserved sign. This is also proved by the output GET /myshare/mydir?restype=directory HTTP/1.1.
Commonly speaking, if the url misses the comp parameter, we should get error message AuthenticationFailed because comp is used in generating SharedKey. However you get HTTP/1.1 200 OK with the SharedKey.
Based on the response headers you get, I guess you also missed the comp when constructing the SharedKey, so the SharedKey and url is capable to get directory properties correctly.
When I'm accessing a site through HTTPS and/or with HTTP proxy, cURL in Linux provides the -v/--verbose flag to show the CONNECT request to the proxy, as well as the SSL/TLS handshake process (including certificate), like
* Rebuilt URL to: https://www.example.com/
* Trying 192.168.2.1...
* Connected to my-proxy.local (192.168.2.1) port 8080 (#0)
* Establish HTTP proxy tunnel to www.example.com:443
> CONNECT www.example.com:443 HTTP/1.1
> Host: www.example.com:443
> User-Agent: curl/7.47.0
> Proxy-Connection: Keep-Alive
>
< HTTP/1.1 200 Connection established
<
* Proxy replied OK to CONNECT request
* found 148 certificates in /etc/ssl/certs/ca-certificates.crt
* found 597 certificates in /etc/ssl/certs
* ALPN, offering http/1.1
* SSL connection using TLS1.2 / ECDHE_RSA_AES_128_GCM_SHA256
* server certificate verification OK
* server certificate status verification SKIPPED
* common name: www.example.org (matched)
* server certificate expiration date OK
* server certificate activation date OK
* certificate public key: RSA
* certificate version: #3
* subject: C=US,ST=California,L=Los Angeles,O=Internet Corporation for Assigned Names and Numbers,OU=Technology,CN=www.example.org
* start date: Tue, 03 Nov 2015 00:00:00 GMT
* expire date: Wed, 28 Nov 2018 12:00:00 GMT
* issuer: C=US,O=DigiCert Inc,OU=www.digicert.com,CN=DigiCert SHA2 High Assurance Server CA
* compression: NULL
* ALPN, server accepted to use http/1.1
> GET / HTTP/1.1
> Host: www.example.com
> User-Agent: curl/7.47.0
> Accept: */*
>
< HTTP/1.1 200 OK
< Cache-Control: max-age=604800
< Content-Type: text/html
< Date: Mon, 27 Nov 2017 23:08:55 GMT
< Etag: "359670651+gzip+ident"
< Expires: Mon, 04 Dec 2017 23:08:55 GMT
< Last-Modified: Fri, 09 Aug 2013 23:54:35 GMT
< Server: ECS (ord/4C84)
< Vary: Accept-Encoding
< X-Cache: HIT
< Content-Length: 1270
<
< (body...)
Is there anyway to get similar information when using Invoke-WebRequest? Or should I use another CmdLet? I tried -Debug and -Verbose, both of which shows neither information. Even the raw content only contains the actual request after proxy, that is, in the above example the GET / HTTP/1.1.
In brief, I want to see something like the line
> CONNECT www.example.com:443 HTTP/1.1
and
* SSL connection using TLS1.2 / ECDHE_RSA_AES_128_GCM_SHA256
You can get some of this information from the .RawContent property of Invoke-WebRequest. Unfortunately if you opt for Invoke-RestMethod, PowerShell basically discards all of the HTTP information which you're interested.
For this example, I'll use https://jsonplaceholder.typicode.com/posts, which is a good test REST Endpoint for fiddling with things like this.
First, I'll make a connection to the site and store it in a Variable, $response.
$response = Invoke-WebRequest -uri https://jsonplaceholder.typicode.com/posts
Now I can interrogate and pull out some of the useful fields to get some of the info you're looking for.
$response.BaseResponse
IsMutuallyAuthenticated : False
Cookies : {__cfduid=d84018de2d621df9d53eb52d97cd33a651511881763}
Headers : {Transfer-Encoding, Connection, Vary, Access-Control-Allow-Credentials...}
SupportsHeaders : True
ContentLength : -1
ContentEncoding :
ContentType : application/json; charset=utf-8
CharacterSet : utf-8
Server : cloudflare-nginx
LastModified : 11/28/2017 10:17:27 AM
StatusCode : OK
StatusDescription : OK
ProtocolVersion : 1.1
ResponseUri : https://jsonplaceholder.typicode.com/posts
Method : GET
IsFromCache : False
We can also get some good info in the first 25 lines or so of the RawContent property, as seen here. RawContent is, well, raw, so I apply a split on new-lines then use array indexing as depicted by [0..20] to select the first 21 lines.
$response.RawContent.Split("`n")[0..20]
HTTP/1.1 200 OK
Transfer-Encoding: chunked
Connection: keep-alive
Vary: Origin, Accept-Encoding
Access-Control-Allow-Credentials: true
Pragma: no-cache
X-Content-Type-Options: nosniff
CF-Cache-Status: HIT
CF-RAY: 3c4e3f804f9d82f7-ATL
Cache-Control: public, max-age=14400
Content-Type: application/json; charset=utf-8
Date: Tue, 28 Nov 2017 15:09:23 GMT
Expires: Tue, 28 Nov 2017 19:09:23 GMT
ETag: W/"6b80-Ybsq/K6GwwqrYkAsFxqDXGC7DoM"
Set-Cookie: __cfduid=d84018de2d621df9d53eb52d97cd33a651511881763; expires=Wed, 28-Nov-18 15:09:23 GMT; path=/; domain=.typicode.com; HttpOnly
Server: cloudflare-nginx
Via: 1.1 vegur
X-Powered-By: Express
I agree that it would be nice to be able to get this information back too. I'll open up an issue on the github.com/PowerShell repo and see if we can get something like this added in the future, and the link will be added to this answer.
I'm trying to get the sample webapp play-streaming-scala to run and in some circumstances I get a weird behavior.
I've got the app running directly on port 80 of some host and I'm checking the output with curl -iv --raw http://somehost/scala/eventSource/liveClock.
What I'm expecting is something like this:
* Hostname was NOT found in DNS cache
* Trying 195.176.3.71...
* Connected to somehost (0.0.0.0) port 80 (#0)
> GET /scala/eventSource/liveClock HTTP/1.1
> User-Agent: curl/7.39.0
> Host: somehost
> Accept: */*
>
< HTTP/1.1 200 OK
HTTP/1.1 200 OK
< Transfer-Encoding: chunked
Transfer-Encoding: chunked
< Content-Type: text/event-stream; charset=utf-8
Content-Type: text/event-stream; charset=utf-8
< Date: Wed, 18 Jan 2017 13:24:55 GMT
Date: Wed, 18 Jan 2017 13:24:55 GMT
<
10
data: 14 24 56
10
data: 14 24 56
10
data: 14 24 56
etc., and clearly see the chunks appear one after the other as time goes by.
Now, on some machines, this works well. On some others on campus, this fails. curl only shows this and then stops:
* Trying 195.176.3.71...
* Connected to somehost (0.0.0.0) port 80 (#0)
> GET /scala/eventSource/liveClock HTTP/1.1
> Host: somehost
> User-Agent: curl/7.43.0
> Accept: */*
>
Now the interesting thing is: if I kill the webapp on the host, curl suddenly “catches up” and spits all the chunks together, closing the connection like this:
10
data: 14 35 20
* transfer closed with outstanding read data remaining
* Closing connection 0
curl: (18) transfer closed with outstanding read data remaining
What can be causing the behavior? What on earth is going on and intercepting these events? Is there any way I can “force flush” something from the Play response?
Turns out the local “hidden” proxy set up automatically by OS X's parental controls system is not forwarding chunked responses properly, thus making a system based on Server-Sent Events inoperable. A shame.
I'm trying to set up a 'Swift All In One' system on a Ubuntu 12.04 VM by the link:http://docs.openstack.org/developer/swift/development_saio.html.
I use VMware WorkStation 12 Pro on Win7 64bit system and use 'Host-only' network mode.The VM ip address is "192.168.137.200".
When I run the command on the VM:
curl -v -H 'X-Storage-User: test:tester' -H 'X-Storage-Pass: testing' http://192.168.137.200/auth/v1.0
It works well.
But when I run the command on the host machine(Win7 platform), It fails and returns:
* Could not resolve host: test:tester'; Host not found
* Closing connection #0
curl: (6) Could not resolve host: test:tester'; Host not found
* Could not resolve host: testing'; Host not found
* Closing connection #0
curl: (6) Could not resolve host: testing'; Host not found
* About to connect() to 192.168.137.200 port 80 (#0)
* Trying 192.168.137.200... connected
* Connected to 192.168.137.200 (192.168.137.200) port 80 (#0)
> GET /auth/v1.0 HTTP/1.1
> User-Agent: curl/7.20.1 (amd64-pc-win32) libcurl/7.20.1 OpenSSL/0.9.8n zlib/1.
2.3
> Host: 192.168.137.200
> Accept: */*
>
< HTTP/1.1 401 Unauthorized
< Date: Fri, 25 Mar 2016 05:57:24 GMT
< Content-Length: 131
< Content-Type: text/html; charset=UTF-8
< Www-Authenticate: Swift realm="unknown"
< X-Trans-Id: tx081d67bec35b457bb4cb8-0056f4d343
< Vary: Accept-Encoding
<
<html><h1>Unauthorized</h1><p>This server could not verify that you are authoriz
ed to access the document you requested.</p></html>* Connection #0 to host 192.1
68.137.200 left intact
* Closing connection #0
Then I make another Ubuntu 12.04 VM and try to run the command above on the second VM, it works well.
Try to use X-Auth-User and X-Auth-Key headers instead.https://swiftstack.com/docs/cookbooks/swift_usage/auth.html
I want to disable TRACE/TRACK HTTP methods in JBoss wildfly 8.1 at server level and not in web.xml. How do do it?
Thanks Federico Sierra for the reply. I have tried again, but it seems by default TRACE is not disabled. I am getting 200 OK status code. See below.
# curl -k -v -X TRACE https://localhost:8443 --user admin:Password#123
* About to connect() to localhost port 8443 (#0)
* Trying ::1...
* Connection refused
* Trying 127.0.0.1...
* connected
* Connected to localhost (127.0.0.1) port 8443 (#0)
* successfully set certificate verify locations:
* CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
* SSLv3, TLS handshake, Client hello (1):
* SSLv3, TLS handshake, Server hello (2):
* SSLv3, TLS handshake, CERT (11):
* SSLv3, TLS handshake, Server key exchange (12):
* SSLv3, TLS handshake, Server finished (14):
* SSLv3, TLS handshake, Client key exchange (16):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSL connection using ECDHE-RSA-AES256-SHA384
* Server certificate:
* subject: CN=scspr0021776001.gdl.englab.netapp.com
* start date: 2015-02
* expire date: 2016-02
* common name: scspr0021776001.gdl.englab.netapp.com (does not match 'localhost')
* issuer: CN=scsp
* SSL certificate verify result: self signed certificate (18), continuing anyway.
* Server auth using Basic with user 'admin'
> TRACE / HTTP/1.1
> Authorization: Basic YWRtaW46UGFzc3dvcmRAMTIz
> User-Agent: curl/7.27.0
> Host: localhost:8443
> Accept: */*
>
< HTTP/1.1 200 OK
< Expires: 0
< Cache-Control: no-cache, no-store, must-revalidate
< X-Powered-By: Undertow/1
< Set-Cookie: JSESSIONID=j1Uh0ZpFl08-op1ak9dm6gr7.scspr0021776001; path=; HttpOnly
< Set-Cookie: JSESSIONIDSSO=jIoXps8-b3dByrQektBEZfOP; path=/
< Server: WildFly/8
< Pragma: no-cache
< Date: Wed, 18 Feb 2015 04:24:33 GMT
< Connection: keep-alive
< Content-Type: message/http
< Content-Length: 135
<
TRACE /index.html HTTP/1.1
Authorization: Basic YWRtaW46UGFzc3dvcmRAMTIz
Host: localhost:8443
User-Agent: curl/7.27.0
Accept: */*
* Connection #0 to host localhost left intact
* Closing connection #0
* SSLv3, TLS alert, Client hello (1):
By default in Wildfly TRACE and TRACK methods are disabled for security reasons.
You can check with:
curl -v -X TRACE http://localhost:8080
* Rebuilt URL to: http://localhost:8080/
* Adding handle: conn: 0x1ddb0a0
* Adding handle: send: 0
* Adding handle: recv: 0
* Curl_addHandleToPipeline: length: 1
* - Conn 0 (0x1ddb0a0) send_pipe: 1, recv_pipe: 0
* About to connect() to localhost port 8080 (#0)
* Trying 127.0.0.1...
* Connected to localhost (127.0.0.1) port 8080 (#0)
> TRACE / HTTP/1.1
> User-Agent: curl/7.33.0
> Host: localhost:8080
> Accept: */*
>
< HTTP/1.1 405 Method Not Allowed
< Connection: keep-alive
< X-Powered-By: Undertow/1
* Server WildFly/8 is not blacklisted
< Server: WildFly/8
< Content-Length: 83
< Content-Type: text/html
< Date: Tue, 17 Feb 2015 20:16:29 GMT
<
<html><head><title>Error</title></head><body>405 - Method Not Allowed</body></html>* Connection #0 to host localhost left intact
I managed to disable the HTTP TRACE in Wildfly 8.2 (it should work with Wildfly 8.1, but I haven't tested) with the following added to the webapp's WEB-INF/undertow-handlers.conf:
method[TRACE] -> response-code[value=405]
If you have multiple webapps you'll have to add it to all that don't need to answer to HTTP TRACE requests.