I made a GET request, the response headers are as follows:
Cache-Control: private
Content-Encoding: gzip
Content-Length: 10566
Content-Type: text/html; charset=utf-8
Date: Fri, 28 Jun 2019 03:45:08 GMT
Server: Microsoft-IIS/8.5
Vary: Accept-Encoding
X-AspNet-Version: 4.0.30319
X-Powered-By: ASP.NET
i saved the response to file, the file looks like this:
1f8b 0800 0000 0000 0400 edbd 0760 1c49
9625 262f 6dca 7b7f 4af5 4ad7 e074 a108
8060 1324 d890 4010 ecc1 88cd e692 ec1d
6947 2329 ab2a 81ca 6556 655d 6616 40cc
ed9d bcf7 de7b efbd f7de 7bef bdf7 ba3b
9d4e 27f7 dfff 3f5c 6664 016c f6ce 4ada
//continues...
how to decode it ?
The response is compressed with gzip/deflate as indicated by the reponse header Content-Encoding:
Content-Encoding: gzip
and as indicated by the first few bytes 1f 8b 08 (assuming that your question shows a hex dump of the response).
Most HTTP client libraries can easily deal with gzip/deflate encoding as it is a commonly used compression algorithm to speed up HTTP requests.
If your programming language or environment (unfortunately, you don't mention it), does not support gzip/deflate, then you can change your HTTP request. Unless the HTTP server is not working properly, your request currently includes the HTTP header Accept-Encoding indicating that your code can accept gzip/deflate encoding (which is not the case). So remove this header and the server should stop sending compressed data.
Related
I'm trying to use the rest API for the Firebase Realtime Database to transmit data from a Controllino MAXI (essentially an arduino mega 2560 with an ethernet chip) to the database. But I'm having trouble with the HTTP request. All types of requests fail but I'm interested in the PUT request.
Using this online tool, the PUT request works, here's the raw data:
PUT /.json HTTP/1.1
Host: *rtdb-name*.firebaseio.com
Content-Type: application/json
Content-Length: 26
{"message":"hello world!"}
That request returns this response:
{
"message": "hello world!"
}
And these headers:
Server: nginx
Date: Wed, 03 Feb 2021 17:02:55 GMT
Content-Type: application/json; charset=utf-8
Content-Length: 26
Connection: keep-alive
Access-Control-Allow-Origin: *
Cache-Control: no-cache
Strict-Transport-Security: max-age=31556926; includeSubDomains; preload
And writes the data to the root of the realtime db:
But when I do the same thing on the arduino using the Ethernet library:
char server[] = "rtdb-name.firebaseio.com"
if (client.connect(server,80)){
String data = "{\"message\":\"hello world!\"}";
Serial.println("connected");
client.println("PUT /.json HTTP/1.1");
client.println("Host: *rtdb-name*.firebaseio.com");
client.println("User-Agent: Arduino/1.0");
client.println("Cache-Control: no-cache, no-store, must-revalidate");
client.println("Pragma: no-cache");
client.println("Expires: 0");
client.println("Content-Type: application/json");
client.println("Connection: close");
client.print("Content-Length: ");
client.println(data.length());
client.println();
client.println(data);
while(client.connected()) {
while (client.available()) {
char c = client.read();
Serial.print(c);
}
}
client.stop();
Serial.println("disconnected");
}else{
Serial.println("Failed to connect to server");
}
I get a 404 error:
HTTP/1.1 404 Not Found
Content-Type: text/html; charset=UTF-8
Referrer-Policy: no-referrer
Content-Length: 1566
Date: Wed, 03 Feb 2021 17:06:07 GMT
Connection: close
I'm not entirely sure how to make this work. I think it's because the website uses HTTPS and the Mega can only do HTTP? Any assistance would be appreciated
For realtime datalogging with arduino, nodemcu, consider about using MQTT.
Should have just started with the documentation:
You can use any Firebase Realtime Database URL as a REST endpoint. All
you need to do is append .json to the end of the URL and send a
request from your favorite HTTPS client.
HTTPS is required. Firebase only responds to encrypted traffic so that
your data remains safe.
The Arduino (Nano, UNO, Mega and the like) simply don't have the power to do SSL (HTTPS) which is necessary to communicate with Firebase.
I've gone ahead and created a Netlify function that responds to a HTTP POST request from the Arduino and then that function writes the data to Firebase. I got the idea from this tutorial.
As suggested below you can use an MQTT broker.
I have a server using the google Drive API. I tried with a curl PUT request to upload a simple file (test.txt) at http://myserver/test.txt. As you can see, I did the PUT request at the root of my server. The response I get is the following:
HTTP/1.1 200 OK
X-GUploader-UploadID: AEnB2UqANa4Bj6ilL7z5HZH0wlQi_ufxDiHPtb2zq1Gzcx7IxAEcOt-AOlWsbX1q_lsZUwWt_hyKOA3weAeVpQvPQTwbQhLhIA
ETag: "6e809cbda0732ac4845916a59016f954"
x-goog-generation: 1548877817413782
x-goog-metageneration: 1
x-goog-hash: crc32c=jwfJwA==
x-goog-hash: md5=boCcvaBzKsSEWRalkBb5VA==
x-goog-stored-content-length: 6
x-goog-stored-content-encoding: identity
Content-Type: text/html; charset=UTF-8
Accept-Ranges: bytes
Via: 1.1 varnish
Content-Length: 0
Accept-Ranges: bytes
Date: Wed, 30 Jan 2019 19:50:17 GMT
Via: 1.1 varnish
Connection: close
X-Served-By: cache-bwi5139-BWI, cache-cdg20732-CDG
X-Cache: MISS, MISS
X-Cache-Hits: 0, 0
X-Timer: S1548877817.232336,VS0,VE241
Vary: Origin
Access-Control-Allow-Methods: POST,PUT,PATCH,GET,DELETE,OPTIONS
Access-Control-Allow-Headers: Cache-Control,X-Requested-With,Authorization,Content-Type,Location,Range
Access-Control-Allow-Credentials: true
Access-Control-Max-Age: 300
I know you're not supposed to use the API that way. I did that for testing purposes. I understand every headers returned but can't figure out if my file has been uploaded because I don't have enough knowledge of this API.
My question is very simple :
Just by looking at the response, can you tell me if my file has been uploaded ?
If yes can I retrieve it and how ?
The HTTP status code traditionally indicates, for any given request, if it was successful. The status code in the response is always on the first line:
HTTP/1.1 200 OK
200 type status codes mean success. You should take some time to familiarize yourself with HTTP status codes if you intend to work with HTTP APIs.
Edit: it seems the issue was caused by a dropped cookie. There should have been a session id cookie as well.
For posterity, here's the original question
When sending a request formed as this
GET https://<url>?<parameters>
Cache-Control: max-age=0
Connection: keep-alive
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Charset: iso-8859-1,utf-8,UTF-8
Accept-Encoding: gzip, x-gzip, deflate, x-bzip2
Accept-Language: en-US,en;q=0.5
If-None-Match: "6eb7d55abfd0546399e3245ad3a76090"
User-Agent: Mozilla/5.0 libwww-perl/6.13
Cookie: auth_token=<blah>; __cfduid=<blah>
Cookie2: $Version="1"
I receive the following response
response-type: text/html
charset=utf-8
HTTP/1.1 406 Not Acceptable
Cache-Control: no-cache
Connection: keep-alive
Date: Fri, 12 Feb 2016 18:34:00 GMT
Server: cloudflare-nginx
Content-Type: text/html; charset=utf-8
CF-RAY: 273a62969a9b288e-SJC
Client-Date: Fri, 12 Feb 2016 18:34:00 GMT
Client-Peer: <IP4>:443
Client-Response-Num: 10
Client-SSL-Cert-Issuer: /C=GB/ST=Greater Manchester/L=Salford/O=COMODO CA Limite
d/CN=COMODO ECC Domain Validation Secure Server CA 2
Client-SSL-Cert-Subject: /OU=Domain Control Validated/OU=PositiveSSL Multi-Domai
n/CN=ssl<blah>.cloudflaressl.com
Client-SSL-Cipher: <some value>
Client-SSL-Socket-Class: IO::Socket::SSL
Client-SSL-Warning: Peer certificate not verified
Client-Transfer-Encoding: chunked
Status: 406 Not Acceptable
X-Runtime: 9
I'm not entirely sure why the response is 406 Not Acceptable. When
downloaded with firefox, the file in question in 996 KB (as reported
by Windows 8.1's explorer). It looks like I have a partially
transferred file from my perl script at 991 KB (again, windows
explorer size), so it got MOST of the file before throwing the Not
Acceptable response. Using the same URL pattern and request style, I
was able to successfully download a 36 MB file from the server with
this perl library and request form, so the size of the file should not
be magically past some max (chunk) size. As these files are being
updated on approximately 15-minute intervals, I suppose it's possible
that a write was performed on the server, invalidating the ETag before
all chunks were complete on this file?
I tried adding chunked to Accept-Encoding, but that's not for
Transfer encoding and it appears to have no affect on the server's behavior. Additionally, as I've been able to download larger files
(same format) from the same server, that alone shouldn't be the cause
of my woes. LWP is supposed to be able to handle chunked data
returned by a response to GET (as per this newsgroup post).
The server in question is running nginx with Rack::Lint. The
particular server configuration (which I in no way control), throws
500 errors on its own attempts to send 304: not modified. This
caused me to write a workaround for File::Download (sub
lintWorkAround here), so I'm not above putting blame on the
server in this instance also, if warranted. I don't believe I buggered
up the chunk-handling code from File::Download 0.3 (see diff),
but I suppose that's also possible. Is it possible to request a
particular chunk size from the server?
I'm using LWP and libwww versions 6.13 in perl 5.18.2.
File::Download version is my own 0.4_050601.
So, what else could the 406 error mean? Is there a way to request that
the server temporarily cache/version control the entire file so that I
can download a given ETag'd file once the transfer begins?
Does anyone know how the GitHub ETag when accessing raw content is generated?
As far as I can tell this is not MD5, SHA1 or any common SHA variant;
Example http headers:
HTTP/1.1 200 OK
Server: nginx/1.0.13
Date: Tue, 05 Jun 2012 19:46:08 GMT
Content-Type: text/html; charset=utf-8
Connection: keep-alive
Status: 200 OK
ETag: "aa1da178ae0a43e23ce49a6b8f474738"
The ETag length is 32 characters, suggesting MD5, but this does not match the MD5 checksum of the downloaded file (downloaded using curl).
I am aware that ETags should be treated as opaque identifiers. Still, curious.
My guess would be they are using the stock nginx etag module.
https://github.com/mikewest/nginx-static-etags/blob/master/ngx_http_static_etags_module.c
I'm trying to implement file upload functionality in the iPhone app. Server code is tested and works when files are uploaded from the desktop browser, so I moved to implementing the Objective-C client code. I'm assembling HTTP requests body manually, and despite that it looks correct, it is rejected by the server (server handler unable to extract the parts from multipart content). In desperation I've simplified the form to having only one parameter, but it still does not work.
I've captured the network traffic and I could see that Wireshark could not parse my multipart content as well (have a look at screenshots: Firefox request, iPhone request). I'm pasting it below in hope that you could see the errors I can't see.
Thanks in advance.
Firefox:
POST /cubepaint/actions/gallery/post HTTP/1.1
Host: [...]
User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en-GB; rv:1.9.1.3) Gecko/20090824 Firefox/3.5.3
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-gb,en;q=0.5
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 300
Connection: keep-alive
Authorization: Basic [...]
Content-Type: multipart/form-data; boundary=---------------------------20072377098235644401115438165
Content-Length: 180
-----------------------------20072377098235644401115438165
Content-Disposition: form-data; name="deviceId"
12345
-----------------------------20072377098235644401115438165--
HTTP/1.1 200 OK
Date: Sat, 17 Oct 2009 22:09:21 GMT
Server: Apache/2.2.3 (Debian) DAV/2 SVN/1.4.2 mod_python/3.2.10 Python/2.4.4 mod_ssl/2.2.3 OpenSSL/0.9.8c
Keep-Alive: timeout=15, max=100
Connection: Keep-Alive
Transfer-Encoding: chunked
Content-Type: text/html; charset=UTF-8
iPhone:
POST /cubepaint/actions/gallery/post HTTP/1.1
Host: [...]
User-Agent: Copenhagen/1.0 CFNetwork/459 Darwin/9.8.0
Content-Type: multipart/form-data; boundary=----------0E7B16E6-CD3D-4213-9B42-07DA30822C74
Accept: */*
Accept-Language: en-us
Accept-Encoding: gzip, deflate
Authorization: Basic [...]
Content-Length: 187
Connection: keep-alive
----------0E7B16E6-CD3D-4213-9B42-07DA30822C74
Content-Disposition: form-data; name="deviceId"
00000000-0000-1000-8000-0016CBCC0B61
----------0E7B16E6-CD3D-4213-9B42-07DA30822C74--
HTTP/1.1 200 OK
Date: Sat, 17 Oct 2009 22:04:07 GMT
Server: Apache/2.2.3 (Debian) DAV/2 SVN/1.4.2 mod_python/3.2.10 Python/2.4.4 mod_ssl/2.2.3 OpenSSL/0.9.8c
Keep-Alive: timeout=15, max=100
Connection: Keep-Alive
Transfer-Encoding: chunked
Content-Type: text/html; charset=UTF-8
Your iPhone version indicates keep-alive but doesn't specify a length. Not sure that's enough to cause trouble.
Also, is it possible your server is checking for user-agent strings it recognizes (say, for backward-compatibility mode)?
I'd also compare the two in a text editor that shows CR/LF characters to make sure you're getting proper line endings.
Another thing you could try is create a simple web-page that does a multipart POST and run it from the iPhone browser (instead of the Mac one) then check the headers that go across the wire. Or you could snag a toolkit like ASIHTTPRequest and see what kind of output it generates for multi-part posts (or just use the toolkit instead of trying to write your own).
Good luck
Solved by reading RFC 2046 (MIME specification): boundary between parts of multipart message should contain two leading '-'s, and last boundary should additionally contain two trailing '-'s. The boundary in the request header and request body in the Firefox request differ:
---------------------------20072377098235644401115438165
and
-----------------------------20072377098235644401115438165
The last boundary looks like this:
-----------------------------20072377098235644401115438165--
You really could not see this with the eye when there are so many leading '-'s in the original boundary.