REST Upload to Skydrive without Content-Length - rest

I'm trying to upload a file to Skydrive where I don't a-priori know the Content-Length. With other storage services I can do this with chunked HTTP upload, but Skydrive always complains about Content-Length being invalid.
Here are the full headers I'm sending:
PUT /v5.0/me/skydrive/files/skydrive_test.js?overwrite=ChooseNewName HTTP/1.1
Authorization: Bearer <TOKEN_REDACTED>
host: apis.live.net
content-type: application/javascript
Connection: keep-alive
Transfer-Encoding: chunked
Here's the response I get back:
cache-control: private, no-cache, no-store, must-revalidate
transfer-encoding: chunked
content-type: application/json; charset=UTF-8
server: Live-API/16.4.1731.327 Microsoft-HTTPAPI/2.0
x-content-type-options: nosniff
x-http-live-request-id: API.c6afda25-2d9f-4248-9f49-001ccb3a9007
x-http-live-server: BAYMSG1010836
date: Wed, 15 May 2013 14:33:00 GMT
{ "error": { "code": "request_invalid_content_length",
"message": "The value for the Content-Length header isn't valid." }}
Is there any way I can do this without setting Content-Length (i.e. using chunked encoding)?
I'm using node.js to do this, but it should apply equally with any language using the REST API, hence I haven't tagged this with a particular language.
For example Dropbox offers the Chunked Upload command: https://www.dropbox.com/developers/core/docs#chunked-upload
And Google Drive, even though it says it wants Content-Length, doesn't need it for it's resumable upload API: https://developers.google.com/drive/manage-uploads#resumable
Is there an API I'm missing?
Edit: Things I've tried: Setting Content-Length: 0 results in it working, but the file is zero bytes. Setting Content-length:0 and Transfer-Encoding: chunked, results in the original error above.

Try setting a dummy content-length to see if it is acceptable. Otherwise set the file size.
If you are using node.js you can get the file size you are trying to upload and set the size to content-length in OCTETs. You can get the file size requiring the fs (filesystem) module in node.
var fs = require('fs');
fs.watchFile('some.file', function () {
fs.stat('some.file', function (err, stats) {
console.log(stats.size);
});
});
#Resource

Related

Does google chrome and similar browsers support range headers for standard downloads

My initial response headers - notice the Accept-Ranges header
HTTP/1.1 200 OK
X-Powered-By: Express
Vary: Origin
Access-Control-Allow-Credentials: true
X-RateLimit-Limit: 1
X-RateLimit-Remaining: 0
Date: Thu, 08 Apr 2021 06:14:19 GMT
X-RateLimit-Reset: 1617862461
Accept-Ranges: bytes
Content-Length: 100000000
Content-Type: text/plain; charset=utf-8
Content-Disposition: attachment; filename="some_file.txt"
Connection: keep-alive
Keep-Alive: timeout=5
I then restart the server and click resume download in chrome, but chrome doesn't send back in Range request headers
I'm following the documentation on Mozilla's website
Am I missing a header or misunderstanding how this works, especially with chrome and other browsers? Is there another way I can manually support resuming downloads by sending the right response and understanding the right request? From a technical perspective, if chrome sends back which range it now needs I will be able to resume the download.
According to this article, chrome should support something like this. I just need to be pointed in the right direction.
Ty!
Chrome needs some way to know that the file it's trying to download at that URL is indeed the same file when it tries to resume.
If you add support for an ETag header, this will likely work.

Realtime Database - Arduino and REST API

I'm trying to use the rest API for the Firebase Realtime Database to transmit data from a Controllino MAXI (essentially an arduino mega 2560 with an ethernet chip) to the database. But I'm having trouble with the HTTP request. All types of requests fail but I'm interested in the PUT request.
Using this online tool, the PUT request works, here's the raw data:
PUT /.json HTTP/1.1
Host: *rtdb-name*.firebaseio.com
Content-Type: application/json
Content-Length: 26
{"message":"hello world!"}
That request returns this response:
{
"message": "hello world!"
}
And these headers:
Server: nginx
Date: Wed, 03 Feb 2021 17:02:55 GMT
Content-Type: application/json; charset=utf-8
Content-Length: 26
Connection: keep-alive
Access-Control-Allow-Origin: *
Cache-Control: no-cache
Strict-Transport-Security: max-age=31556926; includeSubDomains; preload
And writes the data to the root of the realtime db:
But when I do the same thing on the arduino using the Ethernet library:
char server[] = "rtdb-name.firebaseio.com"
if (client.connect(server,80)){
String data = "{\"message\":\"hello world!\"}";
Serial.println("connected");
client.println("PUT /.json HTTP/1.1");
client.println("Host: *rtdb-name*.firebaseio.com");
client.println("User-Agent: Arduino/1.0");
client.println("Cache-Control: no-cache, no-store, must-revalidate");
client.println("Pragma: no-cache");
client.println("Expires: 0");
client.println("Content-Type: application/json");
client.println("Connection: close");
client.print("Content-Length: ");
client.println(data.length());
client.println();
client.println(data);
while(client.connected()) {
while (client.available()) {
char c = client.read();
Serial.print(c);
}
}
client.stop();
Serial.println("disconnected");
}else{
Serial.println("Failed to connect to server");
}
I get a 404 error:
HTTP/1.1 404 Not Found
Content-Type: text/html; charset=UTF-8
Referrer-Policy: no-referrer
Content-Length: 1566
Date: Wed, 03 Feb 2021 17:06:07 GMT
Connection: close
I'm not entirely sure how to make this work. I think it's because the website uses HTTPS and the Mega can only do HTTP? Any assistance would be appreciated
For realtime datalogging with arduino, nodemcu, consider about using MQTT.
Should have just started with the documentation:
You can use any Firebase Realtime Database URL as a REST endpoint. All
you need to do is append .json to the end of the URL and send a
request from your favorite HTTPS client.
HTTPS is required. Firebase only responds to encrypted traffic so that
your data remains safe.
The Arduino (Nano, UNO, Mega and the like) simply don't have the power to do SSL (HTTPS) which is necessary to communicate with Firebase.
I've gone ahead and created a Netlify function that responds to a HTTP POST request from the Arduino and then that function writes the data to Firebase. I got the idea from this tutorial.
As suggested below you can use an MQTT broker.

Uploading a file with google cloud API with a PUT at root of server?

I have a server using the google Drive API. I tried with a curl PUT request to upload a simple file (test.txt) at http://myserver/test.txt. As you can see, I did the PUT request at the root of my server. The response I get is the following:
HTTP/1.1 200 OK
X-GUploader-UploadID: AEnB2UqANa4Bj6ilL7z5HZH0wlQi_ufxDiHPtb2zq1Gzcx7IxAEcOt-AOlWsbX1q_lsZUwWt_hyKOA3weAeVpQvPQTwbQhLhIA
ETag: "6e809cbda0732ac4845916a59016f954"
x-goog-generation: 1548877817413782
x-goog-metageneration: 1
x-goog-hash: crc32c=jwfJwA==
x-goog-hash: md5=boCcvaBzKsSEWRalkBb5VA==
x-goog-stored-content-length: 6
x-goog-stored-content-encoding: identity
Content-Type: text/html; charset=UTF-8
Accept-Ranges: bytes
Via: 1.1 varnish
Content-Length: 0
Accept-Ranges: bytes
Date: Wed, 30 Jan 2019 19:50:17 GMT
Via: 1.1 varnish
Connection: close
X-Served-By: cache-bwi5139-BWI, cache-cdg20732-CDG
X-Cache: MISS, MISS
X-Cache-Hits: 0, 0
X-Timer: S1548877817.232336,VS0,VE241
Vary: Origin
Access-Control-Allow-Methods: POST,PUT,PATCH,GET,DELETE,OPTIONS
Access-Control-Allow-Headers: Cache-Control,X-Requested-With,Authorization,Content-Type,Location,Range
Access-Control-Allow-Credentials: true
Access-Control-Max-Age: 300
I know you're not supposed to use the API that way. I did that for testing purposes. I understand every headers returned but can't figure out if my file has been uploaded because I don't have enough knowledge of this API.
My question is very simple :
Just by looking at the response, can you tell me if my file has been uploaded ?
If yes can I retrieve it and how ?
The HTTP status code traditionally indicates, for any given request, if it was successful. The status code in the response is always on the first line:
HTTP/1.1 200 OK
200 type status codes mean success. You should take some time to familiarize yourself with HTTP status codes if you intend to work with HTTP APIs.

OneDrive REST API - Upload - Content of .png file converted to jpeg

When using the OneDrive REST API to upload files, the content of some files with a .png extension is automatically converted to JPEG. I'm able to reproduce the problem with both documented upload methods.
Sample png files:
http://www39.zippyshare.com/v/59255310/file.html
http://www2.zippyshare.com/v/11270772/file.html
For reference the shortened requests:
PUT method
PUT https://apis.live.net/v5.0/folder.<removed>/files/i2.png HTTP/1.1
Pragma: no-cache
<removed - no Content-Type header>
Content-Length: 33579
Host: apis.live.net
When the Content-Type in the header of the PUT request is set to application/octet-stream the following error is returned:
HTTP/1.1 415 Unsupported Media Type
Server: Live-API/19.7.925.4009 Microsoft-HTTPAPI/2.0
<removed>
{
"error": {
"code": "request_body_invalid_media_type",
"message": "The Content-Type header 'application/octet-stream' isn't supported."
}
}
POST method
POST https://apis.live.net/v5.0/folder.<removed>/files HTTP/1.1
Content-Type: multipart/form-data; boundary=735b1931-a2bb-4970-8142-373848528fcb
<removed>
Content-Length: 33767
Host: apis.live.net
--735b1931-a2bb-4970-8142-373848528fcb
Content-Disposition: form-data; name="file"; filename="i2.png"
Content-Type: application/octet-stream
.PNG<removed>
--735b1931-a2bb-4970-8142-373848528fcb--
Other OneDrive APIs
The problem does not happen with the API (skyapi.onedrive.live.com/API/2) Microsoft is using on the OneDrive website and in the Windows desktop client (skydrive.exe).
OneDrive developers:
Instead of returning a 415 error in the PUT request, would it be possible to support requests where the Content-Type: application/octet-stream header is set? In this case the data should be left as is without doing any data conversion. Thank you.
This is documented behaviour as per http://msdn.microsoft.com/en-us/library/dn659726.aspx (see the first note under uploading).
If you want to disable conversion, you can set downsize_photo_uploads=false in your query string.

Unable to assemble multipart/form-data request

I'm trying to implement file upload functionality in the iPhone app. Server code is tested and works when files are uploaded from the desktop browser, so I moved to implementing the Objective-C client code. I'm assembling HTTP requests body manually, and despite that it looks correct, it is rejected by the server (server handler unable to extract the parts from multipart content). In desperation I've simplified the form to having only one parameter, but it still does not work.
I've captured the network traffic and I could see that Wireshark could not parse my multipart content as well (have a look at screenshots: Firefox request, iPhone request). I'm pasting it below in hope that you could see the errors I can't see.
Thanks in advance.
Firefox:
POST /cubepaint/actions/gallery/post HTTP/1.1
Host: [...]
User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en-GB; rv:1.9.1.3) Gecko/20090824 Firefox/3.5.3
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-gb,en;q=0.5
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 300
Connection: keep-alive
Authorization: Basic [...]
Content-Type: multipart/form-data; boundary=---------------------------20072377098235644401115438165
Content-Length: 180
-----------------------------20072377098235644401115438165
Content-Disposition: form-data; name="deviceId"
12345
-----------------------------20072377098235644401115438165--
HTTP/1.1 200 OK
Date: Sat, 17 Oct 2009 22:09:21 GMT
Server: Apache/2.2.3 (Debian) DAV/2 SVN/1.4.2 mod_python/3.2.10 Python/2.4.4 mod_ssl/2.2.3 OpenSSL/0.9.8c
Keep-Alive: timeout=15, max=100
Connection: Keep-Alive
Transfer-Encoding: chunked
Content-Type: text/html; charset=UTF-8
iPhone:
POST /cubepaint/actions/gallery/post HTTP/1.1
Host: [...]
User-Agent: Copenhagen/1.0 CFNetwork/459 Darwin/9.8.0
Content-Type: multipart/form-data; boundary=----------0E7B16E6-CD3D-4213-9B42-07DA30822C74
Accept: */*
Accept-Language: en-us
Accept-Encoding: gzip, deflate
Authorization: Basic [...]
Content-Length: 187
Connection: keep-alive
----------0E7B16E6-CD3D-4213-9B42-07DA30822C74
Content-Disposition: form-data; name="deviceId"
00000000-0000-1000-8000-0016CBCC0B61
----------0E7B16E6-CD3D-4213-9B42-07DA30822C74--
HTTP/1.1 200 OK
Date: Sat, 17 Oct 2009 22:04:07 GMT
Server: Apache/2.2.3 (Debian) DAV/2 SVN/1.4.2 mod_python/3.2.10 Python/2.4.4 mod_ssl/2.2.3 OpenSSL/0.9.8c
Keep-Alive: timeout=15, max=100
Connection: Keep-Alive
Transfer-Encoding: chunked
Content-Type: text/html; charset=UTF-8
Your iPhone version indicates keep-alive but doesn't specify a length. Not sure that's enough to cause trouble.
Also, is it possible your server is checking for user-agent strings it recognizes (say, for backward-compatibility mode)?
I'd also compare the two in a text editor that shows CR/LF characters to make sure you're getting proper line endings.
Another thing you could try is create a simple web-page that does a multipart POST and run it from the iPhone browser (instead of the Mac one) then check the headers that go across the wire. Or you could snag a toolkit like ASIHTTPRequest and see what kind of output it generates for multi-part posts (or just use the toolkit instead of trying to write your own).
Good luck
Solved by reading RFC 2046 (MIME specification): boundary between parts of multipart message should contain two leading '-'s, and last boundary should additionally contain two trailing '-'s. The boundary in the request header and request body in the Firefox request differ:
---------------------------20072377098235644401115438165
and
-----------------------------20072377098235644401115438165
The last boundary looks like this:
-----------------------------20072377098235644401115438165--
You really could not see this with the eye when there are so many leading '-'s in the original boundary.