I have a 6mb txt file in Box.com site.
Now i would like to download the file using api. as it takes time to download, i would like to download it as a gzipped file.
As given here https://developers.box.com/docs/ where we have to add accept-encoding header with the values "gzip, deflate". I have added this header but the file is not downloaded as zip file it has the same size as 6mb, if it is zipped then it should be less than one mb in size.
But it is not happening. The following are the headers passed in REST request.
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/36.0.1985.125 Safari/537.36
Authorization: Bearer ACCESSTOKEN
Accept: */*
Accept-Encoding: gzip,deflate,sdch
Accept-Language: en-US,en;q=0.8,te;q=0.6
The following are the response headers.
Server: nginx
Date: Thu, 24 Jul 2014 16:24:56 GMT
Content-Type: application/octet-stream
Content-Length: 6685772
Connection: keep-alive
Cache-control: private
Accept-Ranges: bytes
Content-Disposition: attachment;filename="abc.log";filename*=UTF-8''
X-Content-Type-Options: nosniff
Accept-Ranges: bytes
Is there anything that I missed here?
I have run into the same problem. However, I have also realized why this may be happening. Quoting the Box SDK documentation:
If the file is available to be downloaded, the response will be a 302
Found to a URL at dl.boxcloud.com. The dl.boxcloud.com URL is not
persistent. Clients will need to follow the redirect in order to
actually download the file. The raw data of the file is returned
unless the file ID is invalid or the user does not have access to it.
The important point to note is that there is a client redirect to a non-persistent URL in order to download the file. When we check the sequence of headers that are passed as a part of the redirect request, you can see that the Accept-Encoding: gzip deflate is missing. I used Fiddler to try this out, you can use any other HTTP proxy or interceptor.
This should be the reason why the files are not getting downloaded using the gzip encoding.
Hope this helps.
Related
I have an application (React SPA) that calls a bunch of servers on different subdomains of the application domain, i.e.:
the web app sits at foo.bar.com,
and talks to api.foo.bar.com and media.foo.bar.com.
When accessing api.foo.bar.com, I get an error from the browser (be it Edge, Chrome, or Firefox) telling me that the origin (foo.bar.com) is different from the value of the Access-Control-Allow-Origin response header. However, by inspection of the response, they are the same:
(I unfortunately have to obfuscate the address.)
Those apps are hosted on Kubernetes; the ingress is NGINX, and it's is not providing CORS (cors-enabled annotation is false). Both applications (api and media) are Express apps, and both have the same CORS configuration allowing the specific origin.
I'm wondering if this has something to do with the redirect - the call to the media... endpoint returns a redirect (302) whose Location is a api... address.
Other than that, I have no clue what could be wrong. Something is, for sure, because all browsers agree that my request should be blocked (on account of the origin).
In all cases, I've checked the address multiple times for typos, ending forward-slashes, etc. I've called OPTIONS on those endpoints with cURL and Postman, using all headers or just a few. They always answer the correct address.
Additional information, as requested:
Preflight request:
OPTIONS /media/1.0.0/rtsp/hls?feedUrl=https%3A%2F%2Flive.monuv.com.br%2Fa1%2F14298.stream%2Fstr27%2Fchunklist.m3u8%3Fm_hash%3DkhV_hCnKG3nhaNCFaYZxBnoMz-99idQVHiQh80ADW78%253D HTTP/2
Host: media.aiXXXXXXXXXXXXXX.com
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:93.0) Gecko/20100101 Firefox/93.0
Accept: */*
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate, br
Access-Control-Request-Method: GET
Access-Control-Request-Headers: feedurl
Referer: https://aiXXXXXXXXXXXXXXXX.com/
Origin: https://aiXXXXXXXXXXXXXXXX.com
DNT: 1
Connection: keep-alive
Sec-Fetch-Dest: empty
Sec-Fetch-Mode: cors
Sec-Fetch-Site: same-site
Pragma: no-cache
Cache-Control: no-cache
TE: trailers
Preflight response:
HTTP/2 204 No Content
date: Fri, 08 Oct 2021 13:33:10 GMT
x-powered-by: Express
access-control-allow-origin: https://aiXXXXXXXXXXXXXXXXXX.com
vary: Origin
access-control-allow-credentials: true
access-control-allow-methods: GET,HEAD,PUT,PATCH,POST,DELETE
access-control-allow-headers: Content-Type, feedUrl
strict-transport-security: max-age=15724800; includeSubDomains
X-Firefox-Spdy: h2
Request
The preflight passes, and the browsers starts a "flight" request:
GET /media/1.0.0/rtsp/hls?feedUrl=https%3A%2F%2Flive.monuv.com.br%2Fa1%2F14298.stream%2Fstr27%2Fchunklist.m3u8%3Fm_hash%3DkhV_hCnKG3nhaNCFaYZxBnoMz-99idQVHiQh80ADW78%253D HTTP/2
Host: media.aiXXXXXXXXXXXXXXXXXXXXX.com
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:93.0) Gecko/20100101 Firefox/93.0
Accept: */*
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate, br
feedUrl: https://live.monuv.com.br/a1/14298.stream/str27/chunklist.m3u8?m_hash=khV_hCnKG3nhaNCFaYZxBnoMz-99idQVHiQh80ADW78%3D
Origin: https://aiXXXXXXXXXXXXXXXX.com
DNT: 1
Connection: keep-alive
Referer: https://aiXXXXXXXXXXXXXXXXX.com/
Cookie: ory_kratos_session=MTYzMzYzODY1OHxEdi1CQkFFQ180SUFBUkFCRUFBQVJfLUNBQUVHYzNSeWFXNW5EQThBRFhObGMzTnBiXXXXXXXXXXXXYVc1bkRDSUFJSHBtUWxsaWFsVlJhWGRTVGxSMmIzZHRkbTFqYm5CUlRWVkdkelpPWkRoWnXXXTyqwgK-0Pe0qtZHjNhfU-YoASjg3istMZi672swQ==
Sec-Fetch-Dest: empty
Sec-Fetch-Mode: cors
Sec-Fetch-Site: same-site
Pragma: no-cache
Cache-Control: no-cache
TE: trailers
Response
HTTP/2 302 Found
date: Fri, 08 Oct 2021 13:33:10 GMT
content-type: text/plain; charset=utf-8
content-length: 129
location: https://api.aiXXXXXXXXXXXXXXXXXX.com/media/1.0.0/hls/streams/19dd149d-f551-4093-b2aa-e5558388d545/hls.m3u8
x-powered-by: Express
access-control-allow-origin: https://aiXXXXXXXXXXXXXXXX.com
vary: Origin, Accept
access-control-allow-credentials: true
strict-transport-security: max-age=15724800; includeSubDomains
X-Firefox-Spdy: h2
At this response, the browser fails saying that the origin don't match the access-control-allow-origin.
(the first image was from Edge, since the log was more clear; this log is from Firefox)
Problem
The error message—I'm using dummy URLs and origins below—from the browser can be a bit confusing:
Access to XMLHttpRequest at 'https://api.example.com/' (redirected from 'https://media.example.com/') from origin 'https://example.com' has been blocked by CORS policy: Response to preflight request doesn't pass access control check: The 'Access-Control-Allow-Origin' header has a value 'https://example.com' that is not equal to the supplied origin.
The key here is that, as sideshowbarker hinted at in his comment, because your first preflighted request to https://media.example.com/ responds with a cross-origin redirect to https://api.example.com/, the browser performs another whole CORS access-control check for that resource. However, because the redirect resulting from the first preflighted request happens to be cross-origin, the browser sets the origin of the second preflight request (which the error message refers to as the "supplied origin"), not as https://example.com, but as the null origin!
Here's a rundown of what is likely happening:
Because https://api.example.com likely doesn't (and shouldn't!) allow the null, the second access-control check fails and you get that annoying CORS error.
Solution
Resist the temptation to allow the null origin on https://api.example.com/, as doing so has serious security ramifications: it amount to voiding the protection that the Same-Origin Policy provides.
Instead, you should get rid of that redirect from https://media.example.com/ to https://api.example.com/ and make your frontend request the https://api.example.com/ resource directly.
Alternatively, if you cannot completely get rid of the redirect but you can change its destination, make it a same-origin redirect (from somewhere https://media.example.org to elsewhere on https://media.example.org).
I am trying to use SAPUI5 in a SPA to display data from a TEIID/Wildfly ODataV4 service. When SAPUI5 ODATA V4 data model is bound to the service, I run into several errors. I thereby connect via proxy (grunt-connect-proxy2) to the odata service. Basic Auth works. The metadata file above the marked service folder in the attached image (screenshot1) is the actual metadata file of the service which, as you can see, is loaded correctly. So no CORS issue or authorization issue.
Screenshot1
The issue seems to be related to an CSRF Token request as far as I understand. Seems that Teiid/Wildfly is not answering the CSRF Token fetch request. Is there a way to configure Wildfly to answer the request or alternatively a way to disable CSRF requests for the odata V4 model? I have seen such an option in the constructor of the odata V2 model. How could a working configuration look like?
The following is a screenshot from the browser log:
I observed a further issue, from which I do not know if it is related to the previous one (there is also a X-CSRF-Token: Fetch involved), or if a have something more missing somewhere. The second issue happens when I use an aggregation binding to bind a odata collection to a sapui5 list. The response looks like
Request URL: http://localhost:9001/odata4/svc/my_nutri_diary/$batch
Request Method: POST Status Code: 406 Not Acceptable Remote Address:
[::1]:9001 Referrer Policy: no-referrer-when-downgrade Response
Headersview source access-control-allow-credentials: true
access-control-allow-origin: http://localhost:9001 cache-control:
no-cache, no-store, must-revalidate connection: close
content-encoding: gzip content-length: 125 content-type:
application/json;odata.metadata=minimal date: Mon, 12 Nov 2018
20:04:30 GMT expires: 0 odata-version: 4.0 pragma: no-cache server:
WildFly/11 x-powered-by: Undertow/1 Request Headersview source Accept:
multipart/mixed Accept-Encoding: gzip, deflate, br Accept-Language: de
Authorization: Basic SU1TVXNlcjpJTVM0Zm9ydW0l Connection: keep-alive
Content-Length: 329 Content-Type: multipart/mixed;
boundary=batch_id-1542053070786-11 Cookie: sidebar_collapsed=false;
cycle_analytics_help_dismissed=1;
__utmz=111872281.1539128843.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none); __utma=111872281.767670437.1539128843.1541866362.1541870562.42 DNT: 1 Host: localhost:9001 MIME-Version: 1.0 OData-MaxVersion: 4.0
OData-Version: 4.0 Origin: http://localhost:9001 Referer:
http://localhost:9001/ User-Agent: Mozilla/5.0 (iPhone; CPU iPhone OS
11_0 like Mac OS X) AppleWebKit/604.1.38 (KHTML, like Gecko)
Version/11.0 Mobile/15A372 Safari/604.1 X-CSRF-Token: Fetch
X-Requested-With: XMLHttpRequest Request Payload
--batch_id-1542053070786-11 Content-Type:application/http Content-Transfer-Encoding:binary GET Profile?$skip=0&$top=100 HTTP/1.1
Accept:application/json;odata.metadata=minimal;IEEE754Compatible=true
Accept-Language:de
Content-Type:application/json;charset=UTF-8;IEEE754Compatible=true
--batch_id-1542053070786-11--
Thanks for your advice!
Best regards,
Christoph
Further note regarding search for a workaround: As I am currently searching for a workaround to be able to use the odata.v4 model, I found the following blog post:
https://blogs.sap.com/2015/08/05/disable-csrf-token-for-odata-calls-using-sap-netweaver-gateway/
However, the approach does not seem to work for the odata.v4 model as it has a different interface. There is no setHeaders() function to set custom headers on the datamodel. I therefore tried to set the header up via
$.ajaxSetup({headers: {'X-Requested-With': 'X'}});
Unfortunately, this also does not work. If someone has the odata.v4 Model running with TEIID or Olingo v4 it would be great if he could give me a feedback on how he had worked around this issue.
I am new to REST so bear with me if I'm missing something obvious.
Any pointer would be much appreciated as I am a bit lost.
Scenario
I needed to post some data to the following REST service: https://api.dotmailer.com/ from my web application https://myapp.com/.
During testing, I was able to post the data from my local pc.
However, as soon as I published the updated application to https://myapp.com/ on a remote server, I was no longer able to post any data.
What I've tried so far
Added rule to the remote server firewall to allow outgoing traffic to use https. Didn't solve the problem.
Disabled the url rewriting rule that change http to https for myapp.com. Didn't solve the problem.
Pasted the URL I use to post my data (https://api.dotmailer.com/v2/address-books/12345/contacts) in a browser on the remote server, entered the correct credentials, but couldn't access it.
the error message said "Unable to open this internet site. The requested site is either unavailable or cannot be found." If I do the same on my local PC I can access the URL.
Monitored the two calls with Fiddler2.
I include the results of the monitoring process below:
CALLS MADE FROM REMOTE SERVER
----------
POST /bla.aspx HTTP/1.1
Host: myapp.com
Connection: keep-alive
Content-Length: 10660
Cache-Control: max-age=0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
Origin: https://myapp.com
User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.153 Safari/537.36
Content-Type: application/x-www-form-urlencoded
DNT: 1
Referer: https://myapp.com/bla.aspx
Accept-Encoding: gzip,deflate,sdch
Accept-Language: en-US,en;q=0.8,it;q=0.6
Cookie: ASP.NET_SessionId=xxx; Myapp=xxx; GUID=xxx
CALLS MADE FROM LOCAL PC
----------
POST /bla.aspx HTTP/1.1
Host: localhost:xxx
Connection: keep-alive
Content-Length: 10656
Cache-Control: max-age=0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
Origin: http://localhost:60675
User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.153 Safari/537.36
Content-Type: application/x-www-form-urlencoded
DNT: 1
Referer: http://localhost:xxx/bla.aspx
Accept-Encoding: gzip,deflate,sdch
Accept-Language: en-US,en;q=0.8,it;q=0.6
Cookie: __eqtUser=xxx; ASP.NET_SessionId=xxx; Myapp=xxx; GUID=xxx
Question
I believe point 3 shows that the cause is some setting on the remote server.
Does anyone know what it could be? Or am I completely off-track?
Update
I spoke with the developer on the receiving end of my calls who can monitor incoming traffic.
He could see my local calls but not the ones submitted from https://myapp.com.
In response to gmlime reply, I've added the following to myapp.com web.config file but didn't help.
<system.webServer>
<httpProtocol>
<customHeaders>
<add name="Access-Control-Allow-Origin" value="*" />
</customHeaders>
</httpProtocol>
</system.webServer>
Should I put it at a higher level in the hierarchy?
Make sure that this gets added to the response:
YourAddHeaderMethod("Access-Control-Allow-Origin", "*");
Many servers deny posting from other domains and can terminate the connection. You can learn more about it from the w3 docs for Access-Conrol-Allow-Origin and Mozzilla covers some scenarios. You may have to check with the server administrator to rule out cross domain problems also.
I am writing a restful API and try to use all available http method but have a problem with PUT method.
When I send http request whith put method, I have "400 Bad request" error.
If I use POST method, I have no problem.
Here is my http PUT request :
Remote Address:::1:8080
Request URL:http://localhost:8080/adminRight
Request Method:PUT
Status Code:400 Mauvaise Requête
Request Headersview parsed
PUT /adminRight HTTP/1.1
Host: localhost:8080
Connection: keep-alive
Content-Length: 37
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.114 Safari/537.36
Origin: chrome-extension://hgmloofddffdnphfgcellkdfbfbjeloo
Content-Type: application/json
Accept: */*
Accept-Encoding: gzip,deflate,sdch
Accept-Language: fr-FR,fr;q=0.8,en-US;q=0.6,en;q=0.4
Cookie: JSESSIONID=41D1CCDF94D3150F0FCA3754E347A4AD
Request Payload
typeList=1&id=2&nom=labelViewerAvance
Response Headersview parsed
HTTP/1.1 400 Mauvaise Requête
Server: Apache-Coyote/1.1
Content-Type: text/html;charset=utf-8
Content-Length: 984
Date: Fri, 30 May 2014 12:55:32 GMT
Connection: close
And here my http POST request :
Remote Address:::1:8080
Request URL:http://localhost:8080/adminRight
Request Method:POST
Status Code:200 OK
Request Headersview parsed
POST /adminRight HTTP/1.1
Host: localhost:8080
Connection: keep-alive
Content-Length: 37
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.114 Safari/537.36
Origin: chrome-extension://hgmloofddffdnphfgcellkdfbfbjeloo
Content-Type: application/json
Accept: */*
Accept-Encoding: gzip,deflate,sdch
Accept-Language: fr-FR,fr;q=0.8,en-US;q=0.6,en;q=0.4
Cookie: JSESSIONID=41D1CCDF94D3150F0FCA3754E347A4AD
Request Payload
typeList=1&id=2&nom=labelViewerAvance
Response Headersview parsed
HTTP/1.1 200 OK
Server: Apache-Coyote/1.1
Content-Type: application/json;charset=utf-8
Content-Length: 2
Date: Fri, 30 May 2014 13:09:03 GMT
What is the difference between PUT and POST syntax? Or maybe, is it one special configuration in my web.xml?
Thanks in advance for your help.
Edit with new information :
My requests are mapped in java with these two methods :
#RequestMapping(value = "/adminRight",
method = RequestMethod.PUT
)
#ResponseBody
public ResponseEntity<String> updateListRights(#RequestParam(value = "typeList") String typeList,
#RequestParam(value = "id") String idList,
#RequestParam(value = "nom") String nomList)
{
and
#RequestMapping(value = "/adminRight",
method = RequestMethod.POST
)
#ResponseBody
public ResponseEntity<String> addNewListRights(#RequestParam(value = "typeList") String typeList,
#RequestParam(value = "id") String idList,
#RequestParam(value = "nom") String nomList)
{
Your Server: Apache-Coyote/1.1 is just a HTTP connector. Behind that connector there is a web server, for example Apache Tomcat. You have to look up the manual of that server and check how you can allow a HTTP method. By Tomcat there is a server.xml file, in that there is something like this:
// Sample Security Constraint
<security-constraint>
<web-resource-collection>
<web-resource-name><strong>restricted methods</strong></web-resource-name>
<url-pattern>/*</url-pattern>
<http-method>PUT</http-method>
<http-method>POST</http-method>
<http-method>DELETE</http-method>
<http-method>OPTIONS</http-method>
<http-method>TRACE</http-method>
</web-resource-collection>
<auth-constraint />
</security-constraint>
You should add PUT and DELETE to that list. If your REST clients are running in browsers and they are served under a different domain, then you have to enable the OPTIONS method either (for CORS preflight requests), and add CORS allow headers as well. By serving browsers you have to add some HTTP response headers as well and set them properly to prevent XSS attacks.
Another security concern that you should hide the version number of the coyote connector.
Btw. using session cookies like Cookie: JSESSIONID=41D1CCDF94D3150F0FCA3754E347A4AD is not RESTful.
I know very little about java request mapping, but by REST you use POST usually to add a new item resource to a collection resource, for example POST /rights in your case, and PUT usually to edit an entire item resource, for example PUT /rights/{id} where {id} should be a unique resource id (probably the same as one of your aggregate ids). In your code I can't see anything related to this URL structure by the PUT request. You may be interested in PATCH as well.
I'm trying to implement file upload functionality in the iPhone app. Server code is tested and works when files are uploaded from the desktop browser, so I moved to implementing the Objective-C client code. I'm assembling HTTP requests body manually, and despite that it looks correct, it is rejected by the server (server handler unable to extract the parts from multipart content). In desperation I've simplified the form to having only one parameter, but it still does not work.
I've captured the network traffic and I could see that Wireshark could not parse my multipart content as well (have a look at screenshots: Firefox request, iPhone request). I'm pasting it below in hope that you could see the errors I can't see.
Thanks in advance.
Firefox:
POST /cubepaint/actions/gallery/post HTTP/1.1
Host: [...]
User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en-GB; rv:1.9.1.3) Gecko/20090824 Firefox/3.5.3
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-gb,en;q=0.5
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 300
Connection: keep-alive
Authorization: Basic [...]
Content-Type: multipart/form-data; boundary=---------------------------20072377098235644401115438165
Content-Length: 180
-----------------------------20072377098235644401115438165
Content-Disposition: form-data; name="deviceId"
12345
-----------------------------20072377098235644401115438165--
HTTP/1.1 200 OK
Date: Sat, 17 Oct 2009 22:09:21 GMT
Server: Apache/2.2.3 (Debian) DAV/2 SVN/1.4.2 mod_python/3.2.10 Python/2.4.4 mod_ssl/2.2.3 OpenSSL/0.9.8c
Keep-Alive: timeout=15, max=100
Connection: Keep-Alive
Transfer-Encoding: chunked
Content-Type: text/html; charset=UTF-8
iPhone:
POST /cubepaint/actions/gallery/post HTTP/1.1
Host: [...]
User-Agent: Copenhagen/1.0 CFNetwork/459 Darwin/9.8.0
Content-Type: multipart/form-data; boundary=----------0E7B16E6-CD3D-4213-9B42-07DA30822C74
Accept: */*
Accept-Language: en-us
Accept-Encoding: gzip, deflate
Authorization: Basic [...]
Content-Length: 187
Connection: keep-alive
----------0E7B16E6-CD3D-4213-9B42-07DA30822C74
Content-Disposition: form-data; name="deviceId"
00000000-0000-1000-8000-0016CBCC0B61
----------0E7B16E6-CD3D-4213-9B42-07DA30822C74--
HTTP/1.1 200 OK
Date: Sat, 17 Oct 2009 22:04:07 GMT
Server: Apache/2.2.3 (Debian) DAV/2 SVN/1.4.2 mod_python/3.2.10 Python/2.4.4 mod_ssl/2.2.3 OpenSSL/0.9.8c
Keep-Alive: timeout=15, max=100
Connection: Keep-Alive
Transfer-Encoding: chunked
Content-Type: text/html; charset=UTF-8
Your iPhone version indicates keep-alive but doesn't specify a length. Not sure that's enough to cause trouble.
Also, is it possible your server is checking for user-agent strings it recognizes (say, for backward-compatibility mode)?
I'd also compare the two in a text editor that shows CR/LF characters to make sure you're getting proper line endings.
Another thing you could try is create a simple web-page that does a multipart POST and run it from the iPhone browser (instead of the Mac one) then check the headers that go across the wire. Or you could snag a toolkit like ASIHTTPRequest and see what kind of output it generates for multi-part posts (or just use the toolkit instead of trying to write your own).
Good luck
Solved by reading RFC 2046 (MIME specification): boundary between parts of multipart message should contain two leading '-'s, and last boundary should additionally contain two trailing '-'s. The boundary in the request header and request body in the Firefox request differ:
---------------------------20072377098235644401115438165
and
-----------------------------20072377098235644401115438165
The last boundary looks like this:
-----------------------------20072377098235644401115438165--
You really could not see this with the eye when there are so many leading '-'s in the original boundary.