I am trying to upload files from compute engine instance (debian) to cloud storage.
at some point gsutil completely stopped working. When running it with -D flag I see those replies:
reply: 'HTTP/1.1 400 Bad Request\r\n'header: Content-Type: application/json; charset=UTF-8header: Content-Encoding: gzipheader: Date: Tue, 27 May 2014 21:09:47 GMTheader: Expires: Tue, 27 May 2014 21:09:47 GMTheader: Cache-Control: private, max-age=0header: X-Content-Type-Options: nosniffheader: X-Frame-Options: SAMEORIGINheader: X-XSS-Protection: 1; mode=blockheader: Server: GSEheader: Alternate-Protocol: 443:quicheader: Transfer-Encoding: chunkedprocess count: 1thread count: 10
Try setting parallel_composite_upload_threshold = 0 under the [GSUtil] section in your boto file.
Related
I am using traefik in kubernetes and I have a service deployed that is returning 103 Early Hint. I can confirm that it is working by directly querying the service, e.g.
curl -D - http://contra-web-app
HTTP/1.1 103 Early Hints
Link: <https://builds.contra.com>; rel="preconnect"; crossorigin
Link: <https://fonts.googleapis.com/css2?family=Inter:wght#400;500;600;700;900&display=swap>; rel="preload"; as="font"
Link: <https://builds.contra.com/3f509d0cc/assets/entry-client-routing.4f895d55.js>; rel="modulepreload"; as="script"; crossorigin
Link: <https://www.googletagmanager.com/gtag/js?id=G-96H5NXQ2PR>; rel="preload"; as="script"
HTTP/1.1 200 OK
cache-control: no-store
referrer-policy: strict-origin-when-cross-origin
x-frame-options: sameorigin
content-type: text/html
content-length: 9062
Date: Tue, 26 Jul 2022 20:34:19 GMT
Connection: keep-alive
Keep-Alive: timeout=72
However, requesting the same service through Traefik just returns 200 response:
curl -H 'host: contra.com' -D - http://contra-traefik.traefik/gajus
HTTP/1.1 200 OK
Cache-Control: no-store
Content-Length: 11441
Content-Type: text/html
Date: Tue, 26 Jul 2022 19:51:48 GMT
Referrer-Policy: strict-origin-when-cross-origin
Set-Cookie: contra_web_app_service=394e7e912ad85b66; Path=/; Secure
Vary: Accept-Encoding
X-Frame-Options: sameorigi
At this point, I am unable to establish whether I am missing a configuration or if Traefik does not support it.
We recently started using Azure CDN however some users reported an issue and we got a screenshot from one of them:
:
We were not able to get any more information on the issue. Any idea what could cause this and how to fix it?
Our origin server returns correctly the file with the following response headers:
Accept-Ranges: bytes
Access-Control-Allow-Origin: https://www.google.com
Cache-Control: max-age=31536000
Content-Encoding: gzip
Content-Length: 1956119
Content-Type: text/css
Date: Tue, 16 Feb 2021 18:44:10 GMT
ETag: "011889dd7ffd61:0"
Expect-CT: max-age=604800, enforce,
Feature-Policy: autoplay 'none'; camera 'none'
Last-Modified: Wed, 10 Feb 2021 18:07:38 GMT
strict-Transport-Security: max-age=31536000; includeSubDomains; preload
Vary: Accept-Encoding
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-XSS-Protection: 1; mode=block
I got a similar problem. I was able to fix this by enabling cache i Dev Tools.
Enable cahce in Dev Tools
I'm trying to upload a big file (9gb) to google storage using Cyberduck.
The login and transfer with small files work. However for this file I'm getting the following error:
GET / HTTP/1.1
Date: Wed, 30 Apr 2014 08:47:34 GMT
x-goog-project-id: 674064471933
x-goog-api-version: 2
Authorization: OAuth SECRET_KEY
Host: storage.googleapis.com:443
Connection: Keep-Alive
User-Agent: Cyberduck/4.4.4 (Mac OS X/10.9) (x86_64)
HTTP/1.1 200 OK
Content-Type: application/xml; charset=UTF-8
Content-Length: 340
Date: Wed, 30 Apr 2014 08:47:35 GMT
Expires: Wed, 30 Apr 2014 08:47:35 GMT
Cache-Control: private, max-age=0
Server: HTTP Upload Server Built on Apr 16 2014 16:50:43 (1397692243)
Alternate-Protocol: 443:quic
GET /vibetracestorage/?prefix=eventsall.csv&uploads HTTP/1.1
Date: Wed, 30 Apr 2014 08:47:35 GMT
x-goog-api-version: 2
Authorization: OAuth SECRET_KEY
Host: storage.googleapis.com:443
Connection: Keep-Alive
User-Agent: Cyberduck/4.4.4 (Mac OS X/10.9) (x86_64)
HTTP/1.1 400 Bad Request
Content-Type: application/xml; charset=UTF-8
Content-Length: 173
Date: Wed, 30 Apr 2014 08:47:36 GMT
Expires: Wed, 30 Apr 2014 08:47:36 GMT
Cache-Control: private, max-age=0
Server: HTTP Upload Server Built on Apr 16 2014 16:50:43 (1397692243)
Alternate-Protocol: 443:quic
Am I missing anything? Thanks.
According to that log you posted, you're placing a GET to "https://storage.googleapis.com/vibetracestorage/?prefix=eventsall.csv&uploads".
I don't know what that "uploads" parameter tacked onto the end is, but it's not a valid parameter for requesting a bucket listing (which is what that request does).
If you place that request by hand, you'll see this error:
<?xml version='1.0' encoding='UTF-8'?><Error><Code>InvalidArgument</Code><Message>Invalid argument.</Message><Details>Invalid query parameter(s): [uploads]</Details></Error>
Also, as a general point of good practice, do not post logs that contain your full Authorization header. That is a very, very bad idea. You may want to delete this question, although those credentials will expire (and perhaps already have).
This is an interoperability issue. In Cyberduck, when connected to S3, multipart uploads are supported as defined by Amazon S3. The request with the uploads parameter is used to find already existing in progress multipart uploads for the same target object that can be resumed.
Make sure to choose Google Storage and not S3 in the protocol dropdown list in the bookmark or connection prompt. Multipart uploads should then be disabled.
I'm hosting a FireFox plugin on Google Cloud Storage. In order to be properly handled by FireFox, the Content-Type needs to be set to application/x-xpinstall
I have uploaded as follows:
gsutil -h "Content-Type: application/x-xpinstall" cp -a public-read \
ActivityInfo.xpi gs://download.activityinfo.org
When accessed from the standard endpoint, everything is correct:
$ curl -s -D - http://commondatastorage.googleapis.com/download.activityinfo.org/ActivityInfo.xpi \
-o /dev/null
HTTP/1.1 200 OK
Server: HTTP Upload Server Built on Feb 13 2013 15:53:33 (1360799613)
Expires: Thu, 28 Feb 2013 12:38:30 GMT
Date: Thu, 28 Feb 2013 11:38:30 GMT
Last-Modified: Thu, 28 Feb 2013 11:38:01 GMT
ETag: "1ee983889c947a204eab4db6902c9a67"
x-goog-generation: 1362051481261000
x-goog-metageneration: 1
Content-Type: application/x-xpinstall
Content-Language: en
x-goog-crc32c: a11b93ab
Accept-Ranges: bytes
Content-Length: 5562
Cache-Control: public, max-age=3600, no-transform
Age: 491
But when I try to access from the custom domain download.activityinfo.org, the header reverts to application/octet-stream
$ curl -s -D - http://download.activityinfo.org/ActivityInfo.xpi -o /dev/null
HTTP/1.1 200 OK
Server: HTTP Upload Server Built on Feb 13 2013 15:53:33 (1360799613)
Expires: Thu, 28 Feb 2013 12:10:24 GMT
Date: Thu, 28 Feb 2013 11:10:24 GMT
Last-Modified: Wed, 27 Feb 2013 20:36:24 GMT
ETag: "1ee983889c947a204eab4db6902c9a67"
x-goog-generation: 1361997384772000
x-goog-metageneration: 2
Content-Type: application/octet-stream
x-goog-crc32c: a11b93ab
Accept-Ranges: bytes
Content-Length: 5562
Cache-Control: public, max-age=3600, no-transform
Age: 2298
I have set the CNAME to c.storage.googleapis.com per the docs
$ nslookup download.activityinfo.org
Non-authoritative answer:
Server: Comtrend.Home
Address: 192.168.1.1
Name: storage.l.googleusercontent.com
Addresses: 2a00:1450:400c:c00::80
173.194.78.128
Aliases: download.activityinfo.org
c.storage.googleapis.com
Is this a bug or do I need to change my configuration?
The two results above have different values in x-goog-generation and x-goog-metageneration, which makes me suspect you have uploaded the object more than once, and you were seeing the results from different versions (which have different values for Content-Type). Do you have versioning enabled for the bucket? If not, then maybe there is some caching going on in one of the paths. Are you still seeing this behavior?
I'm watching 392 repositories on Github. However, the Github API only returns 100. Does anyone have any idea why?
https://github.com/api/v2/json/repos/watched/trivektor
You need to paginate manually using the page parameter. The HTTP Response headers will tell you the next and the last page, if available. Check the headers:
X-Next
X-Last
Examples:
curl -D- https://github.com/api/v2/json/repos/watched/trivektor
HTTP/1.1 200 OK
Server: nginx/1.0.4
Date: Sat, 22 Oct 2011 08:24:45 GMT
Content-Type: application/json; charset=utf-8
Connection: keep-alive
Status: 200 OK
X-RateLimit-Limit: 60
ETag: "c597e396e9f17b91c5c5a7e462ba954f"
X-Next: https://github.com/api/v2/json/repos/watched/trivektor?page=2
X-Last: https://github.com/api/v2/json/repos/watched/trivektor?page=5
Now the 2nd page:
curl -D- https://github.com/api/v2/json/repos/watched/trivektor?page=2
HTTP/1.1 200 OK
Server: nginx/1.0.4
Date: Sat, 22 Oct 2011 08:28:08 GMT
Content-Type: application/json; charset=utf-8
Connection: keep-alive
Status: 200 OK
X-RateLimit-Limit: 60
ETag: "c57d0e97e2062672cb3771467cf2abc7"
X-Next: https://github.com/api/v2/json/repos/watched/trivektor?page=3
X-Last: https://github.com/api/v2/json/repos/watched/trivektor?page=5
X-Frame-Options: deny
X-RateLimit-Remaining: 58
X-Runtime: 353ms
Content-Length: 44966
Cache-Control: private, max-age=0, must-revalidate
And the last one:
curl -D- https://github.com/api/v2/json/repos/watched/trivektor?page=5
HTTP/1.1 200 OK
Server: nginx/1.0.4
Date: Sat, 22 Oct 2011 08:28:30 GMT
Content-Type: application/json; charset=utf-8
Connection: keep-alive
Status: 200 OK
X-RateLimit-Limit: 60
ETag: "11ce44ebc229eab0dc31731b39e10dcf"
X-Frame-Options: deny
X-RateLimit-Remaining: 57
X-Runtime: 93ms
Content-Length: 7056
Cache-Control: private, max-age=0, must-revalidate
Very common for API's to limit the size of a response object to protect against outliers. Given that it's returning a round number, that suggests this is by design. I don't see them discussing paging in their docs, so it might just be a hard cap. Either way, you should just ping github.