Google Cloud Storage upload 10gb filesize error in Cyberduck - google-cloud-storage

I'm trying to upload a big file (9gb) to google storage using Cyberduck.
The login and transfer with small files work. However for this file I'm getting the following error:
GET / HTTP/1.1
Date: Wed, 30 Apr 2014 08:47:34 GMT
x-goog-project-id: 674064471933
x-goog-api-version: 2
Authorization: OAuth SECRET_KEY
Host: storage.googleapis.com:443
Connection: Keep-Alive
User-Agent: Cyberduck/4.4.4 (Mac OS X/10.9) (x86_64)
HTTP/1.1 200 OK
Content-Type: application/xml; charset=UTF-8
Content-Length: 340
Date: Wed, 30 Apr 2014 08:47:35 GMT
Expires: Wed, 30 Apr 2014 08:47:35 GMT
Cache-Control: private, max-age=0
Server: HTTP Upload Server Built on Apr 16 2014 16:50:43 (1397692243)
Alternate-Protocol: 443:quic
GET /vibetracestorage/?prefix=eventsall.csv&uploads HTTP/1.1
Date: Wed, 30 Apr 2014 08:47:35 GMT
x-goog-api-version: 2
Authorization: OAuth SECRET_KEY
Host: storage.googleapis.com:443
Connection: Keep-Alive
User-Agent: Cyberduck/4.4.4 (Mac OS X/10.9) (x86_64)
HTTP/1.1 400 Bad Request
Content-Type: application/xml; charset=UTF-8
Content-Length: 173
Date: Wed, 30 Apr 2014 08:47:36 GMT
Expires: Wed, 30 Apr 2014 08:47:36 GMT
Cache-Control: private, max-age=0
Server: HTTP Upload Server Built on Apr 16 2014 16:50:43 (1397692243)
Alternate-Protocol: 443:quic
Am I missing anything? Thanks.

According to that log you posted, you're placing a GET to "https://storage.googleapis.com/vibetracestorage/?prefix=eventsall.csv&uploads".
I don't know what that "uploads" parameter tacked onto the end is, but it's not a valid parameter for requesting a bucket listing (which is what that request does).
If you place that request by hand, you'll see this error:
<?xml version='1.0' encoding='UTF-8'?><Error><Code>InvalidArgument</Code><Message>Invalid argument.</Message><Details>Invalid query parameter(s): [uploads]</Details></Error>
Also, as a general point of good practice, do not post logs that contain your full Authorization header. That is a very, very bad idea. You may want to delete this question, although those credentials will expire (and perhaps already have).

This is an interoperability issue. In Cyberduck, when connected to S3, multipart uploads are supported as defined by Amazon S3. The request with the uploads parameter is used to find already existing in progress multipart uploads for the same target object that can be resumed.
Make sure to choose Google Storage and not S3 in the protocol dropdown list in the bookmark or connection prompt. Multipart uploads should then be disabled.

Related

Getting HTTP/1.1 404 error on ReadyAPI REST virtual service on specific resource paths

Why is readyAPI responding with a 404 error when the recourse path is set other than /test in a REST Virtual service.
/test
Sun May 29 16:42:53 BST 2022: DEBUG: http-outgoing >>
POST /test HTTP/1.1
Accept-Encoding: gzip,deflate
Content-Type: application/json
Content-Length: 0
Host: localhost:9019
Connection: Keep-Alive
User-Agent: Apache-HttpClient/4.5.2 (Java/16.0.1)
Sun May 29 16:42:53 BST 2022: DEBUG: http-incoming <<
HTTP/1.1 200 OK
Date: Sun, 29 May 2022 15:42:53 GMT
Content-Type: application/xml
Content-Encoding: gzip
Content-Length: 1249
Server: Jetty(9.4.40.v20210413)
[0x1f][0x8b][0x8][0x0][0x0][0x0][0x0][0x0][0x0][0xff][0xcd]W[0xdb]r[0xe2]F[0x10][0xfd][0x15][0xad][0x9e][0xb9]H[0x2]q+[0xca][0x15][0x8c]|[0x8b][0x8d][0xa1][0x0]o[0x92]J[0xb9][0xa8]A[0x1a]`6[0x92]F;3[0xda]][0xd6][0xcb][0xd7][0xe4]G[0xd3][0xad][0x1b][0x17][0xcb]v[0xaa][0xe2][0xad]
O[0xe2]tk[0xa6][0xa7][0xfb][0x9c]#[0xa9]?[0xe7]gZ[0xe0]y[0x82]Jy[0xe6][0x91]/[0xcc][0xab][0x11][0x97]KE~Y[0x7][0x84][0xf9]5[0x97][0x7][0xfd]z[0x1e][0xd7][0xfa][0xb3]x)][0xc1][0x96]T[0xdc][0xd2][0xed][0x19][0x17][0x1e][0x15][0xb][0x97][0x87]+&[0x2][0xa2][0x18][0xf][0x17]4[0xec][0xd7][0x8f][0x93][0xb4][0xfe][0x90][0x87][0x8a][0xb8]j[0xa0][0x14][0x80][0xb1][0xa2][0xc7][0xb][0x1d][0xc1][0x14]7u[0x88]"gO[0xfa][0x9c][0xeb][0xbd]'=[0xdb][[0xef][0xe9][0xe5][0xd5][0xe9][0x15][0xfd]h?H,-[0xeb][0x87][0xd9]h7[0x8c]v[0xab]i[0xfe]xq[0xa1]g[0x85]b[0x1]e[0x95]"^[0x94]
;f[0xff][0xe6][0xdb][0x88][0xbe][0xb4]?,[0xef][0x93]p[0x1d][0x93]5[0xa6]$[0xff][0x3][0xea][0x11][0xdf]g[0xa4]X[0x17][0x2][0x0]K[0xe2]S[0xb9][0xe2][0xc2][0xa5][0xf][0x92][0x8a][0x1b][0xf][0xd0][0x90]$~L1[0x18][0x91]k[0xa8][0x1b]0iXK[0xaa]HMR%[0x98][0xb7][0xa6]2;[0x85][0xa0].[0x8b][0x18][0x15]V[0x19][0xc3][0x12]YYS[0x88]ETH[0x1e][0xce][0x99][0xf2][0x11][0x19]1hlE[0x87]2[0xa5][0xba]'[0x1]B[0xe][0xf6]&[0xa9][0xb5][0x80][0x6]I[0xa3][0x0]K[0xce][0xf8][0xda] [0x2][0xbe]d>[0xbd][0x8f][0x3]h[0x16][0xe4]5[0xec]n[0xd5]0[0x1b][0xd5]N[0xab]e[0xeb][0xbb]J[0xda][0x17],*[0xb9][0xc0][0x83][0x15]3[0xa9][0xe8][0x8a]+[0xe2]O[0x4][0xf7]bWM[0x4]sag[0xab]]3[0xf0]W[0x12][0xbc][0xf8][0xe6][0xfa]Wl[0xa5][0xce][0xf9][0xb7]gy3[0xec][0xdf][0x9c]#[0xc0][0xa8][0xed][0xc1][0x8b]"[0x16][0xae][0x87][0x1b]"p[0x2][0xa6]qt[0xcf][0xc0][0xfb][0x14]K[0x15]$M+"kABo[0x8e]a[0xbd][0xd7]8[0xd9]C[0x91][0xd5][0xca]a[0xd2][0xe5]qrK[0x6];[0x14]+[0xa4][0xde][0xc7][0xc1]<[0x1]e[0xca][0x9d][0x8]90[0x88]"[0x9f]Q8[0xf4][0x8a][0xf8][0x92]Vt7[0x16][0x82][0x86].[0x12][0xf6][0xea]|[0xa2][0xef][0x81][0xd9]6Xr[0xec][0xf3][0xdf]80[0x9f][0xb8][0xd4][0x3][0x9a][0xe1]$,[0xa3]j4[0xab][0x96]aY[0x9a]a[0xf7][0xec].T[0x5])[0xd0]s[0x1f]NV[0x8][0xe5]IW[0xd9]|[0x1d]q2[0xdd][0xb1][0xcf][0xbe]0r<[0xde][0x19][0x9][0xdd][0xfd][0xe] IW[0xb8]c!5!`[0x9a]mm[0xc4]<[0xcf][0xa7]_[0x99][0xbb][0xd1][0xa6]H[0xb][0x97][0xa9]mz[0x8f][0xb7]$[0xee][0x6][0xb][0x4][0x16][0xb8][0xdc][0xc3][0x95][0x86][0xbf][0x99][0xa6]f^^c[0x1e][0xf6]E[0xa4]g[0xc3][0xd1][0xcb][0xac][0xfd][0xff][0xeb]"=
;S[0xb1][0xbd][0x12]<[0x8e][0xa0][0xc6]?_`*[0xa3][0xc3]d-[0xb8][0xfe][0xac][0xb6][0xe3][0x15][0xd6][0x2][0xe9][0xd6][0xc9]
Y[0xe9]{}^[0xc6][0xfe]
fE[0xb1][0xc6][0xbc]![0xe3][0x84][0x1b][0xd8][0xf]DFi[0x8d][0xf][0xb7][0xf7]#[0xe7][0x8f][0xc1][0xe5][0xfc] 1[0xd3][0xc4][0x9e][0xb7]y[0xbe]C[0xb][0x8a][0xc1][0xad][0xf7]<`![0xb0][0xc5][0xd3][0x6]+EE[0xc8]9:M^[0x17]H[0x8][0xa4][0x9b][0x88][0xd3][0x99]8[0xda]5[0xf][0xa8][0xe6]d[0xb1][0xa3][0x1b][0x14][0xb][0xa8][0xf4][0xb9][0x2][0x9e][0xb]4[0x1b][0xd3][0xea][0x19][0xc6][0x1]~[0x11][0xa2]-[0x99][0x9d][0x14]E[0x93][0x99]Q[0x9f]"[0xf5]s[0xaa][0xb6]r[0xaa][0xa2]q[0x10][0x1][0xd4][0x97]*[0xdf][0xaa],[0xc7][0x7][0xec][0xb5][0x8c][0xdc]?[0xb2]f[0x17][0xb3]Q4[0xc0][0xf9][0xb4][[0x96]e5[0x9b][0x15][0x9d]![0x10][0xae][0xf8][0xde]gJ3p[0xa4]F[0xd7]nt[0xd2][0xff][0x13]8[0xe6][0xde][0xb7][0xc][0xbb][0xd5]l[0xb5]P[,[0x0][0xbf]~[0x98][0xde][0x1][0xba]Q*[0x92][0xbd]z=[0x81][0xe4][0x89][0xed][0xd6][0x99]L[0x3][0xf5]=^[0x9f]&[0x93]2L{1[0xbd]p[0x16]#X[0xee]sLB[0x95][0xb0][0xd3][0xac][0xc1][0x8][0xa3][0xcc][0xe7][0xac][0xc2][r[0xd7]:t[0xbd][0xd3][0x9c][0x17][0xfd]J[0xc6][0x89][0xc5][0x88]![0xf7]y,2[0xfa][0xc1][0xd6]([0x0][0x2][0xd6][0xa2]2[0x7][0xc5][0xd6]D[0xe9]e[0xd6][0x7][0xc3][0xe8]Vr[0xe8][0xa8][0x15][0xc5][0x11][0xf4]"[0x9e]+2$[0x91][0xdc]p[0xa5][0xb9][0x82]KY]ro[0xab]-[0xc9][0xfa][0xfd]{[0xb6]D[0x17][0xce]6[0x1d][0xa6]C[0xed][0xd7][0xc1]p|>[0x83][0xc8]W[0x1e][0xc2]|[0xa7][0x9c][0x7][0x97]>[0xec][0x9c][0x1a]+[0xf0][0x84][0xc9][0xd9][0x81][0xef][0x16][0x86][0xcb][0xe4][0xe4]{[0x98][0xfa][0xff][0xec][0xf6][0xe1][0x0][0xce][0x1e]$[0xa5] [0x1a][0xb6][0x12]1[0xcd]h[0x92]<>[0x89][0xcf]$[0xc9]d[0xbb]K[0x3][0x98]=![0xee]_dR[0xc5][0xfe][0xae]Wj %w[0xc7]9[0x5]S[0xce]>[0x83][0xf7][0xcc][0xb4]w[0x8f]h[0x94][0x94][0xa7]/1[0x1][0x11]n[0xf5][0x13]<d[0x97][0xd0]\[0xb2][0xae][0x9a]UA[0xbd][0xc5][0xe1]<[0xe4][0xc9]s[0xee][0xdf]=P[0xe0][0x99][0xad]b|[0x9d][0x1a][0xea][0xbb]][0xa5]T#[0xf6][0x9b]*[0xb2][0xf]T[0xd4]4m[0xc3],QQ[0xdb][0xee]t;[0xcd][0xb6][0xf5]_[0x19][0xd1]5;U[0x13][0xce]\[0x9d][0xdd]]^[0xdd]\[0xce][0xcf][0xc7][0xbf][0xdf][0xdc]_[0xbd]"'[0xfb]m5[0xd9]o[0x8b][0xe9]U[0xc5]4[0xa1] [0xb3]\1[0xe5][0xe5]>[0x97]Oq#[0xb9][0xa3][0xe1][0x8b][0xcc][0xcf]k[0xd4][0xcf]PJ[0xa6][0x8a]C[0xa1][0xe4]y[0xef][0xa1][0x94]|[0xb6][0xfb]|[0xdc][0x1][0x9a][0xfe][0x8a][0x84][0x9a][0xbb][0xc7]$>[0x82][0x17][0x84][0xf4]=[0xfb][0x9a]D[0x11][0xf8][0x12][0x13]j[0xe3][0x91][0xed][0x87][0xf][0xfa][0xa1][0xc4][0xaa][0x8b][0x17][0x87][0xf5][0xe][0xda]z[0xc4]b[0x12]~[0xcd][0x9f][0xf3][0xf0]&[0x88][0xb8]PN[0xc][0xef][0x12][0x90]o[0xec]vp[0xc3]n[0xd7][0xaf][0xef][0xbf][0xb4][0xc3][0xef][0xa8][0xa3]o[0xa4]z[0xd9][0xf7]T[0x1d]>[0xe4][0xfe][0x1][0xe1][0x96]s{[0xcd][0x0][0x0]
/sendEmail
Sun May 29 16:47:12 BST 2022: DEBUG: http-outgoing >>
POST /sendEmail HTTP/1.1
Accept-Encoding: gzip,deflate
Content-Type: application/json
Content-Length: 0
Host: localhost:9019
Connection: Keep-Alive
User-Agent: Apache-HttpClient/4.5.2 (Java/16.0.1)
Sun May 29 16:47:12 BST 2022: DEBUG: http-incoming <<
HTTP/1.1 404 Not Found
Date: Sun, 29 May 2022 15:47:12 GMT
Transfer-Encoding: chunked
Server: Jetty(9.4.40.v20210413)
0
Are there any specific rules to set the resource paths?
I intend to set the resource path as /messaging/v1/messageDefinitionSends/ for the first action(SendEmail) but it's failing.
second action SendToken is behaving the same, if the path is set as /test it response with a 200 ok status but returns a 404 error on setting the path to /v2/token.
Groovy scripts are used for creating the response.
any explanation for this behavior in ReadyAPI?
I think you might not be setting up your operations correctly. It seems like you're reusing the same endpoint but giving them different names and that is likely the reason why you're getting behavior like this.
What I'd recommend is taking a look at ReadyAPI's documentation on how to add in new operations. I think that once you set up the operations correctly you should be able to access them properly.
https://support.smartbear.com/readyapi/docs/virtualization/configure/create-operations.html
The issue was with the resource path, it was missing.
I assumed it was automatically added. after updating it started working.

gsutil is stuck without copying any data

I am trying to upload files from compute engine instance (debian) to cloud storage.
at some point gsutil completely stopped working. When running it with -D flag I see those replies:
reply: 'HTTP/1.1 400 Bad Request\r\n'header: Content-Type: application/json; charset=UTF-8header: Content-Encoding: gzipheader: Date: Tue, 27 May 2014 21:09:47 GMTheader: Expires: Tue, 27 May 2014 21:09:47 GMTheader: Cache-Control: private, max-age=0header: X-Content-Type-Options: nosniffheader: X-Frame-Options: SAMEORIGINheader: X-XSS-Protection: 1; mode=blockheader: Server: GSEheader: Alternate-Protocol: 443:quicheader: Transfer-Encoding: chunkedprocess count: 1thread count: 10
Try setting parallel_composite_upload_threshold = 0 under the [GSUtil] section in your boto file.

Content-Type header lost when using custom URL

I'm hosting a FireFox plugin on Google Cloud Storage. In order to be properly handled by FireFox, the Content-Type needs to be set to application/x-xpinstall
I have uploaded as follows:
gsutil -h "Content-Type: application/x-xpinstall" cp -a public-read \
ActivityInfo.xpi gs://download.activityinfo.org
When accessed from the standard endpoint, everything is correct:
$ curl -s -D - http://commondatastorage.googleapis.com/download.activityinfo.org/ActivityInfo.xpi \
-o /dev/null
HTTP/1.1 200 OK
Server: HTTP Upload Server Built on Feb 13 2013 15:53:33 (1360799613)
Expires: Thu, 28 Feb 2013 12:38:30 GMT
Date: Thu, 28 Feb 2013 11:38:30 GMT
Last-Modified: Thu, 28 Feb 2013 11:38:01 GMT
ETag: "1ee983889c947a204eab4db6902c9a67"
x-goog-generation: 1362051481261000
x-goog-metageneration: 1
Content-Type: application/x-xpinstall
Content-Language: en
x-goog-crc32c: a11b93ab
Accept-Ranges: bytes
Content-Length: 5562
Cache-Control: public, max-age=3600, no-transform
Age: 491
But when I try to access from the custom domain download.activityinfo.org, the header reverts to application/octet-stream
$ curl -s -D - http://download.activityinfo.org/ActivityInfo.xpi -o /dev/null
HTTP/1.1 200 OK
Server: HTTP Upload Server Built on Feb 13 2013 15:53:33 (1360799613)
Expires: Thu, 28 Feb 2013 12:10:24 GMT
Date: Thu, 28 Feb 2013 11:10:24 GMT
Last-Modified: Wed, 27 Feb 2013 20:36:24 GMT
ETag: "1ee983889c947a204eab4db6902c9a67"
x-goog-generation: 1361997384772000
x-goog-metageneration: 2
Content-Type: application/octet-stream
x-goog-crc32c: a11b93ab
Accept-Ranges: bytes
Content-Length: 5562
Cache-Control: public, max-age=3600, no-transform
Age: 2298
I have set the CNAME to c.storage.googleapis.com per the docs
$ nslookup download.activityinfo.org
Non-authoritative answer:
Server: Comtrend.Home
Address: 192.168.1.1
Name: storage.l.googleusercontent.com
Addresses: 2a00:1450:400c:c00::80
173.194.78.128
Aliases: download.activityinfo.org
c.storage.googleapis.com
Is this a bug or do I need to change my configuration?
The two results above have different values in x-goog-generation and x-goog-metageneration, which makes me suspect you have uploaded the object more than once, and you were seeing the results from different versions (which have different values for Content-Type). Do you have versioning enabled for the bucket? If not, then maybe there is some caching going on in one of the paths. Are you still seeing this behavior?

Weird behaviour on ICS whole day events on iPhone Calendar

our sports website provides an ICS calendar for our matches, draws etc.
For retrieving the ICS file we use a PHP script which reads the local ics file and then does some optional filtering on the VEVENT records etc. and returns the ICS data.
I have subscribed this ICS webcal via webcal://.... on my iPhone.
I now have the weird behaviour that SOME whole day events (DURATION:P1D) like this
BEGIN:VEVENT
DTSTART;VALUE=DATE:20120623
DURATION:P1D
TRANSP:TRANSPARENT
SUMMARY:Auslosung: VWM: Super Globe
DESCRIPTION:VWM: Super Globe
UID:20110124#thw-provinzial.de
CATEGORIES:THW-Termin
URL:http://www.thw-provinzial.de/thw/
COMMENT:TYPE=VWM
END:VEVENT
span two days in my iPhone Calendar if I subscribe the PHP script via webcal://www.thw-provinzial.de/thw/ics.php?config=all?.
(It is shown on 20120623 and 20120624)
If I subscribe the ics file directly via http://www.thw-provinzial.de/thw/thwdate2.ics the event is shown correctly only on day 20120623.
If I do a
wget http://www.thw-provinzial.de/thw/thwdate2.ics
wget http://www.thw-provinzial.de/thw/ics.php?config=all
and then diff the two outputs the only difference is the X-WR-CALNAME all other content is identical.
Could it be that some header information in the response is confusing the iPhone?
Response header of the thwdate2.ics -here the behvaiour is fine
HTTP/1.0 200 OK
Date: XXXXXX
Server: Apache
Last-Modified: Wed, 13 Jun 2012 20:05:04 GMT
ETag: "6c6f78d-c54d-4c260194d7c00"
Accept-Ranges: bytes
Content-Length: 50509
Vary: Accept-Encoding,User-Agent
Content-Type: text/calendar
Age: 787
Response header of the ics.php - here we have the problem with spanning over 2 days
HTTP/1.0 200 OK
Date: Thu, XXXXXX
Server: Apache
Content-Disposition: inline; filename=thwdates.ics
Pragma: no-cache
Cache-Control: no-cache, must-revalidate
Expires: Sat, 26 Jul 1997 05:00:00 GMT
Last-Modified: Wed, 13 Jun 2012 20:05:04 GMT
Vary: Accept-Encoding,User-Agent
Content-Type: text/calendar; charset=utf-8
Any ideas?

Sporadic 502 error with gwt rpc calls

I have a GWT application that is giving sporadic 502 errors all of a sudden. I have managed to replicate it by opening multiple connections to the application. Eventually I get a 502 error and the response headers for look as follows:
Server: squid/2.6.STABLE5
Date: Fri, 19 Aug 2011 12:08:03 GMT
Content-Type: text/html
Content-Length: 1014
Expires: Fri, 19 Aug 2011 12:08:03 GMT
X-Squid-Error: ERR_ZERO_SIZE_OBJECT 0
X-Cache: MISS from sentinel.bsgza.bsg.co.za
X-Cache-Lookup: MISS from sentinel.bsgza.bsg.co.za:3128
Via: 1.0 sentinel.bsgza.bsg.co.za:3128 (squid/2.6.STABLE5)
Connection: close
The response headers for the successful rpc calls look like this:
Date: Fri, 19 Aug 2011 13:04:37 GMT
Server: Apache/2.2.14 (Ubuntu)
Content-Encoding: gzip
Content-Disposition: attachment
Content-Length: 249
Content-Type: application/json;charset=utf-8
X-Cache: MISS from sentinel.bsgza.bsg.co.za
X-Cache-Lookup: MISS from sentinel.bsgza.bsg.co.za:3128
Via: 1.0 sentinel.bsgza.bsg.co.za:3128 (squid/2.6.STABLE5)
Connection: keep-alive
We have been able to repeat this on a local server too so it is not a network issue
Try not to route your RPC call via a proxy (Squid). Or at least try to configure Squid to not try to cache them, but only to forward.
Update
It's suggested here that such condition might occur with HTTP POST (used by GWT-RPC) by clients behind PPPoA gateways (cable modems) which have wrong MTU set. Do you see this errors from such clients?