I was trying to change storage class of a set of existing objects (over 300 GBs) as advised in this post:
I tried it on one file first:
fyn#pod-arch:~$ gsutil ls -L gs://some-bucket/sub-dir/audioArch.mp3
gs://some-bucket/sub-dir/audioArch.mp3:
Creation time: Fri, 29 Jul 2016 00:52:51 GMT
Update time: Fri, 05 Aug 2016 15:40:51 GMT
Storage class: DURABLE_REDUCED_AVAILABILITY
Content-Language: en
Content-Length: 43033404
Content-Type: audio/mpeg
... ...
fyn#pod-arch:~$ gsutil -m rewrite -s coldline gs://some-bucket/sub-dir/audioArch.mp3
- [1/1 files][ 41.0 MiB/ 41.0 MiB] 100% Done
Operation completed over 1 objects/41.0 MiB.
fyn#pod-arch:~$ gsutil ls -L gs://some-bucket/sub-dir/audioArch.mp3
gs://some-bucket/sub-dir/audioArch.mp3:
Creation time: Sun, 30 Oct 2016 23:49:34 GMT
Update time: Sun, 30 Oct 2016 23:49:34 GMT
Storage class: COLDLINE
Content-Language: en
Content-Length: 43033404
Content-Type: audio/mpeg
... ...
Then I tried it on 15 more, and then on the rest of the objects in a subdir... Works like a charm ☺, although the operation overwrites the Creation & Update times!
I had two follow-up queries though:
Is gsutil rewrite operation billable?
Can Creation time be preserved?
Many thanks.
Cheers!
fynali
Yes, it is billable as a Class A operation (it uses
storage.objects.rewrite, see cloud.google.com/storage/pricing). No,
there's no way to preserve the creation/update time because rewrite
creates a new object generation.
–Travis Hobrla in comment here
Related
I am working with a MongoDB database running in a replica set.
Unfortunately, I noticed that the data appears to be corrupted.
There should be over 10,000 documents in the database. However, there are several thousand records that are not being returned in queries.
The total count DOES show the correct total.
db.records.find().count()
10793
And some records are returned when querying by RecordID (a custom sequence integer).
db.records.find({"RecordID": 10049})
{ "_id" : ObjectId("5dfbdb35c1c2a400104edece")
However, when querying for a records that I know for a fact should exist, it does not return anything.
db.records.find({"RecordID": 10048})
db.records.find({"RecordID": 10047})
db.records.find({"RecordID": 10046})
The issue appears to be very sporadic, and in some cases entire ranges of records are missing. The entire range from RecordIDs 1500 to 8000 is missing.
Questions: What could be the cause of the issue? What can I do to troubleshoot this issue further and recover the corrupted data? I looked into running repairDatabase but that is for standalone instances only.
UPDATE:
More info on replication:
rs.printReplicationInfo()
configured oplog size: 5100.880859375MB
log length start to end: 14641107secs (4066.97hrs)
oplog first event time: Wed Mar 03 2021 05:21:25 GMT-0500 (EST)
oplog last event time: Thu Aug 19 2021 17:19:52 GMT-0400 (EDT)
now: Thu Aug 19 2021 17:20:01 GMT-0400 (EDT)
rs.printSecondaryReplicationInfo()
source: node2-examplehost.com:27017
syncedTo: Thu Aug 19 2021 17:16:42 GMT-0400 (EDT)
0 secs (0 hrs) behind the primary
source: node3-examplehost.com:27017
syncedTo: Thu Aug 19 2021 17:16:42 GMT-0400 (EDT)
0 secs (0 hrs) behind the primary
UPDATE 2:
We did a restore from a backup and somehow it looks like it fixed the issue.
We did a restore from a backup and somehow it looks like it fixed the issue.
We are using Google Cloud CDN with a backend-bucket. Everything works correctly, we see cache-hits etc. but the cache rate is lower than expected. Analyzing it, i recognized that none of the request has an age above 3600. Although our max-age is set to 86400. Setting it to something smaller works. Is this defined behaviour? Are we setting anything wrong?
Here are the headers for one of the files:
HTTP/2 200
x-guploader-uploadid: AEnB2Ur4sV1ou6Av1U8OgQC8iNxgFmLzAbQ4bFQ4mBAYCyBOHviUAfAbkWFUycAUGLYDYbgNSdaw_zdkE6ySLdRTe0vScOh3Tw
date: Wed, 05 Sep 2018 14:40:29 GMT
expires: Thu, 06 Sep 2018 14:40:29 GMT
last-modified: Thu, 02 Mar 2017 15:31:23 GMT
etag: "1293d749638a24bf786a15f2a2a6ca76"
x-goog-generation: 1488468683233688
x-goog-metageneration: 3
x-goog-stored-content-encoding: identity
x-goog-stored-content-length: 89976
content-type: text/plain
x-goog-hash: crc32c=nIbPdQ==
x-goog-hash: md5=EpPXSWOKJL94ahXyoqbKdg==
x-goog-storage-class: STANDARD
accept-ranges: bytes
content-length: 89976
access-control-allow-origin: *
access-control-expose-headers: x-unity-version
access-control-expose-headers: origin
server: UploadServer
age: 3041
cache-control: public, max-age=86400
alt-svc: clear
According to this Cloud CDN Documentation,
Note that a cache entry's expiration time is an upper bound on how long the cache entry remains valid. There is no guarantee that a cache entry will remain in the cache until it expires. Cache entries for unpopular content can be evicted to make room for new content. Regardless of the specified expiration time, cache entries that aren't accessed for 30 days are automatically removed.
That being said, we were able to reproduce the same behavior. Hence,
in order to confirm if it is indeed an expected behavior or an issue, I have created a new issue on the Google Issue Tracker for your convenience.
I created a pure German example of https://github.com/actions-on-google/dialogflow-transactions-nodejs .
When I trigger the action via phone (or web Simulator) and go into the first step "transaction_check_nopayment" I see in the logs that the method is called but in the client I get as response: "Sorry, I didn't get any response."
What could be the problem? how to get closer to the problem?
Here is the log:
2018-01-08T08:03:20.685Z I transactions: Mon, 08 Jan 2018 08:03:20 GMT actions-on-google:debug checkForTransactionRequirements: transactionConfig=undefined, dialogState=undefined
2018-01-08T08:03:20.685Z I transactions: Mon, 08 Jan 2018 08:03:20 GMT actions-on-google:debug fulfillSystemIntent_: intent=actions.intent.TRANSACTION_REQUIREMENTS_CHECK, specType=type.googleapis.com/google.actions.v2.TransactionRequirementsCheckSpec, intentSpec={}, promptPlaceholder=PLACEHOLDER_FOR_TXN_REQUIREMENTS dialogState=undefined
2018-01-08T08:03:20.685Z I transactions: Mon, 08 Jan 2018 08:03:20 GMT actions-on-google:debug buildResponse_: textToSpeech=PLACEHOLDER_FOR_TXN_REQUIREMENTS, expectUserResponse=true, noInputs=undefined
2018-01-08T08:03:20.686Z I transactions: Mon, 08 Jan 2018 08:03:20 GMT actions-on-google:debug isSsml_: text=PLACEHOLDER_FOR_TXN_REQUIREMENTS
2018-01-08T08:03:20.690Z I transactions: Mon, 08 Jan 2018 08:03:20 GMT actions-on-google:debug getUser
2018-01-08T08:03:20.691Z I transactions: Mon, 08 Jan 2018 08:03:20 GMT actions-on-google:debug doResponse_: response={"speech":"PLACEHOLDER_FOR_TXN_REQUIREMENTS","contextOut":[{"name":"_actions_on_google_","lifespan":100,"parameters":{}}],"data":{"google":{"expectUserResponse":true,"noInputPrompts":[],"isSsml":false,"systemIntent":{"intent":"actions.intent.TRANSACTION_REQUIREMENTS_CHECK","data":{"#type":"type.googleapis.com/google.actions.v2.TransactionRequirementsCheckSpec"}}}}}, responseCode=200
2018-01-08T08:03:20.691Z I transactions: Mon, 08 Jan 2018 08:03:20 GMT actions-on-google:debug isNotApiVersionOne_
2018-01-08T08:03:20.691Z I transactions: Mon, 08 Jan 2018 08:03:20 GMT actions-on-google:debug Response {"speech":"PLACEHOLDER_FOR_TXN_REQUIREMENTS","contextOut":[{"name":"_actions_on_google_","lifespan":100,"parameters":{}}],"data":{"google":{"expectUserResponse":true,"noInputPrompts":[],"isSsml":false,"systemIntent":{"intent":"actions.intent.TRANSACTION_REQUIREMENTS_CHECK","data":{"#type":"type.googleapis.com/google.actions.v2.TransactionRequirementsCheckSpec"}}}}}
2018-01-08T08:03:20.693712230Z D transactions: Function execution took 627 ms, finished with status code: 200
2018-01-08T08:03:20.698Z I transactions: Mon, 08 Jan 2018 08:03:20 GMT actions-on-google:debug undefined
When I compare the logs for the working English version and the not working German version for this step I see a difference:
In the English log every lines ends with "transactions hyd4scvzft31". This part I miss in the German logs.
However in the configuration of my German app I also answered "Does your app perform Transactions?" with YES.
In the example on transaction_check_nopayment app.askForTransactionRequirements() is called. May be askForTransactionRequirements() is not supported in German? Is there an overview of the transaction-API documenting the supported languages? Should the transaction API work for a German use case? As payment method I have invoice. When I understand the documentation this is like having no payment in the google-API.
Hidden under Policies and Terms > Other > Special Requirements for Certain Use Cases, Google states 'Currently, the Transactions API is only available in the United States; apps in other countries cannot implement the API or complete transactions until the API is available there.'
Source
I'm trying to upload a big file (9gb) to google storage using Cyberduck.
The login and transfer with small files work. However for this file I'm getting the following error:
GET / HTTP/1.1
Date: Wed, 30 Apr 2014 08:47:34 GMT
x-goog-project-id: 674064471933
x-goog-api-version: 2
Authorization: OAuth SECRET_KEY
Host: storage.googleapis.com:443
Connection: Keep-Alive
User-Agent: Cyberduck/4.4.4 (Mac OS X/10.9) (x86_64)
HTTP/1.1 200 OK
Content-Type: application/xml; charset=UTF-8
Content-Length: 340
Date: Wed, 30 Apr 2014 08:47:35 GMT
Expires: Wed, 30 Apr 2014 08:47:35 GMT
Cache-Control: private, max-age=0
Server: HTTP Upload Server Built on Apr 16 2014 16:50:43 (1397692243)
Alternate-Protocol: 443:quic
GET /vibetracestorage/?prefix=eventsall.csv&uploads HTTP/1.1
Date: Wed, 30 Apr 2014 08:47:35 GMT
x-goog-api-version: 2
Authorization: OAuth SECRET_KEY
Host: storage.googleapis.com:443
Connection: Keep-Alive
User-Agent: Cyberduck/4.4.4 (Mac OS X/10.9) (x86_64)
HTTP/1.1 400 Bad Request
Content-Type: application/xml; charset=UTF-8
Content-Length: 173
Date: Wed, 30 Apr 2014 08:47:36 GMT
Expires: Wed, 30 Apr 2014 08:47:36 GMT
Cache-Control: private, max-age=0
Server: HTTP Upload Server Built on Apr 16 2014 16:50:43 (1397692243)
Alternate-Protocol: 443:quic
Am I missing anything? Thanks.
According to that log you posted, you're placing a GET to "https://storage.googleapis.com/vibetracestorage/?prefix=eventsall.csv&uploads".
I don't know what that "uploads" parameter tacked onto the end is, but it's not a valid parameter for requesting a bucket listing (which is what that request does).
If you place that request by hand, you'll see this error:
<?xml version='1.0' encoding='UTF-8'?><Error><Code>InvalidArgument</Code><Message>Invalid argument.</Message><Details>Invalid query parameter(s): [uploads]</Details></Error>
Also, as a general point of good practice, do not post logs that contain your full Authorization header. That is a very, very bad idea. You may want to delete this question, although those credentials will expire (and perhaps already have).
This is an interoperability issue. In Cyberduck, when connected to S3, multipart uploads are supported as defined by Amazon S3. The request with the uploads parameter is used to find already existing in progress multipart uploads for the same target object that can be resumed.
Make sure to choose Google Storage and not S3 in the protocol dropdown list in the bookmark or connection prompt. Multipart uploads should then be disabled.
I'm hosting a FireFox plugin on Google Cloud Storage. In order to be properly handled by FireFox, the Content-Type needs to be set to application/x-xpinstall
I have uploaded as follows:
gsutil -h "Content-Type: application/x-xpinstall" cp -a public-read \
ActivityInfo.xpi gs://download.activityinfo.org
When accessed from the standard endpoint, everything is correct:
$ curl -s -D - http://commondatastorage.googleapis.com/download.activityinfo.org/ActivityInfo.xpi \
-o /dev/null
HTTP/1.1 200 OK
Server: HTTP Upload Server Built on Feb 13 2013 15:53:33 (1360799613)
Expires: Thu, 28 Feb 2013 12:38:30 GMT
Date: Thu, 28 Feb 2013 11:38:30 GMT
Last-Modified: Thu, 28 Feb 2013 11:38:01 GMT
ETag: "1ee983889c947a204eab4db6902c9a67"
x-goog-generation: 1362051481261000
x-goog-metageneration: 1
Content-Type: application/x-xpinstall
Content-Language: en
x-goog-crc32c: a11b93ab
Accept-Ranges: bytes
Content-Length: 5562
Cache-Control: public, max-age=3600, no-transform
Age: 491
But when I try to access from the custom domain download.activityinfo.org, the header reverts to application/octet-stream
$ curl -s -D - http://download.activityinfo.org/ActivityInfo.xpi -o /dev/null
HTTP/1.1 200 OK
Server: HTTP Upload Server Built on Feb 13 2013 15:53:33 (1360799613)
Expires: Thu, 28 Feb 2013 12:10:24 GMT
Date: Thu, 28 Feb 2013 11:10:24 GMT
Last-Modified: Wed, 27 Feb 2013 20:36:24 GMT
ETag: "1ee983889c947a204eab4db6902c9a67"
x-goog-generation: 1361997384772000
x-goog-metageneration: 2
Content-Type: application/octet-stream
x-goog-crc32c: a11b93ab
Accept-Ranges: bytes
Content-Length: 5562
Cache-Control: public, max-age=3600, no-transform
Age: 2298
I have set the CNAME to c.storage.googleapis.com per the docs
$ nslookup download.activityinfo.org
Non-authoritative answer:
Server: Comtrend.Home
Address: 192.168.1.1
Name: storage.l.googleusercontent.com
Addresses: 2a00:1450:400c:c00::80
173.194.78.128
Aliases: download.activityinfo.org
c.storage.googleapis.com
Is this a bug or do I need to change my configuration?
The two results above have different values in x-goog-generation and x-goog-metageneration, which makes me suspect you have uploaded the object more than once, and you were seeing the results from different versions (which have different values for Content-Type). Do you have versioning enabled for the bucket? If not, then maybe there is some caching going on in one of the paths. Are you still seeing this behavior?