Using this command from SSH I can upload a whole folder into Google Cloud Storage:
gsutil cp -R folder_big gs://bucket_name
Those are files inside the folder:
I don't want to click individually on each file to make it public.
How do I make the folder (and all files inside) automatically public on upload?
You could do:
gsutil cp -a public-read -R folder_big gs://bucket_name
Note: if it's a large folder you would likely get a substantial performance improvement if you use the multi-threading option:
gsutil -m cp -a public-read -R folder_big gs://bucket_name
Related
I have a bunch of already gzipped files in GCS that I'd like to download but keep compressed. When I try to download the files running command: gsutil -m cp -r gs://my-bucket-name/path/to/dir/, it downloads the files then immediately unzips them.
The files appear to have Content-Encoding:gzip in their metadata, and gsutil cp seems to have the default behavior that files with this encoding will automatically decompress when served.
How can I just download the files as-is without it being automatically decompressed?
You can use the option Cache-Control: no-transform as indicated here.
As an example:
gsutil -m -h "Cache-Control: no-transform" cp -r gs://YOUR-BUCKET/ .
I'm using gsutil and I need to copy a large number of files/subdirectories from a directory on a windows server to a Google Cloud Storage Bucket.
I have checked the documentation but somehow I can't seem to get the syntax right - I'm trying something along these lines:
c:\test>gsutil -m cp -r . gs://mytestbucket
But I keep getting the message:
CommandException: No URLs matched: .
What am I doing wrong here?
Regards
Morten Hjorth Nielsen
Try gsutil -m cp -r * gs://mytestbucket
Or gsutil -m cp -r *.* gs://mytestbucket
Or if your local directory is called test go one dir up and type: gsutil -m cp -r test gs://mytestbucket
Not sure which syntax you need on Windows, but probably the first.
Scenario: there are multiple folders and many files stored in storage bucket that is accessible by project team members. Instead of downloading individual files one at a time (which is very slow and time consuming), is there a way to download entire folders? Or at least multiple files at once? Is this possible without having to use one of the command consoles? Some of the team members are not tech savvy and need to access these files as simple as possible. Thank you for any help!
I would suggest downloading the files with gsutil. However if you have a large number of files to transfer you might want to use the gsutil -m option, to perform a parallel (multi-threaded/multi-processing) copy:
gsutil -m cp -R gs://your-bucket .
The time reduction for downloading the files can be quite significant. See this Cloud Storage documentation for complete information on the GCS cp command.
If you want to copy into a particular directory, note that the directory must exist first, as gsutils won't create it automatically. (e.g: mkdir my-bucket-local-copy && gsutil -m cp -r gs://your-bucket my-bucket-local-copy)
I recommend they use gsutil. GCS's API deals with only one object at a time. However, its command-line utility, gsutil, is more than happy to download a bunch of objects in parallel, though. Downloading an entire GCS "folder" with gsutil is pretty simple:
$> gsutil cp -r gs://my-bucket/remoteDirectory localDirectory
To download files to local machine need to:
install gsutil to local machine
run Google Cloud SDK Shell
run the command like this (example, for Windows-platform):
gsutil -m cp -r gs://source_folder_path "%userprofile%/Downloads"
gsutil rsync -d -r gs://bucketName .
works for me
when using gsutil -m rsync -p -d -r
the ownership became root
Any idea how to run gsutil rsync just like rsync -a?
thanks
Peter
gsutil rsync doesn't currently support preserving POSIX file attributes in the cloud.
It's not guaranteed that the uid/gid on the system that uploaded a file is even valid on the system that downloaded the file. So (at least for now), you'll need to manage your file permissions manually.
Is there a command using GSUTIL that will allow me to share publicly everything in a specific Bucket? Right now, I'm forced to go through and check "share publicly" individually on EVERY SINGLE FILE in the console.
The best way to do this is:
gsutil -m acl ch -u 'AllUsers:R' gs://your-bucket/**
will update ACLs for each existing object in the bucket.
If you want newly created objects in this bucket to also be public, you should also run:
gsutil defacl ch -u 'AllUsers:R' gs://your-bucket
This question was also asked here but the answer recommends using acl set public-read which has the downside of potentially altering your existing ACLs.
$> gsutil acl ch -g All:R -r gs://bucketName
gsutil is the command-line utility for GCS.
"acl ch" means "Modify an ACL."
"-g All:R" means "include read permissions for all users."
"-r" means "recursively"
and the rest is the path.
If you have a whole lot of files and you want MORE SPEED, you can use -m to mean "and also do this multithreaded!", like so:
$> gsutil -m acl ch -g All:R -r gs://bucketName