I want to copy files from my bucket but only files/directories that not exist on local drive.
Is it possible?
I tried something like
gsutil -m rsync -n -r "MyBUCKET" "my_local_dir"
but something is wrong.
You should try using the recently-added -i flag, which ignores existing files on the destination. That flag was added in this commit and should be available as of gsutil v4.59.
Related
I'm using gsutil and I need to copy a large number of files/subdirectories from a directory on a windows server to a Google Cloud Storage Bucket.
I have checked the documentation but somehow I can't seem to get the syntax right - I'm trying something along these lines:
c:\test>gsutil -m cp -r . gs://mytestbucket
But I keep getting the message:
CommandException: No URLs matched: .
What am I doing wrong here?
Regards
Morten Hjorth Nielsen
Try gsutil -m cp -r * gs://mytestbucket
Or gsutil -m cp -r *.* gs://mytestbucket
Or if your local directory is called test go one dir up and type: gsutil -m cp -r test gs://mytestbucket
Not sure which syntax you need on Windows, but probably the first.
when using gsutil -m rsync -p -d -r
the ownership became root
Any idea how to run gsutil rsync just like rsync -a?
thanks
Peter
gsutil rsync doesn't currently support preserving POSIX file attributes in the cloud.
It's not guaranteed that the uid/gid on the system that uploaded a file is even valid on the system that downloaded the file. So (at least for now), you'll need to manage your file permissions manually.
I'm hosting publicly available static resources in a google storage bucket, and I want to use the gsutil rsync command to sync our local version to the bucket, saving bandwidth and time. Part of our build process is to pre-gzip these resources, but gsutil rsync has no way to set the Content-Encoding header. This means we must run gsutil rsync, then immediately run gsutil setmeta to set headers on all the of gzipped file types. This leaves the bucket in a BAD state until that header is set. Another option is to use gsutil cp, passing the -z option, but this requires us to re-upload the entire directory structure every time, and this includes a LOT of image files and other non-gzipped resources that wastes time and bandwidth.
Is there an atomic way to accomplish the rsync and set proper Content-Encoding headers?
Assuming you're starting with gzipped source files in source-dir you can do:
gsutil -h content-encoding:gzip rsync -r source-dir gs://your-bucket
Note: If you do this and then run rsync in the reverse direction it will decompress and copy all the objects back down:
gsutil rsync -r gs://your-bucket source-dir
which may not be what you want to happen. Basically, the safest way to use rsync is to simply synchronize objects as-is between source and destination, and not try to set content encodings on the objects.
I'm not completely answering the question but I came here as I was wondering the same thing trying to achieve the following:
how to deploy efficiently a static website to google cloud storage
I was able to find an optimized way for deploying my static web site from a local folder to a gs bucket
Split my local folder into 2 folders with the same hierarchy, one containing the content to be gzip (html,css,js...), the other the other files
Gzip each file in my gzip folder (in place)
Call gsutil rsync in for each folder to the same gs destination
Of course, it is only a one way synchronization and deleted local files are not deleted remotely
For the gzip folder the command is
gsutil -m -h Content-Encoding:gzip rsync -c -r src/gzip gs://dst
forcing the content encoding to be gzippped
For the other folder the command is
gsutil -m rsync -c -r src/none gs://dst
the -m option is used for parallel optimization. The -c option is needed to force using checksum validation (Why is gsutil rsync re-downloading all our files?) as I was touching each local file in my build process. the -r option is used for recursivity.
I even wrote a script for it (in dart): http://tekhoow.blogspot.fr/2016/10/deploying-static-website-efficiently-on.html
Using this command from SSH I can upload a whole folder into Google Cloud Storage:
gsutil cp -R folder_big gs://bucket_name
Those are files inside the folder:
I don't want to click individually on each file to make it public.
How do I make the folder (and all files inside) automatically public on upload?
You could do:
gsutil cp -a public-read -R folder_big gs://bucket_name
Note: if it's a large folder you would likely get a substantial performance improvement if you use the multi-threading option:
gsutil -m cp -a public-read -R folder_big gs://bucket_name
When I do rsync this is my command:
rsync -a source dest
I am using dest as my web root /var/www/
so some folder which are set to chmod 777 were no longer with 777 permission.
does rsync change folder permission as well?
What is best way to sync two local folders in same server.? Will rsync delete any changes done in destination and use the source files?
The manual page for rsync says this:
-a, --archive archive mode; equals -rlptgoD (no -H,-A,-X)
Among those options is -p, about which it says:
-p, --perms preserve permissions
So, yes, rsync is making the permissions on dest match those on source in this case. If that is not desired, then read the manual page and decide what options are more appropriate to your need than rsync -a, and use those instead. In the simplest case, add the --no-perms flag after -a to disable the permission preservation.