I am trying to copy all files from cloud storage bucket recursively and I am having problem with the -m flag as I have investigated.
The command that I am running
gsutil -m cp -r gs://{{ src_bucket }} {{ bucket_backup }}
I am getting something like this:
CommandException: 1 file/object could not be transferred.
where the number of files/objects differs every time.
After investigation I have tried to reduce number of threads/processes which used with the -m option, but this has not helped, so I am looking for some advice about this. I have 170 MiB data on the bucket which is approximately 300k files. I need to download them as fast as possible
UPD:
Logs with -L flag
[Errno 2] No such file or directory: '<path>/en_.gstmp' -> '<path>/en'
6 errors like that.
The root of the issue might be that both directory and file of the same name exist in the GCS bucket. Try executing the command with -L flag, so you will get additional logs on the execution and you will be able to find the file that is causing this error.
I would suggest you delete that file and make sure there is no directory in the bucket of that name and then upload this file to the bucket again.
Also check if any of the directory created with Jar name. Delete them and processed the copy files.
And check if the required file is already at destination and delete the file at destination and execute copy again.
There are alternatives to copy, for example, it is possible to transfer files using rsync, as described here.
You can also check similar threads: thread1 , thread2 & thread3
I'm unable to use the rm command to remove files remotely from another directory. I'm a beginner so I apologise for my inability to elaborate properly.
Here's what I'm trying to do:
I'm trying to delete all .srt files from a sub directory. It works when I cd into the specific directory like so:
Command 1:
cd /users/jakubdonovan/library/cloudstorage/iCloud\ drive/the-modern-python3-bootcamp/target_folder
Command 2:
rm *.srt
However, let's say I want to quickly delete a specific file type from a folder without first using the "cd" command, like so:
rm *.srt /users/jakubdonovan/library/cloudstorage/iCloud\ drive/the-modern-python3-bootcamp/target_folder
It returns with "No matches for wildcard '*.srt'. See help expand."
Which is strange because I can use the touch, cp and and all the other commands remotely without a problem.
Is there a way to make the command "rm *.filetype" remove all the files with that specific filetype from a folder and all its subfolders in one swoop?
If you would like to rm in a sub-directory you just have to specify that sub-directory in the command.
rm /path/to/folder/*.filetype
or if you know that the folder is inside your current directory you can try...
rm ./folder/*.filetype
Using puttys pscp -r folder\to\copy\* user#server:/path/to/copy/folder/to it only copies the content of path\to\copy\folder\* and does not include the "main" folder which the subfiles and subdirectories are in.
What I need is that the folder itself is also copied such that I get a folder with the same name as the one I copied with the content inside.
I know I just can create a parent-folder for the one I want to copy and parse that as the path\to\copy\folder\* but that is not the case
Just use pscp -r folder\to\copy user#server:/path/to/copy/folder/to.
Lets say i have this dirs and files:
Dir1/File1 - Local on my ansible host
Dir1/SubDir1/File1
Dir2/SubDir2/ - Remote Server
Dir3/SubDir3/File1
...
How is in ansible the right way to copy the local file to all remote subdirs and to specify the remote dirs as wildcard.
Before copying it should do a backup of the existing files in their dirs.
As result i want to find File1 in all SubDirs.
Regards
To create the backup of the remote file before you replace the file , you can use the 'backup' parameter in copy module. And for using wildcards you can use the fileglob parameter.
You can refer the following links.
http://docs.ansible.com/ansible/copy_module.html
http://docs.ansible.com/ansible/playbooks_loops.html#id17
I am new to Capistranoand I saw there is shared folder and also option :linked_files. I think shared folder is used to keep files between releases. But my question is, how do files end up being in the shared folder?
Also, if I want to symlink another directory to the current directory e.g. static folder at some path, how do I put it at the linked_dirs ?
Lastly how to set chmod 755 to linked_files and linked_dirs.
Thank you.
Folders inside your app are symlinks to folders in the shared directory. If your app writes to log/production.log, it will actually write to ../shared/log/production.log. That's how the files end up being in the shared folder.
You can see how this works by looking at the feature specs or tests in Capistrano.
If you want to chmod these shared files, you can just do it once directly over ssh since they won't ever be modified by Capistrano after they've been created.
To add a linked directory, in your deploy.rb:
set :linked_dirs, %w{bin log tmp/backup tmp/pids tmp/cache tmp/sockets vendor/bundle}
or
set :linked_dirs, fetch(:linked_dirs) + %w{public/system}
Capistrano 3.5+
Capistrano 3.5 introduced append for array fields. From the official docs, you should use these:
For Shared Files:
append :linked_files, %w{config/database.yml}
For Shared Directories:
append :linked_dirs, %w{bin log public/uploads vendor/bundle}
I've written a task for Capistrano 3 to upload your config files to the shared folder of each of your servers, it'll check these directories in order:
config/deploy/config/:stage/*.yml
config/deploy/config/*.yml
And upload all config files found. It'll only upload the files if they've changed. Note also that if you have the same file on both directories then the second one will be ignored.
Here's the code: https://gist.github.com/Jesus/448d618c83fb0445ebbf
One last thing, this task is just uploading the config. files to your remote shared folder, you still need to set linked_files in config/deploy.rb, eg:
set :linked_files, %w{config/database.yml config/aws.yml}
UPDATE:
If you're using Git, you'll probably want to ignore these files:
echo "config/deploy/config/*" >> .gitignore
There are 3 simple steps you can follow to put a file that you don't want to change in consecutive releases; add your file to linked_files list.
set :linked_files, fetch(:linked_files, []).push('config.php')
Select all the files that you want to share. Put this file from your local to remote server through scp
scp config.php deployer#amazon:~/capistrano/shared/config.php
Now, deploy through the command given below:
bundle exec cap staging deploy
of course, staging can be changed as per requirements may be production,sandbox etc.
One more thing, because you don't want your team members to commit such files. So, put this file to your .gitignore file. And push it to git remote repo.
For Capistrano 3.5+, as specified in official doc :
append :linked_dirs, ".bundle", "tmp"
For me non of the above worked so I ended up adding two functions to the end of the deployment process:
namespace :your_company do
desc "remove index.php"
task :rm_files do
on roles(:all) do
execute "rm -rf #{release_path}/index.php"
end
end
end
namespace :your_company do
desc "add symlink to index.php"
task :add_files do
on roles(:all) do
execute "ln -sf #{shared_path }/index.php #{release_path}/index.php"
end
end
end
after "deploy:finished", "your_company:rm_files"
after "deploy:finished", "your_company:add_files"