Capistrano - How to put files in the shared folder? - capistrano

I am new to Capistranoand I saw there is shared folder and also option :linked_files. I think shared folder is used to keep files between releases. But my question is, how do files end up being in the shared folder?
Also, if I want to symlink another directory to the current directory e.g. static folder at some path, how do I put it at the linked_dirs ?
Lastly how to set chmod 755 to linked_files and linked_dirs.
Thank you.

Folders inside your app are symlinks to folders in the shared directory. If your app writes to log/production.log, it will actually write to ../shared/log/production.log. That's how the files end up being in the shared folder.
You can see how this works by looking at the feature specs or tests in Capistrano.
If you want to chmod these shared files, you can just do it once directly over ssh since they won't ever be modified by Capistrano after they've been created.
To add a linked directory, in your deploy.rb:
set :linked_dirs, %w{bin log tmp/backup tmp/pids tmp/cache tmp/sockets vendor/bundle}
or
set :linked_dirs, fetch(:linked_dirs) + %w{public/system}

Capistrano 3.5+
Capistrano 3.5 introduced append for array fields. From the official docs, you should use these:
For Shared Files:
append :linked_files, %w{config/database.yml}
For Shared Directories:
append :linked_dirs, %w{bin log public/uploads vendor/bundle}

I've written a task for Capistrano 3 to upload your config files to the shared folder of each of your servers, it'll check these directories in order:
config/deploy/config/:stage/*.yml
config/deploy/config/*.yml
And upload all config files found. It'll only upload the files if they've changed. Note also that if you have the same file on both directories then the second one will be ignored.
Here's the code: https://gist.github.com/Jesus/448d618c83fb0445ebbf
One last thing, this task is just uploading the config. files to your remote shared folder, you still need to set linked_files in config/deploy.rb, eg:
set :linked_files, %w{config/database.yml config/aws.yml}
UPDATE:
If you're using Git, you'll probably want to ignore these files:
echo "config/deploy/config/*" >> .gitignore

There are 3 simple steps you can follow to put a file that you don't want to change in consecutive releases; add your file to linked_files list.
set :linked_files, fetch(:linked_files, []).push('config.php')
Select all the files that you want to share. Put this file from your local to remote server through scp
scp config.php deployer#amazon:~/capistrano/shared/config.php
Now, deploy through the command given below:
bundle exec cap staging deploy
of course, staging can be changed as per requirements may be production,sandbox etc.
One more thing, because you don't want your team members to commit such files. So, put this file to your .gitignore file. And push it to git remote repo.

For Capistrano 3.5+, as specified in official doc :
append :linked_dirs, ".bundle", "tmp"

For me non of the above worked so I ended up adding two functions to the end of the deployment process:
namespace :your_company do
desc "remove index.php"
task :rm_files do
on roles(:all) do
execute "rm -rf #{release_path}/index.php"
end
end
end
namespace :your_company do
desc "add symlink to index.php"
task :add_files do
on roles(:all) do
execute "ln -sf #{shared_path }/index.php #{release_path}/index.php"
end
end
end
after "deploy:finished", "your_company:rm_files"
after "deploy:finished", "your_company:add_files"

Related

How to download all bucket files. (The issue with the -m flag gsutil)

I am trying to copy all files from cloud storage bucket recursively and I am having problem with the -m flag as I have investigated.
The command that I am running
gsutil -m cp -r gs://{{ src_bucket }} {{ bucket_backup }}
I am getting something like this:
CommandException: 1 file/object could not be transferred.
where the number of files/objects differs every time.
After investigation I have tried to reduce number of threads/processes which used with the -m option, but this has not helped, so I am looking for some advice about this. I have 170 MiB data on the bucket which is approximately 300k files. I need to download them as fast as possible
UPD:
Logs with -L flag
[Errno 2] No such file or directory: '<path>/en_.gstmp' -> '<path>/en'
6 errors like that.
The root of the issue might be that both directory and file of the same name exist in the GCS bucket. Try executing the command with -L flag, so you will get additional logs on the execution and you will be able to find the file that is causing this error.
I would suggest you delete that file and make sure there is no directory in the bucket of that name and then upload this file to the bucket again.
Also check if any of the directory created with Jar name. Delete them and processed the copy files.
And check if the required file is already at destination and delete the file at destination and execute copy again.
There are alternatives to copy, for example, it is possible to transfer files using rsync, as described here.
You can also check similar threads: thread1 , thread2 & thread3

Transfer folder using pscp (Putty) from Windows to Linux

Using puttys pscp -r folder\to\copy\* user#server:/path/to/copy/folder/to it only copies the content of path\to\copy\folder\* and does not include the "main" folder which the subfiles and subdirectories are in.
What I need is that the folder itself is also copied such that I get a folder with the same name as the one I copied with the content inside.
I know I just can create a parent-folder for the one I want to copy and parse that as the path\to\copy\folder\* but that is not the case
Just use pscp -r folder\to\copy user#server:/path/to/copy/folder/to.

Prevent downtime using lftp mirror

I'm using lftp to deploy a website via Travis CI. There is a build process before the deployment, for that reason a build directory is present and pushed to the root of the ftp server.
lftp $FTP_URL -e "glob -d mirror build . --reverse --delete-first --parallel=10 && exit"
It works quite well, but I dislike to have a downtime / temporary PHP parse errors because of missing files on my website. What is the best way to work arround that issue?
My first approach was an option to set a temporary directory, but the lftp man page says there is only a options for temporary files. I still tried the option but it didn't help.
My second approach was to use "mirror build temp" to use a temporary folder and then replace the root with it. The problem here is, that I cannot exclude the temp folder while deleting the old files and folders like rm -rf *.
For small changes not involving adding/removing php files set xfer:use-temp-file should be sufficient. Also don't use --remove-first, as it causes lftp to delete obsolete files before uploading.
For larger changes I'd create a separate directory for each version of the site and redirect the web server to the directory using .htaccess mod_rewrite or some other configuration file. This technique will allow atomic switch to the new version (and back if needed). Besides, you will be able to do final pre-production testing of the new version if you redirect to the new version conditionally based on your IP address or using some other rule.
If you don't want to re-upload whole site for each new version and the FTP server supports FXP with itself, then you can copy old version to a new directory using mirror old_directory ftp://user#example.com/new_directory, then update the new directory using mirror -eR local_dir new_directory.
This is a zero downtown pattern - each placeholder should be replaced:
lftp $FTP_URL -e "mirror {SOURCE} {TARGET}-new-{TIMESTAMP} --reverse --delete-first;
mv {TARGET} {TARGET}-old-{TIMESTAMP};
mv {TARGET}-new-{TIMESTAMP} {TARGET};
rm -rf {TARGET}-old-{TIMESTAMP};
exit"

Capistrano v3 not able to cleanup old releases

Since I'm running my rails app as root, it creates files that are owned by root in the tmp directory. Because of this
cap production deploy:cleanup
can't remove old releases because it is not run as root.
I've looked at the capistrano v3 code, but I don't see a way to run the cleanup command as root. Is this option missing or is this problem occurring because I'm doing something wrong in another place of the deployment flow.
I start the app as root because I need to bind to port 80.
What you can also do is triggering a task just before cleaning up the old release :
namespace :deploy do
before :cleanup, :cleanup_permissions
desc 'Set permissions on old releases before cleanup'
task :cleanup_permissions do
on release_roles :all do |host|
releases = capture(:ls, '-x', releases_path).split
if releases.count >= fetch(:keep_releases)
info "Cleaning permissions on old releases"
directories = (releases - releases.last(1))
if directories.any?
directories.each do |release|
within releases_path.join(release) do
execute :sudo, :chown, '-R', 'deployuser', 'path/to/your/files/writtend/by/root'
end
end
else
info t(:no_old_releases, host: host.to_s, keep_releases: fetch(:keep_releases))
end
end
end
end
end
Note that you'll need to give your deployment user the right to execute this specific sudo command (with a sudoers definition file.
I've looked at the capistrano v3 code, but I don't see a way to run the cleanup command as root. Is this option missing or is this problem occurring because I'm doing something wrong in another place of the deployment flow.
There is no secret sauce in Capistrano, we rely on you having correctly set up the permissions for your deploy user as documented at http://www.capistranorb.com/
Removing directories requires write permissions on the parent directory, that is to say, given the following directory structure:
/var/www/releases/
\- 20131015180000
\- 20131015181500
\- 20131015183000
You need write permission on the /var/www/releases/ directory, as the list of files and directory in that directory, is stored in the directory.
From a similar StackSverflow question:
In UNIX and Linux, the ability to remove a file is not determined by the access bits of that file. It is determined by the access bits of the directory which contains the file.
From the Wikipedia article on Unix File Permissions:
The write permission grants the ability to modify a file. When set for a directory, this permission grants the ability to modify entries in the directory. This includes creating files, deleting files, and renaming files.
One of the things you may want to do is to create a group called app or web on your linux box and add root and the deploy user to the same group. Then, as part of your deployment, chmod the release_path permissions to g+s which will ensure that any new files created by root user are group writable.
You should then be able to remove the old folders as deploy user.
I was running into similar issues, so, to confirm, logged into my Web server via SSH, and tried rm -rf [directory], which also failed due to the same permissions issues, even logged in as admin. Running chmod -R 755 [directory]/, then rm -rf [directory]/ did work, though.
To fix it, in the project's silverstripe.rake file, I changed the command being run from:
execute :chown, "-R [user]:[group] /path/to/project"
to:
execute :chmod, "-R 755 /path/to/project"
So far, no more issues with deleting the oldest release when running cap [release name] deploy

How do I change the temporary directory that Capistrano uses?

How would I change the temporary dir that Capistrano uses?
Example: Instead of /tmp, I want to use /home/user/tmp
My current VPS has /tmp mounted as noexec, which gives me permission denied errors while trying to run cap production deploy.
In Capistrano 3,
set :tmp_dir, '/home/user/tmp'
Are you talking about the remote tmp directory? If yes, here an example::
set :copy_remote_dir, deploy_to
This will change the default tmp directory where the archive was copied on the remote server to the deployment directory instead.
For the ones who are still using Capistrano 2, tmp_dir does not exist. However you can use copy_dir instead:
set :copy_dir, '/home/user/tmp'
Link to the source code: https://github.com/capistrano/capistrano/blob/legacy-v2/lib/capistrano/recipes/deploy/strategy/copy.rb#L275