Since I'm running my rails app as root, it creates files that are owned by root in the tmp directory. Because of this
cap production deploy:cleanup
can't remove old releases because it is not run as root.
I've looked at the capistrano v3 code, but I don't see a way to run the cleanup command as root. Is this option missing or is this problem occurring because I'm doing something wrong in another place of the deployment flow.
I start the app as root because I need to bind to port 80.
What you can also do is triggering a task just before cleaning up the old release :
namespace :deploy do
before :cleanup, :cleanup_permissions
desc 'Set permissions on old releases before cleanup'
task :cleanup_permissions do
on release_roles :all do |host|
releases = capture(:ls, '-x', releases_path).split
if releases.count >= fetch(:keep_releases)
info "Cleaning permissions on old releases"
directories = (releases - releases.last(1))
if directories.any?
directories.each do |release|
within releases_path.join(release) do
execute :sudo, :chown, '-R', 'deployuser', 'path/to/your/files/writtend/by/root'
end
end
else
info t(:no_old_releases, host: host.to_s, keep_releases: fetch(:keep_releases))
end
end
end
end
end
Note that you'll need to give your deployment user the right to execute this specific sudo command (with a sudoers definition file.
I've looked at the capistrano v3 code, but I don't see a way to run the cleanup command as root. Is this option missing or is this problem occurring because I'm doing something wrong in another place of the deployment flow.
There is no secret sauce in Capistrano, we rely on you having correctly set up the permissions for your deploy user as documented at http://www.capistranorb.com/
Removing directories requires write permissions on the parent directory, that is to say, given the following directory structure:
/var/www/releases/
\- 20131015180000
\- 20131015181500
\- 20131015183000
You need write permission on the /var/www/releases/ directory, as the list of files and directory in that directory, is stored in the directory.
From a similar StackSverflow question:
In UNIX and Linux, the ability to remove a file is not determined by the access bits of that file. It is determined by the access bits of the directory which contains the file.
From the Wikipedia article on Unix File Permissions:
The write permission grants the ability to modify a file. When set for a directory, this permission grants the ability to modify entries in the directory. This includes creating files, deleting files, and renaming files.
One of the things you may want to do is to create a group called app or web on your linux box and add root and the deploy user to the same group. Then, as part of your deployment, chmod the release_path permissions to g+s which will ensure that any new files created by root user are group writable.
You should then be able to remove the old folders as deploy user.
I was running into similar issues, so, to confirm, logged into my Web server via SSH, and tried rm -rf [directory], which also failed due to the same permissions issues, even logged in as admin. Running chmod -R 755 [directory]/, then rm -rf [directory]/ did work, though.
To fix it, in the project's silverstripe.rake file, I changed the command being run from:
execute :chown, "-R [user]:[group] /path/to/project"
to:
execute :chmod, "-R 755 /path/to/project"
So far, no more issues with deleting the oldest release when running cap [release name] deploy
Related
I was trying to export files along with their revision history inside my depot folder from 2015.2 to 2019 perforce server.Also , I would want perforce to create new user on my new server corresponding to the commiter/submitter on my original 2015 repo.
Perforce replicate looked like overkill for my current task and then I came across this read on perforce's website that mentioned P4 zip.
This looked like it will solve my problem, but the article has a few issues I could not understand.
Let's say I am moving data from server1_ip:port --> server2_ip:port
I am currently following these steps
Making zip of folder to be copied using
p4 remote my_remote_spec , setting
Address: server1_ip:port
DepotMap://depot/... //depot2/...
p4 -p server1_ip:port zip -o test.zip -r my_remote_spec -A //depot/.... But on this step I get permission denied error. This is weird to me because the user although not super/admin has access to files i ask to get zipped.
Also, when i did try with a super user, i could not find test.zip even though i was not prompted any errors.
Isn't the above command supposed to generate a zip file inside the directory which i run it from?
Is the unzip command supposed to be run after a p4 login from user of second server?
Lastly, from the document why is a third port , 1667 mentioned in the transfer of files from server running on 1666 and 1777.
on this step I get permission denied error. This is weird to me because the user although not super/admin has access to files i ask to get zipped.
This is expected:
C:\Perforce\test>p4 help zip
zip -- Package a set of files and their history for use by p4 unzip
...
The zip command requires super permission granted by p4 protect.
Isn't the above command supposed to generate a zip file inside the directory which i run it from?
Similar to p4 admin checkpoint, the zip file is written to the server machine (relative to the server root, if you don't specify an absolute path), rather than being transferred to the local client directory. This is not explicitly stated in the documentation (which seems like an oversight), but if you look in the root directory of the server where you ran the zip, you should find your test.zip there.
Is the unzip command supposed to be run after a p4 login from user of second server?
Yes, any time you run a command against a particular server, you will need to be logged in to that server. In the case of p4 unzip you will need at least admin permission on the second server.
Lastly, from the document why is a third port , 1667 mentioned in the transfer of files from server running on 1666 and 1777.
I'm pretty sure that's a typo; whoever wrote the article started off using ports 1666 and 1777, changed their mind halfway through, and didn't proofread. :)
I'm using lftp to deploy a website via Travis CI. There is a build process before the deployment, for that reason a build directory is present and pushed to the root of the ftp server.
lftp $FTP_URL -e "glob -d mirror build . --reverse --delete-first --parallel=10 && exit"
It works quite well, but I dislike to have a downtime / temporary PHP parse errors because of missing files on my website. What is the best way to work arround that issue?
My first approach was an option to set a temporary directory, but the lftp man page says there is only a options for temporary files. I still tried the option but it didn't help.
My second approach was to use "mirror build temp" to use a temporary folder and then replace the root with it. The problem here is, that I cannot exclude the temp folder while deleting the old files and folders like rm -rf *.
For small changes not involving adding/removing php files set xfer:use-temp-file should be sufficient. Also don't use --remove-first, as it causes lftp to delete obsolete files before uploading.
For larger changes I'd create a separate directory for each version of the site and redirect the web server to the directory using .htaccess mod_rewrite or some other configuration file. This technique will allow atomic switch to the new version (and back if needed). Besides, you will be able to do final pre-production testing of the new version if you redirect to the new version conditionally based on your IP address or using some other rule.
If you don't want to re-upload whole site for each new version and the FTP server supports FXP with itself, then you can copy old version to a new directory using mirror old_directory ftp://user#example.com/new_directory, then update the new directory using mirror -eR local_dir new_directory.
This is a zero downtown pattern - each placeholder should be replaced:
lftp $FTP_URL -e "mirror {SOURCE} {TARGET}-new-{TIMESTAMP} --reverse --delete-first;
mv {TARGET} {TARGET}-old-{TIMESTAMP};
mv {TARGET}-new-{TIMESTAMP} {TARGET};
rm -rf {TARGET}-old-{TIMESTAMP};
exit"
On CentOS I would like to give the apache user permissions to "ant release" on a home dir it does not own how do I do that? the ant release I am using is as part of the android SDK - I have a dir /home/myuser/android_project/ and ant relase runs fine from there but I would like to give apache the permissions it needs to run it so I can run as as
<?php shell_exec('/home/myuser/android_project/ant release') ?>.
The gotcha
Also there is an issue since I sign the ant release I would like to have the password handled perhaps in a file that php can somehow magically "sign" the ant release.
Now.
Note: to Mr Tinker: Hold the horses - I know that this is might fall foul of the forum topic police, but in my considered opinion it is a unix issue. i.e. I know how PHP does shell_exec I need no programming help. I know how to run ant release manually so I need no installation help: I would like to sew together these two disparate manual "things" within linux (the CentOS server) so I believe 100% this is a unix issue
As you've already stated, you need to give the apache user permission to execute the /home/myuser/android_project/ant file.
tl;dr : run the following command (be warned, it might not be the most secure thing in the world):
chmod 777 /home/myuser/android_project/ant
If you're interested in why this might fix your problem, continue to read below.
First, you need to get some more information.
Run the following command:
ls -l /home/myuser/android_project/ant
The ls -l command will give you the read, write, and execute permissions for the specified file, along with the ownership information. The first column contains the permission information. The 3rd column indicates the owning user, and the 4th column indicates the owning group.
For example:
$ ls -l /etc/passwd
-rw-r--r--. 1 root root 2177 Aug 26 21:23 /etc/passwd
^^^
|----------- All Users & Groups
^^^
|-------------- Specified Group Owner
^^^
|------------------ Specified User Owner
This can be interpreted as user root and group root owning the /etc/passwd file.
The permissions are read as groups of 3 rwx characters. The first group is for owning user, the 2nd for owning group, and the 3rd for everyone else on the system. The permissions in this example mean that the root user can read and write to the file, the root group can read, and everyone else can read.
Now, each group of permissions can be represented as an octal digit:
--- == 0
--x == 1
-w- == 2
-wx == 3
r-- == 4
r-x == 5
rw- == 6
rwx == 7
You now have enough information to understand why the chmod 777 command above worked. Basically you will have given everyone on the system permission to read, write, and execute that ant file.
Ideally, you would only give the minimum permissions required to allow apache to execute the file, I'll leave that much as an exercise to the reader.
I am new to Capistranoand I saw there is shared folder and also option :linked_files. I think shared folder is used to keep files between releases. But my question is, how do files end up being in the shared folder?
Also, if I want to symlink another directory to the current directory e.g. static folder at some path, how do I put it at the linked_dirs ?
Lastly how to set chmod 755 to linked_files and linked_dirs.
Thank you.
Folders inside your app are symlinks to folders in the shared directory. If your app writes to log/production.log, it will actually write to ../shared/log/production.log. That's how the files end up being in the shared folder.
You can see how this works by looking at the feature specs or tests in Capistrano.
If you want to chmod these shared files, you can just do it once directly over ssh since they won't ever be modified by Capistrano after they've been created.
To add a linked directory, in your deploy.rb:
set :linked_dirs, %w{bin log tmp/backup tmp/pids tmp/cache tmp/sockets vendor/bundle}
or
set :linked_dirs, fetch(:linked_dirs) + %w{public/system}
Capistrano 3.5+
Capistrano 3.5 introduced append for array fields. From the official docs, you should use these:
For Shared Files:
append :linked_files, %w{config/database.yml}
For Shared Directories:
append :linked_dirs, %w{bin log public/uploads vendor/bundle}
I've written a task for Capistrano 3 to upload your config files to the shared folder of each of your servers, it'll check these directories in order:
config/deploy/config/:stage/*.yml
config/deploy/config/*.yml
And upload all config files found. It'll only upload the files if they've changed. Note also that if you have the same file on both directories then the second one will be ignored.
Here's the code: https://gist.github.com/Jesus/448d618c83fb0445ebbf
One last thing, this task is just uploading the config. files to your remote shared folder, you still need to set linked_files in config/deploy.rb, eg:
set :linked_files, %w{config/database.yml config/aws.yml}
UPDATE:
If you're using Git, you'll probably want to ignore these files:
echo "config/deploy/config/*" >> .gitignore
There are 3 simple steps you can follow to put a file that you don't want to change in consecutive releases; add your file to linked_files list.
set :linked_files, fetch(:linked_files, []).push('config.php')
Select all the files that you want to share. Put this file from your local to remote server through scp
scp config.php deployer#amazon:~/capistrano/shared/config.php
Now, deploy through the command given below:
bundle exec cap staging deploy
of course, staging can be changed as per requirements may be production,sandbox etc.
One more thing, because you don't want your team members to commit such files. So, put this file to your .gitignore file. And push it to git remote repo.
For Capistrano 3.5+, as specified in official doc :
append :linked_dirs, ".bundle", "tmp"
For me non of the above worked so I ended up adding two functions to the end of the deployment process:
namespace :your_company do
desc "remove index.php"
task :rm_files do
on roles(:all) do
execute "rm -rf #{release_path}/index.php"
end
end
end
namespace :your_company do
desc "add symlink to index.php"
task :add_files do
on roles(:all) do
execute "ln -sf #{shared_path }/index.php #{release_path}/index.php"
end
end
end
after "deploy:finished", "your_company:rm_files"
after "deploy:finished", "your_company:add_files"
I wrote a little CDN server that rebuilds its registry pool when new pool-content-packages are installed into that registry pool.
Instead of having each pool-content-package call the init.d of the cdn-server, I'd like to use triggers. That way it would restart the server only once at the end of an installation run, after all packages were installed.
What have I to do to use triggers in my packages with debhelper support?
What you are looking for is dpkg-triggers.
One solution with use of debhelper to build the debian packages is this:
Step 1)
Create file debian/<serverPackageName>.triggers (replace <serverPackageName> with name of your server package).
Step 1a)
Define a trigger that watch the directory of your pool. The content of file would be:
interest /path/to/my/pool
Step 1b)
But you can also define a named trigger, which have to be fired explicit (see step 3).
content of file:
interest cdn-pool-changed
The name of the trigger cdn-pool-changed is free. You can take what ever you want.
Step 2)
Add handler for trigger to file debian/<serverPackageName>.postinst (replace <serverPackageName> with name of your server package).
Example:
#!/bin/sh
set -e
case "$1" in
configure)
;;
triggered)
#here is the handler
/etc/init.d/<serverPackageName> restart
;;
abort-upgrade|abort-remove|abort-deconfigure)
;;
*)
echo "postinst called with unknown argument \`$1'" >&2
exit 1
;;
esac
#DEBHELPER#
exit 0
Replace <serverPackageName> with name of your server package.
Step 3) (only for named triggers, step 1b) )
Add in every content package the file debian/<contentPackageName>.triggers (replace <contentPackageName> with names of your content packages).
content of file:
activate cdn-pool-changed
Use same name for trigger you defined in step 1.
More detailed Information
The best description for dpkg-triggers I could found is "How to use dpkg triggers". The corresponding git repository with examples you can get here:
git clone git://anonscm.debian.org/users/seanius/dpkg-triggers-example.git
I had a need and read and re-read the docs many times. I think that the process is not clearly explain or rather what goes where is not clearly explained. Here I hope to clarify the use of Debian package triggers.
Service with Configuration Directory
A service reading its settings in a specific directory can mark that directory as being of interest.
Say I create a new service which reads settings from /usr/share/my-service/config/...
That service gets two additions:
In its debian directory I add my-service.triggers
And here are the contents:
# my-service.triggers
interest /usr/share/my-service/config
This means if any other package installs or removes a file from that directory, the trigger enters its "needs to be run" state.
In its debian directory I also add my-service.postinst
And I have a script as follow to check whether the trigger happened and run a process as required:
# my-service.postinst
if [ "$1" = "triggered" ]
then
if [ "$2" = "/usr/share/my-service/config" ]
then
# this may or may not be what you need to do, but this is often
# how you handle a change in your service config files
#
systemctl restart my-service
fi
exit 0
fi
That's it.
Now packages adding extensions to your service can add their own configuration file(s) under /usr/share/my-service/config (or a directory under /etc/my-service/my-service.d/... or /var/lib/my-service/..., although that last one should be reserved for dynamic files, not files installed from a package) and dpkg automatically calls your postinst script with:
postinst triggered /usr/share/my-service/config
# where /usr/share/my-service/config is your <interest-path>
This call happens only once and after all the packages were installed, hence the advantage of having a trigger in the first place. This way each package does not need to know that it has to restart my-service and it does not happen more than once, which could cause all sorts of side effects (i.e. the service tries to listen on a TCP port and get error: address already in use).
IMPORTANT: keep in mind that the postinst should include a line with #DEBHELPER#.
So you do not have to do anything special in other packages. Only make sure to install the configuration files in the correct directory and dpkg picks up from there (i.e. in my example under /usr/share/my-service/config).
I have an extension to BIND9 called ipmgr which makes use of .ini files saved in a specific folder. It uses the files to generate DNS zones (way less errors that way! and it includes support for getting letsencrypt certificates and settings for dmarc/dkim). This package uses this case: a simple directory where configuration files get installed. Other packages do not need to do anything other than install files in the right place (/usr/share/ipmgr/zones, for this package).
Service without a Configuration Folder
In some (rare?) cases, you may need to trigger something in a service which is not driven by the installation of a new configuration file.
In this case, you can use an arbitrary name (it should include your package name to make sure it is unique since this name is global to the entire Debian/Ubuntu system).
To make this one work, you need three files, one of which is a trigger in the other packages.
State the Interest
As above, we have an interest. In this case, the interest is stated as a name on its own. The dpkg system distinguish between a name and a path because a name cannot include a slash (/) character. Names are limited to ASCII except control characters and spaces. I would suggest you stick to a-z, 0-9 and dashes (-).
# my-service.triggers
interest my-service-settings
This is useful if you cannot simply track a folder. For example, the settings could come from a network connection that a package offers once installed.
Listen for the Triggers
Again, as above, you need a postinst script in your Service Package. This captures the trigger and allows you to run a command. The script is the same, only you test for the name instead of the folder (note that you can have any number of triggers, so you could also have both: a folder as above and a special name as here).
# my-service.postinst
if [ "$1" = "triggered" ]
then
if [ "$2" = "my-service-settings" ]
then
# this may or may not what you need to do, but this is often
# how you handle a change in your service config files
#
systemctl restart my-service
fi
exit 0
fi
The Trigger
As mentioned above, we need a third file. An arbitrary name is not going to be triggered automatically by dpkg. It wouldn't know whether your other package needs to trigger something just like that (although it is fairly automated as it is already).
So in other packages, you create a trigger file which looks like this:
# other-package.triggers
activate my-service-settings
Now we recognize the name, it is the same as the interest stated above.
In other words, if the trigger needs to run for something other than just the installation of files in a given location, use a special name and add this triggers file with the activate keyword.
Other Features
I have not tested the other features of the dpkg-trigger(1) tool. There are other keywords support in the triggers files:
interest
interest-await
interest-noawait
activate
activate-await
activate-noawait
The deb-triggers manual page has additional information about those. I am not too sure what the await/noawait implies other than the trigger may happen at any time when nowait is used.
Automatic Trigger Added
The build system on Ubuntu (probably Debian too) automatically adds a triggers file with the following when your package includes a library:
$ cat triggers
# Triggers added by dh_makeshlibs/11.1.6ubuntu2
activate-noawait ldconfig
I suggest you exercise caution if your package includes libraries. If you have your own triggers file, I do not know whether this addition will still happen automatically.
This also shows us a special case where it wants to use the noawait. If I understand correctly, it has to run the ldconfig trigger ASAP so your commands will work as expected after the unpack. Otherwise ldd will not know anything about your newly installed library.