Unicorn continues to use old code following deploy + restart - capistrano

Question

task :restart_unicorn, :except => { :no_release => true } do
run "#{try_sudo} kill -s USR2 $(cat /var/www/app_name/shared/pids/unicorn.pid)"
end
When I do sudo kill -s USR2 $(cat /var/www/app_name/shared/pids/unicorn.pid) on the server, what happens is that a new unicorn master is created, and the old one has (old) appended to its name. The old one is never killed, but even if I kill it on my own, the new unicorn instance is still showing the old code from before my deployment. The new instance has the same creation time as the old -- its as if its just being duplicated. If I stop the instance, and the start it again, it works, but I would like to be able to do zero-downtime deploys.
Any help is appreciated.
EDIT
After following the sugest Ilya O., I created a capistrano task that does this:
old_pid = get_pid('/var/www/appname/shared/pids/unicorn.pid')
run "#{try_sudo} kill -s SIGUSR2 $(cat /var/www/appname/shared/pids/unicorn.pid)"
/var/www/app/current/tmp/pids/unicorn.pid)"
run "#{try_sudo} kill -s SIGWINCH #{old_pid}"
This runs SIGUSR2 on the pid, and kills the old unicorn process. The problem is that all my application server is never updated to my recently deployed code, this task just seems to copy my the old unicorn enviroment into a new process. It works fine if I simply kill the master process and then start unicorn fresh again, but then there is a minute or so of dropped requests.

Answer 1

Do you have preload_app set to true in your Unicorn config? If so, you need to send SIGUSR2 and then SIGQUIT to the original master process after the new process are up and running.
What may also be happening is that the original master process has died but the children are still serving requests. You can try sending SIGUSR2, and after the new master process has spawned, send SIGWINCH (to the old master) to kill the old unicorn child processes, and then SIGQUIT on the old master process. You should now have only the "new" unicorn processes serving requests.
EDIT: Try doing SIGQUIT instead of SIGWINCH. I think the old process might be spawning workers after SIGWINCH.

Answer 2

I'm not sure why are you putting all that work on capistrano, usually on capistrano something like this is enough:
set :unicorn_pid, "unicorn.my_website.pid"
desc "Zero-downtime restart of Unicorn"
task :restart, roles: :app do
if remote_file_exists?("/tmp/#{unicorn_pid}")
puts "Killing /tmp/#{unicorn_pid}"
run "kill -s USR2 `cat /tmp/#{unicorn_pid}`"
else
run "cd #{current_path} ; RAILS_ENV=#{rails_env} bundle exec unicorn_rails -c #{unicorn_config} -D"
end
end
and then the real code for killing the old unicorn process is actually on the unicorn configuration
# config/unicorn.rb
# Set environment to development unless something else is specified
env = ENV["RAILS_ENV"] || "production"
# See http://unicorn.bogomips.org/Unicorn/Configurator.html for complete documentation.
worker_processes 2 # amount of unicorn workers to spin up
APP_PATH = "/u/apps/my_website/current"
listen "/tmp/my_website.socket"
preload_app true
timeout 30 # restarts workers that hang for 30 seconds
pid "/tmp/unicorn.my_website.pid"
# By default, the Unicorn logger will write to stderr.
# Additionally, ome applications/frameworks log to stderr or stdout,
# so prevent them from going to /dev/null when daemonized here:
stderr_path APP_PATH + "/log/unicorn.stderr.log"
stdout_path APP_PATH + "/log/unicorn.stdout.log"
if env == "production"
# Help ensure your application will always spawn in the symlinked
# "current" directory that Capistrano sets up.
working_directory APP_PATH
# feel free to point this anywhere accessible on the filesystem
user 'deploy', 'deploy' # 'user', 'group'
shared_path = "/u/apps/my_website/shared"
stderr_path "#{shared_path}/log/unicorn.stderr.log"
stdout_path "#{shared_path}/log/unicorn.stdout.log"
end
before_fork do |server, worker|
# the following is highly recomended for Rails + "preload_app true"
# as there's no need for the master process to hold a connection
if defined?(ActiveRecord::Base)
ActiveRecord::Base.connection.disconnect!
end
# When sent a USR2, Unicorn will suffix its pidfile with .oldbin and
# immediately start loading up a new version of itself (loaded with a new
# version of our app). When this new Unicorn is completely loaded
# it will begin spawning workers. The first worker spawned will check to
# see if an .oldbin pidfile exists. If so, this means we've just booted up
# a new Unicorn and need to tell the old one that it can now die. To do so
# we send it a QUIT.
#
# This enables 0 downtime deploys.
old_pid = "/tmp/unicorn.my_website.pid.oldbin"
if File.exists?(old_pid) && server.pid != old_pid
begin
Process.kill("QUIT", File.read(old_pid).to_i)
rescue Errno::ENOENT, Errno::ESRCH
# someone else did our job for us
end
end
end
after_fork do |server, worker|
# Unicorn master loads the app then forks off workers - because of the way
# Unix forking works, we need to make sure we aren't using any of the parent's
# sockets, e.g. db connection (since "preload_app true")
if defined?(ActiveRecord::Base)
ActiveRecord::Base.establish_connection
end
# if preload_app is true, then you may also want to check and
# restart any other shared sockets/descriptors such as Memcached,
# and Redis. TokyoCabinet file handles are safe to reuse
# between any number of forked children (assuming your kernel
# correctly implements pread()/pwrite() system calls)
end
I use that setup most of the time and my application is reload without any issue, you probably already have all of that since its mostly the default configuration, but if you are missing something, give it a try ;)

Answer 3

Unicorn guys have a public init.d script for this, you should probably add that in your system and use it.
You will find 2 different ways to restart unicorn : restart and upgrade.
I personnaly add this script in /etc/init.d/unicorn, set this executable then you can use sudo service unicorn upgrade in your capistrano recipe.
Init script : http://unicorn.bogomips.org/examples/init.sh