I put my cron task in a module, and then in my Sinatra server.
module Cron
scheduler = Rufus::Scheduler.new
scheduler.every "30m", :first => :now do
run_cmd('git pull')
puts "pulled the repo!!!"
end
end
class MyServer < Sinatra::Base
include Cron
end
The entry point for the app is unicorn (unicorn config/config.ru -p 9393 -c config/unicorn.rb), and in unicorn.rb, there's this line
worker_processes 7
Because of this, git pull is running seven times every 30 minutes, and pulled the repo!!! is printed seven times.
Is there a way I can run this task only in one thread? I tried putting it in unicorn.rb above worker_processes 7 line but I'm not sure if that's the best place for this code to live in.
Unicorn is a multi-process (not multi-threaded) Rack server. There's no native support for executing a specific code path only in one of the worker processes.
However, you can work around that by saving the worker number after fork into an environment variable and then checking its value in your application code.
In config/unicorn.rb use
after_worker_ready do |server, worker|
ENV["WORKER_NR"] = worker.nr.to_s
end
In your Sinatra app do:
if unicorn_worker_nr == "0"
scheduler.every "30m", :first => :now do
...
end
end
def unicorn_worker_nr
ENV["WORKER_NR"]
end
Related
So every example I've looked up indicates this is how one is supposed to do it but I think I may have found a bug unless there's another way to do this.
I'm using upload! to upload assets to a remote list of servers. The task looks like this:
desc "Upload grunt compiled css/js."
task :upload_assets do
on roles(:all) do
%w{/htdocs/css /htdocs/js}.each do |asset|
upload! "#{fetch(:local_path) + asset}", "#{release_path.to_s + '/' + asset}", recursive: true
end
end
end
If local_path is defined as an absolute path such as:
set :local_path:, '/home/dcmbrown/projects/ABC'
This works fine. However if I do the following:
set :local_path:, '~/projects/ABC'
I end up getting the error:
The deploy has failed with an error: Exception while executing on ec2-54-23-88-125.us-west-2.compute.amazon.com: No such file or directory - ~/projects/ABC/htdocs/css
It's not a ' vs " issue as I've tried both (and I didn't think capistrano paid attention to that anyway).
Is this a bug? Is there a work around? Am I just doing it wrong?
I ended up discovering the best way to do this is to actually use path expansion! (headsmack)
irb> File.expand_path('~dcmbrown/projects/ABC')
=> "/home/dcmbrown/projects/ABC"
Of course what I'd like is to do automatic path expansion but you can't have everything. I think I was mostly dumbstruck that it didn't automatically; so much so I spent a couple of hours trying to figure out why it didn't work and ended up wasting time asking here. :(
I don't think the error is coming from the remote server, it just looks like it since it's running that upload command in the context of a deploy.
I just created a single cap task to just do an upload using the "~" character and it also fails with
cap aborted!
SSHKit::Runner::ExecuteError: Exception while executing as deploy#XXX: No such file or directory # rb_file_s_stat - ~/Projects/testapp/public/404.html
It appears to be a Ruby issue not Capistrano as this also fails in a Ruby console
~/Projects/testapp $ irb
2.2.2 :003 > File.stat('~/Projects/testapp/public/404.html')
Errno::ENOENT: No such file or directory # rb_file_s_stat - ~/Projects/testapp/public/404.html
from (irb):3:in `stat'
from (irb):3
from /Users/supairish/.rvm/rubies/ruby-2.2.2/bin/irb:11:in `<main>'
I'm migrating some shell scripts to Chef recipes. Some of these scripts are fairly involved, so just to make life easier in the short term and to avoid introducing bugs in rewriting everything in Chef/Ruby, I'd like to just run some of them as-is. They're all well-written and idempotent, so honestly there's no rush, but of course, the eventual goal is to rewrite them.
One cool feature of Ruby is its __END__ keyword/method: Lines below __END__ will not be executed. Those lines will be available via the special filehandle DATA.
It would be cool to ship the shell scripts as-is inside the the recipe after __END__, maybe something like the following, which I placed in chef-repo/cookbooks/ruby-data-test/recipes/default.rb:
file = Tempfile.new(File.basename(__FILE__))
file << DATA.read
bash file.path
file.unlink
__END__
echo "Hello, world"
However when I run this (with chef-solo -c solo.rb --override-runlist 'recipe[ruby-data-test]'), I get the following error:
[2014-10-03T17:14:56+00:00] ERROR: uninitialized constant Chef::Recipe::DATA
I'm pretty new to Chef, but I'm guessing the above is something about Chef wrapping my recipe in a class, and there's something simple preventing me from accessing DATA. Since it's "global" (?) I tried putting a dollar sign ($DATA) in front of it but that failed with:
NoMethodError
-------------
undefined method `read' for nil:NilClass
So the question is: How do I access DATA in my Chef recipe? Thanks!
It appears you don't have access to DATA, but you can fake it by reading in the current file yourself and splitting on __END__, like Sinatra does.
I ended up making a Chef LWRP for reuse. I don't know if I'll actually end up using this, but I wanted to figure it out. Like I said, I'm a Chef/Ruby noob, so any better ideas or suggestions welcome!
ruby_data_test/recipes/default.rb:
ruby_data_test_execute_ruby_data __FILE__
__END__
#!/bin/bash
set -o errexit
date
echo "Hello, world"
ruby_data_test/resources/execute_ruby_data.rb:
actions :execute_ruby_data
default_action :execute_ruby_data
attribute :source, :name_attribute => true, :required => true
attribute :args, :kind_of => Array
attribute :ignore_errors, :kind_of => [TrueClass, FalseClass], :default => false
ruby_data_test/providers/execute_ruby_data.rb:
def whyrun_supported?
true
end
use_inline_resources
action :execute_ruby_data do
converge_by("Executing #{#new_resource}") do
Chef::Log.info("Executing #{#new_resource}")
file_who_called_me = #new_resource.source
io = ::IO.respond_to?(:binread) ? ::IO.binread(file_who_called_me) : ::IO.read(file_who_called_me)
app, data = io.gsub("\r\n", "\n").split(/^__END__$/, 2)
data.lstrip!
file = Tempfile.new('execute_ruby_data')
file << data
file.chmod(0755)
file.close
exit_status = ::Open3.popen2e(file.path, *#new_resource.args) do |stdin, stdout_and_stderr, wait_thr|
stdout_and_stderr.each { |line| puts line }
wait_thr.value # exit status
end
if exit_status != 0 && !#new_resource.ignore_errors
throw RuntimeError
end
end
end
Here's the output:
$ chef-solo -c solo.rb --override-runlist 'recipe[ruby_data_test]'
Starting Chef Client, version 11.12.4
[2014-10-03T21:50:29+00:00] WARN: Run List override has been provided.
[2014-10-03T21:50:29+00:00] WARN: Original Run List: []
[2014-10-03T21:50:29+00:00] WARN: Overridden Run List: [recipe[ruby_data_test]]
Compiling Cookbooks...
Converging 1 resources
Recipe: ruby_data_test::default
* ruby_data_test_execute_ruby_data[/root/chef/chef-repo/cookbooks/ruby_data_test/recipes/default.rb] action execute_ruby_dataFri Oct 3 21:50:29 UTC 2014
Hello, world
- Executing ruby_data_test_execute_ruby_data[/root/chef/chef-repo/cookbooks/ruby_data_test/recipes/default.rb]
Running handlers:
Running handlers complete
Chef Client finished, 1/1 resources updated in 1.387608 seconds
I'm working with djcelery and supervisor.
I was running a celery with supervisor and everything worked fine, once I realized that I needed to change it to celery multi everything broke up.
If I run celeryd_multi in a terminal it works but always run in background, like supervisor need that the command run in foreground there where the problem is.
This is my celery.ini:
[program:celery_{{ division }}]
command = {{ virtualenv_bin_dir }}/python manage.py celeryd_multi start default mailchimp -c:mailchimp 3 -c:default 5 --loglevel=info --logfile={{ log_dir }}/celery/%n.log --pidfile={{ run_dir }}/celery/%n.pid --schedule=/home/celery/celerybeat-schedule --settings={{ django_settings_python_path }}
autorestart = false
autostart = false
directory = {{ repo_dir }}/{{ division }}
user=celery
numprocs = 1
redirect_stderr = True
stopwaitsecs = 10
startsecs = 10
priority = 997
startretries = 3
Here is the command I put in the terminal that works fine
python manage.py celeryd_multi start default mailchimp -c:mailchimp 3 -c:default 5 --loglevel=info --logfile=/var/log/celery/%n.log --pidfile=/var/log/celery/%n.pid --schedule=/home/celery/celerybeat-schedule --settings=lively.settings_gunicorn
Reply from the main developer of celery (23 Mar 2012):
I'm not aware of any simple solution to start multiple celeryd instances
using supervisor, but you can always use one config for each. Maybe someone
else has a solution for it though.
The generic-init.d/celeryd script uses celeryd-multi to start multiple servers,
and you can use the CELERYD_NODES /etc/init.d/celeryd variable to set a number, or
a list of worker names.
I'm not sure how supervisord could be made to work with celeryd-multi, as it would
probably need to know which of the resulting PIDs should be monitored or not (maybe a supervisord
plugin could be written?)
-- Ask Solem
Daemontools has a utility called fghack designed to make a background process "stay" in the foreground.
pidsig supposedly is similar, but proxies signals:
http://permalink.gmane.org/gmane.comp.sysutils.supervision.general/2010
https://github.com/chexum/pidsig/blob/master/pidsig.c
I believe both are basically wrappers that wait on the backgrounded children.
I am defining my server setup like this:
task :test do
role(:frontend) {[server1,server2,server3, {:user=> "frontend-user", :options => {:log_location=>"HOW DO I READ THIS??"}}]}
role(:backend) {...}
role(:db) {...}
role(:mq) {...}
end
task :staging do
role(:frontend) {[server1,server2,server3, {:user=> "frontend-user", :options => {:log_location=>"HOW DO I READ THIS??"}}]}
role(:backend) {...}
role(:db) {...}
role(:mq) {...}
end
task :prod do
role(:frontend) {[server1,server2,server3, {:user=> "frontend-user", :options => {:log_location=>"HOW DO I READ THIS??"}}]}
role(:backend) {...}
role(:db) {...}
role(:mq) {...}
end
This is to embrace all the complexity of a legacy enterpricey system.
Now, from a task, I want to read the log_location.
Task example:
namespace :log do
desc "list all log files"
task :list do
run %(ls -1 #{log_location}/*/*.log)
end
end
The problem is that the variable log_location is undefined.
/.rvm/gems/ruby-2.0.0-p0/gems/capistrano-2.14.2/lib/capistrano/configuration/namespaces.rb:193:in
method_missing': undefined local variable or methodlog_location'
for
# (NameError)
How do I access that variable?
Is there a smarter/simpler way of setting this custom variable?
I'm sorry to say you can't read that. The blocks passed to task() aren't executed in a server context, thus the block in effect doesn't know what server it's operating on.
The classical workaround for this over the years has been to upload a config file which looks something like this:
---
hostname1:
log_file_location: "/var/log/hostname1/foo/bar"
hostname2:
log_file_location: "/var/log/hostname2/foo/bar"
(or similar) and use the machines hostname when loading the configuration.
I know this isn't a great workaround, thus in the forthcoming (see the v3 branch at Github) version of Capistrano there's a feature which looks like this:
host1 = SSHKit::Host.new 'user#example.com'
host2 = SSHKit::Host.new 'user#example.org'
host1.properties = {log_file_location: "/foo/bar"}
host2.properties.log_file_location = "/bar/baz"
on hosts do |host|
target = "/var/www/sites/"
if host.hostname =~ /org/
target += "dotorg"
else
target += "dotcom"
end
execute! :head, '-n 20', host.properties.log_file_location
execute! :git, :clone, "git#git.#{host.hostname}", target
end
(SSHKit Examples) - SSHKit is the new backend driver for Capistrano.
The v3 branch probably isn't ready for prime time yet, we're having a lot of success internally but the documentation is pretty ahem non existent. However the code is quite literally an oder of magnitude less imposing, and I think you'll find quite readable.
You need this: https://github.com/capistrano/capistrano/wiki/2.x-Multistage-Extension
It means that you can isolate stage specific code in separate files named after the stage. If you want to test for the stage name in the shared deploy.rb you can do that too, like this:
Put this in your deploy.rb
task :show_stage do
puts(stage)
end
Test from command line
$ cap staging show_stage
staging
Actually, I was able to pull out the log_location variable, but ended up with a solution that had one restriction:
I am using log location for one environment only. This is no problem in my current project, since I run the capistrano task against one role at a time.
For testing this setup, I made this task:
namespace :support do
desc "Test if the log location variable is correctly fetched from configuration"
task :test_log_location do
find_servers_for_task(current_task).each do |server|
# puts server.host
# puts server.port
# puts server.user
# puts server.options
result = "LOG LOCATION: #{server.options[:log_location]}"
#puts result
logger.info result
end
end
end
Then, for my tasks in the :log namespace, I defined the variable with set :log_location and also define the :current_role variable:
namespace :log do
def set_log_location
#set_log_location
#puts fetch(:log_location)
log_location = nil
options = nil
find_servers_for_task(current_task).each do |server|
# puts server.host
# puts server.port
# puts server.user
# puts server.options
options = server.options
log_location = server.options[:log_location]
#log_location = server.options[:current_role]
end
msg1="FATAL: you need to specify 'ROLES=frontend,backend,mq' (or one of them) from command line"
msg2="FATAL: Could not get log_location from environment/server options. I can only see these options: #{options}"
raise msg1 if ENV['ROLES'].nil?
raise msg2 if log_location.nil?
set :log_location, log_location
set :current_role, ENV['ROLES'].split(',').first
logger.info %(CURRENT_ROLE #{fetch(:current_role)})
logger.info %(THE LOG LOCATION IS: #{fetch(:log_location)})
end
end
Finally, I used a separate method to fully qualify the log path (needed for my setup -- also in the :log namespace):
def log_location
log_names = {
:frontend => "*/play.log",
:backend => "*Weblogic*/*.{log,out}"
}
loc = "#{fetch(:log_location)}/#{log_names[fetch(:current_role).to_sym]}"
logger.info "using the log location of '#{loc}'"
loc
end
Now, each task can use the specific log location like this:
desc "list all log files"
task :list do
set_log_location
run %(ls -l #{log_location})
end
I am sure this can be done more elegant, but it works for me
I am running Rails 2.3.5.
In my project I have in lib/tasks the following rake task (test_task.rake):
desc 'test_synchro_task'
task :test_synchro_task => :environment do
# A bunch of things that are properly executed (basically I am inserting
# in the database)...
# ...
# and once the above is done, I want the following to be executed,
# the sphinx index to be rebuild, but it is NOT :
system("cd /sites/project/app")
system("RAILS_ENV='staging' rake ts:index")
end
I trigger the execution of the task via a crontab containing the following entry:
13 14 * * * cd /sites/project/app && /sites/ruby/bin/rake RAILS_ENV=staging test_task
which id correctly called and executed except for 2 system lines in the task.
Please note that when I place those 2 system lines in a ruby test.rb file in my project script directory, and run it manually using the ruby command:
ruby test.rb
those 2 system commands are properly executed and the index is rebuilt correctly.
In my rake task I tried replacing those 2 system lines by:
%x["cd /sites/project/app"]
%x["RAILS_ENV='staging' rake ts:index"]
or by
#cmd="cd /sites/project/app; RAILS_ENV='staging' rake ts:index"
`#{#cmd}`
but the rake ts:index is still not executed.
Any idea why?
Many thanks.
Yves
Problem resolved:
1- In the ruby script, I found that the $?.exitstatus was 127, which is "command not found".
2- This hinted me to a PATH problem occurring in the context of cron.
3- Found that post: http://dewful.com/?p=157 titled "Ruby - Cron Not Working For Ruby Script".
4- Added the PATH in the crontab and everything works fine.
Yves