I have several tasks which all must check that the machines serving as roles have a certain file with certain contents. The logic is reasonable to separate into a prerequisite, or a callback.
task t1, :roles => [:r1] do
...
end
task t2, :roles => [:r2,:r3] do
...
end
before <what?> do
# must only run on :r1 when triggered by t1,
# and only on :r2 and :r3 when triggered by t2!
<ensure role given to parent task has a given file>
end
How do we do that in Capistrano?
It turns out that a before callback can invoke a regular def, in which case it runs for the roles of the parent task. If, however, you call a task there, and that task has no roles, all roles will be used to run it. The real question is where are the dependencies across tasks...
Related
For $REASONS, I created a Rake task to paper over rake test. However, when I use the wrapper task, config/environments/test.rb is never required and, as a result, undesirable things happen (emails are sent out, the database is dropped, etc.).
(FWIW, Rails.env and ENV['RAILS_ENV'] are still set to test in the degenerate case.)
namespace :organization do
desc "Run unit tests."
task :unit do
puts Rails.env # test
Rake::Task["test"].invoke
end
end
you need to pass the :environment in rake tasks for it to work.
namespace :organization do
desc "Run unit tests."
task unit: [:environment] do
puts Rails.env # test
Rake::Task["test"].invoke
end
end
In PSake, is there an option to request that dependent tasks are not not run. For example:
In a build script, Build.ps1:
Task T1 {}
Task T2 -depends T1 {}
Is it possible to do Invoke-psake .\Build.ps1 T2 -ignoredependencies?
The reason for this is that with large projects, testing of tasks that have large number of dependencies (directly or indirectly) takes a long time since all the the dependencies are rebuilt but generally do not need.
Consider the following Rake tasks:
task deploy => [:package] do
end
task package => [:build] do
end
task build do
end
Is there a way to invoke Rake on the command line to execute the package and deploy tasks, but not the build task?
Short answer, no.
The way I usually go about this is instead of using the dependant task notion like you have above:
task deploy => [:package] do
end
I create an alias task for whatever action that is to be completed:
task all => [:build, :package, :deploy]
task fastDeploy => [:package, :deploy]
task deploy do
end
task package do
end
task build do
end
It's not very elegant, but I do find it to be more readable and you can visibly see the dependency of tasks on other tasks instead of the kind of spaghetti code structure the dependant notion can result in... when you have a lot of task it can be awkward to debug the logic to figure what's gone wrong and where at times.
Hope this helps.
How do you disable db:migrate when doing a cap deploy:cold with Capistrano?
In config/deploy.rb the only reference to deploy:migrate is commented out but it's still attempting to do:
bundle exec rake RAILS_ENV=production db:migrate
I got success by overriding the deploy:migrate method in my config/deploy.rb.
namespace :deploy do
desc "No ActiveRecord override"
task :migrate do
end
end
When re-defining a task in Capistrano v2, the original task was replaced. However the Rake DSL on which Capistrano v3 is built is additive. According to documentation.
Under most circumstances, you will simply want to use clear_actions, which removes the specified task’s behaviour, but does not alter it’s dependencies or comments:
namespace :deploy do
Rake::Task["migrate"].clear_actions
task :migrate do
puts "no migration"
end
end
I had the same problem. That's why I override it in the Rakefile. Like this:
namespace :db do
desc "db:migration fakes"
task :migrate => :environment do
p 'No. We will not migrate!'
end
end
Here you can add more logic if you like. You could for example trigger a real migration on certain environments.
I have distinct Rabbit queues each dedicated to a special kind of order processing:
# tasks.py
#celery.task
def process_order_for_product_x(order_id):
pass # elided ...
#celery.task
def process_order_for_product_y(order_id):
pass # elided ...
# settings.py
CELERY_QUEUES = {
"black_hole": {
"binding_key": "black_hole",
"queue_arguments": {"x-ha-policy": "all"}
},
"product_x": {
"binding_key": "product_x",
"queue_arguments": {"x-ha-policy": "all"}
},
"product_y": {
"binding_key": "product_y",
"queue_arguments": {"x-ha-policy": "all"}
},
We have a policy of enforcing explicit routing by setting CELERY_DEFAULT_QUEUE = 'black_hole' and then never consuming from black_hole.
Each of these tasks may use celery's canvas primitives, like so:
# tasks.py
#celery.task
def process_order_for_product_x(order_id):
# These can run in parallel
stage_1_group = group(do_something.si(order_id),
do_something_else.si(order_id))
# These can run in parallel
another_group = group(do_something_at_end.si(order_id),
do_something_else_at_end.si(order_id))
# These run in a linear sequence
process_task = chain(
stage_1_group,
do_something_dependent_on_stage_1.si(order_id),
another_group)
process_task.apply_async()
Supposing I want specific uses of celery.group, celery.chord, celery.chord_unlock, and other canvas tasks to flow through the queue for its corresponding product, rather than getting trapped in a black_hole, is there a way to invoke each particular canvas task with either a custom task name or custom routing_key?
For reasons I won't go into I would prefer to not send all celery.* tasks to a catch-all celery_canvas queue, which is what I am doing in the meantime.
This method allows you to route Celery canvas tasks to the queue of a callback task.
It is possible to specify a custom class-based task router for Celery as described here.
Let's focus on the celery.chord_unlock task. Its signature is defined here.
def unlock_chord(self, group_id, callback, ...):
The second positional argument is the signature of the chord callback task.
Task signatures in Celery are basically dicts, so that gives us an opportunity to access task options, including the task queue name.
Here is an example:
class CeleryRouter(object):
def route_for_task(self, task, args=None, kwargs=None):
if task == 'celery.chord_unlock':
callback_signature = args[1]
options = callback_signature.get('options')
if options:
queue = options.get('queue')
if queue:
return {'queue': queue}
Add it to the Celery config:
CELERY_ROUTES = (CeleryRouter(),
I'm currently using Celery in my project. For some scenarios I need task to chain though different queues:
chain(get_staff.s(url), save_staff.s(dt, partner_id, url))()
Those two functions declared like so:
#task(queue='celery_gevent')
def get_staff(source_url):
#task # send to default queue
def save_staff(suggests, dt, partner, url):
btw, celery_gevent is handled by worker with gevent pool to make http requests.
This example, how you can specify queue implicitly. Also you can explicitly put task in a different queue by specifying additional params, like so:
In [1]: add.apply_async([4,5])
Out[1]: <AsyncResult: bda3dedd-c2c4-44db-be8e-6a97e718f8b0>
$ sudo rabbitmqctl list_queues
Listing queues ...
celery 1
...done.
In [2]: add.apply_async([4,5], queue='your_product')
Out[2]: <AsyncResult: 934f6161-298b-468b-9716-3da6fae58fa5>
$ sudo rabbitmqctl list_queues
Listing queues ...
celery 1
your_product 1
...done.
You can run whole canvas in custom queue:
process_task.apply_async(queue='your_queue')
Try to specify queue_name inside #task decorator. This should help.
Links:
http://docs.celeryproject.org/en/latest/reference/celery.app.task.html
http://docs.celeryproject.org/en/latest/_modules/celery/app/task.html#Task.apply_async