I have a rakefile that executes some (but not all) of it's file tasks even if the files of interest have already been built. The frustrating thing, is that paring down my rake file to a MWE resolves the problem---even though I haven't altered anything wrt the filetask definition, how the files are being selected, the dependencies, or anything else. It seems that simply removing other (file)tasks from the rakefile remedies the problem.
I realize this is a really awful question, but does anyone have ideas about what might be going on here? I'd post sample code, but my MWE works as expected and I don't have any sense for what is causing the problem in the full rake file. All I can think to do is demonstrate that my MWE is literally an excerpt from the full Rakefile, unaltered...
➜ solutionmaps cat mwe/Rakefile|sed '/^$/d'|tee a
require 'rake'
require 'rake/clean'
require 'pathname'
HOME = ENV['HOME']
SHARED_ATLAS = "#{HOME}/MRI/Manchester/data/CommonBrains/MNI_EPI_funcRes.nii"
TXT = Rake::FileList["txt/nodestrength/??.mni"]
AFNI_RAW = TXT.pathmap("afni/nodestrength/%n_raw+tlrc.HEAD")
AFNI_RAW.zip(TXT).each do |target,source|
file target => [source] do
sh("3dUndump -master #{SHARED_ATLAS} -xyz -datum float -prefix #{target.sub("+tlrc.HEAD","")} #{source}")
end
CLOBBER.push(target)
CLOBBER.push(target.sub(".HEAD",".BRIK"))
CLOBBER.push(target.sub(".HEAD",".BRIK.gz"))
end
➜ solutionmaps perl -ne 'print if ($seen{$_} .= #ARGV) =~ /10$/' Rakefile mwe/Rakefile|sed '/^$/d'|tee b
require 'rake'
require 'rake/clean'
require 'pathname'
HOME = ENV['HOME']
SHARED_ATLAS = "#{HOME}/MRI/Manchester/data/CommonBrains/MNI_EPI_funcRes.nii"
TXT = Rake::FileList["txt/nodestrength/??.mni"]
AFNI_RAW = TXT.pathmap("afni/nodestrength/%n_raw+tlrc.HEAD")
AFNI_RAW.zip(TXT).each do |target,source|
file target => [source] do
sh("3dUndump -master #{SHARED_ATLAS} -xyz -datum float -prefix #{target.sub("+tlrc.HEAD","")} #{source}")
end
CLOBBER.push(target)
CLOBBER.push(target.sub(".HEAD",".BRIK"))
CLOBBER.push(target.sub(".HEAD",".BRIK.gz"))
end
➜ solutionmaps diff a b
➜ solutionmaps
And that my mwe works as expected (that is, it does not execute the file task).
➜ mwe rake --trace --dry-run afni/nodestrength/02_raw+tlrc.HEAD
** Invoke afni/nodestrength/02_raw+tlrc.HEAD (first_time, not_needed)
** Invoke txt/nodestrength/02.mni (first_time, not_needed)
But the full rakefile does not.
rake --trace --dry-run afni/nodestrength/02_raw+tlrc.HEAD
** Invoke afni/nodestrength/02_raw+tlrc.HEAD (first_time)
** Invoke txt/nodestrength/02.mni (first_time, not_needed)
** Execute (dry run) afni/nodestrength/02_raw+tlrc.HEAD
➜ solutionmaps ls afni/nodestrength/02_raw+tlrc.HEAD
afni/nodestrength/02_raw+tlrc.HEAD
Finally happened across a possible answer:
Rake determines that a file task needs to be run if the file doesn’t exist or if any of the prerequisite file tasks are newer.
Quoted from: http://madewithenvy.com/ecosystem/articles/2013/rake-file-tasks/
Since my Rakefiles are under heavy development, and my Filetasks are all pretty interrelated, this is probably why my Rakefile always wanted to rebuild everything.
Related
So every example I've looked up indicates this is how one is supposed to do it but I think I may have found a bug unless there's another way to do this.
I'm using upload! to upload assets to a remote list of servers. The task looks like this:
desc "Upload grunt compiled css/js."
task :upload_assets do
on roles(:all) do
%w{/htdocs/css /htdocs/js}.each do |asset|
upload! "#{fetch(:local_path) + asset}", "#{release_path.to_s + '/' + asset}", recursive: true
end
end
end
If local_path is defined as an absolute path such as:
set :local_path:, '/home/dcmbrown/projects/ABC'
This works fine. However if I do the following:
set :local_path:, '~/projects/ABC'
I end up getting the error:
The deploy has failed with an error: Exception while executing on ec2-54-23-88-125.us-west-2.compute.amazon.com: No such file or directory - ~/projects/ABC/htdocs/css
It's not a ' vs " issue as I've tried both (and I didn't think capistrano paid attention to that anyway).
Is this a bug? Is there a work around? Am I just doing it wrong?
I ended up discovering the best way to do this is to actually use path expansion! (headsmack)
irb> File.expand_path('~dcmbrown/projects/ABC')
=> "/home/dcmbrown/projects/ABC"
Of course what I'd like is to do automatic path expansion but you can't have everything. I think I was mostly dumbstruck that it didn't automatically; so much so I spent a couple of hours trying to figure out why it didn't work and ended up wasting time asking here. :(
I don't think the error is coming from the remote server, it just looks like it since it's running that upload command in the context of a deploy.
I just created a single cap task to just do an upload using the "~" character and it also fails with
cap aborted!
SSHKit::Runner::ExecuteError: Exception while executing as deploy#XXX: No such file or directory # rb_file_s_stat - ~/Projects/testapp/public/404.html
It appears to be a Ruby issue not Capistrano as this also fails in a Ruby console
~/Projects/testapp $ irb
2.2.2 :003 > File.stat('~/Projects/testapp/public/404.html')
Errno::ENOENT: No such file or directory # rb_file_s_stat - ~/Projects/testapp/public/404.html
from (irb):3:in `stat'
from (irb):3
from /Users/supairish/.rvm/rubies/ruby-2.2.2/bin/irb:11:in `<main>'
I'm migrating some shell scripts to Chef recipes. Some of these scripts are fairly involved, so just to make life easier in the short term and to avoid introducing bugs in rewriting everything in Chef/Ruby, I'd like to just run some of them as-is. They're all well-written and idempotent, so honestly there's no rush, but of course, the eventual goal is to rewrite them.
One cool feature of Ruby is its __END__ keyword/method: Lines below __END__ will not be executed. Those lines will be available via the special filehandle DATA.
It would be cool to ship the shell scripts as-is inside the the recipe after __END__, maybe something like the following, which I placed in chef-repo/cookbooks/ruby-data-test/recipes/default.rb:
file = Tempfile.new(File.basename(__FILE__))
file << DATA.read
bash file.path
file.unlink
__END__
echo "Hello, world"
However when I run this (with chef-solo -c solo.rb --override-runlist 'recipe[ruby-data-test]'), I get the following error:
[2014-10-03T17:14:56+00:00] ERROR: uninitialized constant Chef::Recipe::DATA
I'm pretty new to Chef, but I'm guessing the above is something about Chef wrapping my recipe in a class, and there's something simple preventing me from accessing DATA. Since it's "global" (?) I tried putting a dollar sign ($DATA) in front of it but that failed with:
NoMethodError
-------------
undefined method `read' for nil:NilClass
So the question is: How do I access DATA in my Chef recipe? Thanks!
It appears you don't have access to DATA, but you can fake it by reading in the current file yourself and splitting on __END__, like Sinatra does.
I ended up making a Chef LWRP for reuse. I don't know if I'll actually end up using this, but I wanted to figure it out. Like I said, I'm a Chef/Ruby noob, so any better ideas or suggestions welcome!
ruby_data_test/recipes/default.rb:
ruby_data_test_execute_ruby_data __FILE__
__END__
#!/bin/bash
set -o errexit
date
echo "Hello, world"
ruby_data_test/resources/execute_ruby_data.rb:
actions :execute_ruby_data
default_action :execute_ruby_data
attribute :source, :name_attribute => true, :required => true
attribute :args, :kind_of => Array
attribute :ignore_errors, :kind_of => [TrueClass, FalseClass], :default => false
ruby_data_test/providers/execute_ruby_data.rb:
def whyrun_supported?
true
end
use_inline_resources
action :execute_ruby_data do
converge_by("Executing #{#new_resource}") do
Chef::Log.info("Executing #{#new_resource}")
file_who_called_me = #new_resource.source
io = ::IO.respond_to?(:binread) ? ::IO.binread(file_who_called_me) : ::IO.read(file_who_called_me)
app, data = io.gsub("\r\n", "\n").split(/^__END__$/, 2)
data.lstrip!
file = Tempfile.new('execute_ruby_data')
file << data
file.chmod(0755)
file.close
exit_status = ::Open3.popen2e(file.path, *#new_resource.args) do |stdin, stdout_and_stderr, wait_thr|
stdout_and_stderr.each { |line| puts line }
wait_thr.value # exit status
end
if exit_status != 0 && !#new_resource.ignore_errors
throw RuntimeError
end
end
end
Here's the output:
$ chef-solo -c solo.rb --override-runlist 'recipe[ruby_data_test]'
Starting Chef Client, version 11.12.4
[2014-10-03T21:50:29+00:00] WARN: Run List override has been provided.
[2014-10-03T21:50:29+00:00] WARN: Original Run List: []
[2014-10-03T21:50:29+00:00] WARN: Overridden Run List: [recipe[ruby_data_test]]
Compiling Cookbooks...
Converging 1 resources
Recipe: ruby_data_test::default
* ruby_data_test_execute_ruby_data[/root/chef/chef-repo/cookbooks/ruby_data_test/recipes/default.rb] action execute_ruby_dataFri Oct 3 21:50:29 UTC 2014
Hello, world
- Executing ruby_data_test_execute_ruby_data[/root/chef/chef-repo/cookbooks/ruby_data_test/recipes/default.rb]
Running handlers:
Running handlers complete
Chef Client finished, 1/1 resources updated in 1.387608 seconds
we have a build environ which sources a script for few environment settings as follows
if test -f build_env
then
# Source the config file
. build_env
but some how this seems to be failing in RHEL6
sh-4.1$ . build_env
sh: .: build_env: file not found
while in RHEL4 it works
sh-3.00$ . buildenv
sh-3.00$
what could be the issue ?
Try . ./build_env (notice './').
Also, your if syntax is doubtful, though I can't be sure until you fix formatting in the question. I would write it like this:
if test -f build_env; then . ./build_env; fi
I am running Rails 2.3.5.
In my project I have in lib/tasks the following rake task (test_task.rake):
desc 'test_synchro_task'
task :test_synchro_task => :environment do
# A bunch of things that are properly executed (basically I am inserting
# in the database)...
# ...
# and once the above is done, I want the following to be executed,
# the sphinx index to be rebuild, but it is NOT :
system("cd /sites/project/app")
system("RAILS_ENV='staging' rake ts:index")
end
I trigger the execution of the task via a crontab containing the following entry:
13 14 * * * cd /sites/project/app && /sites/ruby/bin/rake RAILS_ENV=staging test_task
which id correctly called and executed except for 2 system lines in the task.
Please note that when I place those 2 system lines in a ruby test.rb file in my project script directory, and run it manually using the ruby command:
ruby test.rb
those 2 system commands are properly executed and the index is rebuilt correctly.
In my rake task I tried replacing those 2 system lines by:
%x["cd /sites/project/app"]
%x["RAILS_ENV='staging' rake ts:index"]
or by
#cmd="cd /sites/project/app; RAILS_ENV='staging' rake ts:index"
`#{#cmd}`
but the rake ts:index is still not executed.
Any idea why?
Many thanks.
Yves
Problem resolved:
1- In the ruby script, I found that the $?.exitstatus was 127, which is "command not found".
2- This hinted me to a PATH problem occurring in the context of cron.
3- Found that post: http://dewful.com/?p=157 titled "Ruby - Cron Not Working For Ruby Script".
4- Added the PATH in the crontab and everything works fine.
Yves
Is there a way using Rake to watch depedencies for changes and execute tasks automatically?
For example, given the following Rakefile:
file 'main.o' => ["main.c", "greet.h"] do
sh "cc -c -o main.o main.c"
end
file 'greet.o' => ['greet.c'] do
sh "cc -c -o greet.o greet.c"
end
file "hello" => ["main.o", "greet.o"] do
sh "cc -o hello main.o greet.o"
end
It would be great if when I changed greet.c or main.c then hello would automatically execute.
AFAIK rake doesn't support watching files and directories with a continuous compilation approach.
The buildr project implements a continuous compilation task for java which you can get inspired by to do the job for your project.
Also, it looks like guard might help there. See the guard github profile for implementations using guard. There's even a Rake extension to do what you want!
There is also rerun https://github.com/alexch/rerun. You can do rerun rake to watch and run the default task.