We use this pattern for our hosts (and linux users):
coreapp_customer_stageID#server
I want to run fabric commands on a list of hosts which match a pattern.
Example: I want to run "date" on all hosts of core app "foocms" of customer "c1".
I could use roles, but there are a lot of customers ... A glob matching way would be good.
you can use this
#task
def set_hosts(lookup_param):
'''
'''
hosts=get_all_systems() # that needs to be implemented by you.
regex=re.compile('^%s' % lookup_param.replace('*', '.*'))
sub_hosts=[host for host in hosts if regex.match(host)]
if not sub_hosts:
raise ValueError('No host matches %s. Available: %s' % (lookup_param, hosts))
env.hosts = sub_hosts
#task
def date():
run('date')
Example: fab set_hosts:coreapp_customer1_* date
Taken from: http://docs.fabfile.org/en/1.7/usage/execution.html#the-alternate-approach
Related
I want to shut down celery workers specifically. I was using app.control.broadcast('shutdown'); however, this shutdown all the workers; therefore, I would like to pass the destination parameter.
When I run ps -ef | grep celery, I can see the --hostname on the process.
I know that the format is {CELERYD_NODES}{NODENAME_SEP}{hostname} from the utility function nodename
destination = ''.join(['celery', # CELERYD_NODES defined at /etc/default/newfies-celeryd
'#', # from celery.utils.__init__ import NODENAME_SEP
socket.gethostname()])
Is there a helper function which returns the nodename? I don't want to create it myself since I don't want to hardcode the value.
I am not sure if that's what you're looking for, but with control.inspect you can get info about the workers, for example:
app = Celery('app_name', broker=...)
app.control.inspect().stats() # statistics per worker
app.control.inspect().registered() # registered tasks per each worker
app.control.inspect().active() # active workers/tasks
so basically you can get the list of workers from each one of them:
app.control.inspect().stats().keys()
app.control.inspect().registered().keys()
app.control.inspect().active().keys()
for example:
>>> app.control.inspect().registered().keys()
dict_keys(['worker1#my-host-name', 'worker2#my-host-name', ..])
Is there a way to get the full s3 url path of a metaflow artifact, which was stored in a step?
I looked at Metaflow's DataArtifact class but didn't see an obvious s3 path property.
Yep, you can do
Flow('MyFlow')[42]['foo'].task.artifacts.bar._object['location']
where MyFlow is the name of your flow, 42 is the run ID, foo is the step under consideration and bar is the artifact from that step.
Based on #Savin's answer, I've written a helper function to get the S3 URL of an artifact given a Run ID and the artifact's name:
from metaflow import Flow, Metaflow, Run
from typing import List, Union
def get_artifact_s3url_from_run(
run: Union[str, Run], name: str, legacy_names: List[str] = [], missing_ok: bool = False
) -> str:
"""
Given a MetaFlow Run and a key, scans the run's tasks and returns the artifact's S3 URL with that key.
NOTE: use get_artifact_from_run() if you want the artifact itself, not the S3 URL to the artifact.
This allows us to find data artifacts even in flows that did not finish. If we change the name of an artifact,
we can support backwards compatibility by also passing in the legacy keys. Note: we can avoid this by resuming a
specific run and adding a node which re-maps the artifact to another key. This will assign the run a new ID.
Args:
missing_ok: whether to allow an artifact to be missing
name: name of the attribute to look for in task.data
run: a metaflow.Run() object, or a run ID
legacy_names: backup names to check
Returns:
the value of the attribute. If attribute is not found
Raises:
DataartifactNotFoundError if artifact is not found and missing_ok=False
ValueError if Flow not found
ValueError if Flow is found but run ID is not.
"""
namespace(None) # allows us to access all runs in all namespaces
names_to_check = [name] + legacy_names
if isinstance(run, str):
try:
run = Run(run)
except Exception as e:
# run ID not found. see if we can find other runs and list them
flow = run.split(sep="/")[0]
try:
flow = Flow(flow)
raise ValueError(f"Could not find run ID {run}. Possible values: {flow.runs()}") from e
except Exception as e2:
raise ValueError(f"Could not find flow {flow}. Available flows: {Metaflow().flows}") from e2
for name_ in names_to_check:
for step_ in run:
for task in step_:
print(f"task {task} artifacts: {task.artifacts} \n \n")
if task.artifacts is not None and name_ in task.artifacts:
# https://stackoverflow.com/a/66361249/4212158
return getattr(task.artifacts, name_)._object["location"]
if not missing_ok:
raise DataArtifactNotFoundError(
f"No data artifact with name {name} found in {run}. Also checked legacy names: {legacy_names}"
)
I have a pretty simple snakemake pipeline that takes an input file does three subsequent steps to produce one output. Each individual job is very quick. Now I want to apply this pipeline to >10k files on an SGE cluster. Even if I use group to have one job for each three rules per input file, I would still submit >10k cluster jobs. Is there a way to instead submit limited number of cluster jobs (lets say 100) and distribute all tasks equally between them?
An example would be something like
rule A:
input: {prefix}.start
output: {prefix}.A
group "mygroup"
rule B:
input: {prefix}.A
output: {prefix}.B
group "mygroup"
rule C:
input: {prefix}.B
output: {prefix}.C
group "mygroup"
rule runAll:
input: expand("{prefix}.C", prefix = VERY_MANY_PREFIXES)
and then run it with
snakemake --cluster "qsub <some parameters>" runAll
You could process all the 10k files in the same rule using a for loop (not sure if this is what Manavalan Gajapathy has in mind). For example:
rule A:
input:
txt= expand('{prefix}.start', prefix= PREFIXES),
output:
out= expand('{prefix}.A', prefix= PREFIXES),
run:
io= zip(input.txt, output.out)
for x in io:
shell('some_command %s %s' %(x[0], x[1]))
and the same for rule B and C.
Look also at snakemake local-rules
The only solution I can think of would be to declare rules A, B, and C to be local rules, so that they run in the main snakemake job instead of being submitted as a job. Then you can break up your runAll into batches:
rule runAll1:
input: expand("{prefix}.C", prefix = VERY_MANY_PREFIXES[:1000])
rule runAll2:
input: expand("{prefix}.C", prefix = VERY_MANY_PREFIXES[1000:2000])
rule runAll3:
input: expand("{prefix}.C", prefix = VERY_MANY_PREFIXES[2000:3000])
...etc
Then you submit a snakemake job for runAll1, another for runAll2, and so on. You do this fairly easily with a bash loop:
for i in {1..10}; do sbatch [sbatch params] snakemake runAll$i; done;
Another option which would be more scalable than creating multiple runAll rules would be to have a helper python script that does something like this:
import subprocess
for i in range(0, len(VERY_MANY_PREFIXES), 1000):
subprocess.run(['sbatch', 'snakemake'] + ['{prefix}'.C for prefix in VERY_MANY_PREFIXES[i:i+1000]])
I have nearly a hundred data sources in a WebSphere Application Server (WAS) and due to office relocation, the IP of the database servers have changed and I need to update the datasource IP addresses in my WAS too.
Considering it error-prone to update hundred IPs through admin console.
Is there any way that I can make the change by updating config files or running a script? My version of WAS is 7.0.
You should be able to use the WAS Admin Console's built-in "command assistance" to capture simple code snippets for listing datasources and changing them by just completing those operations in the UI once.
Take those snippets and create a new jython script to list and update all of them.
More info on command assistance:
http://www.ibm.com/developerworks/websphere/library/techarticles/0812_rhodes/0812_rhodes.html
wsadmin scripting library:
https://github.com/wsadminlib/wsadminlib
You can achieve this using wsadmin scripting. Covener has the right idea with using the admin console's command assistance to do the update once manually (to get the code) and then dump that into a script that you can automate.
The basic idea is that a datasource has a set of properties nested under it, one of which is the ip address. So you want to write a script that will query for the datasource, find its nested property set, and iterate over the property set looking for the 'ipAddress' property to update.
Here is a function that will update the value of the "ipAddress" property.
import sys
def updateDataSourceIP(dsName, newIP):
ds = AdminConfig.getid('/Server:myServer/JDBCProvider:myProvider/DataSource:' + dsName + '/')
propertySet = AdminConfig.showAttribute(ds, 'propertySet')
propertyList = AdminConfig.list('J2EEResourceProperty', propertySet).splitlines()
for prop in propertyList:
print AdminConfig.showAttribute(prop, 'name')
if (AdminConfig.showAttribute(prop, 'name') == 'ipAddress'):
AdminConfig.modify(prop, '[[value '" + newIP + "']]')
AdminConfig.save();
# Call the function using command line args
updateDataSourceIP(sys.argv[0], sys.argv[1])
To run this script you would invoke the following from the command line:
$WAS_HOME/bin/wsadmin.sh -lang jython -f /path/to/script.py myDataSource 127.0.0.1
**Disclaimer: untested script. I don't know the name of the "ipAddress" property off the top of my head, but if you run this once it will print out all the properties on your ds, so you can get it there
Some useful links:
Basics about jython scripting
Modifying config objects using wsadmin
As an improvement to aguibert's script, to avoid having to provide all 100 datasource names and to update it to correct the containment path of the configuration id, consider this script which will update all datasources, regardless of the scope at which they're defined. As always, backup your configuration before beginning and once you're satisified the script is working as expected, replace the AdminConfig.reset() with save(). Note, these scripts will likely not work properly if you're using connection URLs in your configuration.
import sys
def updateDataSourceIP(newIP):
datasources = AdminConfig.getid('/DataSource:/').splitlines()
for datasource in datasources:
propertySet = AdminConfig.showAttribute(datasource, 'propertySet')
propertyList = AdminConfig.list('J2EEResourceProperty', propertySet).splitlines()
for prop in propertyList:
if (AdminConfig.showAttribute(prop, 'name') == 'serverName'):
oldip = AdminConfig.showAttribute(prop, 'value')
print "Updating serverName attribute of datasource '" + datasource + "' from " + oldip + " to " + sys.argv[0]
AdminConfig.modify(prop, '[[value ' + newIP + ']]')
AdminConfig.reset();
# Call the function using command line arg
updateDataSourceIP(sys.argv[0])
The script should be invoked similarly to the above, but without the datasource parameter, the only parameter is the new hostname or ip address:
$WAS_HOME/bin/wsadmin.sh -lang jython -f /path/to/script.py 127.0.0.1
I am defining my server setup like this:
task :test do
role(:frontend) {[server1,server2,server3, {:user=> "frontend-user", :options => {:log_location=>"HOW DO I READ THIS??"}}]}
role(:backend) {...}
role(:db) {...}
role(:mq) {...}
end
task :staging do
role(:frontend) {[server1,server2,server3, {:user=> "frontend-user", :options => {:log_location=>"HOW DO I READ THIS??"}}]}
role(:backend) {...}
role(:db) {...}
role(:mq) {...}
end
task :prod do
role(:frontend) {[server1,server2,server3, {:user=> "frontend-user", :options => {:log_location=>"HOW DO I READ THIS??"}}]}
role(:backend) {...}
role(:db) {...}
role(:mq) {...}
end
This is to embrace all the complexity of a legacy enterpricey system.
Now, from a task, I want to read the log_location.
Task example:
namespace :log do
desc "list all log files"
task :list do
run %(ls -1 #{log_location}/*/*.log)
end
end
The problem is that the variable log_location is undefined.
/.rvm/gems/ruby-2.0.0-p0/gems/capistrano-2.14.2/lib/capistrano/configuration/namespaces.rb:193:in
method_missing': undefined local variable or methodlog_location'
for
# (NameError)
How do I access that variable?
Is there a smarter/simpler way of setting this custom variable?
I'm sorry to say you can't read that. The blocks passed to task() aren't executed in a server context, thus the block in effect doesn't know what server it's operating on.
The classical workaround for this over the years has been to upload a config file which looks something like this:
---
hostname1:
log_file_location: "/var/log/hostname1/foo/bar"
hostname2:
log_file_location: "/var/log/hostname2/foo/bar"
(or similar) and use the machines hostname when loading the configuration.
I know this isn't a great workaround, thus in the forthcoming (see the v3 branch at Github) version of Capistrano there's a feature which looks like this:
host1 = SSHKit::Host.new 'user#example.com'
host2 = SSHKit::Host.new 'user#example.org'
host1.properties = {log_file_location: "/foo/bar"}
host2.properties.log_file_location = "/bar/baz"
on hosts do |host|
target = "/var/www/sites/"
if host.hostname =~ /org/
target += "dotorg"
else
target += "dotcom"
end
execute! :head, '-n 20', host.properties.log_file_location
execute! :git, :clone, "git#git.#{host.hostname}", target
end
(SSHKit Examples) - SSHKit is the new backend driver for Capistrano.
The v3 branch probably isn't ready for prime time yet, we're having a lot of success internally but the documentation is pretty ahem non existent. However the code is quite literally an oder of magnitude less imposing, and I think you'll find quite readable.
You need this: https://github.com/capistrano/capistrano/wiki/2.x-Multistage-Extension
It means that you can isolate stage specific code in separate files named after the stage. If you want to test for the stage name in the shared deploy.rb you can do that too, like this:
Put this in your deploy.rb
task :show_stage do
puts(stage)
end
Test from command line
$ cap staging show_stage
staging
Actually, I was able to pull out the log_location variable, but ended up with a solution that had one restriction:
I am using log location for one environment only. This is no problem in my current project, since I run the capistrano task against one role at a time.
For testing this setup, I made this task:
namespace :support do
desc "Test if the log location variable is correctly fetched from configuration"
task :test_log_location do
find_servers_for_task(current_task).each do |server|
# puts server.host
# puts server.port
# puts server.user
# puts server.options
result = "LOG LOCATION: #{server.options[:log_location]}"
#puts result
logger.info result
end
end
end
Then, for my tasks in the :log namespace, I defined the variable with set :log_location and also define the :current_role variable:
namespace :log do
def set_log_location
#set_log_location
#puts fetch(:log_location)
log_location = nil
options = nil
find_servers_for_task(current_task).each do |server|
# puts server.host
# puts server.port
# puts server.user
# puts server.options
options = server.options
log_location = server.options[:log_location]
#log_location = server.options[:current_role]
end
msg1="FATAL: you need to specify 'ROLES=frontend,backend,mq' (or one of them) from command line"
msg2="FATAL: Could not get log_location from environment/server options. I can only see these options: #{options}"
raise msg1 if ENV['ROLES'].nil?
raise msg2 if log_location.nil?
set :log_location, log_location
set :current_role, ENV['ROLES'].split(',').first
logger.info %(CURRENT_ROLE #{fetch(:current_role)})
logger.info %(THE LOG LOCATION IS: #{fetch(:log_location)})
end
end
Finally, I used a separate method to fully qualify the log path (needed for my setup -- also in the :log namespace):
def log_location
log_names = {
:frontend => "*/play.log",
:backend => "*Weblogic*/*.{log,out}"
}
loc = "#{fetch(:log_location)}/#{log_names[fetch(:current_role).to_sym]}"
logger.info "using the log location of '#{loc}'"
loc
end
Now, each task can use the specific log location like this:
desc "list all log files"
task :list do
set_log_location
run %(ls -l #{log_location})
end
I am sure this can be done more elegant, but it works for me