Passing globalMap variables to all the subJobs in a Job
I am working with multiple jobs that share 60% of same code. Every time I've to update something, I end up changing same stuff in all the jobs.
Here is how it looks:
Existing Job 1 - Node ETV -> Node 1 -> Node 2 -> Node 3 -> Node ABC -> Node 5 -> Node 6
Existing Job 2 - Node KJL -> Node 1 -> Node 2 -> Node 3 -> Node MNP -> Node 5 -> Node 6
Existing Job 3 - Node LDG -> Node 1 -> Node 2 -> Node 3 -> Node XYZ -> Node 5 -> Node 6
I want to make them all work like:
Create Sub Job 1, which is combination of Node 1 -> Node 2 -> Node 3
and
create Sub Job 2, which is combination of Node 5 -> Node 6
And then rewrite the jobs as following:
Updated Job 1 - Node ETV -> Sub Job 1 -> Node ABC -> Sub Job 2
Updated Job 2 - Node KJL -> Sub Job 1 -> Node MNP -> Sub Job 2
Updated Job 3 - Node LDG -> Sub Job 1 -> Node XYZ -> Sub Job 2
The issue is, apparently the scope of globalMap is limited and in no way it is 'global'. The values created in Node ETV aren't accessible within Sub Job 1. Can't seem to get past this thing. There must be create variables which are accessible everywhere within the job.
You can use context variables instead of global variables. When you use tRunJob to call your subjob, you can check the option "transmit whole context" . This way your subjob will automatically retrieve context values from the job. You'll just have to declare your context variables with the same name in all the jobs.
Related
In Spring Integration, I want to disable a poller by setting the autoStartup=false on the InboundChannelAdapter. But with the following setup, none of my pollers are firing on either my Tomcat instance 1 nor Tomcat instance 2. I have two Tomcat instances with the same code deployed. I want the pollers to be disabled on one of the instances since I do not want the same job polling on the two Tomcat instances concurrently.
Here is the InboundChannelAdapter:
#Bean
#InboundChannelAdapter(value = "irsDataPrepJobInputWeekdayChannel", poller = #Poller(cron="${batch.job.schedule.cron.weekdays.irsDataPrepJobRunner}", maxMessagesPerPoll="1" ), autoStartup = "${batch.job.schedule.cron.weekdays.irsDataPrepJobRunner.autoStartup}")
public MessageSource<JobLaunchRequest> pollIrsDataPrepWeekdayJob() {
return () -> new GenericMessage<>(requestIrsDataPrepWeekdayJob());
}
The property files are as follows. Property file for Tomcat instance 1:
# I wish for this job to run on Tomcat instance 1
batch.job.schedule.cron.riStateAgencyTransmissionJobRunner=0 50 14 * * *
# since autoStartup defaults to true, I do not provide:
#batch.job.schedule.cron.riStateAgencyTransmissionJobRunner.autoStartup=true
# I do NOT wish for this job to run on Tomcat instance 1
batch.job.schedule.cron.weekdays.irsDataPrepJobRunner.autoStartup=false
# need to supply as poller has a cron placeholder
batch.job.schedule.cron.weekdays.irsDataPrepJobRunner=0 0/7 * * * 1-5
Property file for Tomcat instance 2:
# I wish for this job to run on Tomcat instance 2
batch.job.schedule.cron.weekdays.irsDataPrepJobRunner=0 0/7 * * * 1-5
# since autoStartup defaults to true, I do not provide:
#batch.job.schedule.cron.weekdays.irsDataPrepJobRunner.autoStartup=true
# I do NOT wish for this job to run on Tomcat instance 2
batch.job.schedule.cron.riStateAgencyTransmissionJobRunner.autoStartup=false
# need to supply as poller has a cron placeholder
batch.job.schedule.cron.riStateAgencyTransmissionJobRunner=0 50 14 * * *
The properties files are passed as a VM option, e.g. "-Druntime.scheduler=dev1". I cannot disable the poller on one of the JVMs using "-" as the cron expression -- something similar to the ask here: Poller annotation with cron expression should support a special disable character
My goal of being able to call the job manually from either Tomcat instance 1 or Tomcat instance 2 is working. My problem with the setup mentioned above, is that none of the pollers are firing as per their cron expression.
Consider to investigate a leader election pattern: https://docs.spring.io/spring-integration/docs/current/reference/html/messaging-endpoints.html#leadership-event-handling.
This way you have those endpoints in non-started state by default and in the same role. The election is going to chose a leader and start only this one.
Of course there has to be some shared external service to control leadership.
We have a 3 node Kafka connect cluster running version 5.5.4 in distributed mode. We are observing a strange issue regarding connector's task status.
The REST calls to node 1 and 2 are returning different results.
The first node returned this result:
{
"connector":{
"state":"RUNNING",
"worker_id":"x.com:8083"
},
"name":"connector",
"type":"source",
"tasks":[
]
}
Yes the task is empty where as the other node returned this result:
{
"connector":{
"state":"RUNNING",
"worker_id":"x.com:8083"
},
"name":"connector...",
"type":"source",
"tasks":[
{
"id":0,
"state":"RUNNING",
"worker_id":"x.com:8083"
}
]
}
As mentioned in this doc https://docs.confluent.io/home/connect/userguide.html#kconnect-internal-topics, I have configured group.id, config.storage.topic, offset.storage.topic and status.storage.topic with identical values in all 3 nodes.
I did go through connect-statuses-0 data directory and the file sizes for log, index and timestamp are all identical in node 1 and node 2. I don't know what is the .snapshot file but I see only one with root user/group in first node where as I see 2 of them in the 2nd node. One owned by root user/group and the other owned by our custom created user. Not sure this has anything to do with this problem.
Please guide me in identifying the root cause for this problem. If I do need to check any configuration, please let me know.
I deployed a cluster in AWS 3 AZ, I want to start one master on each AZ. Everything else works except I cannot start one master in one AZ.
Here is my validation:
INSTANCE GROUPS
NAME ROLE MACHINETYPE MIN MAX SUBNETS
bastions Bastion t2.micro 1 1 utility-us-east-1a,utility-us-east-1c,utility-us-east-1d
master-us-east-1a Master m3.medium 1 1 us-east-1a
master-us-east-1c Master m3.medium 2 2 us-east-1c
master-us-east-1d Master m3.medium 1 1 us-east-1d
nodes Node m4.xlarge 3 3 us-east-1a,us-east-1c,us-east-1d
workers Node m4.2xlarge 2 2 us-east-1a,us-east-1c,us-east-1d
NODE STATUS
NAME ROLE READY
ip-10-0-100-34.ec2.internal node True
ip-10-0-107-127.ec2.internal master True
ip-10-0-120-160.ec2.internal node True
ip-10-0-35-184.ec2.internal node True
ip-10-0-39-224.ec2.internal master True
ip-10-0-59-109.ec2.internal node True
ip-10-0-87-169.ec2.internal node True
VALIDATION ERRORS
KIND NAME MESSAGE
InstanceGroup master-us-east-1c InstanceGroup "master-us-east-1c" did not have enough nodes 0 vs 2
Validation Failed
And if I use rolling update, it shows there is one master not started:
NAME STATUS NEEDUPDATE READY MIN MAX NODES
bastions Ready 0 1 1 1 0
master-us-east-1a Ready 0 1 1 1 1
master-us-east-1c Ready 0 0 1 1 0
master-us-east-1d Ready 0 1 1 1 1
nodes Ready 0 3 3 3 3
workers Ready 0 2 2 2 2
What shall I do to bring that machine up?
I solved this problem. It is because the m3.medium type (the default one in kops for master) is no longer available in that AZ. Change it to m4.large makes it work.
I am trying to deploy rhosp10, and when getting to "openstack overcloud deploy" phase, I get these errors:
Error: only 0 of 1 requested ironic nodes are tagged to profile control (for flavor control)
Recommendation: tag more nodes using ironic node-update <NODE ID> replace properties/capabilities=profile:control,boot_option:local
Error: only 0 of 5 requested ironic nodes are tagged to profile compute (for flavor compute)
Recommendation: tag more nodes using ironic node-update <NODE ID> replace properties/capabilities=profile:compute,boot_option:local
Not enough nodes - available: 0, requested: 6
Configuration has 3 errors, fix them before proceeding. Ignoring these errors is likely to lead to a failed deploy.
However, I configured 1 node to use control profile, and 5 to use compute profile. For example:
[stack#rhosp-1-director ~]$ openstack baremetal node show 4e153e0a-4c7b-4ee9-afb5-9036e263949b|grep prop
| properties | {u'cpu_arch': u'x86_64', u'root_device': {u'serial': u'600508b1001c7b0731bc32edbb3a8369'}, u'cpus': u'48', u'capabilities': u'profile:control,boot_option:local', u'memory_mb': u'131072', u'local_gb': u'744'} |
[stack#rhosp-1-director ~]$ openstack baremetal node show 4989038d-de10-4365-8051-44fd42fd0ec7|grep prop
| properties | {u'cpu_arch': u'x86_64', u'root_device': {u'serial': u'600508b1001c73b9fa55f385cd1a4008'}, u'cpus': u'48', u'capabilities': u'profile:compute,boot_option:local', u'memory_mb': u'131072', u'local_gb': u'744'} |
Another thing is that the following command yields no output:
openstack overcloud profiles list
I am following their manual from https://access.redhat.com/documentation/en/red-hat-openstack-platform/10-beta/single/director-installation-and-usage/#sect-Registering_Nodes_for_the_Overcloud step by step, so don't know what I'm doing wrong.
problem ended up being ironic auto cleaning. introspection never completed ok. not sure why, but disabling auto cleaning in ironic.conf right after undercloud install, followed by a reboot (for all ironic services to update this property), followed by the next steps, was successful (including introspection).
In MATLAB, when I use the numlabs instruction in the Command Window after calling parpool(4), it is 1. But, when I use the same instruction in the parallel command window it is 4, Why?
>> parpool(4)
Starting parallel pool (parpool) using the 'local' profile ... connected to 4 workers.
ans =
Pool with properties:
Connected: true
NumWorkers: 4
Cluster: local
AttachedFiles: {}
IdleTimeout: 30 minute(s) (30 minutes remaining)
SpmdEnabled: true
>> numlabs
ans =
1
>> pmode start
Starting pmode using the 'local' profile ... connected to 4 workers.
numlabs 4 4 4 4
numlabs returns the "total number of workers operating in parallel on current job". If you type the command after starting a pool, there obviously won't be any ongoing job. Also from the numlabs documentation:
In an spmd block, numlabs on each worker returns the parallel pool size.
However, inside a parfor-loop, numlabs always returns a value of 1.
If you just want the number of workers, you can use gcp (get current pool):
hp = gcp;
hp.NumWorkers
I think that is normal behavior, since numlabs should return 1 when not used inside a spmd block or during a parallel job. What you have done above is only open the parallel pool, but not asking it to do a parallel job.
I can't test right now, but if you call something like this:
spmd
some statement with numlabs
end
you will see printed in the Command Window that you are indeed using the 4 available workers.