How to run a talend job in multiple instances at the same time with different context group?
You can't run the same Talend Job in multiple context groups as a group is a collection of context variables that are assigned to a Job.
You can run multiple instances of the same Job in a different context, by passing the context at runtime.
e.g. --context=Prod
This assumes that you have considered all other conflicts that may occur, for example, directories and files that they job may use.
I would suggest, if you have not already done this, to externalise your context values so that, when you pass your context at runtime, values are dynamically loaded and you can have different values for different context.
Once your job is building as a jar, you can have multiple instance at the same time.
Related
My problem is the following:
I need to run a docker container for every user in a platform, all these containers use the same docker image, but you can pass some environment variables like user_id,queue_id,etc in order for them to know which user they belong to
I know that I can create tasks and pass environment variables to them, but for me seems not OK to have thousand of identical tasks using the same docker image but different variables...suppose I want to update the docker image or make some change... in this way I would have to iterate and change every single task
My idea was create a single task and then, run different services using this task and passing different environment variables, so I would have some "username_x_service_task1" for the username_x running the task1
but this doesn't seems possible, at least not using boto3, the only thing I can define in the service are tags
my questions is:
this approach is correct?...or must I create different tasks for different users although the image is the same?
would be possible, inside my running service, to access to its custom tags, so I'd use the tasks to pass user_id queue_id and other required information?
is there any better way to do this :sweat_smile: ?
thank you guys.
Yes this is possible. You would define Container Overrides at the service level to override the environment variables set in the task definition, or to add additional environment variables.
I have used one multi instance subprocess which includes an workflow with human task. When executing, its creating the number of human tasks as to the number of elements present inside the collection object. But all tasks have same process instance id. How the relation is working between parent process and multi instance subprocess?
If there are multiple elements in collection list, then it will create those many tasks inside the multi instance sub process. As all the tasks have same process instance id, how to identify the respective process variable values for each task and the uniqueness of each flow afterwards? And is there a way to make it create an different instance id for each task of the multi instance subprocess?
I did not get all the question, but I will try to answer what I got:
Human tasks have their own task instance id
What is collection object? If you mean tasks in bpmn model, then it is as expected: process instance flow starts after start node and when it reaches a human task, it will create an task instance with id. You can see it in the tasks in UI and with api you can claim, work on, complete , populate data etc.
it is wise to have a separate/different variable for every tasks that can execute in parallel. Then the input will be kept in distinguished data placeholders and you can use it accordingly.
you can create a different instance(task instance) for each task or have repeatable tasks
well the answer was to put the multi-instance into a sub-process, this will allow me to have a separate process instance id per each element of the my List (the input of the multi-instance )
I am now working on Talend Open Studio. I have many jobs.
I need to modify the content of my unique context repository, for instance, to add a new context variable. thus, i wish to spread this new context variable in all the jobs I have.
For now, I had to open each job and manually to add manually the context variable I want to spread into the jobs:
Is there a way to directly spread a context variable in all my jobs from the context repository I have modified ?
This is a bug even I have come across. On some occasions, just updating and saving the context groups propagates the updates across all the jobs using that context group. At some other times, it does not.
As per the link below, Shong from Talend team says that this is supposed to be manual as not all context variables are needed in all jobs. However, I personally feel, this should be the other way round. Whatever is not needed should be manually removed, else all updates should be reflected in all jobs.
https://www.talendforge.org/forum/viewtopic.php?id=19199
I am using Spring-batch 3.0.4 stable. While submitting a job I add some specific parameters to its execution, say, a tag. Jobs information is persisted in the DB.
Later on I will need to retrieve all the executions marked with a particular tag.
Currently I see 2 options:
Get all job instances with org.springframework.batch.core.explore.JobExplorer#findJobInstancesByJobName. For each instance get all available executions with org.springframework.batch.core.explore.JobExplorer#getJobExecutions. Filter the resulting collection of executions checking its JobParameters.
Write my own JdbcTemplate-based DAO implementation to run the select query.
While the former option seems pretty inefficient, the latter one suggests writing extra code to deal with the Spring-specific database tables structure.
Is there any option I am missing here?
The result of SqlWorkflowInstanceStore.WaitForEvents does not tell me what type of workflow is runnable. The constructor of WorkflowApplication takes a workflow definition, and at a minimum, I need to be able to store a workflow ID in the store and query it, so that I can determine which workflow definition to load for the WorkflowApplication.
I also don't want to create a SqlWorkflowInstanceStore for each custom workflow type, since there may be thousands of different workflows.
I thought about trying to use WorkflowServiceHost, but not every workflow has a Receive activity and I don't think it is feasible to have thousands of WorkflowServiceHosts running, each supporting a different workflow type.
Ideally, I just want to query the database for a runnable workflow, determine its workflow definition ID, load the appropriate XAML from a workflow definition table, instantiate WorkflowApplication with the workflow definition, and call LoadRunnableInstance().
I would like to have a way to correlate which workflow is related to a given HasRunnableWorkflowEvent raised by the SqlWorkflowInstanceStore (along with the custom workflow definition ID), or have an alternate way of supporting potentially thousands of different custom workflow types created at runtime. I must also load balance the execution of workflows across multiple application servers.
There's a free product from Microsoft that does pretty much everything you say there, and then some. Oh, and it's excellent too.
Windows Server AppFabric. No, not Azure.
http://www.microsoft.com/windowsserver2008/en/us/app-main.aspx
-Oisin