I have a program, it loads a few tasks from a file prepared by user and start executing them according the scheduling shown in the file.
Example: taskFile.txt
Task1: run every hour
Task2: run every 2 seconds
...
TaskN: run every monday at 10:00
This first part is Ok, i solved by using ScheduledExecutorService and i am very satisfied. The tasks are load and run as they should.
Now, let's image that the user, by GUI (at runtime), decides that Task2 should run every minute, and he wants to remove Task3.
I cannot find any way to access one specific task in the pool, in order to remove/modify it.
So I cannot update tasks at runtime. When user changes a task, I can only modify the taskFile.txt and restart the application, in order to reload all tasks according the newly updated taskFile.txt.
Do you know any way to access a single task in order to modify/delete it?
Or even, a way to remove one given task, so i can insert a new one in the pool, with the modifications wanted by the user.
Thanks
This is not elegant, but works.
Let's suppose you need 10 threads, and sometimes you need to manage a specific thread.
Instead to have a pool with 10 thread, use 10 pools with one thread for each, keep them in your favourite data structure, and act on the pool_1 when you want to modify thread_1.
It's possible to remove the older Runnable from the pool and put a new one with the needed changes.
Otherways, anything put in the pool became anonymous and will be not directly manageable.
If somebody has a better solution...
Related
I'm using mill to build a pipeline that
cleans up a bunch of CSV files (producing new files)
loads them into a database
does more work in the database (create views, etc)
runs queries to extract some files.
Should the tasks associated with steps 2 and 3 be producing something analogous to PathRef? If so, what? They aren't producing a file on the disk but nevertheless should not be repeated unless the inputs change. Similarly, tasks associated with step 3 should run if tasks in step 2 are run again.
I see in the documentation for targets that you can return a case class and that re-evaluation depends on the .hashCode of the target's return value. But I'm not sure what to do with that information.
And a related question: Does mill hash the code in each task? It seems to be doing the right thing if I change the code for one task but not others.
A (cached) task in mill is re-run, when the build file (build.sc or it's dependencies/includes) or inputs/dependencies of that task change. Whenever you construct a PathRef, a checksum of the path content is calculated and used as hashCode. This makes it possible to detect changes and only act if anything has changed.
Of course there are exceptions, e.g. input tasks (created with T.input or T.sources) and commands (created with T.command) will always run.
It is in general a good idea to return something from a task. A simple String or Int will do, e.g. to show it in the shell with mill show myTask or post-process it later. Although I think a task running something in an external database should be implemented as a command or input task (which might check, when running, if it really needs something to do), you can also implement it as cached task. But please keep in mind, that the state will only be correct if no other process/user is changing the database in between.
That aside, You could return a current database schema/data version or a last change date in step 2. Make sure, it changes whenever the database was modified. Each change of that return value would trigger step 3 and other dependent tasks. (Of course, step 3 needs to depend on step 2.) As a bonus, you could return the same (previous/old) value in case you did not change the database, which would then avoid any later work in dependent tasks.
In my locustfile I defined test_on_start and test_on_stop events to read a file needed for the test and to write detailed statistics in a CSV at the end of the test. when running in distributed mode, these events occur on the master, not the worker. I am assembling a list of detailed stats for each task in a task sequence and at the end of the test writing a CSV file when the test stops. I found this stackoverflow question which references a setup and teardown. I added these to my class User(HttpUser): but they appear to not be executed.
How can I mimic these events when the test is running on a worker in distributed mode?
Is there a better way?
I am using User on_start and on_stop already - my on_start calls a function to select a random user from a list which was created when the #events.test_start.add_listener is fired, which only happens on the master and not on the workers, so the worker doesn't have any user login data.
It seems counter productive to open the file, read it, select a user at random and close it every time the User on_start method is called. User on_start also sets up the iteration list [] which is where i store the times per task.
When the task sequence is done, meaning the last task is executed, i do a self.interrupt() which runs on_stop, which is where I take the iteration times, and put them into a second list, which is later written using the CSV module. maybe it would be better to just write the data to the CSV during on_stop
The setup/teardown for individual Users has been removed (because they were confusing, as it was run on the first instance of that User class, and when people set properties on that instance got very confused by the fact that later instances didnt get that). Tbh, I wish they had just been replaced by class methods...
The User still has on_start/stop methods though, and if you combine that with a flag it may be able to do what you want. Something like this:
class MyUser(HttpUser):
stopped = False
...
def on_stop(self):
if not MyUser.stopped:
MyUser.stopped = True
# write your csv
# this doesnt guarantee that all your Users are finished though.
https://docs.locust.io/en/stable/writing-a-locustfile.html#on-start-and-on-stop-methods
I have two scripts that do the same thing but for different companies, and during the process they both use the same tables.
It's imperative that only one script runs at once, as sometimes the timings vary greatly, and they are scheduled rather close together purposely. My question is, what is the best method to ensure these scripts do not run together? I tried to have a global field, set to 1 at the beginning of the script, and 0 at the end, so when the 2nd script runs, if global field = 1 - exit script -
This did not work, as both these scripts are scheduled server side, and I have read that the GLOBAL variable is local in this instance.
I assume, we are talking about FileMaker Server schedules.
Global variable will be reset every time you run a scheduled script. Every script will run on it's own session. You can not use them to ensure the scripts do not clash.
As far as I know, FileMaker Server does not run two schedules at the same time. The second script will be delayed until the first one finishes.
FileMaker Server can run simultaneous schedules if they are script schedules, thus an overlap can occur.
What you need to do is set a field that is not a global, so that the schedules can check against the value of that field.
A single record table would be ideal for this.
Make sure that you commit after setting the field, or you may get record locking issues.
Create an OS-level script that uses the fmsadmin command line to run one script, then run the second.
Set the FM Server schedule to run the OS script (which then runs the PSoS scripts).
My company has a couple of joblets that we put in new jobs to do things like initialization of variables, get system information from the database and sending out error / warning emails. The issue we are running into is that if we go ahead and start creating the components of a job and realize that we forgot to include these 3 joblets, we have to basically re-create the job to ensure that the joblets are added first so they run first.
Is there any way to force these joblets to run first and possibly also in a certain order before moving on to the contents of the job being created? Please let me know if there is any information you may need that I'm missing as I have only been using Talend for a few days. The rest of the team has not been using it too much longer than I have, so they do not have the answer I'm looking for either. Thanks in advance!
In Joblets you can use the components Trigger_Input and Trigger_Output as connection-points for on subjob OK triggers. So you can connect joblets and other components in a job with triggers. Thus enforcing execution order.
But you cannot get a on subjob OK trigger from a tPreJob. I am thinking on triggering from a tPreJob to a tWarn (on component OK) and then from tWarn to the joblet (on subjob OK).
I need to execute a sequence of steps a specific number of times.. any pointers on what is the best way to do this in Spring Batch. I am able to implement executing a single step 'x' times. but my requirement is to execute a set of steps - based on a condition 'x' times.Any pointers will help.
Thanks
Lakshmi
You could put all steps in a job an start the whole job several times. There are different ways, how a job actually is launched in spring-batch. have a look at joboperator and launcher and then simply implement a loop around the launching of the job.
You can do this after the whole spring-context is initialized, so there will be no overhead concerning that. But you must by attention about the scope of your beans, especially the reader and writers.
Depending on your needs concerning failurehandling and restart, you also have pay attention how you manage the execution context of your job and steps.
You can simulate a loop with SB using a JobExecutionDecider:
Put it in front of all steps.
Store x in job execution context and check for x value into
decider: move to 'END' if x equals desidered value or increment it
and move to first step of set.
After last step move back to start (the decider).