How does history replay work in cadence?
I have a workflow which calls two activity sequentially.
Say, the first activity got completed and the second has 100 no of lines of code. If the app server restarts when executing the 50th line of the code in activity2, is it exactly starts the execution from the 50th line. If yes, what magic is happening inside cadence?
#Override
public String composeGreeting(String greeting, String name) throws Exception {
FileWriter fw =
new FileWriter(
"/Users/kumble-004/Documents/Uber_Cadence/Sample_Projects/TestCadence/src/com/company/"+name+".txt");
System.out.println(
DateTimeFormatter.ofPattern("yyyy/MM/dd HH:mm:ss").format(LocalDateTime.now())
+ " [Activity] started");
long time = System.currentTimeMillis() + 240000;
int i = 0, j=1;
while (System.currentTimeMillis() != time) {
if(i++ %10000000 == 0) {
fw.write("print - " + j++ + " " +
DateTimeFormatter.ofPattern("yyyy/MM/dd HH:mm:ss").format(LocalDateTime.now()) +"\n");
}
}
fw.close();
System.out.println(
DateTimeFormatter.ofPattern("yyyy/MM/dd HH:mm:ss").format(LocalDateTime.now())
+ " [Activity] ended");
return greeting + " " + name + "!";
}
}
I have the above code in my hello activity. this code will run for 4 minutes and it will be writing data in a file when a condition meets
I started a workflow and have quit the cadence server after printing [Activity] started. I didn't start it just stops it. But after 4 minutes it is exactly printing [Activity] Ended in the console. I am wondering how is this possible because I stop the server but code is executing, data is writtening in file.
While I am checking it via cadence UI it shows that the last history is
ActivityTaskStarted. And I started my server. After 15 mins(beacuse scheduleToCloseTimeoutSeconds is 15 mins) Activity returns with event ActivityTaskTimedOut and the whole whorkflow has failed due to this timeout.
Kindly explain what is happening when restarting cadence server ?
If the app server restarts when executing the 50th line of the code in activity2, is it exactly starts the execution from the 50th line
No, it will not resume from 50th line of activity automatically for you.
Replay is only happening for Workflow. It is relaying on History to replay and rebuild the memory stack. Everything happens in the workflow is stored in the history:
Every step in the workflow code, generates a bunch of results called "Decision"
Activity/ChildWorkflow results
External events like Signals
Timers
Etc.
For more details, please refer to the doc about replay history and What exactly is a Cadence decision task?
But after 4 minutes it is exactly printing [Activity] Ended in the console. I am wondering how is this possible because I stop the server but code is executing, data is writtening in file.
That's because your activity worker is still running. The code you are running is purely activity code.
However, it activity results will not be able reported to server when the server is down. Which means the history will lose it and workflow may reschedule another activity(if retry is enabled).
Please refer to the doc about activity timeout and retry
Related
From the documentation for blockLast():
Subscribe to this Flux and block indefinitely until the upstream signals its last value or completes. Returns that value, or null if the Flux completes empty. In case the Flux errors, the original exception is thrown (wrapped in a RuntimeException if it was a checked exception).
Let's say for an example code sample:
Flux
.range(0, 1000)
.doOnNext(i -> System.out.println("i = " + i + "Thread: " + Thread.currentThread().getName()))
.flatMap(i -> {
System.out.println("end"+ i + " Thread: " + Thread.currentThread().getName());
return Mono.just(i);
}).blockLast();
If I were to understand this based off the documentation's own description, I'd think blockLast means to block the publisher (in this case till all 1000 integers are emitted successfully, last one included).
After which .flatMap(..) is called, one at a time (since we don't specifically force parallel processing.
However I see the following in the console when run:
i = 0Thread: main
end0 Thread: main
i = 1Thread: main
end1 Thread: main
i = 2Thread: main
end2 Thread: main
i = 3Thread: main
end3 Thread: main
i = 4Thread: main
end4 Thread: main
i = 5Thread: main
Isnt i = 0Thread: main supposed to run till i = 1000Thread: main first then .flatMap gets executed?
i.e.
i = 0Thread: main
i = 1Thread: main
i = 2Thread: main
i = 3Thread: main
i = 4Thread: main
.
.
end1 Thread: main
end2 Thread: main
end3 Thread: main
The behavior is exactly the same if .subscribe() is used. I'm kinda confused here.
The observed behaviour is fine. A Flux describes a sequence of operations that are executed as elements are emitted.
So, in your example, each integer generated by range is immediately processed by the next operation in chain, i.e. flatMap here.
It is the same behaviour as with standard java.util.stream.Stream API.
The reason for that behaviour is double:
Avoid buffering all elements between each processing step
A data source can emit an infinite number of messages. And it can also emit messages with various frequency (with constant delay, or not, very fast or very slow, etc.). So, a stream API is designed to process and return each element as soon as it is received, independently of the messages before or after it.
And about blockLast specifically: internally, it subscribe to the flux, and wait for completion or error signal to return or throw an error to the user.
I am trying to use the RxJava caching mechanism ( RxJava2 ) but i can't seem to catch how it works or how can i control the cached contents since there is the cache operator.
I want to verify the cached data with some conditions before emitting the new data.
for example
someObservable.
repeat().
filter { it.age < maxAge }.
map(it.name).
cache()
How can i check and filter the cache value and emit it if its succeeds and if not then i will request a new value.
since the value changes periodically i need to verify if the cache is still valid before i can request a new one.
There is also ObservableCache<T> class but i can't find any resources of using it.
Any help would be much appreciated. Thanks.
This is not how replay/ cache works. Please read the #replay/ #cache documentation first.
replay
This operator returns a ConnectableObservable, which has some methods (#refCount/ #connect/ #autoConnect) for connecting to the source.
When #replay is applied without an overload, the source subscription is multicasted and all emitted values sind connection will be replayed. The source subscription is lazy and can connect to the source via #refCount/ #connect/ #autoConnect.
Returns a ConnectableObservable that shares a single subscription to the underlying ObservableSource that will replay all of its items and notifications to any future Observer.
Applying #relay without any connect-method (#refCount/ #connect/ #autoConnect) will not emit any values on subscription
A Connectable ObservableSource resembles an ordinary ObservableSource, except that it does not begin emitting items when it is subscribed to, but only when its connect method is called.
replay(1)#autoConnect(-1) / #refCount(1) / #connect
Applying replay(1) will cache the last value and will emit the cached value on each subscription. The #autoConnect will connect open an connection immediately and stay open until a terminal event (onComplete, onError) happens. #refCount is smiular, but will disconnect from the source, when all subscriber disappear. The #connect opreator can be used, when you need to wait, when alle subscriptions have been done to the observable, in order not to miss values.
usage
#replay(1) -- most of the it should be used at the end of the observable.
sourcObs.
.filter()
.map()
.replay(bufferSize)
.refCount(connectWhenXSubsciberSubscribed)
caution
applying #replay without a buffer-limit or expiration date will lead to memory-leaks, when you observale is infinite
cache / cacheWithInitialCapacity
Operators are similar to #replay with autoConnect(1). The operators will cache every value and replay on each subsciption.
The operator subscribes only when the first downstream subscriber subscribes and maintains a single subscription towards this ObservableSource. In contrast, the operator family of replay() that return a ConnectableObservable require an explicit call to ConnectableObservable.connect().
Note: You sacrifice the ability to dispose the origin when you use the cache Observer so be careful not to use this Observer on ObservableSources that emit an infinite or very large number of items that will use up memory. A possible workaround is to apply takeUntil with a predicate or another source before (and perhaps after) the application of cache().
example
#Test
fun skfdsfkds() {
val create = PublishSubject.create<Int>()
val cacheWithInitialCapacity = create
.cacheWithInitialCapacity(1)
cacheWithInitialCapacity.subscribe()
create.onNext(1)
create.onNext(2)
create.onNext(3)
cacheWithInitialCapacity.test().assertValues(1, 2, 3)
cacheWithInitialCapacity.test().assertValues(1, 2, 3)
}
usage
Use cache operator, when you can not control the connect phase
This is useful when you want an ObservableSource to cache responses and you can't control the subscribe/dispose behavior of all the Observers.
caution
As with replay() the cache is unbounded and could lead to memory-leaks.
Note: The capacity hint is not an upper bound on cache size. For that, consider replay(int) in combination with ConnectableObservable.autoConnect() or similar.
further reading
https://blog.danlew.net/2018/09/25/connectable-observables-so-hot-right-now/
https://blog.danlew.net/2016/06/13/multicasting-in-rxjava/
If your event source (Observable) is an expensive operation, such as reading from a database, you shouldn't use Subject to observe the events, since that will repeat the expensive operation for each subscriber. Caching can also be risky with infinite streams due to "OutOfMemory" exceptions. A more appropriate solution may be ConnectableObservable, which only performs the source operation once, and broadcasts the updated value to all subscribers.
Here is a code sample. I didn't bother creating an infinite periodic stream or including error handling to keep the example simple. Let me know if it does what you need.
class RxJavaTest {
private final int maxValue = 50;
private final ConnectableObservable<Integer> source =
Observable.<Integer>create(
subscriber -> {
log("Starting Event Source");
subscriber.onNext(readFromDatabase());
subscriber.onNext(readFromDatabase());
subscriber.onNext(readFromDatabase());
subscriber.onComplete();
log("Event Source Terminated");
})
.subscribeOn(Schedulers.io())
.filter(value -> value < maxValue)
.publish();
void run() throws InterruptedException {
log("Starting Application");
log("Subscribing");
source.subscribe(value -> log("Subscriber 1: " + value));
source.subscribe(value -> log("Subscriber 2: " + value));
log("Connecting");
source.connect();
// Add sleep to give event source enough time to complete
log("Application Terminated");
sleep(4000);
}
private Integer readFromDatabase() throws InterruptedException {
// Emulate long database read time
log("Reading data from database...");
sleep(1000);
int randomValue = new Random().nextInt(2 * maxValue) + 1;
log(String.format("Read value: %d", randomValue));
return randomValue;
}
private static void log(Object message) {
System.out.println(
Thread.currentThread().getName() + " >> " + message
);
}
}
Here's the output:
main >> Starting Application
main >> Subscribing
main >> Connecting
main >> Application Terminated
RxCachedThreadScheduler-1 >> Starting Event Source
RxCachedThreadScheduler-1 >> Reading data from database...
RxCachedThreadScheduler-1 >> Read value: 88
RxCachedThreadScheduler-1 >> Reading data from database...
RxCachedThreadScheduler-1 >> Read value: 42
RxCachedThreadScheduler-1 >> Subscriber 1: 42
RxCachedThreadScheduler-1 >> Subscriber 2: 42
RxCachedThreadScheduler-1 >> Reading data from database...
RxCachedThreadScheduler-1 >> Read value: 37
RxCachedThreadScheduler-1 >> Subscriber 1: 37
RxCachedThreadScheduler-1 >> Subscriber 2: 37
RxCachedThreadScheduler-1 >> Event Source Terminated.
Note the following:
Events only start firing once connect() is called on the source, not when observers subscribe to the source.
Database calls are only made once per event update
Filtered values are not emitted to subscribers
All subscribers are executed in the same thread
Application terminates before the events are processed due to concurrency. Normally your app will run in an event loop, so your app will remain responsive during slow operations.
I need to achieve the ability to monitor and be able to cancel an ALREADY RUNNING job on queue.
There's a lot of answers about deleting QUEUED jobs, but not on an already running one.
This is the situation: I have a "job", which consists of HUNDREDS OF THOUSANDS rows on a database, that need to be queried ONE BY ONE against a web service.
Every row needs to be picked up, queried against a web service, stored the response and its status updated.
I had that already working as a Command (launching from / outputting to console), but now I need to implement queues in order to allow piling up more jobs from more users.
So far I've seen Horizon (which doesn't runs on Windows due to missing process control libs). However, in some demos seen around it lacks (I believe) a couple things I need:
Dynamically configurable timeout (the whole job may take more than 12 hours, depending on the number of rows to process on the selected job)
Ability to CANCEL an ALREADY RUNNING job.
I also considered the option to generate EACH REQUEST as a new job instead of seeing a "job" as the whole collection of rows (this would overcome the timeout thing), but that would give me a Horizon "pending jobs" list of hundreds of thousands of records per job, and that would kill the browser (I know Redis can handle this without itching at all). Further, I guess is not possible to cancel "all jobs belonging to X tag".
I've been thinking about hitting an API route, fire the job and decouple it from the app, but I'm seeing that this requires forking processes.
For the ability to cancel, I would implement a database with job_id, and when the user hits an API to cancel a job, I'd mark it as "halted". On every loop I would check its status and if it finds "halted" then kill itself.
If I've missed any aspect just holler and I'll add it or clarify about it.
So I'm asking for an advice here since I'm new to Laravel: how could I achieve this?
So I finally came up with this (a bit clunky) solution:
In Controller:
public function cancelJob()
{
$jobs = DB::table('jobs')->get();
# I could use a specific ID and user owner filter, etc.
foreach ($jobs as $job) {
DB::table('jobs')->delete($job->id);
}
# This is a file that... well, it's self explaining
touch(base_path(config('files.halt_process_signal')));
return "Job cancelled - It will stop soon";
}
In job class (inside model::chunk() function)
# CHECK FOR HALT SIGNAL AND [OPTIONALLY] STOP THE PROCESS
if ($this->service->shouldHaltProcess()) {
# build stats, do some cleanup, log, etc...
$this->halted = true;
$this->service->stopProcess();
# This FALSE is what it makes the chunk() method to stop looping
return false;
}
In service class:
/**
* Checks the existence of the 'Halt Process Signal' file
*
* #return bool
*/
public function shouldHaltProcess() :bool
{
return file_exists($this->config['files.halt_process_signal']);
}
/**
* Stop the batch process
*
* #return void
*/
public function stopProcess() :void
{
logger()->info("=== HALT PROCESS SIGNAL FOUND - STOPPING THE PROCESS ===");
$this->deleteHaltProcessSignalFile();
return ;
}
It doesn't looks quite elegant, but it works.
I've surfed the whole web and many goes for Horizon or other tools that doesn't fit my case.
If anyone has a better way to achieve this, it's welcome to share.
Laravel queue have 3 important config:
1. retry_after
2. timeout
3. tries
See more: https://laravel.com/docs/5.8/queues
Dynamically configurable timeout (the whole job may take more than 12
hours, depending on the number of rows to process on the selected job)
I think you can config timeout + retry_after about 24h.
Ability to CANCEL an ALREADY RUNNING job.
Delete job in jobs table
Delete process by process id in your server
Hope it help you :)
I have a few questions regarding quartz.net.
What is it that keeps track of if there has been a missfire situation i Quartz.net?
What happens in the following scenarios:
If a job is run but cannot finnish due to some bug, does that count as a missfire or not?
What happens if i republish the solution, is the tracking reset?
Is there a way to receive information on what the scheduler has done and not been able to do?
I have the following code in my Run method:
IJobDetail dailyUserMailJob = new JobDetailImpl("DailyUserMailJob", null, typeof(Jobs.TestJob));
ITrigger trigger = TriggerBuilder.Create()
.WithIdentity("trigger1", "group1")
.WithCronSchedule("0 0 4 1 * ?", x => x.WithMisfireHandlingInstructionFireAndProceed())
.Build();
this.Scheduler.ScheduleJob(dailyUserMailJob, trigger);
this.Scheduler.Start();
The job is supposed to run the first every month on 4 am.
When testing I have set the system clock so that the jobb is missed for one month. According to the documentation when using WithMisfireHandlingInstructionFireAndProceed the job should be run the first thing that happens, but it dosent. Is there something wrong with the code or could it be some other reason the job is not run when using WithMisfireHandlingInstructionFireAndProceed() ?
If a job is missed, there is logic to bring it back. However, there is a "window" on how far back to go.
<add key="quartz.jobStore.misfireThreshold" value="60000"/>
You can increase this value.
If you have an ADOStore, misfires are persisted. Thus "if the power goes out", when restarting...you can recover from misfires.
If you have a RamStore...if "the power goes out", everything was in memory to begin with..so you won't get mis-fire handling, because everything was "in memory" and the memory is lost.
..
If you use Sql Server (AdoStore) and put a Profiler/Trace on it, you'll see the engine "poll" for misfires.......with a "go back this far in time" based on the misfireThreshold.
See this link:
http://nurkiewicz.blogspot.com/2012/04/quartz-scheduler-misfire-instructions.html
for more detailed info. Which has a "withMisfireHandlingInstructionFireAndProceed" note.
I am wanting to know a button is rendered on main window UI or not. This button rendering is depending on server response result (written in Objective C). If server response comes perfectly it becomes render perfectly (VISIBLE) otherwise it is not present there (INVISIBLE). And whenever it becomes visible I always tap on it for further next process.
I wrote code
UIATarget.localTarget().pushTimeout(200);
//My code
UIATarget.localTarget().popTimeout();
By the above code I have to wait till 200 sec but my concern is I want to wait but whenever object is on screen I don't want keep me busy in WAITING MODE.
How will I write code in automation?
Thanks
Ok, this might give you idea how to follow-up:
For your view implement an accessibilityValue method which returns a JSON formatted value:
- (NSString *)accessibilityValue
{
return [NSString stringWithFormat:
#"{'MyButtonisVisible':%#}",
self.MyButton.isHidden ? #"false" : #"true"];
}
Then somehow you can access it from your test javascript:
var thisproperty = eval("(" + element.value() + ")");
if (thisproperty.MyButtonisVisible) {
UIATarget.localTarget().tap({"x":100, "y":100});
}
Hope that helps.
If you make the name different when you enable the button you can do this:
var awesomeButton = target.frontMostApp().mainWindow().buttons()[0];
UIATarget.localTarget().pushTimeout(200);
awesomeButton.withName("My Awesome Button");
if (awesomeButton.isVisible()) {
UIALogger.logError("Error no awesome button!");
}
UIATarget.localTarget().popTimeout();
withName will repeatedly test the name and control will return to your script once the name matches or when the time out is reached.
Per Apple's Doc
withName:
Tests if the name attribute of the element has the given string value. If the match fails, the test is retried until the current timeout expires.
Timeout Periods:
If the action completes during the timeout period, that line of code returns, and your script can proceed. If the action doesn’t complete during the timeout period, an exception is thrown.
https://developer.apple.com/library/etc/redirect/xcode/ios/e808aa/documentation/DeveloperTools/Conceptual/InstrumentsUserGuide/UsingtheAutomationInstrument/UsingtheAutomationInstrument.html#//apple_ref/doc/uid/TP40004652-CH20