Rx Extensions - Proper way to use delay to avoid unnecessary observables from executing? - coffeescript

I'm trying to use delay and amb to execute a sequence of the same task separated by time.
All I want is for a download attempt to execute some time in the future only if the same task failed before in the past. Here's how I have things set up, but unlike what I'd expect, all three downloads seem to execute without delay.
Observable.amb([
Observable.catch(redditPageStream, Observable.empty()).delay(0 * 1000),
Observable.catch(redditPageStream, Observable.empty()).delay(30 * 1000),
Observable.catch(redditPageStream, Observable.empty()).delay(90 * 1000),
# Observable.throw(new Error('Failed to retrieve reddit page content')).delay(10000)
# Observable.create(
# (observer) ->
# throw new Error('Failed to retrieve reddit page content')
# )
]).defaultIfEmpty(Observable.throw(new Error('Failed to retrieve reddit page content')))
full code can be found here. src
I was hoping that the first successful observable would cancel out the ones still in delay.
Thanks for any help.

delay doesn't actually stop the execution of what ever you are doing it just delays when the events are propagated. If you want to delay execution you would need to do something like:
redditPageStream.delaySubscription(1000)
Since your source is producing immediately the above will delay the actual subscription to the underlying stream to effectively delay when it begins producing.
I would suggest though that you use one of the retry operators to handle your retry logic though rather than rolling your own through the amb operator.
redditPageStream.delaySubscription(1000).retry(3);
will give you a constant retry delay however if you want to implement the linear backoff approach you can use the retryWhen() operator instead which will let you apply whatever logic you want to the backoff.
redditPageStream.retryWhen(errors => {
return errors
//Only take 3 errors
.take(3)
//Use timer to implement a linear back off and flatten it
.flatMap((e, i) => Rx.Observable.timer(i * 30 * 1000));
});
Essentially retryWhen will create an Observable of errors, each event that makes it through is treated as a retry attempt. If you error or complete the stream then it will stop retrying.

Related

How to trigger handle_info due to timeout in erlang?

I am using a gen_server behaviour and trying to understand how can handle_info/2 be triggered from a timeout occurring in a handle_call for example:
-module(server).
-export([init/1,handle_call/3,handle_info/2,terminate/2).
-export([start/0,stop/0]).
init(Data)->
{ok,33}.
start()->
gen_server:start_link(?MODULE,?MODULE,[]).
stop(Pid)->
gen_server:stop(Pid).
handle_call(Request,From,State)->
Return={reply,State,State,5000},
Return.
handle_info(Request,State)->
{stop,Reason,State}.
terminate(Reason,State)->
{ok,S}=file:file_open("D:/Erlang/Supervisor/err.txt",[read,write]),
io:format(S,"~s~n",[Reason]),
ok.
What i want to do:
I was expecting that if I launch the server and would not use gen_server:call/2 for 5 seconds (in my case) then handle_info would be called, which would in turn issue the stop thus calling terminate.
I see it does not happen this way, in fact handle_info is not called at all.
In examples such as this i see the timeout is set in the return of init/1.What I can deduce is that it handle_info gets triggered only if I initialize the server and issue nothing (nor cast nor call for N seconds).If so why I can provide Timeout in the return of both handle_cast/2 and handle_call/3 ?
Update:
I was trying to get the following functionality:
If no call is issued in X seconds trigger handle_info/2
If no cast is issued in Y seconds trigger handle_info/2
I thought this timeouts can be set in the return of handle_call and handle_cast:
{reply,Reply,State,X} //for call
{noreply,State,Y} //for cast
If not, when are those timeouts triggered since they are returns?
To initiate timeout handling from gen_server:handle_call/3 callback, this callback has to be called in the first place. Your Return={reply,State,State,5000}, is not executed at all.
Instead, if you want to “launch the server and would not use gen_server:call/2 for 5 seconds then handle_info/2 would be called”, you might return {ok,State,Timeout} tuple from gen_server:init/1 callback.
init(Data)->
{ok,33,5000}.
You cannot set the different timeouts for different calls and casts. As stated by Alexey Romanov in comments,
Having different timeouts for different types of messages just isn’t something any gen_* behavior does and would have to be simulated by maintaining them inside state.
If one returns {reply,State,Timeout} tuple from any handle_call/3/handle_cast/2, the timeout will be triggered if the mailbox of this process is empty after Timeout.
i suggest you read source code:gen_server.erl
% gen_server.erl
% line 400
loop(Parent, Name, State, Mod, Time, HibernateAfterTimeout, Debug) ->
Msg = receive
Input ->
Input
after Time ->
timeout
end,
decode_msg(Msg, Parent, Name, State, Mod, Time, HibernateAfterTimeout, Debug, false).
it helps you to understand the parameter Timeout

(Laravel 5) Monitor and optionally cancel an ALREADY RUNNING job on queue

I need to achieve the ability to monitor and be able to cancel an ALREADY RUNNING job on queue.
There's a lot of answers about deleting QUEUED jobs, but not on an already running one.
This is the situation: I have a "job", which consists of HUNDREDS OF THOUSANDS rows on a database, that need to be queried ONE BY ONE against a web service.
Every row needs to be picked up, queried against a web service, stored the response and its status updated.
I had that already working as a Command (launching from / outputting to console), but now I need to implement queues in order to allow piling up more jobs from more users.
So far I've seen Horizon (which doesn't runs on Windows due to missing process control libs). However, in some demos seen around it lacks (I believe) a couple things I need:
Dynamically configurable timeout (the whole job may take more than 12 hours, depending on the number of rows to process on the selected job)
Ability to CANCEL an ALREADY RUNNING job.
I also considered the option to generate EACH REQUEST as a new job instead of seeing a "job" as the whole collection of rows (this would overcome the timeout thing), but that would give me a Horizon "pending jobs" list of hundreds of thousands of records per job, and that would kill the browser (I know Redis can handle this without itching at all). Further, I guess is not possible to cancel "all jobs belonging to X tag".
I've been thinking about hitting an API route, fire the job and decouple it from the app, but I'm seeing that this requires forking processes.
For the ability to cancel, I would implement a database with job_id, and when the user hits an API to cancel a job, I'd mark it as "halted". On every loop I would check its status and if it finds "halted" then kill itself.
If I've missed any aspect just holler and I'll add it or clarify about it.
So I'm asking for an advice here since I'm new to Laravel: how could I achieve this?
So I finally came up with this (a bit clunky) solution:
In Controller:
public function cancelJob()
{
$jobs = DB::table('jobs')->get();
# I could use a specific ID and user owner filter, etc.
foreach ($jobs as $job) {
DB::table('jobs')->delete($job->id);
}
# This is a file that... well, it's self explaining
touch(base_path(config('files.halt_process_signal')));
return "Job cancelled - It will stop soon";
}
In job class (inside model::chunk() function)
# CHECK FOR HALT SIGNAL AND [OPTIONALLY] STOP THE PROCESS
if ($this->service->shouldHaltProcess()) {
# build stats, do some cleanup, log, etc...
$this->halted = true;
$this->service->stopProcess();
# This FALSE is what it makes the chunk() method to stop looping
return false;
}
In service class:
/**
* Checks the existence of the 'Halt Process Signal' file
*
* #return bool
*/
public function shouldHaltProcess() :bool
{
return file_exists($this->config['files.halt_process_signal']);
}
/**
* Stop the batch process
*
* #return void
*/
public function stopProcess() :void
{
logger()->info("=== HALT PROCESS SIGNAL FOUND - STOPPING THE PROCESS ===");
$this->deleteHaltProcessSignalFile();
return ;
}
It doesn't looks quite elegant, but it works.
I've surfed the whole web and many goes for Horizon or other tools that doesn't fit my case.
If anyone has a better way to achieve this, it's welcome to share.
Laravel queue have 3 important config:
1. retry_after
2. timeout
3. tries
See more: https://laravel.com/docs/5.8/queues
Dynamically configurable timeout (the whole job may take more than 12
hours, depending on the number of rows to process on the selected job)
I think you can config timeout + retry_after about 24h.
Ability to CANCEL an ALREADY RUNNING job.
Delete job in jobs table
Delete process by process id in your server
Hope it help you :)

How to make Protractor's browser.wait() more verbose?

In Protractor tests I call many times browser.wait method for example to wait once the particular element will appear on the screen or it will be clickable.
In many cases tests passes on my local machine, but does not on other.
I receive very generic information about the timeout which doesn't help me a lot to debug / find a source of issue.
Is it possible to make a browser.wait more verbose, for example:
if at least defaultTimeoutInterval will elapse when waiting for particular element, will it be possible to console.log information about the element that it tried to wait for,
take a screenshot when the timeout error occurs,
provide full call stack when timeout appears in browser.wait
If the main issue is that you don't know for which element the wait timed out, I would suggest writing a helper function for wait and use it instead of wait, something like:
wait = function(variable, variableName,waitingTime){
console.log('Waiting for ' + variableName);
browser.wait(protractor.ExpectedConditions.elementToBeClickable(variablename),waitingTime);
console.log('Success');
}
Because protractor stops executing test after first fail, if wait timed out, console won't print success message after failing to load a certain element.
For screenshots I suggest trying out protractor-jasmine2-screenshot-reporter, it generates an easily readable html report with screenshots and debug information on failed tests (for example, in which code line the failure occured).
Look into using protractor's Expected Condition, you can specify what to wait for and how long to wait for it.
For screenshots there are npm modules out there that can take a screenshot when a test fails. This might help.
browser.wait returns a promise, so catch the error and print/throw something meaningful like:
await browser.wait(ExpectedConditions.visibilityOf(css), waitingTime).catch((error) =>
{
throw new CustomError(`Could not find ${css} ${error.message}`)
});

Offer to queue with some initial display

I want to offer to queue a string sent in load request after some initial delay say 10 seconds.
If the subsequent request is made with some short interval delay(1 second) then everything works fine, but if it is made continuously like from a script then there is no delay.
Here is the sample code.
def load(randomStr :String) = Action { implicit request =>
Source.single(randomStr)
.delay(10 seconds, DelayOverflowStrategy.backpressure)
.map(x =>{
println(x)
queue.offer(x)
})
.runWith(Sink.ignore)
Ok("")
}
I am not entirely sure that this is the correct way of doing what you want. There are some things you need to reconsider:
a delayed source has an initial buffer capacity of 16 elements. You can increase this with addAttributes(initialBuffer)
In your case the buffer cannot actually become full because every time you provide just one element.
Who is the caller of the Action? You are defining a DelayOverflowStrategy.backpressure strategy but is the caller able to handle this?
On every call of the action you are creating a Stream consisting of one element, how is the backpressure here helping? It is applied on the stream processing and not on the offering to the queue

Proper way to stop Akka Streams on condition

I have been successfully using FileIO to stream the contents of a file, compute some transformations for each line and aggregate/reduce the results.
Now I have a pretty specific use case, where I would like to stop the stream when a condition is reached, so that it is not necessary to read the whole file but the process finishes as soon as possible. What is the recommended way to achieve this?
If the stop condition is "on the outside of the stream"
There is a advanced building-block called KillSwitch that you could use to do this: http://doc.akka.io/japi/akka/2.4.7/akka/stream/KillSwitches.html The stream would get shut down once the kill switch is notified.
It has methods like abort(reason) / shutdown etc, see here for it's API: http://doc.akka.io/japi/akka/2.4.7/akka/stream/SharedKillSwitch.html
Reference documentation is here: http://doc.akka.io/docs/akka/2.4.8/scala/stream/stream-dynamic.html#kill-switch-scala
Example usage would be:
val countingSrc = Source(Stream.from(1)).delay(1.second,
DelayOverflowStrategy.backpressure)
val lastSnk = Sink.last[Int]
val (killSwitch, last) = countingSrc
.viaMat(KillSwitches.single)(Keep.right)
.toMat(lastSnk)(Keep.both)
.run()
doSomethingElse()
killSwitch.shutdown()
Await.result(last, 1.second) shouldBe 2
If the stop condition is inside the stream
You can use takeWhile to express any condition really, though sometimes take or limit may be also enough "take 10 lnes".
If your logic is very advanced, you could build a special stage that handles that special logic using statefulMapConcat that allows to express literally anything - so you could complete the stream whenever you want to "from the inside".