I'm using an Event var watcher to implement an internal queue. When the producer thread adds something to the queue (just an array) it will change the value of a watched variable to signal that an element was added.
How can you do the same with AnyEvent? It doesn't seem to support variable watching. Do I have to use pipes and use an IO watcher (i.e. the producer writes a byte on one end of the pipe when it has added an element.)
I'd also be interested to know how to do this with Coro.
It sounds as if you are using variable watching as means of transferring control back to the consumer. In AnyEvent, this can be done with condition variables by calling $cv->send() from the producer and $cv->recv() in the consumer. You could consider send()ing the item that you'd otherwise have put in the queue, but calling send without parameters should be an allowed way of notifying the consumer.
I figured out the paradigm to use:
my #queue;
my $queue_watcher;
sub add_item {
push(#queue, $_[0]);
$queue_watcher ||= AnyEvent->timer(after => 0, cb => \&process_queue);
}
sub process_queue {
... # remove zero or more elements from #queue
if (#queue) {
$queue_watcher = AnyEvent->timer(after => 0, cb => \&process_queue);
} else {
undef $queue_watcher;
}
}
Basically $queue_watcher is defined and active if and only if #queue is not empty.
Related
I have paged interface. Given a starting point a request will produce a list of results and a continuation indicator.
I've created an observable that is built by constructing and flat mapping an observable that reads the page. The result of this observable contains both the data for the page and a value to continue with. I pluck the data and flat map it to the subscriber. Producing a stream of values.
To handle the paging I've created a subject for the next page values. It's seeded with an initial value then each time I receive a response with a valid next page I push to the pages subject and trigger another read until such time as there is no more to read.
Is there a more idiomatic way of doing this?
function records(start = 'LATEST', limit = 1000) {
let pages = new rx.Subject();
this.connect(start)
.subscribe(page => pages.onNext(page));
let records = pages
.flatMap(page => {
return this.read(page, limit)
.doOnNext(result => {
let next = result.next;
if (next === undefined) {
pages.onCompleted();
} else {
pages.onNext(next);
}
});
})
.pluck('data')
.flatMap(data => data);
return records;
}
That's a reasonable way to do it. It has a couple of potential flaws in it (that may or may not impact you depending upon your use case):
You provide no way to observe any errors that occur in this.connect(start)
Your observable is effectively hot. If the caller does not immediately subscribe to the observable (perhaps they store it and subscribe later), then they'll miss the completion of this.connect(start) and the observable will appear to never produce anything.
You provide no way to unsubscribe from the initial connect call if the caller changes its mind and unsubscribes early. Not a real big deal, but usually when one constructs an observable, one should try to chain the disposables together so it call cleans up properly if the caller unsubscribes.
Here's a modified version:
It passes errors from this.connect to the observer.
It uses Observable.create to create a cold observable that only starts is business when the caller actually subscribes so there is no chance of missing the initial page value and stalling the stream.
It combines the this.connect subscription disposable with the overall subscription disposable
Code:
function records(start = 'LATEST', limit = 1000) {
return Rx.Observable.create(observer => {
let pages = new Rx.Subject();
let connectSub = new Rx.SingleAssignmentDisposable();
let resultsSub = new Rx.SingleAssignmentDisposable();
let sub = new Rx.CompositeDisposable(connectSub, resultsSub);
// Make sure we subscribe to pages before we issue this.connect()
// just in case this.connect() finishes synchronously (possible if it caches values or something?)
let results = pages
.flatMap(page => this.read(page, limit))
.doOnNext(r => this.next !== undefined ? pages.onNext(this.next) : pages.onCompleted())
.flatMap(r => r.data);
resultsSub.setDisposable(results.subscribe(observer));
// now query the first page
connectSub.setDisposable(this.connect(start)
.subscribe(p => pages.onNext(p), e => observer.onError(e)));
return sub;
});
}
Note: I've not used the ES6 syntax before, so hopefully I didn't mess anything up here.
Here is the sample code flow:
class FSMActor{
when(Idle) {
case Event(Start, Uninitialized) =>
case Event(InitMap(inMap), t # EvaluteRuleMap(v, c)) =>
logger.info(s"State = $stateName, Event = Event(_, InitMap(inMap))")
goto(EVALRule) using t.copy(ruleMap = inMap)
}
when(EVALRule) {
case Event(RowMap(m), t # EvaluteRuleMap(v, c)) =>
logger.debug("input row map m " + m)
**if ( <somecondition> ) { // If comment this if-else block, I could see rowMaps being received.
logger.debug(s"Moving to State Trigger x=$x")
goto(TriggerRule) using t.copy(ruleMap = x.get)
} else {
logger.debug(s"staying in EVALRule, x = $x")
stay
}**
}
when(TriggerRule) {
case Event(_, _) => ....
}
}
}
When the control in in "EVALRule" state, It will keep receiving streams maps(RowMap) and based on some computation, it moves to trigger rule.
Unfortunately for some weird reason, some of the incoming RowMaps are not being received at "case Event(RowMap(m), t # EvaluteRuleMap(v, c)) =>" and
If I comment the code bock (bolded in the above code) then I could see all incoming rowmaps being received.
Could anyone let me know why is so? I've been trying to find the cause but couldn't get it to.
Appreciate your help, thanks.
When if ( <somecondition> ) is true, you are moving to the TriggerRule state. In that state you are looking for messages of type EVENT instead of Event (all caps). So the message is not handled by the FSM.
In general, when missing messages in FSM, the best way to debug is to write a whenUnhandled block with a log/print statement to see what messages are not handled by the states you have defined.
There was some issue with message handling in the code itself, we debugged it and fixed the issue, now its working seamlessly.
I have never setup up a queueing system before. I decided to give it a shot. It seems the queueing system is working perfectly. However, it doesn't seem the data is being sent correctly. Here is my code.
...
$comment = new Comment(Input::all());
$comment->user_id = $user->id;
$comment->save();
if ($comment->isSaved())
{
$voters = $comment->argument->voters->unique()->toArray();
Queue::push('Queues\NewComment',
[
'comment' => $comment->load('argument', 'user')->toArray(),
'voters' => $voters
]
);
return Response::json(['success' => true, 'comment' => $comment->load('user')->toArray()]);
}
...
The class that handles this looks like this:
class NewComment {
public function fire($job, $data)
{
$comment = $data['comment'];
$voters = $data['voters'];
Log::info($data);
foreach ($voters as $voter)
{
if ($voter['id'] != $comment['user_id'])
{
$mailer = new NewCommentMailer($voter, $comment);
$mailer->send();
}
}
$job->delete();
}
}
This works beautifully on my local server using the synchronous queue driver. However, on my production server, I'm using Beanstalkd. The queue is firing like it is supposed to. However, I'm getting an error like this:
[2013-12-19 10:25:02] production.ERROR: exception 'ErrorException' with message 'Undefined index: voters' in /var/www/mywebsite/app/queues/NewComment.php:10
If I print out the $data variable passed into the NewComment queue handler, I get this:
[2013-12-19 10:28:05] production.INFO: {"comment":{"incrementing":true,"timestamps":true,"exists":true}} [] []
I have no clue why this is happening. Anyone have an idea how to fix this.
So $voters apparently isn't being put into the queue as part of the payload. I'd build the payload array outside of the Queue::push() function, log the contents, and see exactly what is being put in.
I've found if you aren't getting something out that you expect, chances are, it's not being put in like you expect either.
While you are at it, make sure that the beanstalkd system hasn't got old data stuck in it that is incorrect. You could add a timestamp into the payload to help make sure it's the latest data, and arrange to delete or bury any jobs that don't have the appropriate information - checked before you start to process them. Just looking at a count of items in the beanstalkd tubes should make it plain if there are stuck jobs.
I've not done anything with Laravel, but I have written many tasks for other Beanstalkd and SQS-backed systems, and the hard part is when the job fails, and you have to figure out what went wrong, and how to avoid just redoing the same failure over and over again.
What I ended up doing was sticking with simple numbers. I only stored the comment's ID on the queue and then did all the processing in my queue handler class. That was the easiest way to do it.
You will get data as expected in the handler by wrapping the data in an array:
array(
array('comment' => $comment->load('argument', 'user')->toArray(),
'voters' => $voters
)
)
I have a bunch of events coming in and I have to execute ALL of them without a loss, but I want to make sure that they are buffered and consumed at the appropriate time slots. Anyone have a solution?
I can't find any operators in Rx that can do that without the loss of the events (Throttle - looses events). I've also considered Buffered, Delay, etc... Can't find a good solution.
I've tried to put a timer in the middle, but somehow it doesn't work at all:
GetInitSequence()
.IntervalThrottle(TimeSpan.FromSeconds(5))
.Subscribe(
item =>
{
Console.WriteLine(DateTime.Now);
// Process item
}
);
public static IObservable<T> IntervalThrottle<T>(this IObservable<T> source, TimeSpan dueTime)
{
return Observable.Create<T>(o =>
{
return source.Subscribe(x =>
{
new Timer(state =>
o.OnNext((T)state), x, dueTime, TimeSpan.FromMilliseconds(-1));
}, o.OnError, o.OnCompleted);
});
}
The question is not 100% clear so I'm making some presumptions.
Observable.Delay is not what you want because that will create a delay from when each event arrives, rather than creating even time intervals for processing.
Observable.Buffer is not what you want because that will cause all events in each given interval to be passed to you, rather than one at a time.
So I believe you're looking for a solution that creates some sort of metronome that ticks away, and gives you an event per tick. This can be naively constructed using Observable.Interval for the metronome and Zip for connecting it to your source:
var source = GetInitSequence();
var trigger = Observable.Interval(TimeSpan.FromSeconds(5));
var triggeredSource = source.Zip(trigger, (s,_) => s);
triggeredSource.Subscribe(item => Console.WriteLine(DateTime.Now));
This will trigger every 5 seconds (in the example above), and give you the original items in sequence.
The only problem with this solution is that if you don't have any more source elements for (say) 10 seconds, when the source elements arrive they will be immediately sent out since some of the 'trigger' events are sitting there waiting for them. Marble diagram for that scenario:
source: -a-b-c----------------------d-e-f-g
trigger: ----o----o----o----o----o----o----o
result: ----a----b----c-------------d-e-f-g
This is a very reasonable issue. There are two questions here already that tackle it:
Rx IObservable buffering to smooth out bursts of events
A way to push buffered events in even intervals
The solution provided is a main Drain extension method and secondary Buffered extension. I've modified these to be far simpler (no need for Drain, just use Concat). Usage is:
var bufferedSource = source.StepInterval(TimeSpan.FromSeconds(5));
The extension method StepInterval:
public static IObservable<T> StepInterval<T>(this IObservable<T> source, TimeSpan minDelay)
{
return source.Select(x =>
Observable.Empty<T>()
.Delay(minDelay)
.StartWith(x)
).Concat();
}
I know this could just be too simple, but would this work?
var intervaled = source.Do(x => { Thread.Sleep(100); });
Basically this just puts a minimum delay between values. Too simplistic?
Along the lines of Enigmativity's answer, if all you want to do is just Delay all of the values by a TimeSpan, I cant see why Delay is not the operator you want
GetInitSequence()
.Delay(TimeSpan.FromSeconds(5)) //ideally pass an IScheduler here
.Subscribe(
item =>
{
Console.WriteLine(DateTime.Now);
// Process item
}
);
How about Observable.Buffer? This should return all the events in the 1s window as a single event.
var xs = Observable.Interval(TimeSpan.FromMilliseconds(100));
var bufferdStream = xs.Buffer(TimeSpan.FromSeconds(5));
bufferdStream.Subscribe(item => { Console.WriteLine("Number of events in window: {0}", item.Count); });
It might be what you're asking isnt that clear. What is your code supposed to do? It looks like you're just delaying by creating a timer for each event. It also breaks the semantics of the observable as the next and complete could occur before the next.
Note this is also only as accurate at the timer used. Typically the timers are accurate to at most 16ms.
Edit:
your example becomes, and item contains all the events in the window:
GetInitSequence()
.Buffer(TimeSpan.FromSeconds(5))
.Subscribe(
item =>
{
Console.WriteLine(DateTime.Now);
// Process item
}
);
Using reactive extension, it is easy to subscribe 2 times to the same observable.
When a new value is available in the observable, both subscribers are called with this same value.
Is there a way to have each subscriber get a different value (the next one) from this observable ?
Ex of what i'm after:
source sequence: [1,2,3,4,5,...] (infinite)
The source is constantly adding new items at an unknown rate.
I'm trying to execute a lenghty async action for each item using N subscribers.
1st subscriber: 1,2,4,...
2nd subscriber: 3,5,...
...
or
1st subscriber: 1,3,...
2nd subscriber: 2,4,5,...
...
or
1st subscriber: 1,3,5,...
2nd subscriber: 2,4,6,...
I would agree with Asti.
You could use Rx to populate a Queue (Blocking Collection) and then have competing consumers read from the queue. This way if one process was for some reason faster it could pick up the next item potentially before the other consumer if it was still busy.
However, if you want to do it, against good advice :), then you could just use the Select operator that will provide you with the index of each element. You can then pass that down to your subscribers and they can fiter on a modulus. (Yuck! Leaky abstractions, magic numbers, potentially blocking, potentiall side effects to the source sequence etc)
var source = Obserservable.Interval(1.Seconds())
.Select((i,element)=>{new Index=i, Element=element});
var subscription1 = source.Where(x=>x.Index%2==0).Subscribe(x=>DoWithThing1(x.Element));
var subscription2 = source.Where(x=>x.Index%2==1).Subscribe(x=>DoWithThing2(x.Element));
Also remember that the work done on the OnNext handler if it is blocking will still block the scheduler that it is on. This could affect the speed of your source/producer. Another reason why Asti's answer is a better option.
Ask if that is not clear :-)
How about:
IObservable<TRet> SomeLengthyOperation(T input)
{
return Observable.Defer(() => Observable.Start(() => {
return someCalculatedValueThatTookALongTime;
}, Scheduler.TaskPoolScheduler));
}
someObservableSource
.SelectMany(x => SomeLengthyOperation(input))
.Subscribe(x => Console.WriteLine("The result was {0}", x);
You can even limit the number of concurrent operations:
someObservableSource
.Select(x => SomeLengthyOperation(input))
.Merge(4 /* at a time */)
.Subscribe(x => Console.WriteLine("The result was {0}", x);
It's important for the Merge(4) to work, that the Observable returned by SomeLengthyOperation be a Cold Observable, which is what the Defer does here - it makes the Observable.Start not happen until someone Subscribes.