Rx how to merge observables and only fire if all provided an onnext in a specified timeframe - system.reactive

I have a number of observable event streams that are all providing events with timestamps. I don't care about the events individually, I need to know when they all fired within a specified timeframe.
For example:
Button one was clicked (don't care)
Button two was clicked (don't care)
Button one was clicked and within 5 seconds button two was clicked (I need this)
I tried "and then when" but I get old events and can't figure out how to filter them out if it is not within the time window.
Thanks!
Edit:
I attempted to create a marble diagram to clarify what I am trying to achieve...
I have a bunch of random event streams represented in the top portion. Some events fire more often then others. I only want to capture the group of events that fired within a specified time window. In this example I used windows of 3 seconds. The events I want are highlighted in dark black all other events should be ignored. I hope this helps better explain the problem.

Is sequence important? If so, the usual way of dealing with "this and then that" is using flatMapLatest. You can achieve your other constraints by applying them to stream passed to flatMapLatest. Consider the following example:
const fromClick = x => Rx.Observable.fromEvent(document.getElementById(x), 'click');
const a$ = fromClick('button1');
const b$ = fromClick('button2');
const sub = a$.flatMapLatest(() => b$.first().timeout(5000, Rx.Observable.never()))
.subscribe(() => console.log('condition met'));
Here we're saying "for each click on button 1, start listening to button 2, and return either the first click or nothing (if we hit the timeout). Here's a working example: https://jsbin.com/zosipociti/edit?js,console,output

You want to use the Sample operator. I'm not sure if you want .NET or JS sample code, since you tagged both.
Edit:
Here's a .NET sample. metaSample is an observable of 10 child observables. Each of the child observables has numbers going from 1 to 99 with random time-gaps between each number. The time gaps are anywhere between 0 to 200 milliseconds.
var random = new Random();
IObservable<IObservable<int>> metaSample = Observable.Generate(1, i => i < 10, i => i + 1, i =>
Observable.Generate(1, j => j < 100, j => j + 1, j => j, j => TimeSpan.FromMilliseconds(random.Next(200))));
We then Sample each of the child operators every one second. This gives us the latest value that occurred in that one second window. We then merge those sampled streams together:
IObservable<int> joined = metaSample
.Select(o => o.Sample(TimeSpan.FromSeconds(1)))
.Merge();
A marble diagram for 5 of them could look like this:
child1: --1----2--3-4---5----6
child2: -1-23---4--5----6-7--8
child3: --1----2----3-4-5--6--
child4: ----1-2--34---567--8-9
child5: 1----2--3-4-5--------6-
t : ------1------2------3-
------------------------------
result: ------13122--45345--5768--
So the after 1 second, it grabs the latest from each child and dumps it, after 2 seconds the same. After 3 seconds, notice that child5 hasn't emitted anything, so there's only 4 numbers emitted. Obviously with our parameters that's impossible, but that's demonstrated as how Sample would work with no events.

This is the closest I have come to accomplishing this task... There has to be a cleaner way with groupjoin but I can't figure it out!
static void Main(string[] args)
{
var random = new Random();
var o1 = Observable.Interval(TimeSpan.FromSeconds(2)).Select(t => "A " + DateTime.Now.ToString("HH:mm:ss"));
o1.Subscribe(Console.WriteLine);
var o2 = Observable.Interval(TimeSpan.FromSeconds(3)).Select(t => "B " + DateTime.Now.ToString("HH:mm:ss"));
o2.Subscribe(Console.WriteLine);
var o3 = Observable.Interval(TimeSpan.FromSeconds(random.Next(3, 7))).Select(t => "C " + DateTime.Now.ToString("HH:mm:ss"));
o3.Subscribe(Console.WriteLine);
var o4 = Observable.Interval(TimeSpan.FromSeconds(random.Next(5, 10))).Select(t => "D " + DateTime.Now.ToString("HH:mm:ss"));
o4.Subscribe(Console.WriteLine);
var joined = o1
.CombineLatest(o2, (s1, s2)=> new { e1 = s1, e2 = s2})
.CombineLatest(o3, (s1, s2) => new { e1 = s1.e1, e2 = s1.e2, e3 = s2 })
.CombineLatest(o4, (s1, s2) => new { e1 = s1.e1, e2 = s1.e2, e3 = s1.e3, e4 = s2 })
.Sample(TimeSpan.FromSeconds(3));
joined.Subscribe(e => Console.WriteLine($"{DateTime.Now}: {e.e1} - {e.e2} - {e.e3} - {e.e4}"));
Console.ReadLine();
}

Related

How to move the x-axis position using dc.js?

I have this:
But I want the x axis to run along y=0, e.g.
And ideally I'd either have the tick labels on top i.e. in the blue bit, or where they were at the bottom of the chart.
EDIT: how the chart is created.
I'm using something like:
var
ndx = crossfilter(data),
dataDimension = ndx.dimension(d => d.period),
ordinals = data.map(d => d.period),
lossGroup = dataDimension.group().reduceSum(d => d.loss),
offsets = lossGroup.all().map(d => -d.value),
chart;
// The data is like {
// period: Date (start of period, e.g. month),
// start: Integer (number at start of period/end of last period)
// loss: Integer (number lost during period)
// gain: Integer (number gained during period)
// }
chart = dc.barChart(chartElement)
.dimension(dataDimension)
// The first group is the loss
.group(lossGroup)
.stack(dataDimension.group().reduceSum(d => d.start), 'carryforward')
.stack(dataDimension.group().reduceSum(d => d.gain), 'gain')
.stackLayout(d3.layout.stack().offset(layers => offsets))
.x(d3.scale.ordinal(ordinals).domain(ordinals))
.xUnits(dc.units.ordinal)
.elasticY(true)
.renderLabel(false)
// The first group is the loss
.title(item => 'Loss: ' + item.value)
.title('carryforward', item => 'Sustained: ' + item.value)
.title('gain', item => 'Gain: ' + item.value)
.renderTitle(true);
dc.renderAll();
Hmm, I guess you are using .stackLayout() to get the negative stacking. Since dc.js doesn't “look inside” this setting, I don't think there is anything built-in to offset the axis. You would probably need to use a pretransition handler to move it.
As for moving the tick labels, you could use .title() instead, like in this example. And then set the tick label to empty.
Best I can think of, not really an answer but more than a comment. :-)

Variable not updated

This is perhaps very basic but I am certainly missing something here.
It is all within a method, where I increment/alter a variable within a child/sub scope (it could be within a if block, or like here, within a map)
However, the result is unchanged variable. e.g. here, sum remains zero after the map. whereas, it should come up to 3L.
What am I missing ?
val resultsMap = scala.collection.mutable.Map.empty[String, Long]
resultsMap("0001") = 0L
resultsMap("0003") = 2L
resultsMap("0007") = 1L
var sum = 0L
resultsMap.mapValues(x => {sum = sum + x})
// I first wrote this, but then got worried and wrote more explicit version too, same behaviour
// resultMap.mapValues(sum+=_)
println("total of counts for txn ="+sum) // sum still 0
-- Update
I have similar behaviour where a loop is not updating the variable outside the loop. looking for text on variable scoping, but not found the golden source yet. all help is appreciated.
var cnt : Int = 0
rdd.foreach(article => {
if (<something>) {
println(<something>) // being printed
cnt += 1
println("counter is now "+cnt) // printed correctly
}
})
You should proceed like this:
val sum = resultsMap.values.reduce(_+_)
You just get your values and then add them up with reduce.
EDIT:
The reason sum stays unchanged is that mapValues produces a view, which means (among other things) the new map won't be computed unless the resulting view is acted upon, so in this case - the code block updating sum is simply never executed.
To see this - you can "force" the view to "materialize" (compute the new map) and see that sum is updated, as expected:
var sum = 0L
resultsMap.mapValues(x => {sum = sum + x}).view.force
println("SUM: " + sum) // prints 3
See related discussion here: Scala: Why mapValues produces a view and is there any stable alternatives?

RX and buffering

I'm trying to obtain the following observable (with a buffer capacity of 10 ticks):
Time 0 5 10 15 20 25 30 35 40
|----|----|----|----|----|----|----|----|
Source A B C D E F G H
Result A E H
B F
C G
D
Phase |<------->|-------|<------->|<------->|
B I B B
That is, the behavior is very similar to the Buffer observable with the difference that the buffering phase is not in precise time slot, but starts at the first symbol pushed in the idle phase. I mean, in the example above the buffering phases start with the 'A', 'E', and 'H' symbols.
Is there a way to compose the observable or do I have to implement it from scratch?
Any help will be appreciated.
Try this:
IObservable<T> source = ...;
IScheduler scheduler = ...;
IObservable<IList<T>> query = source
.Publish(obs => obs
.Buffer(() => obs.Take(1).IgnoreElements()
.Concat(Observable.Return(default(T)).Delay(duration, scheduler))
.Amb(obs.IgnoreElements())));
The buffer closing selector is called once at the start and then once whenever a buffer closes. The selector says "The buffer being started now should be closed duration after the first element of this buffer, or when the source completes, whichever occurs first."
Edit: Based on your comments, if you want to make multiple subscriptions to query share a single subscription to source, you can do that by appending .Publish().RefCount() to the query.
IObservable<IList<T>> query = source
.Publish(obs => obs
.Buffer(() => obs.Take(1).IgnoreElements()
.Concat(Observable.Return(default(T)).Delay(duration, scheduler))
.Amb(obs.IgnoreElements())));
.Publish()
.RefCount();

Consolidating a data table in Scala

I am working on a small data analysis tool, and practicing/learning Scala in the process. However I got stuck at a small problem.
Assume data of type:
X Gr1 x_11 ... x_1n
X Gr2 x_21 ... x_2n
..
X GrK x_k1 ... x_kn
Y Gr1 y_11 ... y_1n
Y Gr3 y_31 ... y_3n
..
Y Gr(K-1) ...
Here I have entries (X,Y...) that may or may not exist in up to K groups, with a series of values for each group. What I want to do is pretty simple (in theory), I would like to consolidate the rows that belong to the same "entity" in different groups. so instead of multiple lines that start with X, I want to have one row with all values from x_11 to x_kn in columns.
What makes things complicated however is that not all entities exist in all groups. So wherever there's "missing data" I would like to pad with for instance zeroes, or some string that denotes a missing value. So if I have (X,Y,Z) in up to 3 groups, the type I table I want to have is as follows:
X x_11 x_12 x_21 x_22 x_31 x_32
Y y_11 y_12 N/A N/A y_31 y_32
Z N/A N/A z_21 z_22 N/A N/A
I have been stuck trying to figure this out, is there a smart way to use List functions to solve this?
I wrote this simple loop:
for {
(id, hitlist) <- hits.groupBy(_.acc)
h <- hitlist
} println(id + "\t" + h.sampleId + "\t" + h.ratios.mkString("\t"))
to able to generate the tables that look like the example above. Note that, my original data is of a different format and layout,but that has little to do with the problem at hand, thus I have skipped all steps regarding parsing. I should be able to use groupBy in a better way that actually solves this for me, but I can't seem to get there.
Then I modified my loop mapping the hits to ratios and appending them to one another:
for ((id, hitlist) <- hits.groupBy(_.acc)){
val l = hitlist.map(_.ratios).foldRight(List[Double]()){
(l1: List[Double], l2: List[Double]) => l1 ::: l2
}
println(id + "\t" + l.mkString("\t"))
//println(id + "\t" + h.sampleId + "\t" + h.ratios.mkString("\t"))
}
That gets me one step closer but still no cigar! Instead of a fully padded "matrix" I get a jagged table. Taking the example above:
X x_11 x_12 x_21 x_22 x_31 x_32
Y y_11 y_12 y_31 y_32
Z z_21 z_22
Any ideas as to how I can pad the table so that values from respective groups are aligned with one another? I should be able to use _.sampleId, which holds the "group membersip" for each "hit", but I am not sure how exactly. ´hits´ is a List of type Hit which is practically a wrapper for each row, giving convenience methods for getting individual values, so essentially a tuple which have "named indices" (such as .acc, .sampleId..)
(I would like to solve this problem without hardcoding the number of groups, as it might change from case to case)
Thanks!
This is a bit of a contrived example, but I think you can see where this is going:
case class Hit(acc:String, subAcc:String, value:Int)
val hits = List(Hit("X", "x_11", 1), Hit("X", "x_21", 2), Hit("X", "x_31", 3))
val kMax = 4
val nMax = 2
for {
(id, hitlist) <- hits.groupBy(_.acc)
k <- 1 to kMax
n <- 1 to nMax
} yield {
val subId = "x_%s%s".format(k, n)
val row = hitlist.find(h => h.subAcc == subId).getOrElse(Hit(id, subId, 0))
println(row)
}
//Prints
Hit(X,x_11,1)
Hit(X,x_12,0)
Hit(X,x_21,2)
Hit(X,x_22,0)
Hit(X,x_31,3)
Hit(X,x_32,0)
Hit(X,x_41,0)
Hit(X,x_42,0)
If you provide more information on your hits lists then we could probably come with something a little more accurate.
I have managed to solve this problem with the following code, I am putting it here as an answer in case someone else runs into a similar problem and requires some help. The use of find() from Noah's answer was definitely very useful, so do give him a +1 in case this code snippet helps you out.
val samples = hits.groupBy(_.sampleId).keys.toList.sorted
for ((id, hitlist) <- hits.groupBy(_.acc)) {
val ratios =
for (sample <- samples)
yield hitlist.find(h => h.sampleId == sample).map(_.ratios)
.getOrElse(List(Double.NaN, Double.NaN, Double.NaN, Double.NaN, Double.NaN, Double.NaN))
println(id + "\t" + ratios.flatten.mkString("\t"))
}
I figure it's not a very elegant or efficient solution, as I have two calls to groupBy and I would be interested to see better solutions to this problem.

How to switch from one sequence of events to another?

I would like that first event to arrive will cause some work. Later I would like to throttle down work a little bit. Until now I came with the following code:
var events = Observable.FromEventPattern<...>(...);
var throttled = events.Throttle(TimeSpan.FromSeconds(1));
events.Take(1).Subscribe((x) =>
{
DoWork(x);
throttled.Subscribe((y) => DoWork(y);
});
Is there a more elegant way of expressing it?
Apparently it's quite simple:
var events = Observable.FromEventPattern<...>(...);
var throttled = events.Throttle(TimeSpan.FromSeconds(1));
events.Take(1).Concat(throttled).Subscribe((x) => DoWork(x));
Concat will wait for the first sequence to finish and than move subscription to the second.
Another common way is to use select-many. This allows you to pass data from the 1st sequence to the 2nd sequence, it also allows the two sequences to be of different types:
In query comprehension syntax;
var q = from x in xs
from y in ys
select new {x, y};
-or- you can use the standard linq operators as extension methods (losing access to the x value however)
xs.SelectMany(x=>ys)
.Select(y=>y)