Rx.Net - Get stock price changes and process them - system.reactive

Question

The problem I'm trying to solve
Get stock ticks
Always consider latest stock price
Each x second take a snapshot of ticks and send for processing
So I have an Observable source of stock ticks. It sends only the ticks for stocks I'm interested in. What I need to do is to receive these stock prices, and after each x seconds (for the sake of example let's say every 3 seconds) send a snapshot of prices for processing. If within 3 seconds I receive 2 ticks for the same stock, I only need the latest tick. This processing is compute heavy, so if possible I would like to avoid sending same stock price for processing twice.
To bring an example.
Let's say at the beginning of sequence I received 2 ticks -> MSFT:1$, GOOG:2$.
In the next 3 seconds I receive nothing, so MSFT & GOOG ticks should be sent for processing.
Now the next second I receive new ticks -> MSFT:1$, GOOG:3$, INTL:3$
Again let's assume within next 3 seconds nothing comes in.
Here, since MSFT price didn't change (it's still 1$), only GOOG & INTL should be sent for processing.
And this repeats throughout a day.
Now I think Rx helps to solve this kind of problems in easy & elegant way. But I'm having a problem to have the proper queries.
This is what I have so far, will try to explain what it does and what's the issue with it
var finalQuery =
from priceUpdate in **Observable<StockTick>**
group priceUpdate by priceUpdate.Stock into grouped
from combined in Observable.Interval(TimeSpan.FromSeconds(3))
.CombineLatest(grouped, (t, pei) => new { PEI = pei, Interval = t })
group combined by new { combined.Interval } into combined
select new
{
Interval = combined.Key.Interval,
PEI = combined.Select(c => new StockTick(c.PEI.Stock, c.PEI.Price))
};
finalQuery
.SelectMany(combined => combined.PEI)
.Distinct(pu => new { pu.Stock, pu.Price })
.Subscribe(priceUpdate =>
{
Process(priceUpdate);
});
public class StockTick
{
public StockTick(string stock, decimal price)
{
Stock = stock;
Price = price;
}
public string Stock {get;set;}
public decimal Price {get;set;}
}
So this gets the stock price, groups it by stock, then combines latest from this grouped sequence with Observable.Interval. This way I'm trying to ensure only latest ticks for a stock are processed and it fires up every 3 seconds.
Then again it groups it up by interval this time, as a result I have group of sequences for each 3 second intervals that passed.
And as a last step, I flatten this sequence to sequence of stock price updates using SelectMany and also I'm applying Distinct to ensure same price for the same stock is not processed twice.
There are 2 issues with this query I don't like. First is I don't really like double group by's - is there any way to avoid it ? Second - with this approach I have to process prices one by one, what I really would like to have is snapshots - that is within 3 seconds whatever I have I will bundle and send for processing, but can't figure out how to bundle.
I'll be happy for suggestions to solve this problem other way, but I would prefer to stay within Rx, unless there is really something much much better.

Answer 1

A couple things:
You'll want to take advantage of the Sample operator:
You probably want DistinctUntilChanged instead of Distinct. If you use Distinct, then if MSFT goes from $1, to $2 then back to $1, you won't get an event on the third tick.
I imagine your solution will look something like this:
IObservable<StockTick> source;
source
.GroupBy(st => st.Stock)
.Select(stockObservable => stockObservable
.Sample(TimeSpan.FromSeconds(3))
.DistinctUntilChanged(st => st.Price)
)
.Merge()
.Subscribe(st => Process(st));
EDIT (Distinct performance problems):
Each Distinct operator has to maintain within it, the full distinct history. If you have a high-priced stock, for example AMZN, which so far today has ranged from $958-$974, then you could end up with a lot of data. That's ~1600 possible data points that have to sit in memory until you unsubscribe from the Distinct. It also will eventually degrade performance, as each AMZN tick has to be compared to the 1600-ish present data points before going through. If this is for a long-running process (spanning multiple trading days), then you'll end up with even more data points.
Given N stocks, you have N Distinct operators that need to operate accordingly. Multiply that behavior by N stocks, and you have an ever-increasing problem.