I started using StreamInsight and I'm using it as a wcf service.
I've already tried seeking help in "A Hitchhiker's Guide to Microsoft StreamInsight Queries" and tried the examples as well as the examples in codeplex.
My problem is this:
My event producer feeds the adapter with AlertEvent's:
public sealed class AlertEvent
{
public DateTime Date { get; set; }
public long IDGroup { get; set; }
public bool IsToNormalize { get; set; }
public bool IsError { get; set; }
}
When a AlertEvent has IsError = false, the flag IsToNormalize is true;
The behaviour that I'm trying to achieve is when I receive a stream with IsError, I want to see if, in the next 'x' minutes, arrives any alertEvent with IsToNormalize. I then send to output the IsError AlarmEvent that started the search.
What I've done is, when I receive a input that correspond to the filter, I extend its lifetime in 'x' minutes and create a TumblingWindow to see if in that period another AlertEvent arrives with the other flag (using a ExtensionMethod to iterate through all payloads in the window).
var stream= from item in input
where item.IsError
group item by item.IdGroup into eachGroup
from window in eachDigital.AlterEventDuration(e => TimeSpan.FromMinutes((double)1.5)).TumblingWindow(TimeSpan.FromSeconds(15), HoppingWindowOutputPolicy.ClipToWindowEnd)
select new
{
Id = eachDigital.Key,
NormEvents = window.HasNormalizationEvents(),
Count = window.Count()
};
Then, to get the AlarmEvent that triggered the TumblingWindow, I've made a join with the original input.
var resultStream = from e1 in stream
join e2 in input
on e1.Id equals e2.DigitalTag
where e1.NormEvents != 0
select e2;
This isn't working at all... :/ Any ideas to help solving this issue?
Another doubt that I have is if there'll be create a new window with a new startDate for each input that passes the filter or will it be created only one tumblingWindow.
Thanks.
Try this:
// Move all error events
// to the point where the potential timeout would occur.
var timedOut = input
.Where(e => e.IsError == true)
.ShiftEventTime(e => e.StartTime + TimeSpan.FromMinutes(5));
// Extend all events IsToNormalize by the timeout.
var following = input
.Where(e => e. IsToNormalize == true)
.AlterEventDuration(e => TimeSpan.FromMinutes(5));
// Mask away the potential timeout events by the extended events.
// - If IsToNormalize did not occur within the timeout, then the shifted error event
// will remain unchanged by the left-anti-semi-join and represent
// the desired output.
var result = from t in timedOut
where (from c in following
where t.IdGroup == c.IdGroup
select c).IsEmpty()
select t; // or some projection on t
Related
I have a LINQ query
var age = new int[]{1,2,3};
dbContext.TA.WHERE(x=> age.Contains( x.age)).ToList()
In an online article #11 (https://medium.com/swlh/entity-framework-common-performance-mistakes-cdb8861cf0e7) mentioned it is not a good practice as it creates many execution plan at the SQL server.
In this case, how should LINQ be revised so that I can do the same thing but minimize the amount of execution plans generated?
(note that I have no intention to convert it into a stored procedure and pass & join with the UDT as again it requires too many effort to do so)
That article offers some good things to keep in mind when writing expressions for EF. As a general rule that example is something to keep in mind, not a hard "never do this" kind of rule. It is a warning over writing queries that allow for multi-select and to avoid this when possible as it will be on the more expensive side.
In your example with something like "Ages", having a hard-coded list of values does not cause a problem because every execution uses the same list. (until the app is re-compiled with a new list, or you have code that changes the list for some reason.) Examples where it can be perfectly valid to use this is with something like Statuses where you have a status Enum. If there are a small number of valid statuses that a record can have, then declaring a common array of valid statuses to use in an Contains clause is fine:
public void DeleteEnquiry(int enquiryId)
{
var allowedStatuses = new[] { Statuses.Pending, Statuses.InProgress, Statuses.UnderReview };
var enquiry = context.Enquiries
.Where(x => x.EnquiryId == enquiryId && allowedStatuses.Contains(x.Status))
.SingleOrDefault();
try
{
if(enquiry != null)
{
enquiry.IsActive = false;
context.SaveChanges();
}
else
{
// Enquiry not found or invalid status.
}
}
catch (Exception ex) { /* handle exception */ }
}
The statuses in the list aren't going to change so the execution plan is static for that context.
The problem is where you accept something like a parameter with criteria that include a list for a Contains clause.
it is highly unlikely that someone would want to load data where a user could select ages "2, 4, and 6", but rather they would want to select something like: ">=2", or "<=6, or "2>=6" So rather than creating a method that accepts a list of acceptable ages:
public IEnumerable<Children> GetByAges(int[] ages)
{
return _dbContext.Children.Where(x => ages.Contains( x.Age)).ToList();
}
You would probably be better served with ranging the parameters:
private IEnumerable<Children> GetByAgeRange(int? minAge = null, int? maxAge = null)
{
var query = _dbContext.Children.AsQueryable();
if (minAge.HasValue)
query = query.Where(x => x.Age >= minAge.Value);
if (maxAge.HasValue)
query = query.Where(x => x.Age <= maxAge.Value);
return query.ToList();
}
private IEnumerable<Children> GetByAge(int age)
{
return _dbContext.Children.Where(x => x.Age == age).ToList();
}
I've got a simple program here that displays the number of letters in various words. It works as expected.
static void Main(string[] args) {
var word = new Subject<string>();
var wordPub = word.Publish().RefCount();
var length = word.Select(i => i.Length);
var report =
wordPub
.GroupJoin(length,
s => wordPub,
s => Observable.Empty<int>(),
(w, a) => new { Word = w, Lengths = a })
.SelectMany(i => i.Lengths.Select(j => new { Word = i.Word, Length = j }));
report.Subscribe(i => Console.WriteLine($"{i.Word} {i.Length}"));
word.OnNext("Apple");
word.OnNext("Banana");
word.OnNext("Cat");
word.OnNext("Donkey");
word.OnNext("Elephant");
word.OnNext("Zebra");
Console.ReadLine();
}
And the output is:
Apple 5
Banana 6
Cat 3
Donkey 6
Elephant 8
Zebra 5
I used the Publish().RefCount() because "wordpub" is included in "report" twice. Without it, when a word is emitted first one part of the report would get notified by a callback, and then the other part of report would be notified, double the notifications. That is kindof what happens; the output ends up having 11 items rather than 6. At least that is what I think is going on. I think of using Publish().RefCount() in this situation as simultaneously updating both parts of the report.
However if I change the length function to ALSO use the published source like this:
var length = wordPub.Select(i => i.Length);
Then the output is this:
Apple 5
Apple 6
Banana 6
Cat 3
Banana 3
Cat 6
Donkey 6
Elephant 8
Donkey 8
Elephant 5
Zebra 5
Why can't the length function also use the same published source?
This was a great challenge to solve!
So subtle the conditions that this happens.
Apologies in advance for the long explanation, but bear with me!
TL;DR
Subscriptions to the published source are processed in order, but before any other subscription directly to the unpublished source. i.e. you can jump the queue!
With GroupJoin subscription order is important to determine when windows open and close.
My first concern would be that you are publish refcounting a subject.
This should be a no-op.
Subject<T> has no subscription cost.
So when you remove the Publish().RefCount() :
var word = new Subject<string>();
var wordPub = word;//.Publish().RefCount();
var length = word.Select(i => i.Length);
then you get the same issue.
So then I look to the GroupJoin (because my intuition suggests that Publish().Refcount() is a red herring).
For me, eyeballing this alone was too hard to rationalise, so I lean on a simple debugging too I have used dozens of times of the years - a Trace or Log extension method.
public interface ILogger
{
void Log(string input);
}
public class DumpLogger : ILogger
{
public void Log(string input)
{
//LinqPad `Dump()` extension method.
// Could use Console.Write instead.
input.Dump();
}
}
public static class ObservableLoggingExtensions
{
private static int _index = 0;
public static IObservable<T> Log<T>(this IObservable<T> source, ILogger logger, string name)
{
return Observable.Create<T>(o =>
{
var index = Interlocked.Increment(ref _index);
var label = $"{index:0000}{name}";
logger.Log($"{label}.Subscribe()");
var disposed = Disposable.Create(() => logger.Log($"{label}.Dispose()"));
var subscription = source
.Do(
x => logger.Log($"{label}.OnNext({x.ToString()})"),
ex => logger.Log($"{label}.OnError({ex})"),
() => logger.Log($"{label}.OnCompleted()")
)
.Subscribe(o);
return new CompositeDisposable(subscription, disposed);
});
}
}
When I add the logging to your provided code it looks like this:
var logger = new DumpLogger();
var word = new Subject<string>();
var wordPub = word.Publish().RefCount();
var length = word.Select(i => i.Length);
var report =
wordPub.Log(logger, "lhs")
.GroupJoin(word.Select(i => i.Length).Log(logger, "rhs"),
s => wordPub.Log(logger, "lhsDuration"),
s => Observable.Empty<int>().Log(logger, "rhsDuration"),
(w, a) => new { Word = w, Lengths = a })
.SelectMany(i => i.Lengths.Select(j => new { Word = i.Word, Length = j }));
report.Subscribe(i => ($"{i.Word} {i.Length}").Dump("OnNext"));
word.OnNext("Apple");
word.OnNext("Banana");
word.OnNext("Cat");
word.OnNext("Donkey");
word.OnNext("Elephant");
word.OnNext("Zebra");
This will then output in my log something like the following
Log with Publish().RefCount() used
0001lhs.Subscribe()
0002rhs.Subscribe()
0001lhs.OnNext(Apple)
0003lhsDuration.Subscribe()
0002rhs.OnNext(5)
0004rhsDuration.Subscribe()
0004rhsDuration.OnCompleted()
0004rhsDuration.Dispose()
OnNext
Apple 5
0001lhs.OnNext(Banana)
0005lhsDuration.Subscribe()
0003lhsDuration.OnNext(Banana)
0003lhsDuration.Dispose()
0002rhs.OnNext(6)
0006rhsDuration.Subscribe()
0006rhsDuration.OnCompleted()
0006rhsDuration.Dispose()
OnNext
Banana 6
...
However when I remove the usage Publish().RefCount() the new log output is as follows:
Log without only Subject
0001lhs.Subscribe()
0002rhs.Subscribe()
0001lhs.OnNext(Apple)
0003lhsDuration.Subscribe()
0002rhs.OnNext(5)
0004rhsDuration.Subscribe()
0004rhsDuration.OnCompleted()
0004rhsDuration.Dispose()
OnNext
Apple 5
0001lhs.OnNext(Banana)
0005lhsDuration.Subscribe()
0002rhs.OnNext(6)
0006rhsDuration.Subscribe()
0006rhsDuration.OnCompleted()
0006rhsDuration.Dispose()
OnNext
Apple 6
OnNext
Banana 6
0003lhsDuration.OnNext(Banana)
0003lhsDuration.Dispose()
...
This gives us some insight, however when the issue really becomes clear is when we start annotating our logs with a logical list of subscriptions.
In the original (working) code with the RefCount our annotations might look like this
//word.Subsribers.Add(wordPub)
0001lhs.Subscribe() //wordPub.Subsribers.Add(0001lhs)
0002rhs.Subscribe() //word.Subsribers.Add(0002rhs)
0001lhs.OnNext(Apple)
0003lhsDuration.Subscribe() //wordPub.Subsribers.Add(0003lhsDuration)
0002rhs.OnNext(5)
0004rhsDuration.Subscribe()
0004rhsDuration.OnCompleted()
0004rhsDuration.Dispose()
OnNext
Apple 5
0001lhs.OnNext(Banana)
0005lhsDuration.Subscribe() //wordPub.Subsribers.Add(0005lhsDuration)
0003lhsDuration.OnNext(Banana)
0003lhsDuration.Dispose() //wordPub.Subsribers.Remove(0003lhsDuration)
0002rhs.OnNext(6)
0006rhsDuration.Subscribe()
0006rhsDuration.OnCompleted()
0006rhsDuration.Dispose()
OnNext
Banana 6
So in this example, when word.OnNext("Banana"); is executed the chain of observers is linked in this order
wordPub
0002rhs
However, wordPub has child subscriptions!
So the real subscription list looks like
wordPub
0001lhs
0003lhsDuration
0005lhsDuration
0002rhs
If we annotate the Subject only log we see where the subtlety lies
0001lhs.Subscribe() //word.Subsribers.Add(0001lhs)
0002rhs.Subscribe() //word.Subsribers.Add(0002rhs)
0001lhs.OnNext(Apple)
0003lhsDuration.Subscribe() //word.Subsribers.Add(0003lhsDuration)
0002rhs.OnNext(5)
0004rhsDuration.Subscribe()
0004rhsDuration.OnCompleted()
0004rhsDuration.Dispose()
OnNext
Apple 5
0001lhs.OnNext(Banana)
0005lhsDuration.Subscribe() //word.Subsribers.Add(0005lhsDuration)
0002rhs.OnNext(6)
0006rhsDuration.Subscribe()
0006rhsDuration.OnCompleted()
0006rhsDuration.Dispose()
OnNext
Apple 6
OnNext
Banana 6
0003lhsDuration.OnNext(Banana)
0003lhsDuration.Dispose()
So in this example, when word.OnNext("Banana"); is executed the chain of observers is linked in this order
1. 0001lhs
2. 0002rhs
3. 0003lhsDuration
4. 0005lhsDuration
As the 0003lhsDuration subscription is activated after the 0002rhs, it wont see the "Banana" value to terminate the window, until after the rhs has been sent the value, thus yielding it in the still open window.
Whew
As #francezu13k50 points out the obvious and simple solution to your problem is to just use word.Select(x => new { Word = x, Length = x.Length });, but as I think you have given us a simplified version of your real problem (appreciated) I understand why this isn't suitable.
However, as I dont know what your real problem space is I am not sure what to suggest to you to provide a solution, except that you have one with your current code, and now you should know why it works the way it does.
RefCount returns an Observable that stays connected to the source as long as there is at least one subscription to the returned Observable. When the last subscription is disposed, RefCount disposes it's connection to the source, and reconnects when a new subscription is being made. It might be the case with your report query that all subscriptions to the 'wordPub' are disposed before the query is fulfilled.
Instead of the complicated GroupJoin query you could simply do :
var report = word.Select(x => new { Word = x, Length = x.Length });
Edit:
Change your report query to this if you want to use the GroupJoin operator :
var report =
wordPub
.GroupJoin(length,
s => wordPub,
s => Observable.Empty<int>(),
(w, a) => new { Word = w, Lengths = a })
.SelectMany(i => i.Lengths.FirstAsync().Select(j => new { Word = i.Word, Length = j }));
Because GroupJoin seems to be very tricky to work with, here is another approach for correlating the inputs and outputs of functions.
static void Main(string[] args) {
var word = new Subject<string>();
var length = new Subject<int>();
var report =
word
.CombineLatest(length, (w, l) => new { Word = w, Length = l })
.Scan((a, b) => new { Word = b.Word, Length = a.Word == b.Word ? b.Length : -1 })
.Where(i => i.Length != -1);
report.Subscribe(i => Console.WriteLine($"{i.Word} {i.Length}"));
word.OnNext("Apple"); length.OnNext(5);
word.OnNext("Banana");
word.OnNext("Cat"); length.OnNext(3);
word.OnNext("Donkey");
word.OnNext("Elephant"); length.OnNext(8);
word.OnNext("Zebra"); length.OnNext(5);
Console.ReadLine();
}
This approach works if every input has 0 or more outputs subject to the constraints that (1) outputs only arrive in the same order as the inputs AND (2) each output corresponds to its most recent input. This is like a LeftJoin - each item in the first list (word) is paired with items in the right list (length) that subsequently arrive, up until another item in the first list is emitted.
Trying to use regular Join instead of GroupJoin. I thought the problem was that when a new word was created there was a race condition inside Join between creating a new window and ending the current one. So here I tried to elimate that by pairing every word with a null signifying the end of the window. Doesn't work, just like the first version did not. How is it possible that a new window is created for each word without the previous one being closed first? Completely confused.
static void Main(string[] args) {
var lgr = new DelegateLogger(Console.WriteLine);
var word = new Subject<string>();
var wordDelimited =
word
.Select(i => Observable.Return<string>(null).StartWith(i))
.SelectMany(i => i);
var wordStart = wordDelimited.Where(i => i != null);
var wordEnd = wordDelimited.Where(i => i == null);
var report = Observable
.Join(
wordStart.Log(lgr, "word"), // starts window
wordStart.Select(i => i.Length),
s => wordEnd.Log(lgr, "expireWord"), // ends current window
s => Observable.Empty<int>(),
(l, r) => new { Word = l, Length = r });
report.Subscribe(i => Console.WriteLine($"{i.Word} {i.Length}"));
word.OnNext("Apple");
word.OnNext("Banana");
word.OnNext("Cat");
word.OnNext("Zebra");
word.OnNext("Elephant");
word.OnNext("Bear");
Console.ReadLine();
}
I want to pass more than one record between two forms. The user opens Form-A, selects multiple records and then clicks a button that opens Form-B.
In Form-B there are two (or more) StringEdit controls and they should display values from the selected records.
I know how to pass only one record, to do that I use the following code in a method of Form-B:
if (element.args().parmEnumType() == enumNum(NoYes)
&& element.args().parmEnum() == NoYes::Yes)
{
myTable = element.args().record();
stringEdit.text(myTable.Field);
}
How should I change my code so that I can set the text of another StringEdit control to the field value of the next record that the user selected?
To do this, you can use an args to pass the records, which you will need to prepare in your Form-A, as below (taking the SalesTable as example);
int recordsCount;
SalesTable salesTable;
container con;
Args args = newArgs();
// gets the total records selected
recordsCount = salesTable_ds.recordsMarked().lastIndex();
salesTable = salesTable_ds.getFirst(1);
while(salesTable)
{
// storing recid of selected record in container
con = conIns(con,1, salesTable.RecId);
salesTable = SampleTable_ds.getNext(); // moves to next record
}
// passing container converted to string
args.parm(con2Str(con,','));
Then on your Form-B, you will need to override the init() method to read the args you created,
In order to retrieve passed arguments in the recipient from. Override the init() method of new form as shown
public void init()
{
container con;
int i;
super();
// string to container
con = str2con(element.args().parm(),'','');
// for sorting
for(i = 1;i<= conLen(con) ;i++)
{
salesTable_ds.query().dataSourceTable(Tablenum(SalesTable)).addRange(fieldNum(SalesTable,RecId)).value(SysQuery::value(conPeek(con,i)));
}
}
Hope it helps.
Taken from Ax-Forum
This will typically imply looping through the selected records in Form-A and changing the query in Form-B.
The idiomatic way to loop through selected records involves a for loop and the getFirst and getNext methods of the datasource:
SalesLine sl;
FormDataSourcs ds = _salesLine.dataSource();
for (sl = ds.getFirst(true) ? ds.getFirst(true) : ds.cursor(); sl; sl = ds.getNext())
{
//Do your thing, add a query range
}
The MultiSelectionHelper class may be of use here, as it does the thing:
public void init()
{
MultiSelectionHelper ms;
super();
if (element.args() && element.args().caller() && element.args().record())
{
this.query().dataSourceTable(tableNum(SalesLine)).clearDynalinks();
ms = MultiSelectionHelper::createFromCaller(element.args().caller());
ms.createQueryRanges(this.query().dataSourceTable(tablenum(SalesLine)), fieldstr(SalesLine, InventTransId));
}
}
Here we have a Observable Sequence... in .NET using Rx.
var aSource = new Subject<int>();
var bSource = new Subject<int>();
var paired = Observable
.Merge(aSource, bSource)
.GroupBy(i => i).SelectMany(g => g.Buffer(2).Take(1));
paired.Subscribe(g => Console.WriteLine("{0}:{1}", g.ElementAt(0), g.ElementAt(1)));
aSource.OnNext(4);
bSource.OnNext(1);
aSource.OnNext(2);
bSource.OnNext(5);
aSource.OnNext(3);
bSource.OnNext(3);
aSource.OnNext(5);
bSource.OnNext(2);
aSource.OnNext(1);
bSource.OnNext(4);
Output:
3:3
5:5
2:2
1:1
4:4
We will get events every time a pair of numbers arrive with the same id.
Perfect! Just what i want.
Groups of two, paired by value.
Next question....
How to get a selectmany/buffer for sequences of values.
So 1,2,3,4,5 arrives at both aSource and bSource via OnNext(). Then fire ConsoleWriteLine() for 1-5. Then when 2,3,4,5,6 arrives, we get another console.writeline(). Any clues anyone?
Immediately, the Rx forum suggests looking at .Window()
http://introtorx.com/Content/v1.0.10621.0/17_SequencesOfCoincidence.html
Which on the surface looks perfect. In my case i need a window of value 4, in this case.
Where in the query sequence does it belong to get this effect?
var paired = Observable.Merge(aSource, bSource).GroupBy(i => i).SelectMany(g => g.Buffer(2).Take(1));
Output
1,2,3,4,5 : 1,2,3,4,5
2,3,4,5,6 : 2,3,4,5,6
Regards,
Daniel
Assuming events arrive randomly at the sources, use my answer to "Reordering events with Reactive Extensions" to get the events in order.
Then use Observable.Buffer to create a sliding buffer:
// get this using the OrderedCollect/Sort in the referenced question
IObservable<int> orderedSource;
// then subscribe to this
orderedSource.Buffer(5, 1);
Here is an extension method that fires when it has n inputs of the same ids.
public static class RxExtension
{
public static IObservable<TSource> MergeBuffer<TSource>(this IObservable<TSource> source, Func<TSource, int> keySelector, Func<IList<TSource>,TSource> mergeFunction, int bufferCount)
{
return Observable.Create<TSource>(o => {
var buffer = new Dictionary<int, IList<TSource>>();
return source.Subscribe<TSource>(i =>
{
var index = keySelector(i);
if (buffer.ContainsKey(index))
{
buffer[index].Add(i);
}
else
{
buffer.Add(index, new List<TSource>(){i});
}
if (buffer.Count==bufferCount)
{
o.OnNext(mergeFunction(buffer[index]));
buffer.Remove(index);
}
});
});
}
}
Calling the extension.
mainInput = Observable.Merge(inputNodes.ToArray()).MergeBuffer<NodeData>(x => x.id, x => MergeData(x), 1);
Thought this would be pretty straight forward, but my value is remaining the same (0).
What I'd like to do is increment my UnreadMessages field when the user receives a message they haven't read and then decrement it when they have. So I thought code like this would work:
var userHelper = new MongoHelper<User>();
//increment
userHelper.Collection.Update(Query.EQ("Id", userId.ToId()), Update.Inc("UnreadMessages", 1));
//decrement
userHelper.Collection.Update(Query.EQ("Id", userId.ToId()), Update.Inc("UnreadMessages", -1));
After running these no errors are thrown but the value doesn't change either. And no I'm not running one after the other as the code above could be interpreted :)
Update
Here's my helper class:
public class MongoHelper<T> : Sandbox.Services.IMongoHelper<T> where T : class
{
public MongoCollection<T> Collection { get; private set; }
public MongoHelper()
{
var con = new MongoConnectionStringBuilder(ConfigurationManager.ConnectionStrings["MongoDB"].ConnectionString);
var server = MongoServer.Create(con);
var db = server.GetDatabase(con.DatabaseName);
Collection = db.GetCollection<T>(typeof(T).Name.ToLower());
}
}
and thanks to Travis' answer I was able to pull this off:
MongoHelper<UserDocument> userHelper = new MongoHelper<UserDocument>();
var user = userHelper.Collection.FindAndModify(Query.EQ("Username", "a"), SortBy.Null, Update.Inc("MessageCount", 1), true).GetModifiedDocumentAs<UserDocument>();
Not sure what your helper does. Here is a working snippet I use:
var query = Query.And(Query.EQ("_id", keyName));
var sortBy = SortBy.Null;
var update = Update.Inc("KeyValue", adjustmentAmount);
var result = collection.FindAndModify(query, sortBy, update, true);
So, "query" finds the document, update does the increment, and FindAndModify puts them together and actually hits the database.