I want to create a sequence that emits a value (let's say 'true') whenever a source sequence emits a value. Then, when the source sequence is idle for a period of time, emits 'false'. Essentially, I need to know when the source sequence is 'idle' for some time.
Source: ---1-----5-------2-------(timeout)--------8-----3------>
| | | | | |
Output: ---true--true----true----false------------true--true--->
In actual fact, I don't need the repeated occurrences of true, so this would be even better:
Source: ---1-----5-------2-------(timeout)---------8-----3------>
| | |
Output: ---true------------------false-------------true--------->
I've seen this answer, but to be honest I don't really understand how it's working. It seems like there should be a simpler answer.
What's worse is that I'm sure I have solved this exact problem before, but I can't remember how! Can anyone help out here?
It's quite simple with a Switch. Try this:
var source = new Subject<int>();
var query =
source
.Select(x =>
Observable
.Timer(TimeSpan.FromSeconds(1.0))
.Select(y => false)
.StartWith(true))
.Switch();
query.Subscribe(Console.WriteLine);
source.OnNext(1);
Thread.Sleep(TimeSpan.FromSeconds(0.5));
source.OnNext(5);
Thread.Sleep(TimeSpan.FromSeconds(0.5));
source.OnNext(2);
Thread.Sleep(TimeSpan.FromSeconds(1.5));
source.OnNext(8);
Thread.Sleep(TimeSpan.FromSeconds(0.5));
source.OnNext(3);
That gives me:
True
True
True
False
True
True
False
Related
I have a streaming pipeline connected with a PubSub subscription (with around 2MLN elements every hour. I need to collect them in a group and then extract some information.
def expand(self, pcoll):
return (
pcoll |
beam.WindowInto(FixedWindows(10),
trigger=AfterAny(AfterCount(2000), AfterProcessingTime(30)),
allowed_lateness=10,
trigger=AfterAny(
AfterCount(8000),
AfterProcessingTime(30),
AfterWatermark(
early=AfterProcessingTime(60),
late=AfterProcessingTime(60)
)
),
allowed_lateness=60 * 60 * 24,
accumulation_mode=AccumulationMode.DISCARDING)
| "Group by Key" >> beam.GroupByKey()
I try my best to NOT miss any data. But I found out that I have around 4% missing data.
As you can see in the code I trigger anytime I hit 8k elements or every 30 seconds.
Allowing lateness 1 day, and it should trigger both if the pipeline is analyzing early or late events.
Still missing those 4% though. So, is there a way to know if the pipeline is discarding some data? How many elements? For which reason?
Thank you so much in advance
First, I see you have two triggers in the sample code, I assume this is a typo, though.
It looks you are dropping elements due to no using Repeatedly, so all elements after the first trigger get lost. There's an official doc on this from Beam.
Allow me to post an example:
test_stream = (TestStream()
.add_elements([
TimestampedValue('in_time_1', 0),
TimestampedValue('in_time_2', 0)])
.advance_watermark_to(9)
.advance_processing_time(9)
.add_elements([TimestampedValue('late_but_in_window', 8)])
.advance_watermark_to(10)
.advance_processing_time(10)
.add_elements([TimestampedValue('in_time_window2', 12)])
.advance_watermark_to(20) # Past window time
.advance_processing_time(20)
.add_elements([TimestampedValue('late_window_closed', 9),
TimestampedValue('in_time_window2_2', 12)])
.advance_watermark_to_infinity())
class RecordFn(beam.DoFn):
def process(
self,
element=beam.DoFn.ElementParam,
timestamp=beam.DoFn.TimestampParam):
yield ("key", (element, timestamp))
options = PipelineOptions()
options.view_as(StandardOptions).streaming = True
with TestPipeline(options=options) as p:
records = (p | test_stream
| beam.ParDo(RecordFn())
| beam.WindowInto(FixedWindows(10),
allowed_lateness=0,
# trigger=trigger.AfterCount(1),
trigger=trigger.Repeatedly(trigger.AfterCount(1)),
accumulation_mode=trigger.AccumulationMode.DISCARDING)
| beam.GroupByKey()
| beam.Map(print)
)
If we have trigger trigger.Repeatedly(trigger.AfterCount(1)), all elements are fired as they come, with no dropped element (but late_window_closed which is expected as it was late):
('key', [('in_time_1', Timestamp(0)), ('in_time_2', Timestamp(0))]) # this two are together since they arrived together
('key', [('late_but_in_window', Timestamp(8))])
('key', [('in_time_window2', Timestamp(12))])
('key', [('in_time_window2_2', Timestamp(12))])
If we use trigger.AfterCount(1) (no repeatedly), we only get the first elements that arrived in the pipeline:
('key', [('in_time_1', Timestamp(0)), ('in_time_2', Timestamp(0))])
('key', [('in_time_window2', Timestamp(12))])
Note that both in_time_(1,2) appear in the first fired pane because the arrived at the same time (0), were one of them appear later it would have been dropped.
I am just starting with GraphFrames, and though I am following the documentation, I am not able to get any result from the aggregateMessages function (it returns an empty dataframe). Here is a simplified example of my problem: I GraphFrames object called testGraph such that my vertexRDD consists of only a single vertex Y with no vertex attributes, and my edgeRDD consists of two records like this:
| src | dst | min_ts1 | min_ts2 |
| X | Y | 20 | null |
| Y | X | null | -10 |
Now, I want to implement a simple algorithm that sends the value of min_ts1 to dst, and sends min_ts2 to the src. The code I am using to implement this algorithm is :
import org.graphframes.lib.AggregateMessages
import org.apache.spark.sql.functions._
val AM = AggregateMessages
val msgToSrc = AM.edge("min_ts2)
val msgToDst = AM.edge("min_ts1")
val delay = testGraph
.aggregateMessages
.sendToSrc(msgToSrc)
.sendToDst(msgToDst)
.agg(sum(AM.msg).as("avg_time_delay"))
I realize there are some null values here, but regardless I would expect the message passing algorithm to do the following: look at the first record, and send a message of 20 to Y and a message of null to X. Then look at the second record, and send a message of null to X and a message of -10 to Y. Finally I would expect the result to show that the sum of messages for Y is 10, and for there to be no record for X in the result, since it was not included in the vertexRDD. And if X were included in the vertexRDD, I would expect the result to be simply null, since both of the messages were null.
However, what I am getting is an empty RDD. Could someone please help me understand why I am getting an empty result?
Ok, it appears that the reason for this issue is indeed that I did not have X in my VertexRDD. I guess even if there are edges going to and from that vertex in my edgeRDD and my aggregatemessages depend only on edge attributes, the algorithm is not able to send those messages.
I am working on a thingspeak code on matlab analysis for my weather station which checks last 24 readings and then gives alert on the basis of given conditions, and I gave this condition but I guess I am messing up with something hence getting wrong results. I want the answer to be overall logical 1 or 0. I get 1's for even the values that should not give me one and the answer for both variables is a 24*1 logical array. But even then the tweets are not being generated as well. Here's my code;
t =thingSpeakRead(293182,'Fields',1,'NumPoints',24,'OutputFormat','matrix');
h =thingSpeakRead(293182,'Fields',2,'NumPoints',24,'OutputFormat','matrix');
DangerAlert = ((t>42.5)&(t<43.5)&(h>17)&(h<21))|(((t>40.5)&(t<43.5))&((h>21)&(h<27)))|((t>39.5)&(t<43.5)&(h>27)&(h<31)) | ((t>38.5)&(t<43.5)&(h>31)&(h<37))| ((t>37.5)&(t<42.5)&(h>37)&(h<41))| ((t>36.5)&(t<40.5)&(h>41)&(h<47))| ((t>35.5)&(t<39.5)&(h>47)&(h<51))| ((t>34.5)&(t<38.5)&(h>51)&(h<57))| ((t>33.5)&(t<38.5)&(h>57)&(h<68))| ((t>33.5)&(t<37.5)&(h>63)&(h<68)) | ((t>32.5)&(t<38.5)&(h>68)&(h<73)) | ((t>31.5)&(t<35.5)&(h>73)&(h<83))| ((t>30.5)&(t<33.5)&(h>83)&(h<88)) | ((t>29.5)&(t<33.5)&(h>83)&(h<93))| ((t>29.5)&(t<32.5)&(h>93)&(h<100))
HeatStrokeAlert=((t>42.5)&(t<43.5)&(h>37)&(h<41)) | ((t>40.5)&(t<2.5)&(h>41)&(h<47)) | ((t>39.5)&(t<41.5)&(h>47)|(h<51))| ((t>38.5)&(t<40.5)&(h>51)&(h<57))| ((t>38.5)&(t<39.5)&(h>57)&(h<63))| ((t>37.5)&(t<38.5)&(h>63)&(h<68))| ((t>36.5)&(t<38.5)&(h>68)&(h<78))| ((t>35.5)&(t<37.5)&(h>73)&(h<83)) | ((t>34.5)&(t<36.5)&(h>83)&(h<88)) | ((t>33.5)&(t<36.5)&(h>88)&(h<93)) | ((t>33.5)&(t<35.5)&(h>93)&(h<97)) | ((t>32.5)&(t<34.5)&(h>97))
if DangerAlert
webwrite('http://api.thingspeak.com/apps/thingtweet/1/statuses/update','api_key', 'XXXXXXXXXXXXX', 'status', 'Alert!Dangerously High temperature tomorrow!')
end
if HeatStrokeAlert
webwrite('http://api.thingspeak.com/apps/thingtweet/1/statuses/update','api_key', 'XXXXXXXXX', 'status', 'Alert!Heat Stroke alert tomorrow!')
end
I know the blunder is minor.But this needs to solve.
Your range values for t go from 29.5 to 43.5, and for h go from 17 to 100. So any value you put in between those numbers will give you a 1, because you are using the OR statements ||. So if ANY one of those is true, it will come back true (=1).
Also, for the website, make sure you follow these directions:
https://www.mathworks.com/help/matlab/ref/webwrite.html
Make sure you have a ThinkSpeak account, and try changing your URL to match their format:
[thingSpeakURL 'update'];
So add 'update' string and use brackets.
Also, set your if statement expression to one. So:
if DangerAlert = 1
I'm using powershell for completing webform. I have dropdown there and can select it by value:
$dropdown.value = '3236'
It works fine, but I need to select by text. I read this answer and tried this code:
($dropdown | where {$_.innerHTML -eq "sometext"}).Selected = $true
It works too, but because dropdown has too many options (probably several thousands), it takes several minutes to select, which is not acceptable.
How can I improve this?
btw, I use powershell 2.0
Okay, I found solution, maybe it isn't perfect, but I got performance improved in hundreds times, instead several minutes it takes less than second.
I created the following function:
function getValue($innerHTML, $Name)
{
$innerHTML = $innerHTML.Substring($innerHTML.IndexOf($Name) - 11, 10)
return $innerHTML.Substring($innerHTML.IndexOf('=') + 1)
}
Function takes innerHTML of dropdown as string, search for option name and return value instead.
and use function the following way:
$drop.Value = getValue $drop.innerHTML 'somename'
First off, for full disclosure I'm a n00b in RX, but I'm learning daily now...
I need to build an Observable that's going to enable a button (or automatically start an action) as long as a stream of another incoming observable averaged signals is coming in within a certain range. As far as I've learned so far, I could do that by adding a .Where to an averaged Observable in which I can then check that my observed average values created from an event handler are in fact within a given range...
What I need also however is to have these observable values influence the state of my action/button (allow it to be executed) just until its underlying/inner signals overstep the given range. Is such a thing in RX possible as a reversed .TakeUntil(inverse where clause) which I now think could maybe solve my problem, or should I just reuse the original observable and copy it with a negated .Where clause and then use that as another independent observable...if latter, is there some performance loss by reusing almost identical observables multiple times, just changing few their linq queries...how many observables is too much please?
It seems to me that something like this is sufficient.
var source = Observable.Range(0, 10);
var query =
source
.Buffer(4, 1)
.Select(xs => xs.Average())
.Select(x => x > 5.0 && x <= 7.0);
query.ObservableOn(button).Subscribe(enabled => button.Enabled = enabled);
(I've assumed Windows forms.)
This gives me:
False
False
False
False
True
True
False
False
False
False
I can improve it slightly like this:
var query =
source
.Buffer(4, 1)
.Select(xs => xs.Average())
.Select(x => x > 5.0 && x <= 7.0)
.StartWith(true)
.DistinctUntilChanged();
That then gives me:
True
False
True
False
Please let me know if I've missed anything.