Observable to simulate high frequency signal - system.reactive

I have a server that receives batches of signal samples from clients. Each batch can contain up to 1000 samples and be received once per second (1000 Hz). Once I have received the samples, I want to publish them locally via an Observable at the same frequency they were generated from the signal source device. I have put together the following concept code that accomplishes what I want to do.
LINQPad code:
var frequency = 1000; // samples / second
int sampleCount = 1000;
var samples = Enumerable.Range(1, sampleCount).ToArray();
ManualResetEvent done = new ManualResetEvent(false);
int count = 0;
Stopwatch sw = new Stopwatch();
sw.Start();
samples.ToObservable()
.Do(_ => Thread.Sleep(1000/frequency))
.Subscribe((next) => count++, () => { sw.Stop(); done.Set(); });
done.WaitOne();
sw.ElapsedMilliseconds.Dump("Subscription duration");
count.Dump("Items observed");
I tried other ways to simulate the delay between samples, but at such high frequencies, any timer-based or task-based approaches apparently had too much overhead to keep up. So my question(s) are:
Is this the best approach to delaying a high frequency observable?
In the Do(...) method, it is the 'SubscribeOn' thread that I am putting to sleep, right?
Should I explicitly be calling the SubscribeOn() and ObserveOn() method to help ensure that the observer code does not interfere with publishing of the samples?
Thanks in advance!

Related

What happens if i set 'raspicam_node/image' queue size to 1?

I can't get real-time image from raspberry pi cam.
So I adjusted the Queue Size from 100 to 1.
Then there was an improvement in performance.
I wonder why there was a performance improvement.
int main(int argc, char** argv)
{
ros::init(argc, argv, "lane_driving");
ros::NodeHandle nh, nhp;
image_transport::ImageTransport it(nh);
pub = nhp.advertise<geometry_msgs::Twist>("/cmd_vel", 100);
subAD = nh.subscribe("arrowDetecter", 1,&arrowMessage);
subScan = nh.subscribe("/scan",1,&scanMsgCallback);
subOdom = nh.subscribe("/odom", 1,&odomMsgCallback);
image_transport::Subscriber sub = it.subscribe("/raspicam_node/image", 1, &poseMessageReceived, ros::VoidPtr(), image_transport::TransportHints("compressed"));
while(ros::ok()){
baseCmd.linear.x = 0.01;
pub.publish(baseCmd);
ros::spin();
}
If the number of messages that arrive on the /raspicam_node/image topics in a single ros::spin() is greater than the queue_size, the extra messages will be discarded. What this means is that your callback is probably not running for every image that comes from your camera. So you are probably "dropping" (not analyzing) some of the camera frames. This would give the performance improvement you observed.
Whether or not this is what you want depends on your application.

Kafka Streams - GroupBy - Late Event - persistentWindowStore - WindowBy with Grace Period and Suppress

My purpose to calculate success and fail message from source to destination per second and sum their results in daily bases.
I had two options to do that ;
stream events then group them time#source#destination
KeyValueBytesStoreSupplier streamStore = Stores.persistentKeyValueStore("store-name");
sourceStream.selectKey((k, v) -> v.getDataTime() + KEY_SEPERATOR + SRC + KEY_SEPERATOR + DEST ).groupByKey().aggregate(
DO SOME Aggregation,
Materialized.<String, AggregationObject>as(streamStore)
.withKeySerde(Serdes.String())
.withValueSerde(AggregationObjectSerdes));
After trying this approach above we noticed that state store is getting increase because of number of unique keys are increasing and if i am correct, because of state topics are only "compact" they are never expires.
NumberOfUniqueKeys = 86.400 seconds in a day X SOURCE X DESTINATION
Then we thought that if we do not put a time field in a KEY block, we can reduce state store size. We tried windowing operation as second approach.
using windowing operation with persistentWindowStore, CustomTimeStampExtractor, WindowBy, Suppress
WindowBytesStoreSupplier streamStore = Stores.persistentWindowStore("store-name", Duration.ofHours(6), Duration.ofSeconds(1), false);
sourceStream.selectKey((k, v) -> SRC + KEY_SEPERATOR + DEST)
.groupByKey() .windowedBy(TimeWindows.of(Duration.ofSeconds(1)).grace(Duration.ofSeconds(5)))
.aggregate(
{
DO SOME Aggregation
}, Materialized.<String, AggregationObject>as(streamStore)
.withKeySerde(Serdes.String())
.withValueSerde(AggregationObjectSerdes))
.suppress(Suppressed.untilWindowCloses(Suppressed.BufferConfig.unbounded())).toStream();`
After trying that second approach, we reduced state store size but now we had problem with late arrive events. Then we added grace period with 5 seconds with suppress operation and in addition using grace period and suppress operation did not guarantee to handle all late arrived events, another side effect of suppress operation is a latency because it emits result of aggregation after window grace period.
BTW
using windowing operation caused a getting WARNING message like
"WARN 1 --- [-StreamThread-2] o.a.k.s.state.internals.WindowKeySchema : Warning: window end time was truncated to Long.MAX"
I checked the reason from source code and I found from here
https://github.com/a0x8o/kafka/blob/master/streams/src/main/java/org/apache/kafka/streams/state/internals/WindowKeySchema.java
/**
* Safely construct a time window of the given size,
* taking care of bounding endMs to Long.MAX_VALUE if necessary
*/
static TimeWindow timeWindowForSize(final long startMs,
final long windowSize) {
long endMs = startMs + windowSize;
if (endMs < 0) {
LOG.warn("Warning: window end time was truncated to Long.MAX");
endMs = Long.MAX_VALUE;
}
return new TimeWindow(startMs, endMs);
}
BUT actually it does not make any sense to me that how endMs can be lower than 0...
Questions ?
What if we go through with approach 1, how can we reduce state store size ? In approach 1, It was guaranteed that all event will be processed and there will be no missing event because of latency.
What if we go through with approach 2, how should i tune my logic and catch late arrival data and reduce latency ?
Why do i get Warning message in approach 2 although all time fields are positive in my model ?
What can be other options that you can suggest other then these two approaches ?
I need some expert help :)
BR,
According to mail kafka mail group about warning message
WARNING message like "WARN 1 --- [-StreamThread-2] o.a.k.s.state.internals.WindowKeySchema : Warning: window end time was truncated to Long.MAX"
It was written to me :
You can get this message "o.a.k.s.state.internals.WindowKeySchema :
Warning: window end time was truncated to Long.MAX"" when your
TimeWindowDeserializer is created without a windowSize. There are two
constructors for a TimeWindowDeserializer, are you using the one with
WindowSize?
https://github.com/apache/kafka/blob/trunk/streams/src/main/java/org/apache/kafka/streams/kstream/TimeWindowedDeserializer.java#L46-L55
It calls WindowKeySchema with a Long.MAX_VALUE
https://github.com/apache/kafka/blob/trunk/streams/src/main/java/org/apache/kafka/streams/kstream/TimeWindowedDeserializer.java#L84-L90

Why is the turtlebot not moving continously?

if __name__ == '__main__':
rospy.init_node('gray')
settings = termios.tcgetattr(sys.stdin)
pub = rospy.Publisher('cmd_vel', Twist, queue_size=1)
x = 0
th = 0
node = Gray()
node.main()
We make the publisher(cmd_vel) in main, and run the main function of class gray.
def __init__(self):
self.r = rospy.Rate(10)
self.selecting_sub_image = "compressed" # you can choose image type "compressed", "raw"
if self.selecting_sub_image == "compressed":
self._sub = rospy.Subscriber('/raspicam_node/image/compressed', CompressedImage, self.callback, queue_size=1)
else:
self._sub = rospy.Subscriber('/usb_cam/image_raw', Image, self.callback, queue_size=1)
self.bridge = CvBridge()
init function makes a subscriber, which runs 'callback' when it gets data.
def main(self):
rospy.spin()
Then it runs the spin() function.
v, ang = vel_select(lvalue, rvalue, left_angle_num, right_angle_num, left_down, red_dots)
self.sendv(v, ang)
Inside the callback function, it gets a linear speed and angular speed value, and runs a sendv function to send it to the subscribers.
def sendv(self, lin_v, ang_v):
twist = Twist()
speed = rospy.get_param("~speed", 0.5)
turn = rospy.get_param("~turn", 1.0)
twist.linear.x = lin_v * speed
twist.angular.z = ang_v * turn
twist.linear.y, twist.linear.z, twist.angular.x, twist.angular.y = 0, 0, 0, 0
pub.publish(twist)
and... sendv function sends it to the turtlebot.
It has to move continuously, because if we do not publish data, it still has to move with the speed it got from the last publish. Also, callback function runs every 0.1 seconds, so it keeps sending data.
But it does not move continously. It stops for a few seconds, and go for a very short time, and stops again, and go for a very short time, and so on. The code which selects the speed works correctly, but the code who sents it to the turtlebot does not work well. Can anyone help?
Also, callback function runs every 0.1 seconds.
I believe this is incorrect. I see that you have made a self.r object but never used it anywhere in the code to achieve an update rate of 10hz. If you want to run the main loop at every 0.1 seconds, you will have to call your commands within the following loop (see rospy-rates) before calling rospy.spin():
self.r = rospy.Rate(10)
while not rospy.is_shutdown():
<user commands>
self.r.sleep()
However, this would not help you either since your code is publishing to /cmd_vel within the subscriber callback which gets called only on receiving some data from the subscriber. So basically, your /cmd_vel is not being published at the rate of 10hz but rather at the rate at which you are receiving the data from the subscribed topic ('/raspicam_node/image/compressed'). Since these are image topics, they might be taking a lot of time to be updated hence the delay in your velocity commands to the robot.

Q: [Anylogic] Measuring production throughput rate

I would like to know how to measure the throughput rate of the production line on Anylogic.
Question: Are there any methods to measure the Time Between Departure of the agent at the sink block? >>(I will calculate the throughput rate by inverting the time between departure value.)
At the moment, I just simply calculated the throughput based on Little's law, which I use the average lead time and WIP level of the line. I am not sure that whether the throughput value based on this calculation will be equal to the inverted value of the time between departure or not?
I hope you guys could help me figure it out.
Thanks in advance!
There is a function "time()" that returns the current model time in model time units. Using this function, you may know the times when agent A and agent B left the system, and calculate the difference between these times. You can do this by writing the code like below in the "On exit" field of the "sink" block:
statistic.add(time() - TimeOfPreviousAgent);
TimeOfPreviousAgent = time();
"TimeOfPreviousAgent" is a variable of "double" type;
"statistic" is a "Statistic" element used to collect the measurements
This approach of measuring time in the process flow is described in the tutorial Bank Office.
As an alternative, you can store leaving time of each agent into a collection. Then, you will need to iterate over the samples stored in the collection to find the difference between each pair of samples.
Not sure if this will help but it stems off Tatiana's answer. In the agents state chart you can create variables TimeIn, TimeOut, and TimeInSystem. Then at the Statechart Entry Point have,
TimeIn = time();
And at the Final state have,
TimeOut = time();
TimeInSystem = TimeOut - TimeIn;
To observe these times for each individual agent you can use the following code,
System.out.println("I came in at " + TimeIn + " and exited at " TimeOut + " and spent " + TimeInSystem + " seconds in the system";
Then for statistical analysis you can calculate the min, avg, and max throughputs of all agents by creating in Main variables, TotalTime, TotalAgentsServiced, AvgServiceTime, MaxServiceTime, MinServiceTime and then add a function call it say TrackAvgTimeInSystem ... within the function add argument NextAgent with type double. In the function body have,
TotalTime += NextAgent;
TotalAgentsServiced += 1;
AverageServiceTime = TotalTime/TotalCarsServiced;
if(MinServiceTimeReported == 0)
{
MinServiceTime = NextAgent;
}
else if(NextAgent < MinServiceTime)
{
MinServiceTime = NextAgent;
}
if(NextAgent > MaxServiceTime)
{
MaxServiceTime = NextAgent;
}
Then within your agent's state charts, in the Final State call the function
get_Main().TrackAvgTimeInSystem(TimeInSystem);
This then calculates the min, max, and average throughput of all agents.

In Rx (or RxJava/RxScala), how to make an auto-resetting stateful latch map/filter for measuring in-stream elapsed time to touch a barrier?

Apologies if the question is poorly phrased, I'll do my best.
If I have a sequence of values with times as an Observable[(U,T)] where U is a value and T is a time-like type (or anything difference-able I suppose), how could I write an operator which is an auto-reset one-touch barrier, which is silent when abs(u_n - u_reset) < barrier, but spits out t_n - t_reset if the barrier is touched, at which point it also resets u_reset = u_n.
That is to say, the first value this operator receives becomes the baseline, and it emits nothing. Henceforth it monitors the values of the stream, and as soon as one of them is beyond the baseline value (above or below), it emits the elapsed time (measured by the timestamps of the events), and resets the baseline. These times then will be processed to form a high-frequency estimate of the volatility.
For reference, I am trying to write a volatility estimator outlined in http://www.amazon.com/Volatility-Trading-CD-ROM-Wiley/dp/0470181990 , where rather than measuring the standard deviation (deviations at regular homogeneous times), you repeatedly measure the time taken to breach a barrier for some fixed barrier amount.
Specifically, could this be written using existing operators? I'm a bit stuck on how the state would be reset, though maybe I need to make two nested operators, one which is one-shot and another which keeps creating that one-shot... I know it could be done by writing one by hand, but then I need to write my own publisher etc etc.
Thanks!
I don't fully understand the algorithm and your variables in the example, but you can use flatMap with some heap-state and return empty() or just() as needed:
int[] var1 = { 0 };
source.flatMap(v -> {
var1[0] += v;
if ((var1[0] & 1) == 0) {
return Observable.just(v);
}
return Observable.empty();
});
If you need a per-sequence state because of multiple consumers, you can defer the whole thing:
Observable.defer(() -> {
int[] var1 = { 0 };
return source.flatMap(v -> {
var1[0] += v;
if ((var1[0] & 1) == 0) {
return Observable.just(v);
}
return Observable.empty();
});
}).subscribe(...);