I'm building a Promela model in which one process send a request to N other processes, waits for the replies, and then computes a value. Basically a typical map-reduce style execution flow. Currently my model sends requests in a fixed order. I'd like to generalize this to send a non-deterministic order. I've looked at the select statement, but that appears to select a single element non-deterministically.
Is there a good pattern for achieving this? Here the basic structure of what I'm working with:
#define NUM_OBJECTS 2
chan obj_req[NUM_OBJECTS] = [0] of { mtype, chan };
This is the object process that responds to msgtype messages with some value that it computes.
proctype Object(chan request) {
chan reply;
end:
do
:: request ? msgtype(reply) ->
int value = 23
reply ! value
od;
}
This is the client. It sends a request to each of the objects in order 0, 1, 2, ..., and collects all the responses and reduces the values.
proctype Client() {
chan obj_reply = [0] of { int };
int value
// WOULD LIKE NON-DETERMINISM HERE
for (i in obj_req) {
obj_req[i] ! msgtype(obj_reply)
obj_reply ? value
// do something with value
}
}
And I start up the system like this
init {
atomic {
run Object(obj_req[0]);
run Object(obj_req[1]);
run Client();
}
}
From your question I gather that you want to assign a task to a given process in a randomised order, as opposed to simply assign a random task to an ordered sequence of processes.
All in all, the solution for both approaches is very similar. I don't know whether the one I am going to propose is the most elegant approach, though.
#define NUM_OBJECTS 10
mtype = { ASSIGN_TASK };
chan obj_req[NUM_OBJECTS] = [0] of { mtype, chan, int };
init
{
byte i;
for (i in obj_req) {
run Object(i, obj_req[i]);
}
run Client();
};
proctype Client ()
{
byte i, id;
int value;
byte map[NUM_OBJECTS];
int data[NUM_OBJECTS];
chan obj_reply = [NUM_OBJECTS] of { byte, int };
d_step {
for (i in obj_req) {
map[i] = i;
}
}
// scramble task assignment map
for (i in obj_req) {
byte j;
select(j : 0 .. (NUM_OBJECTS - 1));
byte tmp = map[i];
map[i] = map[j];
map[j] = tmp;
}
// assign tasks
for (i in obj_req) {
obj_req[map[i]] ! ASSIGN_TASK(obj_reply, data[i]);
}
// out-of-order wait of data
for (i in obj_req) {
obj_reply ? id(value);
printf("Object[%d]: end!\n", id, value);
}
printf("client ends\n");
};
proctype Object(byte id; chan request)
{
chan reply;
int in_data;
end:
do
:: request ? ASSIGN_TASK(reply, in_data) ->
printf("Object[%d]: start!\n", id)
reply ! id(id)
od;
};
The idea is have an array which acts like a map from the set of indexes to the starting position (or, equivalently, to the assigned task).
The map is then scrambled through a finite number of swap operations. After that, each object is assigned its own task in parallel, so they can all start more-or-less at the same time.
In the following output example, you can see that:
Objects are being assigned a task in a random order
Objects can complete the task in a different random order
~$ spin test.pml
Object[1]: start!
Object[9]: start!
Object[0]: start!
Object[6]: start!
Object[2]: start!
Object[8]: start!
Object[4]: start!
Object[5]: start!
Object[3]: start!
Object[7]: start!
Object[1]: end!
Object[9]: end!
Object[0]: end!
Object[6]: end!
Object[2]: end!
Object[4]: end!
Object[8]: end!
Object[5]: end!
Object[3]: end!
Object[7]: end!
client ends
timeout
#processes: 11
...
If one wants to assign a random task to each object rather than starting them randomly, then it suffices to change:
obj_req[map[i]] ! ASSIGN_TASK(obj_reply, data[i]);
into:
obj_req[i] ! ASSIGN_TASK(obj_reply, data[map[i]]);
Obviously, data should be initialised to some meaningful content first.
Related
I have frames that are delimited by bytes to start and stop the frame (they do not appear in the stream).
I read a chunk from disk or network socket, i then need to pass to a deserializer but only after I have de-framed the packet first.
Frames may span multiple chunks that have been read, note how frame 3 is split across array 1 and array 2.
Rather than reinvent the wheel for this common problem, do any github or similar projects exist?
I am investigating ReadOnlySequenceSegment<T> from https://www.codemag.com/article/1807051/Introducing-.NET-Core-2.1-Flagship-Types-Span-T-and-Memory-T and will post updates as I work out the requirements.
Update
Further to Stephen Cleary link (thank you!!) to https://github.com/davidfowl/TcpEcho/blob/master/src/Server/Program.cs I have the below.
My data is json, so unlike the original question the delimiter tokens will appear in the stream. Therefore I have to count the array delimitator and only declare a frame when i have found the outermost [ and ] characters.
The below code works, and less manual copies done (not sure if still done behind the scenes - code is quite neater using David Fowl approach).
However I am casting to array instead of using buffer.PositionOf((byte)'[') since I was unable to see how I could call the PositionOf with an offset applied (i.e. scan deeper into the frame past previously found delimiter tokens).
Am i using/butchering the library in a brute force way, or is the below good to go with the array cast?
class Program
{
static async Task Main(string[] args)
{
using var stream = File.Open(args[0], FileMode.Open);
var reader = PipeReader.Create(stream);
while (true)
{
ReadResult result = await reader.ReadAsync();
ReadOnlySequence<byte> buffer = result.Buffer;
while (TryDeframe(ref buffer, out ReadOnlySequence<byte> line))
{
// Process the line.
var str = System.Text.Encoding.UTF8.GetString(line.ToArray());
Console.WriteLine(str);
}
// Tell the PipeReader how much of the buffer has been consumed.
reader.AdvanceTo(buffer.Start, buffer.End);
// Stop reading if there's no more data coming.
if (result.IsCompleted)
{
break;
}
}
// Mark the PipeReader as complete.
await reader.CompleteAsync();
}
private static bool TryDeframe(ref ReadOnlySequence<byte> buffer, out ReadOnlySequence<byte> frame)
{
int frameCount = 0;
int start = -1;
int end = -1;
var bytes = buffer.ToArray();
for (var i = 0; i < bytes.Length; i++)
{
var b = bytes[i];
if (b == (byte)'[')
{
if (start == -1)
start = i;
frameCount++;
}
else if (b == (byte)']')
{
frameCount--;
if (frameCount == 0)
{
end = i;
break;
}
}
}
if (start == -1 || end == -1) // no frame found
{
frame = default;
return false;
}
frame = buffer.Slice(start, end+1);
buffer = buffer.Slice(frame.Length);
return true;
}
}
do any github or similar projects exist?
David Fowler has an echo server that uses Pipelines to implement delimited frames.
I have a problem that I do not know how to handle beautifully with RX.
I have multiple streams that all supposedly contain the same elements
However each stream may lose messages (UDP is involved) or be late/early compared to others. Each of these messages have a sequence number.
Now what I want to achieve is get a single stream out of all those streams, without duplicate and keeping the message order. In other words, the same sequence number should not appear twice and their values only have to increase, never decrease.
When a message was lost on all the streams, I'm OK with losing it (as there is another TCP mechanism involved that allows me to ask explicitly for missing messages).
I am looking to do that in RxJava, but I guess my problem is not specific to Java.
Here's a marble diagram to help visualizing what I want to achieve:
marble diagram
You can see in that diagram that we are waiting for 2 on the first stream to output 3 from the second stream.
Likewise, 6 is only outputted once we receive 6 from the second stream because only at that point can we know for sure that 5 will never be received by any stream.
This is browser code, but I think it should give you a good idea of how you could solve this.
public static IObservable<T> Sequenced<T>(
this IObservable<T> source,
Func<T, int> getSequenceNumber,
int sequenceBegin,
int sequenceRedundancy)
{
return Observable.Create(observer =>
{
// The next sequence number in order.
var sequenceNext = sequenceBegin;
// The key is the sequence number.
// The value is (T, Count).
var counts = new SortedDictionary<int, Tuple<T, int>>();
return source.Subscribe(
value =>
{
var sequenceNumber = getSequenceNumber(value);
// If the sequence number for the current value is
// earlier in the sequence, just throw away this value.
if (sequenceNumber < sequenceNext)
{
return;
}
// Update counts based on the current value.
Tuple<T, int> count;
if (!counts.TryGetValue(sequenceNumber, out count))
{
count = Tuple.Create(value, 0);
}
count = Tuple.Create(count.Item1, count.Item2 + 1);
counts[sequenceNumber] = count;
// If the current count has reached sequenceRedundancy,
// that means any seqeunce values S such that
// sequenceNext < S < sequenceNumber and S has not been
// seen yet will never be seen. So we emit everything
// we have seen up to this point, in order.
if (count.Item2 >= sequenceRedundancy)
{
var removal = counts.Keys
.TakeWhile(seq => seq <= sequenceNumber)
.ToList();
foreach (var seq in removal)
{
count = counts[seq];
observer.OnNext(count.Item1);
counts.Remove(seq);
}
sequenceNext++;
}
// Emit stored values as long as we keep having the
// next sequence value.
while (counts.TryGetValue(sequenceNext, out count))
{
observer.OnNext(count.Item1);
counts.Remove(sequenceNext);
sequenceNext++;
}
},
observer.OnError,
() =>
{
// Emit in order any remaining values.
foreach (var count in counts.Values)
{
observer.OnNext(count.Item1);
}
observer.OnCompleted();
});
});
}
If you have two streams IObservable<Message> A and IObservable<Message> B, you would use this by doing Observable.Merge(A, B).Sequenced(msg => msg.SequenceNumber, 1, 2).
For your example marble diagram, this would look like the following, where the source column shows the values emitted by Observable.Merge(A, B) and the counts column shows the contents of the SortedDictionary after each step of the algorithm. I am assuming that the "messages" of the original source sequence (without any lost values) is (A,1), (B,2), (C,3), (D,4), (E,5), (F,6) where the second component of each message is its sequence number.
source | counts
-------|-----------
(A,1) | --> emit A
(A,1) | --> skip
(C,3) | (3,(C,1))
(B,2) | (3,(C,1)) --> emit B,C and remove C
(D,4) | --> emit D
(F,6) | (6,(F,1))
(F,6) | (6,(F,2)) --> emit F and remove
A similar question came up a while ago and I have a custom merge operator that when given ordered streams, it merges them in order but doesn't do deduplication.
Edit:
If you can "afford" it, you can use this custom merge and then distinctUntilChanged(Func1) to filter out subsequent messages with the same sequence number.
Observable<Message> messages = SortedMerge.create(
Arrays.asList(src1, src2, src3), (a, b) -> Long.compare(a.id, b.id))
.distinctUntilChanged(v -> v.id);
I have the below repro code which demonstrate a problem in a more complex flow:
static void Main(string[] args)
{
var r = Observable.Range(1, 10).Finally(() => Console.WriteLine("Disposed"));
var x = Observable.Create<int>(o =>
{
for (int i = 1; i < 11; i++)
{
o.OnNext(i);
}
o.OnCompleted();
return Disposable.Create(() => Console.WriteLine("Disposed"));
});
var src = x.Publish().RefCount();
var a = src.Where(i => i % 2 == 0).Do(i => Console.WriteLine("Pair:" + i));
var b = src.Where(i => i % 2 != 0).Do(i => Console.WriteLine("Even:" + i));
var c = Observable.Merge(a, b);
using (c.Subscribe(i => Console.WriteLine("final " + i), () => Console.WriteLine("Complete")))
{
Console.ReadKey();
}
}
running this snippet with r as src (var src = r.Publish().RefCount()) will produce all the numbers from 1 till 10,
switching the src to x(like in example) will produce only the pairs, actually the first observable to subscribe unless i change Publish() to Replay().
Why? What is the difference between r and x?
Thanks.
Although I do not have the patience to sort through the Rx.NET source code to find exactly what implementation detail causes this exact behavior, I can provide the following insight:
The difference in behavior your are seeing is caused by a race condition. The racers in this case are the subscriptions of a and b which happen as a result of your subscription to the observable returned by Observable.Merge. You subscribe to c, which in turn subscribes to a and b. a and b are defined in terms of a Publish and RefCount of either x or r, depending on which case you choose.
Here's what's happening.
src = r
In this case, you are using a custom Observable. When subscribed to, your custom observible immediately and synchronously begins to onNext the numbers 1 though 10, and then calls onCompleted. Interestingly enough, this subscription is caused by your Publish().RefCount() Observable when it is subscribe to the first time. It is subscribed to the first time by a, because a is the first parameter to Merge. So, before Merge has even subscribed to b, your subscription has already completed. Merge subscribes to b, which is the RefCount observable. That observable is already completed, so Merge looks for the next Observable to merge. Since there are no more Observables to merge, and because all of the existing Observables have completed, the merged observable completes.
The values onNext'd through your custom observable have traveled through the "pairs" observable, but not the "evens" observable. Therefore, you end up with the following:
// "pairs" (has this been named incorrectly?)
[2, 4, 6, 8, 10]
src = x
In this case, you are using the built-in Range method to create an Observable. When subscribed to, this Range Observable does something that eventually ends up yielding the numbers 1 though 10. Interesting. We haven't a clue what's happening in that method, or when it's happening. We can, however, make some observations about it. If we look at what happens when src = r (above), we can see that only the first subscription takes effect because the observable is yielding immediately and synchronously. Therefore, we can determine that the Range Observable must not be yielding in the same manner, but instead allows the application's control flow to execute the subscription to b before any values are yielded. The difference between your custom Observable and this Range Observable, is probably that the Range Observable is scheduling the yields to happen on the CurrentThread Scheduler.
How to avoid this kind of race condition:
var src = a.Publish(); // not ref count
var a = src.where(...);
var b = src.where(...);
var c = Observable.Merge(a, b);
var subscription = c.Subscribe(i => Console.WriteLine("final " + i), () => Console.WriteLine("Complete"))
// don't dispose of the subscription. The observable creates an auto-disposing subscription which will call dispose once `OnCompleted` or `OnError` is called.
src.Connect(); // connect to the underlying observable, *after* merge has subscribed to both a and b.
Notice that the solution to fixing the subscription to this composition of Observables was not to change how the source observable works, but instead to make sure your subscription logic isn't allowing any race conditions to exist. This is important, because trying to fix this problem in the Observable is simply changing behavior, not fixing the race. Had we changed the source and switched it out later, the subscription logic would still be buggy.
I suspect it's the schedulers. This change causes the two to behave identically:
var x = Observable.Create<int>(o =>
{
NewThreadScheduler.Default.Schedule(() =>
{
for (int i = 1; i < 11; i++)
{
o.OnNext(i);
}
o.OnCompleted();
});
return Disposable.Create(() => Console.WriteLine("Disposed"));
});
Whereas using Scheduler.Immediate gives the same behavior as yours.
I would like to do a little project to do some calculation and add the calculated results in listbox.
My code:
int SumLoop(int lowLimit, int highLimit)
{
int idx;
int totalSum = 0;
for (idx = lowLimit; idx <= highLimit; idx = idx + 1)
{
totalSum += idx;
}
return totalSum;
}
private void button1_Click(object sender, EventArgs e)
{
var test2 = Observable.Interval(TimeSpan.FromMilliseconds(1000)).Select(x=>(int)x).Take(10);
test2.Subscribe(n =>
{
this.BeginInvoke(new Action(() =>
{
listBox1.Items.Add("input:" + n);
listBox1.Items.Add("result:" + SumLoop(n,99900000));
}));
});
}
The result:
input:0
result:376307504
(stop a while)
input:1
result:376307504
(stop a while)
input:2
result:376307503
(stop a while)
input:3
result:376307501
(stop a while)
....
...
..
.
input:"9
result:376307468
If i would like to modify the interval constant from 1000 --> 10,
var test2 = Observable.Interval(TimeSpan.FromMilliseconds(10)).Select(x=>(int)x).Take(10);
The displaying behavior becomes different. The listbox will display all inputs and results just a shot. It seems that it waits all results to complete and then display everything to listbox. Why?
If i would like to keep using this constant (interval:10) and dont want to display everything just a shot. I want to display "Input :0" -->wait for calculation-->display "result:376307504"....
So, how can i do this?
Thankx for your help.
If I understand you correctly you're wanting to run the sum loop off the UI thread, here's how you would do that:
Observable
.Interval(TimeSpan.FromMilliseconds(1000))
.Select(x => (int)x)
.Select(x => SumLoop(x, 99900000))
.Take(10)
.ObserveOn(listBox1) // or ObserveOnDispatcher() if you're using WPF
.Subscribe(r => {
listBox1.Items.Add("result:" + r);
});
You should see the results trickle in on an interval of 10ms + ~500ms.
Instead of doing control.Invoke/control.BeginInvoke, you'll want to call .ObserveOnDispatcher() to get your action invoked on the UI thread:
Observable
.Interval(TimeSpan.FromMilliseconds(1000))
.Select(x=>(int)x)
.Take(10)
.Subscribe(x => {
listBox1.Items.Add("input:" + x);
listBox1.Items.Add("result:" + SumLoop(x, 99900000));
});
You said that if you change the interval from 1000 ms to 10ms, you observe different behavior.
The listbox will display all inputs and results just a shot.
I suspect this is because 10ms is so fast, all the actions you're executing are queued up. The UI thread comes around to execute them, and wham, executes everything that's queued.
In contrast, posting them every 1000ms (one second) allows the UI thread to execute one, rest, execute another one, rest, etc.
In the shared buffer memory problem , why is it that we can have at most (n-1) items in the buffer at the same time.
Where 'n' is the buffer's size .
Thanks!
In an OS development class in college, I had an adjunct teacher that claimed it was impossible to have a software-only solution that could use all N elements in the buffer.
I proved him wrong with something I decided to call the race track solution (inspired by the fact that I like to run track).
On a race track, you are not limited to a 400 meter race; a race can consist of more than one lap. What happens if two runners are neck and neck
in a race? How do you know whether they are tied, or whether one runner has lapped the other? The answer is simple: in a race, we don't monitor a runner's position
on the track; we monitor the distance each runner has traversed. Thus, when two runners are neck and neck, we can disambiguafy between a tie and when one runner has
lapped the other.
So, our algorithm has an N-element array, and manages a 2N race. We don't restart the producer/consumer's counter back to zero until they finish their respective 2N race.
We don't allow the producer to be more than one lap ahead of the consumer, and we don't allow the consumer to be ahead of the producer.
Actually, we only have to monitor the distance between the producer and consumer.
The code is as follows:
Item track[LAP];
int consIdx = 0;
int prodIdx = 0;
void consumer()
{ while(true)
{ int diff = abs(prodIdx - consIdx);
if(0 < diff) //If the consumer isn't tied
{ track[consIdx%LAP] = null;
consIdx = (consIdx + 1) % (2*LAP);
}
}
}
void producer()
{ while(true)
{ int diff = (prodIdx - consIdx);
if(diff < LAP) //If prod hasn't lapped cons
{ track[prodIdx%LAP] = Item(); //Advance on the 1-lap track.
prodIdx = (prodIdx + 1) % (2*LAP);//Advance in the 2-lap race.
}
}
}
It's been a while since I originally solved the problem, so this is according to my best recollection. Hopefully I didn't overlook any bugs.
Hope this helps!
Oops, here's a bug fix:
Item track[LAP];
int consIdx = 0;
int prodIdx = 0;
void consumer()
{ while(true)
{ int diff = prodIdx - consIdx; //When prodIdx wraps to 0 before consIdx,
diff = 0<=diff? diff: diff + (2*LAP); //think in 3 Laps until consIdx wraps to 0.
if(0 < diff) //If the consumer isn't tied
{ track[consIdx%LAP] = null;
consIdx = (consIdx + 1) % (2*LAP);
}
}
}
void producer()
{ while(true)
{ int diff = prodIdx - consIdx;
diff = 0<=diff? diff: diff + (2*LAP);
if(diff < LAP) //If prod hasn't lapped cons
{ track[prodIdx%LAP] = Item(); //Advance on the 1-lap track.
prodIdx = (prodIdx + 1) % (2*LAP);//Advance in the 2-lap race.
}
}
}
Well, theoretically a bounded buffer can hold elements upto its size. But what you are saying could be related to certain implementation quirks like a clean way of figuring out when the buffer is empty/full. This question -> Empty element in array-based bounded buffer deals with a similar thing. See if it helps.
However you can of course have implementations that have all n slots filled up. That's how the bounded buffer problem is defined anyway.