Remove read topic from DDS - publish-subscribe

I have a problem with subscribing the data (using the java platform). When a subscriber subscribes to a topic, that subscribed data must be removed from the DDS. But in my case whenever I subscribe to the data the same data is subscribed many times. The data is not removed from the DDS. I tried with QoS but I don't know how to use it.
Please suggest how I can remove the read data from the DDS.

This behavior is not caused by your QoS settings, but by your method of accessing the DataReader. When you retrieve your data, you are probably calling something like the following read() in a loop:
FooReader.read(
dataSeq, infoSeq, 10,
ANY_SAMPLE_STATE.value,
ANY_VIEW_STATE.value,
ANY_INSTANCE_STATE.value);
The read() method invoked like this will return all currently available samples in your FooReader. After the read(), those samples still remain available in the FooReader, that is how the read() method behaves. Think of a read as a "peek". The next time that you invoke the read() method in this way, you will see all samples that you saw before, unless they have been overwritten by a new update from a DataWriter.
To resolve your issue, you could replace the read() with a take(), like this:
FooReader.take(
dataSeq, infoSeq, 10,
ANY_SAMPLE_STATE.value,
ANY_VIEW_STATE.value,
ANY_INSTANCE_STATE.value);
The take() method is different from the read() method in that it does a destructive read; it not only reads the data but also removes it from FooReader. That way, you will never receive the same sample twice. In fact, if you consistently use take() as opposed to read(), you will never be able to see any sample twice.
Another way to resolve your issue is to stick with read(), but adjust the requested SAMPLE_STATE, from ANY to NOT_READ, like this:
FooReader.read(
dataSeq, infoSeq, 10,
NOT_READ_SAMPLE_STATE.value,
ANY_VIEW_STATE.value,
ANY_INSTANCE_STATE.value);
That way, you will only read samples that you have not read previously. The difference with take() in this case is that the data does remain available in your FooReader, which might be useful if you want to re-read it at a later stage (in which case you need to use the ANY sample state as opposed to NOT_READ to obtain previously read samples).

Related

reactive 4.2.0 net Subject<T> ReplaySubject<T>

I am using ReplaySubject as a pipe when I feed the Observer with OnNext messages and then I subscribe to the message in another part of the code.
I tried ISubject first, by using Subject.Create(observer:, observable:) but this seems to do nothing, or in other words, the Observer is a sink that goes nowhere. Is there some code that will make the Subject do something pipe-like. For example, I want to inject onNext messages with a integral countN and I want the observable to broadcast N-many 0.00E0(s) in sequence.
UPDATE:
It seems I need to implement the ISubject interface. Is there source code for a simple implementation that functions the same as ReplaySubject, with simple, proper memory handling of buffer size and Observer collection and Disposable return object.
Ok, so you use
new Subject<Int32>()
to get a subject that is a pipe-line.
Someone care to explain what Subject.Create does?
Can you inherit from Subject or ReplaySubject? Should you be able to?

Why does `libpq` use polling rather than notification for data fetch?

I am reading libpq reference. It has both of sync and async methods. Bu I discovered something strange.
When I see PQsendQuery function, it seems to send a query and return immediately. And I expected a callback function to get notified, but there was no such thing and the manual says to poll for data availability.
I don't understand why async method is written in polling way. Anyway, as libp is the official client implementation, I believe there should be a good reason for this design. What is that? Or am I missing correct callback stuffs mentioned somewhere else?
In the execution model of a mono-threaded program, the execution flow can't be interrupted by data coming back from an asynchronous query, or more generally a network socket. Only signals (SIGTERM and friends) may interrupt the flow, but signals can't be hooked to data coming in.
That's why having a callback to get notified of incoming data is not possible. The piece of code in libpq that would be necessary to emit the callback would never run if your code doesn't call it. And if you have to call it, that defeats the whole point of a callback.
There are libraries like Qt that provide callbacks, but they're architectured from the ground up with a main loop that acts as an event processor. The user code is organized in callbacks and event-based processing of incoming data is possible. But in this case the library takes ownership of the execution flow, meaning its mainloop polls the data sources. That just shifts the responsibility to another piece of code outside of libpq.
This page is describing how I can get be notified for async result fetch.
http://www.postgresql.org/docs/9.3/static/libpq-events.html#LIBPQ-EVENTS-PROC
PGEVT_RESULTCREATE
The result creation event is fired in response to any query execution
function that generates a result, including PQgetResult. This event
will only be fired after the result has been created successfully.
typedef struct {
PGconn *conn;
PGresult *result; } PGEventResultCreate; When a PGEVT_RESULTCREATE event is received, the evtInfo pointer should be cast to a
PGEventResultCreate *. The conn is the connection used to generate the
result. This is the ideal place to initialize any instanceData that
needs to be associated with the result. If the event procedure fails,
the result will be cleared and the failure will be propagated. The
event procedure must not try to PQclear the result object for itself.
When returning a failure code, all cleanup must be performed as no
PGEVT_RESULTDESTROY event will be sent.

What do the various ISubject implementations do and when would they be used?

I have a fairly good idea of what the Subject class does and when to use it, but I've just been looking through the language reference on msdn and see there are various other ISubject implementations such as:
AsyncSubject
BehaviorSubject
ReplaySubject
As the documentation is pretty thin on the ground, whats the point of each of these types and under what situations would you use them?
These subjects all share a common property - they take some (or all) of what gets posted to them via OnNext and record it and play it back to you - i.e. they take a Hot Observable and make it Cold. This means, that if you Subscribe to any of these more than once (i.e. Subscribe => Unsubscribe => Subscribe again), you'll see at least one of the same value again.
ReplaySubject: Every time you subscribe to the Subject, you get the entire history of what has been posted replayed back to you, as fast as possible (or a subset, like the last n items)
AsyncSubject: Always plays back the last item posted and completes, but only after the source has completed. This Subject is awesome for async functions, since you can write them without worrying about race conditions: even if someone Subscribes after the async method completes, they get the result.
BehaviorSubject: Kind of like ReplaySubject but with a buffer of one, so you always get the last thing that was posted. You also can provide an initial value. Always provides one item instantly on Subscribe.
In light of the latest version (v1.0.2856.0) and to keep this question up to date, there has been a new set of subject classes:
FastSubject, FastBehaviorSubject, FastAsyncSubject and FastReplaySubject
As per the release notes they
are much faster than regular subjects
but:
don’t decouple producer and consumer by an IScheduler
(effectively limiting them to
ImmediateScheduler);
don’t protect against stack overflow;
don’t synchronize input messages.
Fast subjects are used by Publish and
Prune operators if no scheduler is
specified.
In regards to AsyncSubject
This code:
var s = new AsyncSubject<int>();
s.OnNext(1);
s.Subscribe(Console.WriteLine);
s.OnNext(2);
s.OnNext(3);
s.OnCompleted();
prints a single value 3. And it prints same if subscription is moved to after completion. So it plays back not the first, but the last item, plays it after completion (until complete, it does not produce values), and it does not work like Subject before completion.
See this Prune discussion for more info (AsyncSubject is basically the same as Prune)
Paul's answer pretty much nails it. There's a few things worth adding, though:
AsyncSubject works as Paul says, but only after the source completes. Before that, it works like Subject (where "live" values are received by subscribers)
AsyncSubject has changed since I last ran tests against it. It no longer acts as a live subject before completion, but waits for completion before it emits a value. And, as Sergey mentions, it returns the last value, not the first (though I should have caught that as that's always been the case)
AsyncSubject is used by Prune, FromAsyncPattern, ToAsync and probably a few others
BehaviorSubject is used by overloads of Publish that accept an initial value
ReplaySubject is used by Replay
NOTE: All operator references above refer to the publishing set of operators as they were before they were replaced with generalised publish operators in rev 2838 (Christmas '10) as it has been mentioned that the original operators will be re-added

An IOCP documentation interpretation question - buffer ownership ambiguity

Since I'm not a native English speaker I might be missing something so maybe someone here knows better than me.
Taken from WSASend's doumentation at MSDN:
lpBuffers [in]
A pointer to an array of WSABUF
structures. Each WSABUF structure
contains a pointer to a buffer and the
length, in bytes, of the buffer. For a
Winsock application, once the WSASend
function is called, the system owns
these buffers and the application may
not access them. This array must
remain valid for the duration of the
send operation.
Ok, can you see the bold text? That's the unclear spot!
I can think of two translations for this line (might be something else, you name it):
Translation 1 - "buffers" refers to the OVERLAPPED structure that I pass this function when calling it. I may reuse the object again only when getting a completion notification about it.
Translation 2 - "buffers" refer to the actual buffers, those with the data I'm sending. If the WSABUF object points to one buffer, then I cannot touch this buffer until the operation is complete.
Can anyone tell what's the right interpretation to that line?
And..... If the answer is the second one - how would you resolve it?
Because to me it implies that for each and every data/buffer I'm sending I must retain a copy of it at the sender side - thus having MANY "pending" buffers (in different sizes) on an high traffic application, which really going to hurt "scalability".
Statement 1:
In addition to the above paragraph (the "And...."), I thought that IOCP copies the data to-be-sent to it's own buffer and sends from there, unless you set SO_SNDBUF to zero.
Statement 2:
I use stack-allocated buffers (you know, something like char cBuff[1024]; at the function body - if the translation to the main question is the second option (i.e buffers must stay as they are until the send is complete), then... that really screws things up big-time! Can you think of a way to resolve it? (I know, I asked it in other words above).
The answer is that the overlapped structure and the data buffer itself cannot be reused or released until the completion for the operation occurs.
This is because the operation is completed asynchronously so even if the data is eventually copied into operating system owned buffers in the TCP/IP stack that may not occur until some time in the future and you're notified of when by the write completion occurring. Note that with write completions these may be delayed for a surprising amount of time if you're sending without explicit flow control and relying on the the TCP stack to do flow control for you (see here: some OVERLAPS using WSASend not returning in a timely manner using GetQueuedCompletionStatus?) ...
You can't use stack allocated buffers unless you place an event in the overlapped structure and block on it until the async operation completes; there's not a lot of point in doing that as you add complexity over a normal blocking call and you don't gain a great deal by issuing the call async and then waiting on it.
In my IOCP server framework (which you can get for free from here) I use dynamically allocated buffers which include the OVERLAPPED structure and which are reference counted. This means that the cleanup (in my case they're returned to a pool for reuse) happens when the completion occurs and the reference is released. It also means that you can choose to continue to use the buffer after the operation and the cleanup is still simple.
See also here: I/O Completion Port, How to free Per Socket Context and Per I/O Context?

How can I get a callback when there is some data to read on a boost.asio stream without reading it into a buffer?

It seems that since boost 1.40.0 there has been a change to the way that the the async_read_some() call works.
Previously, you could pass in a null_buffer and you would get a callback when there was data to read, but without the framework reading the data into any buffer (because there wasn't one!). This basically allowed you to write code that acted like a select() call, where you would be told when your socket had some data on it.
In the new code the behaviour has been changed to work in the following way:
If the total size of all buffers in the sequence mb is 0, the asynchronous read operation shall complete immediately and pass 0 as the argument to the handler that specifies the number of bytes read.
This means that my old (and incidentally, the method shown in this official example) way of detecting data on the socket no longer works. The problem for me is that I need a way detecting this because I've layered my own streaming classes on-top of the asio socket streams and as such, I cannot just read data off the sockets that my streams will expect to be there. The only workaround I can think of right now is to read a single byte, store it and when my stream classes then request some bytes, return that byte if one is set: not pretty.
Does anyone know of a better way to implement this kind of behaviour under the latest boost.asio code?
My quick test with an official example with boost-1.41 works... So I think it still should work (if you use null_buffers)