I'm trying to combine an IObservable that contains request information with an IObservable that contains response information. Both the request and response information contain an Id and a timestamp in the form of a DateTime. The Id on the Request will match the Id on the Response.
I need to combine the two streams based on Id and calculate the time elapsed between the Request's timestamp and the Response's timestamp. What is the best way to accomplish this:
You could join by coincidence.
from request in requests // 1
join response in responses // 2
on responses.Where(response => response.Id == request.Id) // 1,3
equals Observable.Empty<Unit>() // 2
where request.Id == response.Id // 4
select response.Time - request.Time // 5
Think of it like this:
Create a window for each request.
Project every response into every request window.
When a response matches a request, close that request window.
For each request, ignore all responses that do not match.
Project the time difference for the response that does match.
Update:
Request windows are opened by from and closed by on.
Response windows are opened by join and closed by equals. The response duration specified by equals is Empty, thus responses aren't actually windows - they are treated as point events and simply projected once into whatever request windows are currently open.
#3 requires a Where operator because it defines the duration of each request; i.e., it indicates when a request window closes, which is when a response with a matching ID arrives.
#4 is required because of #3. Think about it - we're cross-joining every response with every request window that's currently open. Request windows only close when a matching response arrives. That means every response that doesn't match is paired with every request, but you only care about the last response - the one that matches.
If you don't have concurrent requests, Zip is probably the operator you are looking for:
http://rxmarbles.com/#zip
Zip will combine two observables in a single sequence that produces elements at a rate that is defined by the slowest observable being combined.
In case you have concurrent requests, you should probably try a different approach that does not require you to combine the two observables. You could for instance store the request ID and DateTime in a ConcurrentDictionary and subscribe to the responses observable where you lookup for the corresponding request from the ConcurrentDictionary (don't forget to remove the entry after reading it).
Related
Is it possible to get multiple (let's say 300) calls with a single GET request? The protocol is HTTP/1.1 to a REST interface.
To be more explicit:
There is a REST-API and i have to call that through GET method. But the problem is that i have to make this request hundreds of times consecutively every day. So instead of many single requests (open connection, retrieve data, close connection) i would like get all hundreds of responses at once. Is it possible to concatenate many GET requests to retrieve all the responses at once?
You can modify your api to run over the given data 100 times and then send the response. Since you can request a get method only once
for e.g you can send an array of values and you can run the logic for each item in an array and get back an array containing all information at once
I have an application that takes user's input personal details (name, dob, etc.) and looks for a match in the database. I am not creating a record, nor editing one.
Now normally I'd use GET, but GET doesn't allow me a body, and all the data that needs to be checked has to be sent up. I run a database query against all the user's inputs to find a match. If the database returns a match then one is found, otherwise it returns an empty array and no match is found.
So can you recommend the correct RESTful method (HTTP) that I might use for this?
Thanks
As the semantics of the payload of a POST operation are defined by the service itself, it should be used in any case where the other operations do not fit. POST is therefore not only used to create new resources but also to trigger certain calculations and stuff like that.
The general question however is, why do you need such a method? If you fear overwriting any changes done by an other client to the resource between you fetching the state of a resource and requesting an update, i.e., you should consider using conditional requests as defined in RFC 7232 where either an ETag hash is calculated for the current state or the Last-Modified value is taken and added to response header. A client could then send the request including the payload to check first including a If-Match or If-Unmodified-Since header requesting the server to apply an update only if the precondition holds. If it fails the server would tell you that with a 412 Precondition Failed error response.
I have REST service and client (Android app) that send POST request to REST service. On client side there are documents (orders) that need to be synchronized with web server. Synchronization means that client sends POST request to REST service for each order. When REST service receive POST request it writes data to database and sends response with status 200 to client. Client receives 200 and mark that order as synchronized.
Problem is when connection is broken after a server sent status 200 response but before client received response. Client doesn't mark order as synchronized. Next time client sends again this order and servers write it again in database so we have same order two times.
What is good practice to deal with this kind of problem?
Problem is when connection is broken after a server sent status 200 response but before client received response. Client doesn't mark order as synchronized. Next time client sends again this order and servers write it again in database so we have same order two times.
Welcome to the world of unreliable messaging.
What is good practice to deal with this kind of problem?
You should review Nobody Needs Reliable Messaging, by Marc de Graauw (2010).
The cornerstone of reliable messaging is idempotent request handling. Idempotent semantics are described this way
A request method is considered "idempotent" if the intended effect on the server of multiple identical requests with that method is the same as the effect for a single such request.
Simply fussing with the request method, however, doesn't get you anything. First, the other semantics in the message may not align with the idempotent request methods, and second the server needs to know how to implement the effect as intended.
There are two basic patterns to idempotent request handling. The simpler of these is set, meaning "overwrite the current representation with the one I am providing".
// X == 6
server.setX(7)
// X == 7
server.setX(7) <- a second, identical request, but the _effect_ is the same.
// X == 7
The alternative is test and set (sometimes called compare and swap); in this pattern, the request has two parts - a predicate to determine is some condition holds, and the change to apply if the condition does hold.
// X == 6
server.testAndSetX(6,7)
// X == 7
server.testAndSetX(6,7) <- this is a no op, because 7 != 6
// X == 7
That's the core idea.
From your description, what you are doing is manipulating a collection of orders.
The same basic idea works there. If you can calculate a unique identifier from the information in the request, then you can treat your collection like a set/key-value store.
// collection.get(Id.of(7)) == Nothing
collection.put(Id.of(7), 7)
// collection.get(Id.of(7)) == Just(7)
collection.put(Id.of(7), 7) <- a second, identical request, but the _effect_ is the same.
// collection.get(Id.of(7)) == Just(7)
When that isn't an option, then you need some property of the collection that will change when your edit is made, encoded into the request
if (collection.size() == 3) {
collection.append(7)
}
A generic way to manage something like this is to consider version numbers -- each time a change is made, the version number is incremented as part of the same transaction
// begin transaction
if (resource.version.get() == expectedVersion) {
resource.version.set(1 + expectedVersion)
resource.applyChange(request)
}
// end transaction
For a real world example, consider JSON Patch, which includes a test operation that can be used as a condition to prevent "concurrent" modification of a document.
What we're describing in all of these test and set scenarios is the notion of a conditional request
Conditional requests are HTTP requests [RFC7231] that include one or more header fields indicating a precondition to be tested before applying the method semantics to the target resource.
What the conditional requests specification gives you is a generic way to describe conditions in the meta data of your requests and responses, so that generic http components can usefully contribute.
Note well: what this works gets us is not a guarantee that the server will do what the client wants. Instead, it's a weaker: that the client can safely repeat the request until it receives the acknowledgement from the server.
Surely your documents must have an unique identifier. The semantically correct way would be to use the If-None-Match header where you send that identifier.
Then the server checks whether a document with that identifier already exists, and will respond with a 412 Precondition Failed if that is the case.
One of possible options would be validation on server side. Order should have some uniqueness parameter: name or id or something else. But this parameter should be send by client also. Then you get this value (e.x. if name is unique and client send it), find this order in database. If order is founded then you don't need to save it into database and should send 409 Conflict response to client. If you din't find such order in database then you save it and send 201 Ok response.
Best practices:
201 Ok for POST
409 Conflict - if resource already exists
Your requests should be idempotent.
From your description, you should be using PUT instead of POST.
Client side generated Ids (guids) and Upsert logic server side, help achieve this.
This way you can implement a retry logic client side for failed requests, without introducing multiple records.
I'm having some trouble solving a issue in my code, I hope you can help me.
I have two modules, A and B. Module A does requests to B, and after a number of cycles B sends a multi-cycle response to A. A can hold up to 8 requests waiting to be responded, and the responses from B don't necessarily come back ordered. That's why we use an ID, to identify the returning data.
To verify this behaviour, I have a scoreboard with several checkers. One of the checkings that I do is if the ID used for a request is free or not. To do that, I keep an associative array with the IDs pending to be responded, and I insert, check and delete items as needed. I control this from two interfaces and monitors, one for the requests and another one for the responses. The response monitor, being the responses more than one cycle long, waits until it has all the data to send a transaction to the scoreboard, where I update my structs.
The problem comes the moment that once that A sees that it is actually getting a valid response from B, frees the ID and can use it for a new request. That is happening in some of my simulations, and since I won't receive the transaction until all the response is complete, block A is doing a new request with an ID I won't know it's legit to use until I get the complete transaction from the monitor.
Any ideas on how to solve this? Thanks!
In the cycle that you see a response from B, why don't you move the request from A into another associative array, one that represents responses that have been initiated. That way you'll have a free slot in original array to handle new requests from A, but now you'll have the new, second array to handle the multi-cycle responses that have already begun.
As part of a REST Service, I need to implement a search call. The basic idea is that the user can POST a new search, and GET the results:
POST http://localhost/api/search
GET http://localhost/api/search?id=123
However, my search may run a few minutes, and return partial results until it is done. That is, the GET-Request would return something like:
status: running
results: a, b, c.
while the next GET-Request might return
status: completed
results: a, b, c, d, e.
This contradicts the semantics of a RESTful GET request. The request should always return the same result when called several times. For example, when the user uses a caching proxy, the full results might never be delivered to the user.
Question: Is there any way to provide a truly RESTful implementation for long running searches with partial results?
While the search is executing, you could set the appropriate response headers (e.g. Expires or max-age) to indicate that the response should not be cached (HTTP/1.1 14.9.3, 13.4).
Once the search result is complete, you could then send a more appropriate Expires / max-age header to allow or extend the cacheability of the result.
The burden would be on the client to re-query the resource until its search status is complete. The client could maybe use the value of the Expires header to determine when it should re-query for updated results.
Alongside of this, you could also use a custom 2XX status code to indicate that the result is not yet complete. Maybe a HTTP/1.1 299 In Progress, or whatever makes sense. The spec indicates that HTTP status codes are extensible.
For the record, your statement:
This contradicts the semantics of a RESTful GET request. The request should always return the same result when called several times.
is not true for GET requests - resources can change. That GET requests are idempotent only means that "...the side-effects of N > 0 identical requests is the same as for a single request". [spec]
A few days ago I happend to stumble upon a blog post over at reddit that deals with your problem. You might want to check it out: Bill Higgin's RESTy long-ops.
Happy reading.
It's not a problem if the first GET request returns partial results, and the second GET request returns the full results. That's because the first GET request doesn't cause the result of the second request to change: That request would have returned the full results, even if the first GET hadn't been issued. "idempotent" doesn't mean identical results. It means that the first GET doesn't affect the second GET.
It would be a problem if the first GET request returned partial results, and the second GET would return the remaining results (first GET returns A, B, C; second GET returns D, E, F). Here the first GET changes the second result, so it's not RESTful.
Maybe not the most elegant answer, but will get around caching proxies: Just don't send the same query twice. Add a timestamp to the query (&time=1318355458). This way, each request is unique (you could also add milliseconds to the time if you're requesting > 1hz).
As for following the doctrine of "The request should always return the same result when called several times", it seems logically contradictory to the goal of returning partial results at different times for the same query.
Could you do a wait instead of a poll if you just want the full results?
Why can't you provide a resource as part of your POST that will get the results PUT to it? You're providing a 'call back' REST interface so, rather than polling, the client process waits for a PUT to the provided resource. You can then either GET the results or the results could be included in the PUT.