I am trying to build a lightstreamer client for Matlab. There do exist a couple of libraries for platforms like JAVA, Python, .Net etc. But unfortunately not Matlab.
However, it turns out that most of these client implementations use the very same text-mode protocol for lightstreamer which is pretty basic HTTP requesting.
I figured out how to establish/close a lightstreamer session. I get the sessionId and I can use this id to subscribe to the data I want to stream. But although I do get a valid response for the subscription call, there is no data pushed.
I use the urlead2 function and the response seems fine:
[output,extras] = urlread2([lightstream_url,'/lightstreamer/control.txt'],'POST',body,headers);
allHeaders =
Response: {'HTTP/1.1 200 OK'}
Server: {'Lightstreamer'}
Content_Type: {'text/plain; charset=iso-8859-1'}
Cache_Control: {'no-store' 'no-cache'}
Pragma: {'no-cache'}
Expires: {'Thu, 1 Jan 1970 00:00:00 GMT'}
Date: {'Wed, 8 Apr 2015 11:15:02 GMT'}
Content_Length: {'4'}
status =
value: 200
msg: 'OK'
isGood =
1
output =
OK
It is correct that the response body contains "OK ", this is documented (documentation, page 20ff.), but there is supposed to be the stream data itself as well, isn't it?
So how do I get the actual data?
Somewhere in your code you should have a create_session.txt/bind_session.txt request, otherwise you should not have a valid session id that is required to obtain an OK answer from a control.txt request (e.g. the following generates the SYNC ERROR, that means that the server does not recognize the specified session: http://push.lightstreamer.com/lightstreamer/control.txt?LS_op=add&LS_session=invalid )
The data stream is not received on the control.txt response, that OK response simply means "OK I have added the subscription to your session".
The data stream is received on the create_session.txt/bind_session.txt response. Sections 4.1 and 4.2 + section 4.5 on the document you linked should explain how the data is received
I've found that opening a polling connection by setting LS_polling=true works fine without needing a listner. urlread2 hangs if you leave LS_polling as the default of false.
Create the session with /lightstreamer/create_session.txt
Request a subscription with /lightstreamer/control.txt
Repeatedly poll the connection to get the data with
/lightstreamer/bind_session.txt
The return from urlread2 will look something like this:
d =
OK
SessionId:S9b09da8ebd6b835aT5316913
ControlAddress:apd119a.marketdatasystems.com
KeepaliveMillis:1000
MaxBandwidth:0.0
RequestLimit:50000
1,1|10162.00|0.00|0.00
2,2|10686.8|TRADEABLE|0.5524861
2,13|1202.6|CLOSED|0.5714285
2,14|5900.51|CLOSED|0.5714285
...
LOOP 1000
Related
I am designing a ReST API which follows the basic CRUD pattern.
My API can receive a request to update a resource which may take a short time to process. Ideally I would like to inform clients that a new version is about to be available and that there is some uncertainty over when the version I have cached actually expires.
So the process I intend to use something like this (improvements welcome):
client: GET /some/item
myapi: 200 OK
last-modified: time-stamp-of-v1
etag: some-hash-relating-to-v1-of-my-item-in-this-format
content: json or whatever
data/for/some/item/v1...
client: PUT /some/item
if-match: some-hash-relating-to-v1-of-my-item-in-this-format
content: json or whatever
data/for/some/item/v2...
myapi: 202 ACCEPTED,
content: json or whatever
time-accepted: time-stamp-after-v1-but-before-v2
your item will be at /some/item
here is a URI /some/taskid to track progress
while upload is pending:
client: GET /some/item
myapi: 200 OK
some/item ...
last-modified: time-stamp-of-v1
etag: some-hash-relating-to-v1-of-my-item-in-this-format
>>>> expires: time-stamp-after-v1-but-before-v2 <<<
>>>> warning: 110 Response is stale <<<<
content: json or whatever
data/for/some/item/v1...
client: GET /some/task/id
myapi: 200 OK
content: json or whatever
time-accepted: time-stamp-after-v1-but-before-v2
your item will be at /some/item
status/of/upload/v2...
after task completed:
client: GET /some/item
myapi: 200 OKAY
some/item/v2 ...
last-modified: time-stamp-of-v2
etag: some-hash-relating-to-v2-of-my-item-in-this-format
content: json or whatever
data/for/some/item/v2...
client: GET /some/task/id
myapi: 303 SEE OTHER
look-here: /some/item
If you are a proxy and know know your content is stale you can put "warning: 110 - response is stale" in the header.
However, in this case the data is not actually invalid yet.
I would like to say that I can guarantee it is valid up until the time I received and passed on the upload request (time-stamp-after-v1-but-before-v2 or later as if I am in contact with the upload server). It hasn't really expired at the time I receive the upload request. I just expect its going to.
(In fact if the request fails it might not be updated at all).
Now the default choice is just to serve the old content and let the client catch up on its own. This has high latency. If possible, I would like to do better.
For example, if the client knows the document is about to expire it could poll more often or it could try to upgrade the connection to a web-socket and get sent an update the moment I get it (would that still count as ReST?)
There is another case where using expired data must be avoided at all costs. For that scenario I think I want to tell the client that the resource is temporarily unavailable. Using the warning and expires fields as I have above seems correct there. Though it might be better to send a 503 with a suitable retry-after header.
So the question is: how should I reply to a GET while the upload of a new version is pending?
In anticipation of answers along the lines of use a messaging framework like AMQP or zeroMQ instead for low latency, I should point out this API is acting as a AMQP gateway/proxy for clients unwilling to use AMQP directly. Information on using webhooks or websockets would be still be interesting.
Some related useful content is:
How to proper design a restful API to invalidate a cache?
https://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html
HTTP status code for temporarily unavailable pages
http://www.albertoleal.me/posts/how-to-prevent-race-conditions-in-restful-apis.html
(the etag prevents races from simultaneously uploads)
Tl;Dr;
While upload is pending send:
client: GET /some/item
myapi: 200 OK
some/item ...
last-modified: time-stamp-of-v1
etag: some-hash-relating-to-v1-of-my-item-in-this-format
expires: time-stamp-after-v1-but-before-v2
stale-while-revalidate: 100
warning: 110 Response is stale
content: json or whatever
data/for/some/item/v1...
At first sight it looks like using Warning is not correct. See https://www.rfc-editor.org/rfc/rfc7234#section-5.5.0
In this case the server is acting as a proxy (though not an HTTP proxy).
It is not disconnected from AMQP and "A proxy MUST NOT send stale responses" unless it is disconnected.
This is annoying as it looked like the right thing to do here.
4.2.4. Serving Stale Responses
A "stale" response is one that either has explicit expiry
information or is allowed to have heuristic expiry calculated, but
is not fresh according to the calculations in Section 4.2.
A cache MUST NOT generate a stale response if it is prohibited by
an explicit in-protocol directive (e.g., by a "no-store" or
"no-cache" cache directive, a "must-revalidate"
cache-response-directive, or an applicable "s-maxage" or
"proxy-revalidate" cache-response-directive; see Section 5.2.2).
**> A cache MUST NOT send stale responses unless it is disconnected
(i.e., it cannot contact the origin server or otherwise find a
forward path) or doing so is explicitly allowed (e.g., by the
max-stale request directive; see Section 5.2.1).**
A cache SHOULD generate a Warning header field with the 110
warn-code (see Section 5.5.1) in stale responses. Likewise, a
cache SHOULD generate a 112 warn-code (see Section 5.5.3) in stale
responses if the cache is disconnected.
A cache SHOULD NOT generate a new Warning header field when
forwarding a response that does not have an Age header field, even if
the response is already stale. A cache need not validate a response
that merely became stale in transit.
Also
4.4. Invalidation
Because unsafe request methods (Section 4.2.1 of [RFC7231]) such as
PUT, POST or DELETE have the potential for changing state on the
origin server, intervening caches can use them to keep their contents
up to date.
**> A cache MUST invalidate the effective Request URI (Section 5.5 of
[RFC7230]) as well as the URI(s) in the Location and Content-Location
response header fields (if present) when a non-error status code is
received in response to an unsafe request method.**
However a warning is required if stale-while-revalidate is used (see https://www.rfc-editor.org/rfc/rfc5861)
The stale-while-revalidate Cache-Control Extension
When present in an HTTP response, the stale-while-revalidate Cache-
Control extension indicates that caches MAY serve the response in
which it appears after it becomes stale, up to the indicated number
of seconds.
stale-while-revalidate = "stale-while-revalidate" "=" delta-seconds
If a cached response is served stale due to the presence of this
extension, the cache SHOULD attempt to revalidate it while still
serving stale responses (i.e., without blocking).
I thought this was unclear so I submitted an errata. This was rejected (though at the time of writing its still showing as reported) on the grounds that the cache control extensions in rfc5861 override the MUST NOT in rfc7234 ("doing so is explicitly allowed" see above).
It is okay to use expires but its not very helpful as it doesn't imply anything.
5.3. Expires
The "Expires" header field gives the date/time after which the
response is considered stale. See Section 4.2 for further discussion
of the freshness model.
**> The presence of an Expires field does not imply that the original
resource will change or cease to exist at, before, or after that
time.**
I'm writing a C# Web API server application, and will send JSON to it via a Mirth HTTP Sender destination. This post is about how to handle error conditions. Specifically, there are three scenarios I want to handle:
Sometimes we take the C# application server offline for a short period for system upgrade or maintenance, and Mirth is unable to connect at all. I want Mirth to queue all messages in order, and when the server is available, process them in the order they were received.
The server receives the request, but rejects it due to a problem with the content of the request, e.g., missing a required field. In accordance with REST conventions, the server will return a 400-level HTTP response. This message would be rejected every time it's submitted, so it should not be re-sent; just log the failure and move on to the next message.
The server receives the request, but something goes wrong on the server, and the server returns an HTTP 500 Server Error response. This would be the appropriate response, for example, when something in the server environment has gone wrong. One real-world example was the time the Web API server was running, but somebody rebooted the database server. REST conventions would suggest we continue to resend the message until the transient problem has been resolved.
For #1, initially I had it queue on failure/always, but it appears the response transformer never runs for messages that were queued (at least, the debug statements never showed in the log). I have turned queueing off, and set it to retry every ten seconds for an hour, and that seems to give the desired behavior. Am I on the right track here, or missing something?
For #2 and #3, returning any HTTP 400 or 500 error invokes the 1-hour retries. What I want is to apply the 1-hour retries for the 500 errors, but not the 400 errors. I’ve tried responseStatus = SENT in the response transformer, but the response transformer only runs once, after the hour has expired, and not for each retry.
This seems like a common problem, yet I’m not finding a solution. How are the rest of you handling this?
You're close!
So by default, the response transformer will only run if there's a response payload to transform. For connection problems, or possibly for 4xx/5xx responses that contain no payload, the response transformer won't execute.
However, if you set your response data types (From the Summary -> Set Data Types dialog, or from the Destinations -> Edit Response, Message Templates tab) to Raw, then the response transformer will execute all the time. The reason being that the Raw data type considers even an empty payload to be "transformable".
So turn queuing back on, and set your response data types to Raw. Then in the response transformer, if you look at the Reference tab there's a category for HTTP Sender:
You'll want the "response status line", that's the "HTTP/1.1 200 OK" line of the response that contains the response code. Here's a response transformer script that forces 4xx responses to error:
if (responseStatus == QUEUED) {
var statusLine = $('responseStatusLine');
if (statusLine) {
var parts = statusLine.split(' ');
if (parts.length >= 2) {
var responseCode = parseInt(parts[1], 10);
// Force 4xx responses to error
if (responseCode >= 400 && responseCode < 500) {
responseStatus = ERROR;
responseStatusMessage = statusLine;
}
}
}
}
I need to access the Date: header when I handle the request, but this seems to be "swallowed" by the framework; any other header (even made up FooBar ones) show up and I can get them, but this gives me None (I'm using Postman to send a simple GET request - everything else works just fine):
println("Date: " + request.headers.get("Date").getOrElse("no date!"))
returns "no date!" no matter how I try to send something sensible.
I'm wondering whether this gets processed before the request object reaches my Action.
I need the actual string value sent, as this should be part of the request's signature - so an equivalent Date object representing the same value would not be of much use (as it needs to be part of the hash, to avoid replay attacks).
Just as a test, I replaced the Date header with a Date-Auth one, and this one shows up just fine:
ArrayBuffer((Date-Auth, ArrayBuffer(Wed, 15 Nov 2014 06:25:24 GMT))
Any ideas or suggestions greatly appreciated!
Are you sure there is a Date Header in your request (tested with tools like firebug or wireshark)?
Browsers do not need to send a Date header.
RFC 2616 (HTTP 1.1) from the Date section (14.18)
Clients SHOULD only send a Date header field in messages that include an entity-body, as in the case of the PUT and POST requests, and even then it is optional. A client without a clock MUST NOT send a Date header field in a request.
I stand corrected - it turns out that Chrome blocks a whole bunch of headers:
http://www.getpostman.com/docs/requests
I wrote a Python Flask test server and, in fact, the Date header is not there.
That page has also a fix, which works just fine with Postman Version 0.10.4.3 and Interceptor(1).
sorry for wasting everyone's time!
1 Incidentally, IMO Postman is the best REST client and has now also some awesome looks, beyond incredible functionality. If you're working with REST APIs, I highly recommend it.
i have this as my dgrid constructor ,
var MyQuickGrid = declare([onDemandGrid,Keyboard,Selection,ColumnHider,ColumnResizer,ColumnReorder]);
and the partial config_opts are
var config_opts = {
loadingMessage: " Loading data...",
noDataMessage: "No results found."
}
.
.
.
lang.mixin(grid_opts,config_opts);
window.grid = new MyQuickGrid(grid_opts,'node_of_intrest');
however when i request data from the server using a JsonRESTStore, and the returned json data is empty i.e "[]" the grid does not display the "noDataMessage", i initially thought this was because of the headers i was returning, since i was returning 200 OK even for empty results set, i changed this to 204 No Content but still nothing seems to be working. I would appreciate a work around, or even a way to know if the grid failed to get results cuz this native feature for some reasons seem to be too smart for me for now.
Mentioned by nbjoerg on IRC
make sure your JsonRest server is setting the proper Content-Range headers in its query responses (e.g. in this case it should be "items 0-0/0").
For more info on how Dojo expects JsonRest endpoints to behave, see http://dojotoolkit.org/reference-guide/1.9/quickstart/rest.html
Here's an example of the headers returned by a JsonRest service for which noDataMessage displays fine:
Connection:Keep-Alive
Content-Length:2
Content-Range:items 0-0/0
Content-Type:application/json
Date:Thu, 19 Sep 2013 12:56:19 GMT
Keep-Alive:timeout=5, max=92
Server:Apache/2.2.12 (Win32) DAV/2 mod_ssl/2.2.12 OpenSSL/0.9.8k mod_autoindex_color PHP/5.3.0
X-Powered-By:PHP/5.3.0
And a screenshot, for good measure:
What is the appropriate way of giving an estimate for request completion when the server returns a 202 - Accepted status code for asynchronous requests?
From the HTTP spec (italics added by me):
202 Accepted
The request has been accepted for processing, but the processing has not been completed. [...]
The entity returned with this response SHOULD include an indication of the request's current status and either a pointer to a status monitor or some estimate of when the user can expect the request to be fulfilled.
Here are some of thoughts:
I have glanced at the max-age directive, but using it would be abusing Cache-Control?
Return the expected wait time in the response body?
Add an application specific X- response header, but the X-headers was deprecated in RFC 6648?
Add a (non X-) specific response header? If so, how should it be named? The SO question Custom HTTP headers : naming conventions gave some ideas, but after the deprecation it only answers how HTTP headers are formatted, not how they should be named.
Other suggestions?
Definitely do not abuse existing HTTP headers for this. Since it's your own server, you get to define what the response looks like. You can (and should) pick whatever response works best for the intended recipient of this information and return the actual information in the response body.
For example, if you are only interested in displaying a human-readable message then you could return text/plain saying "Your request is likely to be processed in the next 30 minutes.".
At the other end of the spectrum, you might want to go all the REST way and return application/json, perhaps formatted like this (I totally made this up on the spot):
{
"status": "pending",
"completion": {
"estimate": "Thu Sep 08 2011 12:00:00 GMT-0400",
"rejected-after": "Fri Sep 09 2011 12:00:00 GMT-0400",
},
"tracking": {
"url": "http://server/status?id=myUniqueId"
}
}
You can use the Location header to specify the URL of the status monitor. Things like current status and estimate can either go in custom headers (which noone but your own software would use), or in the response body (which a web browser would display to a user, at least).
Although not explicitly mentioned specifically for the 202 - Accepted response code, the Retry-After header seems to be a suitable option. From the documentation:
The Retry-After response-header field can be used [...] to indicate how long the service is expected to be unavailable to the requesting client.