I have a problem with streams and the web api.
I return the stream which is consumed by the web api. Currently, i put the socket into a pool after getting the stream. but this cause some errors.
Now, I must putthe socket into the pool AFTER the request ended. (The stream was consumed and is now closed).
Is there a delegate for this or some other best practises?
Example code:
public HttpResponseMessage Get(int fileId)
{
HttpResponseMessage response = null;
response = new HttpResponseMessage(HttpStatusCode.OK);
Stream s = GetFile(id);
response.Content = new StreamContent(fileStream);
}
GetFile(int id)
{
FSClient fs = GetFSClient();
Stream s = fs.GetFileStream(id);
AddFSToPool(fs);
return s;
}
GetFile uses a self-programmed FileServer-Client.
It has an option to reuse FileServer-Connections. This connections will be stored in a pool. (In the pool are only unused FileServer-connections). If the next request calls GetFSClient() it gets an connected one from the pool (and removes it from the pool).
But if another requests comes in and uses a FileServer-Connection which is in the pool (because unused), there is still the problem, that the Stream is possibly in use.
Now I want to do the "put the FSClint into the pool" after the request ended and the stream is fully consumed.
Is there an entry point for that?
Stream is seen as a volatile/temporary resource - no wonder it implements IDisposable.
Also Stream is not thread-safe since it has a Position which means if it is read up to the end, it should be reset back to start and if two Threads reading the stream they will most likely read different chunks.
As such, I would not even attempt to solve this problem. Re-using streams on a web site (inherently multi-user / multi-threaded) not recommended.
UPDATE
As I said, still think that the best option is to re-think the solution but if you need to register something that runs after request finishes, use RegisterForDispose on request:
public HttpResponseMessage Get(HttpRequestMessage req, int fileId)
{
....
req.RegisterForDispose(myStream);
}
Related
What is the recommended way in vert.x to write an Asynchronous request handler?
In this service, a request processing typically involves calling DB, calling external services, etc. I do not want to block the request handling thread however. What is the recommended way to achieve this using vet.x? In a typical asynchronous processing chain, I would use the request handling thread to emit a message to the message bus with the request object. Another handler will pick this message and do some processing such as checking request params. This handler can then emit a new message to the bus which can be picked up by the next handler which will do a remote call. This handler emits a new message with the result of the call which can be picked up by the next handler which will do error checking etc. Next handler would be responsible for creating the response and sending it to the client.
How one can create a similar pipeline using vert.x?
Everything, comprising request handlers for HttpServer, is asynchronous, isn't it?
var server = vertx.createHttpServer(HttpServerOptions())
server.requestHandler { req ->
req.setExpectMultipart(true) // for handling forms
var totalBuffer = Buffer.buffer()
req.handler { buff -> b.appendBuffer(buff) }
.endHandler { // the body has now been fully read
var formAttributes = request.formAttributes()
req.response().putHeader("Content-type","text/html");
req.response().end("Hello HTTP!");
}
// the above is so common that Vertx provides: bodyHandler{totalbuff->..}
}.listen(8080, "127.0.0.1", { res -> if(res.succeeded()) ... });
You just need to (end) write on req.response() on your final handler of your pipeline.
For a more stream-like implementation (i.e., not callback-based), you may use Vert.x Rx/ReactiveStreams API. E.g., you may use Vert.x Web Client for making requests, possibly using its Rx-fied API.
I'm new to Vert.x and trying I am trying to implement a small REST API that stores its data in JSON files on the local file system.
So far I managed to implement the REST API since Vertx is very well documented on that part.
What I'm currently looking for are examples how to build data access objects in Vert.x. How can I implement a Verticle that can perform crud operations on a text file containing JSON?
Can you provide me any examples? Any hints?
UPDATE 1:
By CRUD operations on a file I'm thinking of the following. Imagine there is a REST resource called Records exposed on the the path /api/v1/user/:userid/records/.
In my verticle that starts my HTTP server I have the following routes.
router.get('/api/user/:userid/records').handler(this.&handleGetRecords)
router.post('/api/user/:userid/records').handler(this.&handleNewRecord)
The handler methods handleGetRecords and handleNewRecord are sending a message using the Vertx event bus.
request.bodyHandler({ b ->
def userid = request.getParam('userid')
logger.info "Reading record for user {}", userid
vertx.eventBus().send(GET_TIME_ENTRIES.name(), "read time records", [headers: [userId: userid]], { reply ->
// This handler will be called for every request
def response = routingContext.response()
if (reply.succeeded()) {
response.putHeader("content-type", "text/json")
// Write to the response and end it
response.end(reply.result().body())
} else {
logger.warn("Reply failed {}", reply.failed())
response.statusCode = 500
response.putHeader("content-type", "text/plain")
response.end('That did not work out well')
}
})
})
Then there is another verticle that consumes these messages GET_TIME_ENTRIES or CREATE_TIME_ENTRY. I think of this consumer verticle as a Data Access Object for Records. This verticle can read a file of the given :userid that contains all user records. The verticle is able to
add a record
read all records
read a specific record
update a record
delete a or all records
Here is the example of reading all records.
vertx.eventBus().consumer(GET_TIME_ENTRIES.name(), { message ->
String userId = message.headers().get('userId')
String absPath = "${this.source}/${userId}.json" as String
vertx.fileSystem().readFile(absPath, { result ->
if (result.succeeded()) {
logger.info("About to read from user file {}", absPath)
def jsonObject = new JsonObject(result.result().toString())
message.reply(jsonObject.getJsonArray('records').toString())
} else {
logger.warn("User file {} does not exist", absPath)
message.fail(404, "user ${userId} does not exist")
}
})
})
What I trying to achieve is to read the file like I did above and deserialise the JSON into a POJO (e.g. a List<Records>). This seems much more convenient that working with JsonObject of Vertx. I don't want to manipulate the JsonObject instance.
First of all, your approach using EventBus is fine, in my opinion. It may be a bit slower, because EventBus will serialize/deserialize your objects, but it gives you a very good decoupling.
Example of another approach you can see here:
https://github.com/aesteve/vertx-feeds/blob/master/src/main/java/io/vertx/examples/feeds/dao/RedisDAO.java
Note how every method receives handler as its last argument:
public void getMaxDate(String feedHash, Handler<Date> handler) {
More coupled, but also more efficient.
And for a more classic and straightforward approach, you can see the official examples:
https://github.com/aokolnychyi/vertx-example/blob/master/src/main/java/com/aokolnychyi/vertx/example/dao/MongoDbTodoDaoImpl.java
You can see that here DAO is pretty much synchronous, but since the handlers are still async, it's fine anyway.
I guess the following link will help you out and this is a good example of Vertx crud operations.
Vertx student crud operations using hikari
I have used cache2k with read through in a web application to load blog posts on demand. However, I am concerned about blocking for the read through feature. For example, if multiple threads (requests) ask the cache for the same key, is it possible for the read through method to be called multiple times to load the same key/value into the cache?
I get the impression from the documentation that the read through feature does block concurrent requests for the same key until the load has completed, but may I have mis-read the documentation. I just want to check that this is the behaviour.
The method which initializes the cache looks like this:
private void initializeURItoPostCache()
{
final CacheLoader<String, PostImpl> postFileLoader = new CacheLoader<String, PostImpl>(){
#Override public PostImpl load(String uri)
{
// Fetch the data and create the post object
final PostImpl post = new PostImpl();
//.. code omitted
return post;
}
};
// Initialize the cache with a read-through loader
this.cacheUriToPost = new Cache2kBuilder<String, PostImpl>(){}
.name("cacheBlogPosts")
.eternal(true)
.loader(postFileLoader)
.build();
}
The following method is used to request a post from the cache:
public Post getPostByURI(final String uri)
{
// Check with the index service to ensure the URI is known (valid to the application)
if(this.indexService.isValidPostURI(uri))
{
// We have a post associated with the given URI, so
// request it from the cache
return this.cacheUriToPost.get(uri);
}
return EMPTY_POST;
}
Many thanks in advance, and a happy and prosperous New Year to all.
When multiple requests to the same key will provoke a cache loader call, cache2k will only invoke the loader once. Other threads wait until the load is finished. This behavior is called blocking read through. To cite from the Java Doc:
Blocking: If the loader is invoked by Cache.get(K) or other methods that allow transparent access concurrent requests on the same key will block until the loading is completed. For expired values blocking can be avoided by enabling Cache2kBuilder.refreshAhead(boolean). There is no guarantee that the loader is invoked only for one key at a time. For example, after Cache.clear() is called load operations for one key may overlap.
This behavior is very important for caches, since it protects against the Cache stampede. An example: A high traffic website receives 1000 requests per second. One resource takes quite long to generate, about 100 milliseconds. When the cache is not blocking out the multiple requests when there is a cache miss, there would be at least 100 requests hitting the loader for the same key. "at least" is an understatement, since your machine will probably not handle 100 requests at the same speed then one.
Keep in mind that there is no hard guarantee by the cache. The loader must still be able to perform correctly when called for the same key at the same time. For example blocking read through and Cache.clear() lead to competing requirements. The Cache.clear() should be fast, which means we don't want to wait for ongoing load operations to finish.
I have a ServiceWorker registered on my page and want to pass some data to it so it can be stored in an IndexedDB and used later for network requests (it's an access token).
Is the correct thing just to use network requests and catch them on the SW side using fetch, or is there something more clever?
Note for future readers wondering similar things to me:
Setting properties on the SW registration object, e.g. setting self.registration.foo to a function within the service worker and doing the following in the page:
navigator.serviceWorker.getRegistration().then(function(reg) { reg.foo; })
Results in TypeError: reg.foo is not a function. I presume this is something to do with the lifecycle of a ServiceWorker meaning you can't modify it and expect those modification to be accessible in the future, so any interface with a SW likely has to be postMessage style, so perhaps just using fetch is the best way to go...?
So it turns out that you can't actually call a method within a SW from your app (due to lifecycle issues), so you have to use a postMessage API to pass serialized JSON messages around (so no passing callbacks etc).
You can send a message to the controlling SW with the following app code:
navigator.serviceWorker.controller.postMessage({'hello': 'world'})
Combined with the following in the SW code:
self.addEventListener('message', function (evt) {
console.log('postMessage received', evt.data);
})
Which results in the following in my SW's console:
postMessage received Object {hello: "world"}
So by passing in a message (JS object) which indicates the function and arguments I want to call my event listener can receive it and call the right function in the SW. To return a result to the app code you will need to also pass a port of a MessageChannel in to the SW and then respond via postMessage, for example in the app you'd create and send over a MessageChannel with the data:
var messageChannel = new MessageChannel();
messageChannel.port1.onmessage = function(event) {
console.log(event.data);
};
// This sends the message data as well as transferring messageChannel.port2 to the service worker.
// The service worker can then use the transferred port to reply via postMessage(), which
// will in turn trigger the onmessage handler on messageChannel.port1.
// See https://html.spec.whatwg.org/multipage/workers.html#dom-worker-postmessage
navigator.serviceWorker.controller.postMessage(message, [messageChannel.port2]);
and then you can respond via it in your Service Worker within the message handler:
evt.ports[0].postMessage({'hello': 'world'});
To pass data to your service worker, the above mentioned is a good way. But in case, if someone is still having a hard time implementing that, there is an other hack around for that,
1 - append your data to get parameter while you load service-worker (for eg., from sw.js -> sw.js?a=x&b=y&c=z)
2- Now in service worker, fetch those data using self.self.location.search.
Note, this will be beneficial only if the data you pass do not change for a particular client very often, other wise it will keep changing the loading url of service worker for that particular client and every time the client reloads or revisits, new service worker is installed.
I want to fetch two XML documents from the server and resume processing when both have arrived. Can I fetch them in parallel, or do I have to refrain from issuing the second request until the first has completed?
You can fetch them in parallel, but keep in mind that browsers have a limit on the number of parallel requests, see http://www.browserscope.org/?category=network (choose "Major Versions" in the dropdown on the top left to see more versions). Note especially, that IE < 8 has a limit of 2 connections per hostname!
If you still want to do this, then note that the responses can arrive in any order. So you'll have to implement something that will keep track of the requests/responses (a counter or something more sophisticated), so that you'll know when all responses you need have arrived.
The best solution is often to send just one request that asks for both XML documents, and the server returns them both at once in one response.
Make both requests, then check when either one completes whether the other is done, and continue if it is.
private String responseOne;
private String responseTwo;
public startRequests() {
makeAsyncRequestOne(new AsyncCallback<String>() {
onSuccess(String response) {
this.responseOne = response;
if (responseTwo != null) {
proceed();
}
}
});
makeAsyncRequestTwo(new AsyncCallback<String>() {
onSuccess(String response) {
this.responseTwo = response;
if (responseOne != null) {
proceed();
}
}
});
}
As Chris points out, this may hit a ceiling on maximum concurrent requests to the same hostname, so if you have lots of requests to send at once, you could keep a queue of requests and call the next one in proceed() until the queue is exhausted.
But if you plan on having a lot of concurrent requests, you probably need to redesign your service anyway, to batch operations together.