On Heroku, and using the Play Framework, is it necessary to set up a background job processor (using Akka/RabbitMQ/etc.) in order to perform heavy tasks*?
I know that Play offers the ability to perform tasks asynchronously within requests, but would this be enough to avoid requiring a background job processor? On non-Heroku-deployed standalone Play apps, the asynchronicity features make it possible to do everything all in one process, but on Heroku apps, it seems like it would not be enough: according to the book Professional Heroku Programming, (page 254, in the Developing with Ruby section), the web dyno is blocked between the time a request is received and a response is delivered, and all other requests are queued during this time.
If a background job processor is necessary, are there any examples? I've seen examples of Play and Akka, and Play and Heroku, but not all three together.
*(By heavy tasks, I generally mean potentially long-running tasks that require an answer, such as the result of a complex database query or web-service call to be given to the end user, as opposed to fire-and-forget things like sending emails).
You don't need an explicit worker when using Play. The common pattern with Play 2 is to use an Async response in a controller and Akka for longer running processes. All the examples on the Play Framework website should work out of the box on Heroku.
Related
I am developing a web application which uses celery for task distribution and management. Web application also uses machine learning and deep learning algorithms to do predictions. These predictive models are deployed on separate server as separate application and their predictive functions are integrated with celery as individual task.
For example, X (user) wants to know forecast of stock price and submits query to web application. Web application will initiate celery tasks with X's query payload. This Celery tasks after performing certain operations submits tasks to other server where machine learning / deep learning 's celery is working and start waiting for other server tasks to be completed and response to be received.
This increased our performance to 10 folds as compare to when we deployed RESTful endpoints for machine learning predictive models using Flask. For deep learning, we need to move to Tensorflow and integrate it with celery. After thorough research, it was concluded to use Tensorflow Serving and call predictive functions inside celery tasks on machine learning server.
Other approach, was to deploy TensorFlow models as separate end points using Sanic and rather than web application's celery submitting tasks directly to other server celery, now it will directly execute and call RESTful API Endpoint which will be asynchronous as well.
What do you suggest, what would work best for us in this scenario? What benefit can celery provide over RESTful API or vice versa?
For online processing of (mostly) serial requests within a cloud application, using Celery to queue up batches for inference doesn't seem like a good design choice, nor does the use of Flask, or any combination thereof. Tensorflow serving is a performant solution that handles dynamic batching for you, no need to put tasks on a queue that will then be sent to tf-serving only to again be queued prior to being processed.
Shameless plug: there is another package called virtex, built on top of asyncio, which offers a serving solution that is entirely agnostic to how your computation is implemented. It's very easy to use and highly performant. The server runs in a single process with an event loop handling requests. With this framework you define your request processing with three callback functions (which amounts to refactoring your inference code into three functions whose signatures are mildly constrained), and the virtex engine chains them together on its event loop. The virtex client can send requests in bundled or serial fashion. For online applications requiring low latency, in which the request/model-input ratio is roughly 1, it's the most performant http serving solution that I have tested (it's been benchmarked against tf-serving and bert-serving for Resnet50V2 and BERT-base, respectively).
I'm getting into utilising work queues for my app and can easily understand their benefit: computationally or time-intensive tasks (like sending an email) can be put in a queue and run asynchronously to the request that initiated the task so that it can get on with responding to the user.
For example in my Laravel app I am using redis queueing. I push a job into onto the queue and a separate artisan process which is running on the server (artisan queue:listen) is listening on the queue for new jobs for it to execute.
My confusion is when it comes to the actually queuing system. Surely the queue worker is just a list of jobs with any pertinent data serialised and passed through. The job itself is still computed by the application.
So this leads me to wonder about the benefit and cost of large-scale queue workers like Iron.io or Amazon SQS. These services cost a fair amount for what seems like a fairly straightforward and computationally minimal task. Even with thousands of jobs a minute a simple redis or beanstalkd queue on the same server will surely be handled far easier than the actual jobs themselves. Having a hosted system seems like it'll slow down the process more due to the latency between servers.
Can anyone explain the workings of a work queue and how these hosted services improve the performance of such an application. I imagine the benefit is only felt once an app has scaled sufficiently but more insight would be helpful.
I am trying to understand implementations/options for server-side Websocket endpoints - particularly in Perl using PSGI/Plack and I have a question: Why are all server-side websocket implementations based around event-driven PSGI servers (Twiggy, Tatsumaki, etc.)?
I get that websocket communication is asynchronous, but a non-event driven PSGI server (say Starman) could spawn an asynchronous listener to handle the websocket side of things. I have seen (but not understood) PHP implementations of Websocket servers, so why cant the same be done with PSGI without having to change the server to an event driven one?
Underlying network logic to deal with sockets depends on platform, OS and particular software implementations.
Most common three methods are:
pulling - there is blocking constant "asking" if socket has some data. This method is well bad, as it will block execution of main thread for as long as it waits for some data.
thread per socket - each new connection involves creating new thread and asking each socket in blocking manner happens within that thread. So it wont block main thread with logic. This method is bad as creating thread for each connection is too expensive for memory, and can be around 1Mb or RAM based on OS and other criteria.
async - uses system features to "notify" your process when there is something. So you can react once your app is ready (in case of single threaded app) or even react in separate thread straight away. This method is well efficient as it saves RAM, and allows your app to work without need of waiting or asking for data. It utilises existing functionalities that most OS and platforms provide.
Taking this in account, you indeed can create single process functional way to deal with sockets traffic. But that is not efficient at all as been proven previously. That is why fully async models are major today, as most languages and platforms do support such paradigm.
I want client machines on internet who subscribe to my server to donate their idle cpu cycles.(Like SETI#Home)
They would take jobs(work-units) from server to process, and send back results to the server. (This is the most simple description). The framework i need should allow me to define a job/task. Rest of things like communication, job execution/tracking, client binaries update etc. should be managed by framework.
I evaluated Alchemi.NET a bit, but its not actively maintained, seems half-baked.
BOINC has API in C, but i want a .NET or JAVA framework.
I am looking at Manjrasoft's ANEKA , but it seems to work only for LAN clouds.
There must be some such frameworks available. I need expert recommendations!
I'm hardly an expert but I do have a little experience with distributed computing using MPI (with C). What you're talking about does not sound like grid computing, rather a master/slave system. That is your server is the master and it directs all the clients (slaves).
I know very little about .net programming so I'll speak in general.
There are a lot of web frame works out there and probably most have the facilities you'll need. That is your client will be able to upload files with the content they have gathered (or they could just use http get/post), because you don't have to worry about UI issues you can handle everything probably through one action (assuming an action based web framework). Then the server can return a response via JSON or XML which the client can use to take further direction on. JSON is the right choice if the system is very simple and probably a good choice for prototyping.
For the application upgrade I would consider this a separate issue (although it should be a simple matter for the server to return this to the client).
BOINC is the framework that most naturally meets your volunteer computing requirements, and it's stable and highly scalable -- I'd make sure you've considered it completely before ruling it out.
If you need to deliver something to a short deadline then I'd consider working up a simple supervisor (or scheduler) - worker pattern, just to get the application off the ground. The supervisor would be responsible for chunking up the data and making it available over http. Workers (your client app) would connect to a supervisor server; download a chunk of work; complete the chunk; and upload the results to the supervisor.
The main trick is to get the state machine thrashed out properly, so that you can accurately track what state each work chunk is in. I'd have the supervisor persist state in a database in the background.
Does your first release need to be internal, or is it for public consumption?
While that's working I'd get started on looking at getting up to speed with BOINC and planning a migration.
My recommendation
Work dist:
Have a receiver of requests, that places messages in a message queue, like rabbit mq
Have a host of workers, listening to the same queue, taking work from it and acking it when done.
When done, send a message on another queue, containing an URI to a known location, such as your network drive. The target is your parsed data.
The receiver listens to these "completed" messages. Fetches the data from the URI.
Done.
RabbitMQ comes with great CLR APIs.
The same reasoning works well with Microsoft Azure and their AbbFabric Queue. A plus is that is scales extremely well.
Hot Versioning
http://topshelf-project.com/
It gives a folder where you can drop binaries, which are then run. It manages versioning of these as well as running them as windows services.
Deployment
You can deploy the binaries with robocopy/xcopy and "net use Q: pwd \server\share", "net delete Q:"
Continuous Integration
Teamcity
After working with MsBuild extensively, I would recommend scripting it with psake and running the build with PowerShell. If you get advanced with PowerShell you also have WinRM available to you from your build scripts, which is really smooth.
Use the git/subversion commit number as the 0.0.0.x, x in the previous version number, and you will have automatic versioning that is "shared" across "Debug"/"Production" builds.
The Azure way
Work dist:
Same as above but with AppFabric Queue instead of RabbitMQ.
Hot Versioning
By swapping "Staging" and "Production" instances around, you avoid the downtime.
Deployment
You can either tap into the Azure Tools for Visual Studio's MsBuild tasks as can be read about here or you could use the PowerShell AzureSnapIns with a similar setup as above for Continuous Integration.
Continuous Integration
Same as above.
How about .net's ClickOnce installer to manage auto updated client binaries.
http://msdn.microsoft.com/en-us/library/t71a733d.aspx
I'm not sure of a "jobs framework" per-say, but Microsoft's Sync framework to support rolling your own jobs syncing with clients?
http://msdn.microsoft.com/en-us/sync/default
I want to understand basics of Event Driven web server, I know one of them is Tornado, but any other information is much appreciated.
Thanks
There's a nice analogy of this described here:
http://daverecycles.tumblr.com/post/3104767110/explain-event-driven-web-servers-to-your-grandma
Relevant text from the above link:
Explain “Event-Driven” Web Servers to Your Grandma You’ve heard the
term event-driven, event-based, or evented when it comes to web
servers. Node.js is based on evented I/O. nginx is an asynchronous
event-driven web server.
But what does the term mean? Here’s an easy way to think about it.
Let’s think of a web server as a pizza shop that needs to take orders
over the phone (requests for web pages).
Traditional Web Server
The pizza shop hires operators (processes/threads) to take orders over
the phone. Each operator has one phone line. After the operator is
done taking the order, the operator keeps the customer on the line
until the pizza (response web page) is done baking and then tells them
it’s ready to pick up.
So the pizza shop needs to hire as many operators as the number of
pizzas that may be baked at once in order to serve all customers who
call in.
Event-Driven Web Server
The pizza shop only hires one operator, but has trained the operator
to hang up after taking the order, and call the customer back when the
pizza is ready to be picked up.
Now one operator can serve many customers.
A web server needs to handle concurrent connections. There are many ways to do this, some of them are:
A process per connection.
A process per connection, and have a pool of processes ready to use.
A thread per connection.
A thread per connection, and have a pool of threads ready to use.
A single process, handle every event (accepted connection, data available to read, can write to client, ...) on a callback.
Some combination of the above.
...
At the end, the distinction ends up being in how you store each connection state (explicitly in a context structure, implicitly in the stack, implicitly in a continuation, ...) and how you schedule between connections (let the OS scheduler do it, let the OS polling primitives do it, ...).
Event-driven manner aims at resolving the C10K Problem. It turns the traditional 'push model' into a 'pull model' to create a non-blocking evented I/O. Simply put, the event-driven architecture avoid spawning additional threads and thread context switching overheads, and usually ends up with better performance and less resource consumption.
Some overview from a rails developer, also includes analogy:
http://odysseyonrails.com/articles/8