I want to use Motor with Sanic but I came across the following text in the documentation:
Threading and forking
Multithreading and forking are not supported; Motor is intended to be used in a single-threaded Tornado application. See Tornado’s documentation on running Tornado in production to take advantage of multiple cores.
Does this mean Motor cannot be used with multiple Sanic workers? Has anybody attempted that?
Sanic is not multi-threaded. And, forking is only used to setup the workers. If you instantiate motor using one of the server life cycle listeners, you will have no problem.
https://sanicframework.org/en/guide/basics/listeners.html
Related
I am developing a web application which uses celery for task distribution and management. Web application also uses machine learning and deep learning algorithms to do predictions. These predictive models are deployed on separate server as separate application and their predictive functions are integrated with celery as individual task.
For example, X (user) wants to know forecast of stock price and submits query to web application. Web application will initiate celery tasks with X's query payload. This Celery tasks after performing certain operations submits tasks to other server where machine learning / deep learning 's celery is working and start waiting for other server tasks to be completed and response to be received.
This increased our performance to 10 folds as compare to when we deployed RESTful endpoints for machine learning predictive models using Flask. For deep learning, we need to move to Tensorflow and integrate it with celery. After thorough research, it was concluded to use Tensorflow Serving and call predictive functions inside celery tasks on machine learning server.
Other approach, was to deploy TensorFlow models as separate end points using Sanic and rather than web application's celery submitting tasks directly to other server celery, now it will directly execute and call RESTful API Endpoint which will be asynchronous as well.
What do you suggest, what would work best for us in this scenario? What benefit can celery provide over RESTful API or vice versa?
For online processing of (mostly) serial requests within a cloud application, using Celery to queue up batches for inference doesn't seem like a good design choice, nor does the use of Flask, or any combination thereof. Tensorflow serving is a performant solution that handles dynamic batching for you, no need to put tasks on a queue that will then be sent to tf-serving only to again be queued prior to being processed.
Shameless plug: there is another package called virtex, built on top of asyncio, which offers a serving solution that is entirely agnostic to how your computation is implemented. It's very easy to use and highly performant. The server runs in a single process with an event loop handling requests. With this framework you define your request processing with three callback functions (which amounts to refactoring your inference code into three functions whose signatures are mildly constrained), and the virtex engine chains them together on its event loop. The virtex client can send requests in bundled or serial fashion. For online applications requiring low latency, in which the request/model-input ratio is roughly 1, it's the most performant http serving solution that I have tested (it's been benchmarked against tf-serving and bert-serving for Resnet50V2 and BERT-base, respectively).
I have an existing distributed computing framework built on top of MassTransit and RabbitMQ. There is essentially a manager which responds with work based on requests. Each worker will take a certain amount of items based on the physcial machine specs. The worker then sends completion messages when done. It works rather well and seems to be highly scalable since the only link is the service bus.
I recently evaluated Akka.Net in order to see if that would be a simpler system to implement the same pattern. After looking at it I was somewhat confused at what exactly it is used for. It seems that if I wanted to do something similar the manager would have to know about each worker ahead of time and directly send it work.
I believe I am missing something because that model doesn't seem to scale well.
Service buses like MassTransit are build as reliable messaging services. Ensuring the message delivery is primary concern there.
Actor frameworks also use messages, but this is the only similarity. Messaging is only a mean to achieve goal and it's not as reliable as in case of the service buses. They are more oriented on building high performance, easily distributed system topologies, centered around actors as primary unit of work. Conceptually actor is close to Active Record pattern (however this is a great simplification). They are also very lightweight. You can have millions of them living in memory of the executing machine.
When it comes to performance, Akka.NET is able to send over 30 mln messages/sec on a single VM (tested on 8 cores) - a lot more than any service bus, but the characteristics also differs significantly.
On the JVM we now that akka clusters may rise up to 2400 machines. Unfortunately we where not able to test, what the .NET implementation limits are.
You have to decide what do you really need: a messaging library, an actor framework or a combination of both.
I agree with #Horusiath answer. In addition, I'd say that in most cases you can replace a servicebus for the messaging system of an actor model like akka, but they are not in the same class.
Messaging is just one thing that Akka provides, and while it's a great feature, I wouldn't say it's the main one. When analyzing it as an alternative, you must first look at the benefits of the model itself and then look if the messaging capabilities are good enough for your use case. You can still use a dedicated external servicebus to distribute messages across different clusters and keep akka.net exchanging messages inside clusters for example.
But the point is that if you decide to use Akka.net, you won't be using it only for messaging.
Reading the documentation about the Play Framework and ReactiveMongo leads me to believe that ReactiveMongo works in such a way that it uses few threads and never blocks.
However, it seems that the communication from the Play application to the Mongo server would have to happen on some thread somewhere. How is this implemented? Links to the source code for Play, ReactiveMongo, Akka, etc. would also be very appreciated.
The Play Framework includes some documentation about this on this page about thread pools. It starts off:
Play framework is, from the bottom up, an asynchronous web framework. Streams are handled asynchronously using iteratees. Thread pools in Play are tuned to use fewer threads than in traditional web frameworks, since IO in play-core never blocks.
It then talks a little bit about ReactiveMongo:
The most common place that a typical Play application will block is when it’s talking to a database. Unfortunately, none of the major databases provide asynchronous database drivers for the JVM, so for most databases, your only option is to using blocking IO. A notable exception to this is ReactiveMongo, a driver for MongoDB that uses Play’s Iteratee library to talk to MongoDB.
Following is a note about using Futures:
Note that you may be tempted to therefore wrap your blocking code in Futures. This does not make it non blocking, it just means the blocking will happen in a different thread. You still need to make sure that the thread pool that you are using there has enough threads to handle the blocking.
There is a similar note in the Play documentation on the page Handling Asynchronous Results:
You can’t magically turn synchronous IO into asynchronous by wrapping it in a Future. If you can’t change the application’s architecture to avoid blocking operations, at some point that operation will have to be executed, and that thread is going to block. So in addition to enclosing the operation in a Future, it’s necessary to configure it to run in a separate execution context that has been configured with enough threads to deal with the expected concurrency.
The documentation seems to be saying that ReactiveMongo is non-blocking, so you don't have to worry about it eating up a lot of the threads in your thread pool. But ReactiveMongo has to communicate with the Mongo server somewhere.
How is this communication implemented so that Mongo doesn't use up threads from Play's default thread pool?
Once again, links to the specific files in Play, ReactiveMongo, Akka, etc, would be very appreciated.
Yes, indeed, you still need to use threads to perform any kind of work, including communication with the database. What's important is how exactly this communication happens.
ReactiveMongo "does not use threads" in a sense that it does not use blocking I/O. Usual Java I/O facilities like java.io.InputStream are blocking; this means that reading from such an InputStream or writing to OutputStream blocks the thread until the "other side" provides the required data or is ready to accept it. For network communication this means that threads will be blocked.
However, Java provides NIO API which supports non-blocking and asynchronous I/O. I don't want to get into its details right now, but the basic idea, naturally, is that non-blocking I/O allow not to block threads which need to exchange some data with the outside world: for example, these threads can poll the data source to check if there is some data available, and if there is none, they return to the thread pool and can be used for other tasks. Of course, down there these facilities are provided by the underlying OS.
Exact implementation details of non-blocking I/O is usually hidden inside high-level libraries like Netty because it is not at all nice to use. Netty (which is exactly the library ReactiveMongo uses), for example, provides nice asynchronous callback-like API which is really easy to use but is also powerful and expressive enough to allow building complex I/O-heavy applications with high throughput.
So, ReactiveMongo uses Netty to talk with Mongo database server, and because Netty is an implementation of asynchronous network I/O, ReactiveMongo really does not need to block threads for a long time.
I'm writing this as a follow up to PlayFramework -- Look up actors in another local ActorSystem, but this time targetting the question specifically to the Akka crowd.
The question is simple: Does it make sense to deploy two ActorSystems on the same host (not just on the same host but even on the same JVM), given that there appears to be no way to simply lookup the other system through system.actorSelection unless you remote to localhost?
In other words, since system1.actorSelection("akka://system2/user/my-actor") does not work, but system1.actorSelection("akka.tcp://system2#127.0.0.1:2552/user/my-actor") does, why even consider deploying two systems?
I suspect you're going to ask about a use case, so here's one for you. Assume I have a complex real-time system using Akka and that this system is deployed as autonomous agents on any number of machines. Ideally, I'd like to have fine-grained control of the resources I allocate to this system and I'd like it to be somewhat isolated. Furthermore, assume that I want to write a small control interface (e.g., a REST API) with the specific purpose to provide input and monitor the real-time system. Naturally, I would make that control system another ActorSystem which interacts with the first system. It makes sense, right? I don't want to have actors running in the same ActorSystem as the real-time processing (for isolation, practicality, separate logging, non pollution of resource monitoring, supervision -- that would add one more branch to the hierarchy --, etc.). That control ActorSystem would never be deployed on a separate machine since it goes hand in hand with the real-time system. Yet, the only way for these two systems to communicate is through loopback tcp.
Is what I'm suggesting not the proper/intended way to do things? Am I missing something? Is there a way to do this that I haven't considered? Does my use case even call for using Akka?
Thanks in advance for your input!
Instead of having two separate actor systems, you could have a top level actor for each of the branches and run each branch on a dedicated dispatcher. Each top level actor will have its own error kernel as well. Having 2 actor systems mostly makes sense, when they are not related, but as yours communicate, I would not separate them.
May be this question is not fitting into the stackoverflow FAQ
The reason I am asking this question is that a senior developer came in to the team and started saying that we should move our code to a custom threadpool manager instead of relying on celery to do asynchronous multiprocessing
We love celery because it is so easy. But the argument seems valid, mostly because we do not want to give up on efficiency.
Is it true that since celery uses multiple processes instead of multiple threads, we are loosing on efficiency?
Firstly, look at doc:
On Unix the processes pool will fork, so that child processes start
with the same memory as the parent process.
Secondly, Celery can more. It may use microthreads. Read page about how does it
Finally, we can say that microthreads more efficiency than threads, and threads more efficiency processes. But you must remember it depends also from hardware configuration