We are trying to use Celery for our multitenant application - the application as an example sends email for each tenant.
Producer adds emails to be sent to a queue. We have multiple producers who keep adding to the queue based on the tenant.
At the worker side, we need a way to group all these emails based on the tenant so that we can process them in one big chunk.
Our initial idea was to store all the emails to redis while adding a task to celery to execute for each tenant. The worker then gets all the tasks from redis, assembles and sends them in bulk.
But it is reinventing the wheel and we feel that Celery can make this possible as well.
Is it possible to give a key with each task and Celery can give all these tasks based on the key which can be executed only once per key?
Related
Currently I have many jobs written in java that pull data from various sources, these jobs run every 5 minutes on a virtual machine in crontab, they send the data to a Kafka queue , where this a consumer that pushes the data to the database,
This system is not very reliable as you usually need to open the virtual machine to stop and force jobs to start.
So I need a Job management software for queuing and scheduling heavy jobs that take several minutes to complete, preferably I want a user interface to be able to monitor the jobs easily and force start or force stop them at anytime, the architecture needs to be robust in 2 matters, the first is scheduling many jobs that takes a lot of time, the second matter is to handle job requests in a queue, as sometimes we need to pull data on request from other sources which will run for one time only.
I am currently working on an application which will schedule a task as a timers. The timers can be run on any day of the week with the configuration by the user. Currently it is implemented with bullqueue and redis for storage. Once the timer will execute it will execute an event and further process the business logic. There can be thousands of queue messages in the redis.
I am looking to replace redis with Kafa as I have read it is easy to scale and guarantee of no message loss.
The question is. Is it a good idea to go with Kafa? If yes then how can we schedule a jobs in kafka with the combination of bullqueue. I am new to Kafka and still trying to understand how can we schedule the jobs in Kafka or is it a good architecture setup to go with.
My current application setup is with nestjs, nodejs
Kafka doesn't have any feature like this built-in, so you'd need to combine it with some other timer/queue system for scheduling a KafkaProducer action.
Similarly, Kafka Consumers are typically always running, although, you can start/pause them periodically as well.
In my new work's project, i discovered that instead of directly making post/put API calls from one microservice to another microservice, a microservice would produce a message to kafka, which is then consumed by a single microservice.
For example, Order microservice would publish a record to "pending-order" topic, which would then be consumed by Inventory microservice (no other consumer). In turn, after consuming the record and done some processing, Inventory microservice would produce a record to "processed-order" which would then be consumed only by Order microservice.
Is this a correct use case? Or is it better to just do API calls between microservices in this case?
There are two strong use cases of Kafka in a microservice based application:
You need to do a state change in multiple microservices as part of a single end user activity. If you do this by calling all the appropriate microservice APIs sequentially or parallely, there will be two issues:
Firstly, you lose atomicity i.e. you canNot guarantee "all or nothing" . It's very well possible that the call to microservice A succeeds but call to service B fails and that would lead to inconsistent data permanently. Secondly, in a cloud environment unpredictable latency and network timeouts are not uncommon and so when you make multiple calls as part of a single call, the probability of one of these calls getting delayed or failed is higher impacting user experience. Hence, the general recommendation here is, you write the user action atomically in a Kafka topic as an event and have multiple consumer groups - one for each interested microservice consume the event and make the state change in their own database. If the action is triggered by the user from a UI, you would also need to provide a "read your own write" guarantee where the user would like to see his data immediately after writing. Therefore, you'd need to write the event first in the local database of the first microservice and then do log based event sourcing (using an aporopriate Kafka Connector) to transfer the event data to Kafka. This will enable you to show the data to the user from the local DB. You may also need to update a cache, a search index, a distributed file system etc. and all of these can be done by consuming the Kafka events published by the individual microservices.
It is not very uncommon that you need to pull data from multiple microservice to do some activity or to aggregate data and display to the user. This, in general, is not recommended because of the latency and timeout issue mentioned above. It is usually recommended that we precompute those aggregates in the microservice local DB based on Kafka events published by the other microservices when they were changing their own state. This will allow you to serve the aggregate data to the user much faster. This is called materialized view pattern.
The only point to remember here is writing to Kafka log or broker and reading from it us asynchronous and there maybe a little time delay.
Microservice as consumer, seems fishy to me. You might mean Listeners to that topic would consume the message and maybe they will call your second microservice i.e. Inventory Microservice.
Yes, the model is fine, specially when you want to have asynchronous behavior and loads of traffic handled through it.
Imaging a scenario when you have more than 1 microservice to call from 1 endpoint. Here you need either aggregation layer which aggregates your services and you call it once, or you would like to publish several messages to Kafka which then will do the job.
Also think about Read services, if you need to call a microservice to read some data from somewhere else, then you can't use Kafka.
It all depends on your requirements and design.
Trying to wrap my head around Celery. So far I understand that there is the Client, Broker, and Worker(s). For simplicity, I'm looking at a Dockerized version of this setup.
If I understand correctly, the client enqueues a task on the broker, then the worker continuously attempts to pop from the broker and process the task.
In the example, it seems like both the Client (in this case a Flask app) and the Worker, reference the exact same code. Is that correct?
Does this mean that if each of the components were broken out into their own machines that the code would need to be deployed to both Client and Worker machines at the same time? It seems strange that these pieces would need to access the same task/function to do their respective work.
This is one thing I was initially confused by as well. The tasks' code doesn't have to be present on both, only the worker needs the code to do the actual work. When you say that client enqueues the task on broker which worker then executes, it's crucial to understand how this works. The client only sends a message to the broker, not an actual task. The message contains task name, arguments and other stuff. So the client needs to know just these parameters about the task to enqueue it. Then, it can use send_task to enqueue the task without knowing.
This is how I employ Celery in a simple jobber application where I want to decouple the pieces as much as possible. I have a Flask app that serves as a UI for the jobs which users can manage (create, see the state/progress etc.) The Flask app uses APScheduler to actually run the jobs (where a job is nothing else than a Celery task). Now, I want the client part (Flask app + scheduler) to know only as little as possible about the tasks to run them as jobs. That means their names and arguments they can take. To make it really independent of the tasks' code, I get this information from the workers via the broker. You can see a little bit more background from this issue I initially created.
I'm getting into utilising work queues for my app and can easily understand their benefit: computationally or time-intensive tasks (like sending an email) can be put in a queue and run asynchronously to the request that initiated the task so that it can get on with responding to the user.
For example in my Laravel app I am using redis queueing. I push a job into onto the queue and a separate artisan process which is running on the server (artisan queue:listen) is listening on the queue for new jobs for it to execute.
My confusion is when it comes to the actually queuing system. Surely the queue worker is just a list of jobs with any pertinent data serialised and passed through. The job itself is still computed by the application.
So this leads me to wonder about the benefit and cost of large-scale queue workers like Iron.io or Amazon SQS. These services cost a fair amount for what seems like a fairly straightforward and computationally minimal task. Even with thousands of jobs a minute a simple redis or beanstalkd queue on the same server will surely be handled far easier than the actual jobs themselves. Having a hosted system seems like it'll slow down the process more due to the latency between servers.
Can anyone explain the workings of a work queue and how these hosted services improve the performance of such an application. I imagine the benefit is only felt once an app has scaled sufficiently but more insight would be helpful.