Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 2 years ago.
Improve this question
I need to migrate a service that is on the mainframe to Kafka. The server was hosted on amazon (AWS).
Do I need to worry about something? Can the server not support it?
it will be a credit card transaction.
Here's some back of a napkin maths:
8 billion messages an hour is c.2 million a second (8000000000/60/60)
If you assume 1 kB message size that's c.2GB per second
This demo shows Kafka scaling to 2.7GB/s of ingress - so yes, Kafka can support it. You just need to scale and configure your brokers accordingly.
Related
Closed. This question needs details or clarity. It is not currently accepting answers.
Want to improve this question? Add details and clarify the problem by editing this post.
Closed 9 days ago.
Improve this question
I am using managed Postgres from DigitalOcean. I have the cheapest instance, with 1 CPU, 1GB RAM and 10 GB space. I have a small database (aprox 25 tables), so the resources should be enough. I am using Postgres version number 15.
However, even when not using the database (not querying or inserting), the disk usage continues to go up. I suspect that the logging might be an issue.. from their API I've set the temp_log_size property to a small size, still no success.
Does anybody know what I can do? I don't think that it is possible to access the configuration file directly. Thanks a lot.
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 3 years ago.
Improve this question
This is my problem:
I need to have a queue to which I am going push with unlimited number of messages. Each message will have customer ID attached.
Each message then get consumed, message data will be transformed and sent to a API server.
However, the API server only allows receive a fixed number of messages per customer per second.
My question is: Which queue will be suitable for this requirement?
All of these and then, none.
The service that reads from your queue and talks to the API should be the one responsible for keeping track of the API call rate and slow down (by waiting) when the rate is exceeded.
This is especially suited if these rate limits are complex (per customer, etc.) plus you keep your message broker as an implementation detail.
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 4 years ago.
Improve this question
How to check following condition
How many streams can support before noticing Kafka cluster degradation and how to scale up the cluster
It will hugely depend on what your application is doing, the throughput, and so on. Some general resources to help you:
Elastic Scaling in the Streams API in Kafka
Kafka Streams Capacity planning and sizing
Closed. This question needs details or clarity. It is not currently accepting answers.
Want to improve this question? Add details and clarify the problem by editing this post.
Closed 7 years ago.
Improve this question
In the command line, I can use "--from-beginning" to get all the messages in one topic, but how can I get the same effort when I code a java program ,and I'm using High Level Consumer api.
while creating the consumer properties you can add
props.put("auto.offset.reset", "smallest"); to start reading from the beginning
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 8 years ago.
Improve this question
I am considering using mongo db (it could be postgresql or any other ) as a data warehouse, my concern is that up to twenty or more users could be running queries at a time and this could have serious implications in terms of performance.
My question is what is the best approach to handle this in a cloud based and non cloud based environment? Do cloud based db's automatically handle this? If so would the data be consistent through all instances if a refresh on the data was made? In a non cloud based environment would the best approach be to load balance all instances? Again how would you ensure data integrity for all instances?
thanks in advance
I think auto sharding is what I am looking for
http://docs.mongodb.org/v2.6/MongoDB-sharding-guide.pdf