As per the Google Firebase documentation, Firestore will support upto 1,000,000 simultaneous connections and 10,000 writes per second per database. We have ran a scalalbility test to check the Firestore is viable for our solution and encountering the errors like "Connect to firebase.googleapis.com:443:Connection timed out" during write operations for 40K samples, "firestore.googleapis.com:443:failed to respond" during GET operations while the 40K samples writing. Would like to understand the Firestore limits and its scalalbility
Running JMeter script to write data to Firebase (Firestore Blaze Plan purchased for testing) on multiple VMs and PCs which are connected to wired network to check the scalalbility. The JMeter script write data to Firebase using REST API PATCH and each script on PC/VM writes 5K data in a period 5 minutes. There are total of 8 PC/VM which writes 40K data to Firebase. During this we also GET 500 records in 5 minutes 2 times a day. During this test we are htting the Firebase failures
The errors are "Connect to firebase.googleapis.com:443:Connection timed out" during write operations for 40K samples, "firestore.googleapis.com:443:failed to respond" and Connection reset.
Are you considering the other limits for your specific operations?
According to https://firebase.google.com/docs/firestore/quotas
For example, for the write operation:
Maximum writes per second per database = 10,000 (up to 10 MiB per second)
Maximum write rate to a document= 1 per second
Maximum write rate to a collection in which documents contain sequential values in an indexed field = 500 per second
Maximum number of writes that can be passed to a Commit operation or performed in a transaction = 500
During GET operations you have the limits: Maximum number of exists(), get(), and getAfter() calls per request:
10 for single-document requests and query requests.
20 for multi-document reads, transactions, and batched writes. The previous limit of 10 also applies to each operation.
For example, imagine you create a batched write request with 3 write operations and that your security rules use 2 document access calls to validate each write. In this case, each write uses 2 of its 10 access calls and the batched write request uses 6 of its 20 access calls.
Exceeding either limit results in a permission denied error.
Some document access calls may be cached, and cached calls do not count towards the limits.
I think that some parameter could be causing the abortion of these connections.
Related
I have been noticing a bad behavior with my Postrgre database, but I still can't find any solution or improvement to apply.
The context is simple, let's say I have two tables: CUSTOMERS and ITEMS. During certain days, the number of concurrent customers increase and so the request of items, they can consult, add or remove the amount from them. However in the APM I can see how any new request goes slower than the previous, pointing at the query response from those tables as the highest time consumer.
If the normal execution of the query is about 200 milliseconds, few moments later it can be about 20 seconds.
I understand the lock process in PostgreSQL as many users can be checking over the same item, they could be even affecting the values of it, but the response from the database it's too slow.
So I would like to know if there are ways to improve the performance in the database.
The first time I used PGTune to get the initial settings and it worked well. I have version 11 with 20Gb for RAM, 4 vCPU and SAN storage, the simultaneous customers (no sessions) can reach over 500.
Coundant standatd plan is written as "100 reads / sec, 50 writes / sec, 5 global queries / sec".
Is this IO/s calculate end-to-end request? Or is it based on the query execution plan?
Let's give some examples
Q1.Let's say I use a Bulk operation to create 3 new documents in Cloudant (Bluemix Standard plan).
1 write operation? 3 write operation?
Q2. Query by aggregation(join)-1000 indexed docs with "name, age range, join time" and get as one docs.
1 read? 1000 + 1 read?
Q3. When I am using the standard plan (limit 100 reads / sec), it is assumed that 100 users executed the query in (Q2) at the same time.
How is IO calculated? 1 * 100 reads? (1000 + 1) * reads?
Do some users fail to execute queries because of limitation IO?
There is no data listed properly about Cloudant Price Method.
Can anyone please point me out correctly?
I want to know exactly how the standard plan calculation is measured.
It would be better if you could add a calculation example and answer!
Also answered here, https://developer.ibm.com/answers/questions/525190/how-do-calculate-io-operations-in-cloudant-account/
Bulk operations currently count as 1 W, regardless of the number of docs it contains.
A query is a request to a URL that has one of _design, _find or _search, again unreated to the number of documents actually involved. Note that some of these API endpoins (search) are paged, so it would be 1 Query per requested page of results.
I assume that by "100 users" you mean 100 concurrent connections using the same credentials, as Cloudant's rate limiting is applied per account. If so, the sum total of requests are counted towards the limit. When that bucket is full, any further requests will be cut off, and failed with a 429: Too Many Requests HTTP status code.
As an example, let's say you have a Standard account where you've set the rate limit to allow 100 queries per second. You have 100 concurrent connections hitting _find repeatedly, each query returning 1000 documents. Cloudant will allow 100 queries per second, so on average each of your connections will get 1 query per second fulfilled, and any attempts to push harder than that will results in 429 http errors. With 10 concurrent connections, on average each will get 10 qps etc.
Cloudant rate limits at the http level. No splitting of bulk ops into the constituent parts take place, at least not yet.
Documentation for how this all hangs together can be found here: https://cloud.ibm.com/docs/services/Cloudant?topic=cloudant-pricing#pricing"
The Cloudant offering in the IBM Cloud Catalog has a link to the documentation. In the docs is a description of the plans with additional examples. The docs also have sections that explain how read and write operations are calculated.
The http code 429 is returned by Cloudant to indicate too many requests. The documentation discusses this fact and there are code samples on how to handle it.
In beta, it stated that maximum writes per second per database (at beta) is 10k. However, I am not clear on the context.
Is that means that for example I have 1 million record in BigQuery. When I want to write each row's record from BigQuery into each document, it can write 10k documents in a second?
Or, assume we have 100k concurrent users connecting to the Firestore and everyone of them perform 1 update, so after 1 second, it will have 10k successful write in the Firestore, which will take 10 second for the Firestore to complete all the write from 100k concurrent users?
We are switching to the new Discovery service from the Retrieve and Rank service on Bluemix. This requires quite a bit of code rewrite and I'm getting a strange error message while transferring our answer units to Discovery:
Error: The service is busy processing 20 of your documents. Please wait for some of the documents to complete, then try again.
Is there some kind of rate limit on inserting documents into Discovery? If so, how do we increase or better, eliminate it?
Yes, there is a rate limit for inserting documents into Watson Discovery. Larger environment sizes have higher rate limits, but every size does have a limit.
Background
The big difference between sending a document into Retrieve and Rank compared to Discovery is that sending a document into Retrieve and Rank is synchronous but sending a document into Discovery is asynchronous.
Retrieve and Rank will not respond to an API call to index a document until the processing is complete. Discovery responds as soon as the document has been accepted into the pipeline for processing and processing is likely to take several seconds to complete.
Discovery limits the number of documents in the processing pipeline per service instance and returns a 429 status when the limit is exceeded.
Retrieve and Rank has what amounts to the same kind of restriction: limiting the number of concurrent requests for a single service instance.
I am trying to write large amounts of data to dynamo using AmazonDynamoDBAsyncClient and I am trying to understand what the best practice of handling throttling is?
For example, I have a capacity of 3000 writes and at a given moment I have, let's say, 100,000 records I'd like to write. I don't need them all in immediately, but I am trying to figure what the best way to get them in is.
This application is running in a distributed environment so there maybe 5 executors all trying to do this at the same time. Would the best way to handle this be this way? Where I sleep the write process should we hit the throttle? Or should I be doing something to avoid the throttle completely. In fact, is my code even doing what I think it is, which is retrying the data after waiting a second?
try{
amazonDynamoAsyncDb.updateItemAsync(updateRequest)
}catch{
case e: ThrottlingException => {
Thread.sleep(1000)
//retry here, but how?
}
}
The AWS SDK for Java will retry throttled requests 10 times by default, before throwing a ProvisionedThroughputExceededException. If your items are small (1KB or less) and you are performing the writes from EC2 in the same region as your table you can assume each write will take around 10 ms. That means each thread of processing can do about 100 writes per second. To scale your writes to 3000 writes per second, you would need 30 threads and 30 HTTP connections. 3000 small (1kb) writes per second translates to a data throughput of 2.92 MB per second. Thus, for this write load, it does not appear that EC2 hardware could become a bottleneck. I recommend you do some measurements to figure out how long it takes to write each of your items on average, and scale your threads and HTTP connections appropriately.