I made data migration with help of the Azure Database Migration Service from Mongo3.4 to Azure Cosmos DB. All collections were copied. Then I deployed app and run report inside the application. I was receiving errors in k8s like:
[report-srv-8a49370c7976028acfc037b7b9b69a37b34b8afezmg5r] 2020-09-17T14:12:27.653Z ERROR: [handleControllerHeart] Error handling heart: {"err":{"driver":true,"name":"MongoError","index":0,"code":16500}}
Error=16500, RetryAfterMs=5481, Details='Response status code does not
indicate success: TooManyRequests (429); Substatus: 3200; ActivityId:
********; Reason: ({\r\n "Errors": [\r\n "Request rate is large. More Request Units may be needed, so no changes were made.
Please retry this request later. Learn more:
http://aka.ms/cosmosdb-error-429
Then I increased RUs but the same behavior.
Does anybody have experience with migration from Mongo3.4 to Azure Cosmos DB?
You need to increase Throughput aka RUs (Request Units). You can do it from here, see how much you are using already from here and may be double it then from the dashboard as before see how much you used when you ran your report and then adjust with what you need.
As a result we created indexes in each collection and that gave us a possibility to decrease shared RU.
Increasing RU also helped but queries were very slow.
Related
We are using cloud firestore as our database and getting following error when rate of parallel reads from database increases.
details: "Bandwidth exhausted"
message: "8 RESOURCE_EXHAUSTED: Bandwidth exhausted"
stack: "Error: 8 RESOURCE_EXHAUSTED: Bandwidth exhausted
at callErrorFromStatus (/usr/service/node_modules/#grpc/grpc-js/build/src/call.js:30:26)
at Http2CallStream.call.on (/usr/service/node_modules/#grpc/grpc-js/build/src/call.js:79:34)
at Http2CallStream.emit (events.js:198:15)
at process.nextTick (/usr/service/node_modules/#grpc/grpc-js/build/src/call-stream.js:100:22)
at processTicksAndRejections (internal/process/task_queues.js:79:9)"
We couldn't find what is the rate limits. Could you please let me know what are the read rate limits and in which cases firestore returns Bandwidth exhausted error?
Note: Billing is enabled in our project. The problem is we can't find what limit we are reaching.
The RESOURCE_EXHAUSTED error indicates that the project exceeded either its quota or the region/multi-region capacity, so probably your app is doing more reads than expected given what you described. You can check more details in this documentation.
You can check the free quotas and the standard limits on this link and the pricing for what exceeds those numbers on this link. It's important to note that, if you choose to allow your app to go further than the free quotas, you must enable billing for your Cloud Platform project, here is a how to.
You can also check how much your app is actually using of the quotas on app engine on the section below:
Hope This helps.
If your are reading all data from Firebase, this issue happens to you, I had the same problem for reading all data from firebase, after a while, I figured out if we stop this process and run a new process we can pass the error and continue the job.
then I used child-process and it helped me:
I did write a parent script and a child script,
Parent script runs child script as a child-process,
The child goes through a collection until get [8 RESOURCE_EXHAUSTED] error, then send a message to parent to inform it from the error.
Then parent kills child and create a new one and tells it where to start reading again.
This is a solution that works 100 percent, but it's a little advance and beginners-intermediates may could not able to implement it.
update:
I have written a complete instruction on GitHub Gist for this issue, you can check it:
https://gist.github.com/navidshad/973e9c594a63838d1ebb8f2c2495cf87
I have situation which is almost identical to the one described here: Play framework resource starvation after a few days
My application is simple, Play 2.6 + PostgreSQL + Slick 3.
Also, DB retrieval operations are Slick only and simple.
Usage scenario is that data comes in through one endpoint, gets stored into DB (there are some actors storing some data in async fashion which can fail with default strategy) and is served through rest endpoints.
So far so good.
After few days, every endpoint that has anything to do with database stops responding. Application is server on t3-medium on a single instance connected to RDS instance. Connection count to RDS is always the same and stable, mostly idling.
What I have also noticed is that database actually gets called and query gets executed, but request never ends or gets any data.
Simplest endpoint (POST) is for posting feedback - basically one liner:
feedbackService.storeFeedback(feedback.deviceId, feedback.message).map(_ => Success)
This Success thing is wrapper around Ok("something") so no magic there.
Feedback Service stores one record in DB in a Slick preferred way, nothing crazy there as well.
Once feedback post is called, I notice in psql client that INSERT query has been executed and data really ends up in database, but HTTP request never ends and no success data gets returned. In parallel, calling non DB related endpoints which do return some values like status endpoint goes through without problems.
Production logs don't show anything and restarting helps for a day or two.
I suppose some kind of resource starvation is happening, but which and where is currently beyond me.
I have a SlashDB installation on top of MySQL. Once in a while, especially after a period of time of inactivity, the first call to an API returns:
'ResultProxy' object has no attribute 'execution_options'
The second request (same endpoint) would work normally.
My guess was that SlashDB's connection to MySQL had been terminated due to inactivity. I set the wait_timeout system variable (MySQL) to about 10 hours and it seems to help in some cases.
What does this error mean, and is there a way to prevent it?
This should no longer occur in version 1.0, but in case you still have a problem please post a dump of your database schema with your question.
I've set up an Azure Data Factory pipeline to transfer the data from one table in our SQL Server Database to our new Azure Search service. The transfer job continuously fails giving the following error:
Copy activity encountered a user error at Sink side:
GatewayNodeName=SQLMAIN01,ErrorCode=UserErrorAzuerSearchOperation,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Error
happened when writing data to Azure Search Index
'001'.,Source=Microsoft.DataTransfer.ClientLibrary.AzureSearch,''Type=Microsoft.Rest.Azure.CloudException,Message=Operation
returned an invalid status code
'RequestEntityTooLarge',Source=Microsoft.Azure.Search,'.
From what I've read thus far, Request Entity Too Large error is a standard HTTP error 413 found inside REST API. Of all the research I've done though, nothing helps me understand how I can truly diagnose and resolve this error.
Has anyone dealt with this with specific context to Azure? I would like to find out how to get all of our database data into our Azure Search service. If there are adjustments that can be made on the Azure side to increase the allowed request size, the process for doing so certainly is not readily-available anywhere I've seen on the internet nor in the Azure documentation.
This error means that the batch size written by Azure Search sink into Azure Search is too large. The default batch size is 1000 documents (rows). You can decrease it to a value that balances size and performance by using writeBatchSize property of the Azure Search sink. See Copy Activity Properties in Push data to an Azure Search index by using Azure Data Factory article.
For example, writeBatchSize can be configured on the sink as follows:
"sink": { "type": "AzureSearchIndexSink", "writeBatchSize": 200 }
Is there any way when using the REST API to get the summary of operations that have completed without returning the nodes.
When using the web admin console after doing an operation I get a summary like
1 node inserted
2 relationships inserted
1 node deleted.
In the examples here I notice there is no example of summary information sent back to the client. I would have to return the nodes inserted to know the insert had occurred.
When doing a request over the network often it is a good idea to minimize the data response size. A quick summary would help with this. is it possible to get one from the REST endpoint?
I'm pretty sure this is not possible. It would be a nice addition, though. Have you filed a feature request?