I am planning to migrate all my mongoDb database to azure cosmosDB. For the testing purposes I was trying to test all the queries by making azure cosmosDB with mongo API as the underlying datasource. I was able to execute all the queries by creating some custom indices (Single field / wildcard) in cosmosDB. But a particular query is failing. I couldn't find any noticable reason for the same. The query looks as follows (The query is somewhat too long with $in operator. I suspect that)
db.getCollection("rules").find({
"$and":[
{
"$or":[
{
"subRules.ALL.leftOperand":"GROUP",
"subRules.ALL.rightOperand":{
"$in":[
<4000+ values>
]
}
},
{
"subRules.ALL.leftOperand":"MACHINE",
"subRules.ALL.rightOperand":{
"$in":[
<4000+ values>
]
}
}
]
},
{
"ruleName":{
"$regex":""
}
}
],
"subRules.ANY.leftOperand":{
"$ne":"COFFEE_AMOUNT"
}
})
This exact query works well with mongo server. But when I run this in azure cosmos. I am getting the following error
error: {
"ok" : 0,
"errmsg" : "Error=2, Details='Response status code does not indicate success: BadRequest (400); Substatus: 0; ActivityId: 443df8b1-136f-4018-9d77-63b3442ab7f8; Reason: (Response status code does not indicate success: BadRequest (400); Substatus: 0; ActivityId: 443df8b1-136f-4018-9d77-63b3442ab7f8; Reason: (Response status code does not indicate success: BadRequest (400); Substatus: 0; ActivityId: 443df8b1-136f-4018-9d77-63b3442ab7f8; Reason: (Message: ��errors�ꠈseverity�Error�location�\u000e�start\u0000�end�5\u0002�code�SC3031�message~pTt\u00194�2A�zY�\u0007���r�\\&���2�\u001dƧ��6�]o��r��\f�6�E{�\rz���0�-���a:\u001a�.��t4�\u001c���av�=.�#Tt\u0019\u0014f���2\u0019�N��tPz\u000e��\\\r\nActivityId: 443df8b1-136f-4018-9d77-63b3442ab7f8, Request URI: /apps/27c904fe-36bc-4acd-b55d-2c8494082ce7/services/c88f5dcc-2b90-4935-9281-38171d9dd72a/partitions/398285ee-29c1-4564-8e79-91c184ef65dd/replicas/132769491079094442s/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.17763 cosmos-netstandard-sdk/3.18.0);););",
"code" : 2,
"codeName" : "BadValue"
}
I have tried creating wildcard index on the particular collection. Still it fails.
Nb : When the values in the array of $in operator is few the query executes successfully in cosmosDB as well!!
If anyone can shed some light to this issue it will be much appreciated. Thank you
I think that at this time you already get a workaround, but just in case, you are comparing the right version of mongodb with cosmos? Remember that cosmos is in 4 or less, mongodb actually is in v5.
Anyway, a "$IN" clause with a big amount of values to compare can be a bottleneck, SQL or NOT, have you tried to make your "IN" list a lookupable collection? This maybe work with cosmos
Related
I am trying to create an unique index on some of the collections on our CosmosDB. Per MS document https://learn.microsoft.com/en-us/azure/cosmos-db/mongodb-indexing : Unique indexes can be created only when the collection is empty (contains no documents).
So I cleanup all the data on the collection, however I was encountering the error, the message was :
Error=13, Details='Response status code does not indicate success: Forbidden (403); Substatus: 0; ActivityId: xxx, Reason: (Message: {"serializedCollection":"{"Errors":["The unique index cannot be modified. To change the unique index, remove the collection and re-create a new one."]}","serializedOffer":"","serializedPartitionKeyRanges":[],"serializedPartitions":[],"collectionRemoteStorageSecurityIdentifier":"xxx","collectionChildResourceNameLimitInBytes":-1,"collectionChildResourceContentLengthLimitInKB":-1,"uniqueIndexNameEncodingMode":0,"uniqueIndexReIndexingState":0}
ActivityId: xxx, Request URI: /apps/xxx/xxx/partitions/xxx/replicas/xx, RequestStats:
RequestStartTime: 2020-12-02T23:25:04.2104773Z, RequestEndTime: 2020-12-02T23:25:04.2204125Z, Number of regions attempted:1
ResponseTime: 2020-12-02T23:25:04.2204125Z, StoreResult: StorePhysicalAddress: rntbd://10.0.0.29:11000/apps/xxx/services/xx/partitions/xx/replicas/xx, LSN: 41, GlobalCommittedLsn: 41, PartitionKeyRangeId: , IsValid: True, StatusCode: 403, SubStatusCode: 0, RequestCharge: 1.57, ItemLSN: -1, SessionToken: -1#41, UsingLocalLSN: False, TransportException: null, ResourceType: Collection, OperationType: Replace
, SDK: Microsoft.Azure.Documents.Common/2.11.0, Please see CosmosDiagnostics, Windows/10.0.17763 cosmos-netstandard-sdk/3.3.2);
I then tried to add only the field name , then it was throwing errors like:
Cannot create unique index when collection contains documents
We are using CosomosDB which supports Mongodb 3.6. I was using the command:
db.CollectName.createIndex({"fieldname" : 1}, {"unique" : true } )
Any ideas why I am getting the errors?
Thanks,
Not sure what happened but it suddenly starts to work now. So the step is to clean out the data in the collection. Run the script to add the unique index and then load the data back. I am using Robo 3t to connect to the CosmosDB and run all the commands through Robo3T command line.
I am doing a query like the following, which works fine with MongoDB, but sometimes fails with CosmosDB. I need it to work with both.
(XXX is a placeholder for any string value. All strings have unique values that are redacted for readability, and actual content should be of no significance.)
{
server_index: {
$elemMatch: {
server: "XXX",
index: "XXX",
delete_time: { $exists: false },
path: {
$in: ["XXX", "XXX", "XXX" ]
}
}
}
}
The schema of a document is somewhat like this:
{
...,
server_index: [
{
server: "XXX",
index: "XXX",
delete_time: ISODate(...), // optional
path: "XXX"
},
{...}, // same as above
...
],
...
}
This query sometimes works as expected with CosmosDB as well, but sometimes I also get the following response:
{
_t: "OKMongoResponse",
ok: 0,
code: 115,
errmsg: "Command is not supported",
$err: "Command is not supported"
}
What is especially strange is that the query seemingly succeeds, and the response above is returned by a "valid" cursor as the first document, which then causes my document parser "crash".
I am using the C++ legacy driver. Is this even supported by Cosmos DB?
(According to the developer I inherited this project from, it is, and as always when you inherit projects, it all worked fine according to the previous developer... So this may be due to a change in Cosmos DB, due to the nature of my test data, or who knows what...)
Side note: In MongoDB, there is a multi-key index on server_index, which looks like this:
{
"server_index.delete_time" : 1,
"server_index.server" : 1,
"server_index.index" : 1,
"server_index.path" : 1
}
Is this even supported in CosmosDB?
EDIT: Trying to add this index with Robo 3T silently fails, with no error message whatsoever. The index is simply not added. Nice!
(Please don't ask about the strange database schema. It is like it is for a reason, and believe me, I, too, would like to burn it all down and replace it with something else ... I am open for suggestions for alternative queries, though)
This was probably a server-side problem. It seemed wrong in the first place (error status returned as part of the query result), and it has disappeared after a couple of weeks without me changing anything.
My document have some long field:
{
"_id" : BinData(3,"qUlr7T//axtyJisrSnDRkw=="),
...
"next-run-date" : "2018-12-23T17:22:59.026Z",
"next-run-millis" : NumberLong("1540833792947"),
...
"_class" : "...domain.scheduling.ScheduleReportConfiguration"
}
I use Spring Data MongoDB (v2.0.5) and have a repository:
interface ScheduleReportConfigurationRepository extends MongoRepository<ScheduleReportConfiguration, UUID> {
List<ScheduleReportConfiguration> findByNextRunMillisLessThan( long milliseconds )
}
When I try to execute it a produced query is:
{ "next-run-millis" : { "$lt" : { "$numberLong" : "1544809451462" } } }
Question 1: Why it look like that?? When I try to execute it in DB, it no make sense. Why spring don't produce the query like: { "next-run-millis" : { "$lt" : NumberLong(1544809451462) } }??
In any case, both of these queries don't work. By first query I mean the repository method execution in my app integration test,by second - execution the query in mongo shell. The error:
org.springframework.data.mongodb.UncategorizedMongoDbException:
Query failed with error code 2 and error message 'error processing query:
ns=test.scheduleReportConfigurationTree: next-run-millis $lt 1544809451462
No query solutions' on server mongo:27017; nested exception is
com.mongodb.MongoQueryException: Query failed with error code 2 and error message
'error processing query: ns=test.scheduleReportConfigurationTree: next-run-millis $lt 1544809451462
Question 2: What do I wrong?? How me to make my query worked?? According to the docs, comparing operators should work fine with long.
Thank you for your answers and help.
I just needed to make my field indexed.
I have a Collection containing 1.7 million documents. When executing count() on server side console I get correct results.
meteor:PRIMARY> db.postcodes.find().count();
1737697
meteor:PRIMARY>
Whereas at the browser console I always get zero for count() and for findOne() returns undefined.
insecure package has not been removed. And count() and findOne() are working for other smaller Collections.Not much code is present at the moment. Apart from the default html, js, css. Only a couple of line of code is present. I have model.js living in its own folder (neither in Server nor in Client) that has
PostCodes = new Mongo.Collection('postcodes');
Hello = new Mongo.Collection('hello');
All the Collections I have at the moment is
meteor:PRIMARY> db.getCollectionNames();
[
"hello",
"meteor_accounts_loginServiceConfiguration",
"parttimejobs",
"postcodes",
"system.indexes",
"users"
]
meteor:PRIMARY>
Package I have are
autopublish 1.0.3
ian:accounts-ui-bootstrap-3 1.2.69
insecure 1.0.3
meteor-platform 1.2.2
twbs:bootstrap 3.3.5
Sample document
meteor:PRIMARY> db.postcodes.findOne();
{
"_id" : ObjectId("559933dc4a8617644069fa5b"),
"postcode" : "AB10 1AB",
"latitude" : 57.149079,
"longitude" : -2.096964,
"county" : "",
"district" : "Aberdeen City",
"ward" : "George St/Harbour",
"constituency" : "Aberdeen North",
"loc" : [
-2.096964,
57.149079
]
}
Assuming you are trying to do count() very early on when the app launches, it does not seem like the data is ready (due to the large amount that needs to be loaded).
If you just need the count, I would recommend using a Meteor Method and asynchronously get the count, so you don't have to wait for the the client subscription to be ready for 1.7 million documents.
Below is an example of a method:
On the server:
Meteor.methods({
getSomeCollectionCount: function () {
return someCollection.find().count();
}
});
Then on the client:
Meteor.call('getSomeCollectionCount', function (error, result) {
if(!error) {
// add code to be run when count is ready
console.log('someCollection count: ', result);
}
});
The other way is to use pub-sub or iron router waiton, but if you are just trying to get the count I would not recommend the client subscribing to all 1.7 million documents.
As many mentioned, this was due to large volume of data. Expected results are returned after removing autopublish and creating a channel.
Meteor.publish('postCodesChannel', function(){
return PostCodes.find({postcode: {$regex: /B14 6B[DE]/}});
});
Browser console now returns appropriate values
>PostCodes.find().count();
2
I've a document which has another nested document in it represent a changes (logging). Each change document has a timestamp, field, old and new values. Basically as you can see this document would grow a lot, and I really don't want to get all changes but only a recent one.
What I want to do is make a query and get only those changes which falls in between two time stamps. I am not interested in any other information in document, so I dont want to pull it, just recent changes.
{
.......
"adwordsChanges":[
{
"timestamp":NumberLong("1400162491325"),
"field":"syncState",
"old":null,
"new":"OK"
},
{
"timestamp":NumberLong("1400162491325"),
"field":"keywordId",
"old":null,
"new":NumberLong("23918779329")
},
{
"timestamp":NumberLong("1400162491325"),
"field":"adGroupId",
"old":null,
"new":NumberLong("16972286472")
}
]
}
This is what I've tried!
db.keywords.find(
{
$and :[{"_id" : ObjectId("5374c7a7ac58b0d3b5e970fa")}, {"adwordsChanges.field" : "keywordId"}, {"adwordsChanges.timestamp" : {$gte:NumberLong("11111111"), $lte:NumberLong(99999999999999) }}]
})
Running Andrei's query I am getting compilation error:
assert: command failed: {
"errmsg" : "exception: The top-level _id field is the only field currently supported for exclusion",
"code" : 16406,
"ok" : 0
} : aggregate failed
Error: command failed: {
"errmsg" : "exception: The top-level _id field is the only field currently supported for exclusion",
"code" : 16406,
"ok" : 0
} : aggregate failed
at Error (<anonymous>)
at doassert (src/mongo/shell/assert.js:11:14)
at Function.assert.commandWorked (src/mongo/shell/assert.js:244:5)
at DBCollection.aggregate (src/mongo/shell/collection.js:1149:12)
at (shell):1:13
2014-05-27T15:23:54.945+0100 Error: command failed: {
"errmsg" : "exception: The top-level _id field is the only field currently supported for exclusion",
"code" : 16406,
"ok" : 0
Thanks for any help!
You could use aggregation framework to filter subdocuments, like this:
db.keywords.aggregate({$unwind:"$adwordsChanges"},
{$match:{"adwordsChanges.timestamp" :
{$gte:1400162491325, $lte:23918779329},
"adwordsChanges.field" : "keywordId"}},
{$project:{_id:0,
timestamp:"$adwordsChanges.timestamp",
field:"$adwordsChanges.field",
old:"$adwordsChanges.old",
new:"$adwordsChanges.new"
}});
Explanation of difference between project and group
Initially I thought original document should be returned and group was used to restore original document structure, i.e in this case groupis opposite operation to unwind. After clarification it became clear that only subdocuments were needed, then I replaced group operation with project, so that subdocuments were projected to root level.