I am doing a query like the following, which works fine with MongoDB, but sometimes fails with CosmosDB. I need it to work with both.
(XXX is a placeholder for any string value. All strings have unique values that are redacted for readability, and actual content should be of no significance.)
{
server_index: {
$elemMatch: {
server: "XXX",
index: "XXX",
delete_time: { $exists: false },
path: {
$in: ["XXX", "XXX", "XXX" ]
}
}
}
}
The schema of a document is somewhat like this:
{
...,
server_index: [
{
server: "XXX",
index: "XXX",
delete_time: ISODate(...), // optional
path: "XXX"
},
{...}, // same as above
...
],
...
}
This query sometimes works as expected with CosmosDB as well, but sometimes I also get the following response:
{
_t: "OKMongoResponse",
ok: 0,
code: 115,
errmsg: "Command is not supported",
$err: "Command is not supported"
}
What is especially strange is that the query seemingly succeeds, and the response above is returned by a "valid" cursor as the first document, which then causes my document parser "crash".
I am using the C++ legacy driver. Is this even supported by Cosmos DB?
(According to the developer I inherited this project from, it is, and as always when you inherit projects, it all worked fine according to the previous developer... So this may be due to a change in Cosmos DB, due to the nature of my test data, or who knows what...)
Side note: In MongoDB, there is a multi-key index on server_index, which looks like this:
{
"server_index.delete_time" : 1,
"server_index.server" : 1,
"server_index.index" : 1,
"server_index.path" : 1
}
Is this even supported in CosmosDB?
EDIT: Trying to add this index with Robo 3T silently fails, with no error message whatsoever. The index is simply not added. Nice!
(Please don't ask about the strange database schema. It is like it is for a reason, and believe me, I, too, would like to burn it all down and replace it with something else ... I am open for suggestions for alternative queries, though)
This was probably a server-side problem. It seemed wrong in the first place (error status returned as part of the query result), and it has disappeared after a couple of weeks without me changing anything.
Related
I am planning to migrate all my mongoDb database to azure cosmosDB. For the testing purposes I was trying to test all the queries by making azure cosmosDB with mongo API as the underlying datasource. I was able to execute all the queries by creating some custom indices (Single field / wildcard) in cosmosDB. But a particular query is failing. I couldn't find any noticable reason for the same. The query looks as follows (The query is somewhat too long with $in operator. I suspect that)
db.getCollection("rules").find({
"$and":[
{
"$or":[
{
"subRules.ALL.leftOperand":"GROUP",
"subRules.ALL.rightOperand":{
"$in":[
<4000+ values>
]
}
},
{
"subRules.ALL.leftOperand":"MACHINE",
"subRules.ALL.rightOperand":{
"$in":[
<4000+ values>
]
}
}
]
},
{
"ruleName":{
"$regex":""
}
}
],
"subRules.ANY.leftOperand":{
"$ne":"COFFEE_AMOUNT"
}
})
This exact query works well with mongo server. But when I run this in azure cosmos. I am getting the following error
error: {
"ok" : 0,
"errmsg" : "Error=2, Details='Response status code does not indicate success: BadRequest (400); Substatus: 0; ActivityId: 443df8b1-136f-4018-9d77-63b3442ab7f8; Reason: (Response status code does not indicate success: BadRequest (400); Substatus: 0; ActivityId: 443df8b1-136f-4018-9d77-63b3442ab7f8; Reason: (Response status code does not indicate success: BadRequest (400); Substatus: 0; ActivityId: 443df8b1-136f-4018-9d77-63b3442ab7f8; Reason: (Message: ��errors�ꠈseverity�Error�location�\u000e�start\u0000�end�5\u0002�code�SC3031�message~pTt\u00194�2A�zY�\u0007���r�\\&���2�\u001dƧ��6�]o��r��\f�6�E{�\rz���0�-���a:\u001a�.��t4�\u001c���av�=.�#Tt\u0019\u0014f���2\u0019�N��tPz\u000e��\\\r\nActivityId: 443df8b1-136f-4018-9d77-63b3442ab7f8, Request URI: /apps/27c904fe-36bc-4acd-b55d-2c8494082ce7/services/c88f5dcc-2b90-4935-9281-38171d9dd72a/partitions/398285ee-29c1-4564-8e79-91c184ef65dd/replicas/132769491079094442s/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.17763 cosmos-netstandard-sdk/3.18.0);););",
"code" : 2,
"codeName" : "BadValue"
}
I have tried creating wildcard index on the particular collection. Still it fails.
Nb : When the values in the array of $in operator is few the query executes successfully in cosmosDB as well!!
If anyone can shed some light to this issue it will be much appreciated. Thank you
I think that at this time you already get a workaround, but just in case, you are comparing the right version of mongodb with cosmos? Remember that cosmos is in 4 or less, mongodb actually is in v5.
Anyway, a "$IN" clause with a big amount of values to compare can be a bottleneck, SQL or NOT, have you tried to make your "IN" list a lookupable collection? This maybe work with cosmos
I have a what I consider to be a pretty straight forward search. I am developing it on MOngoDB locally but deploying it for DocumentDB on AWS. The function that I've written looks something like this:
if (searchCategory) {
return [
{
$match: {
$or: [
{ title: params.searchString },
{ description: params.searchString },
{ workingNotes: params.searchString },
],
active: true,
},
},
];
}
This query used to not have the $match statement around it and it worked fine, but something that we wanted to do was to use reference fields in our model for searching linked items as well. So we have something like this in our model:
appendix: [{type: Schema.Types.ObjectId, ref: "document"}],
Now when I fire off the search query, I get an error back saying "Aggregation stage not supported: '$lookup on multiple join conditions and uncorrelated subquery.'"
Adding the reference field and the $match were the only changes made to the functionality, which worked before we added those.
Based off the documentation I've read about DocumentDB, the $match operator is supported, so it's possible I am missing something else about how to structure the query or handle the reference fields but I've been unable to determine what that might be.
Any help would be appreciated. Thanks!
In MongoDB 3.4 (and still in 3.6 according to the doc), the collection.aggregate([...]) methods returns a cursor. Then we could do collection.aggregate([...]).toArray() or .forEach(). And get or iterate over the array of results.
Since 3.6 (appart the fact that the cursor options is now required, which is not specified in the doc), the following command:
collection.aggregate(
[...],
{ cursor: { batchSize: 10 } }
)
Returns on object with this shape:
{
"cursor": {
"firstBatch": [...],
"id",
"ns"
},
"ok": 1,
"$clusterTime": {...},
"operationTime": Timestamp(1525344553, 1)
}
It's not possible anymore to iterate over the whole results and collection.aggregate([...]).toArray() is not a function.
A found a few issues about this, but nothing really relevant.
The doc is really outdated on this and I'm not even sure that this is the expected behaviours.
Note: I'm running it in the mongoshell, but also encountered these issues with the last node driver.
The problems comes from mongo-hacker...
Say I have an entry in the inventory collection that looks like
{ _id: 1, item: "polarizing_filter", tags: [ "electronics", "camera" ]}
and I issue the command
db.inventory.update(
{ _id: 1 },
{ $addToSet: { tags: "accessories" } }
)
I have an oplog tailer, and would like to know that, specifically, "accessories" has been added to this document's tags field. As far as I can tell, the oplog always normalizes commands to use $set and $unset to maintain idempotency. In this case, the field of the entry describing the update would show something like
{$set : { tags : ["electronics", "camera", "accessories"] } }
which makes it impossible to know which tags were actually added by this update. Is there anyway to do this? I'm also curious about the analogous case in which fields are modified through deletion, e.g. through $pull. Solutions outside of the realm of an oplog tailer are welcome, as well as pointers to documentation of this command normalization process - I can't find it.
Thanks!
This is the case: A webshop in which I want to configure which items should be listed in the sjop based on a set of parameters.
I want this to be configurable, because that allows me to experiment with different parameters also change their values easily.
I have a Product collection that I want to query based on multiple parameters.
A couple of these are found here:
within product:
"delivery" : {
"maximum_delivery_days" : 30,
"average_delivery_days" : 10,
"source" : 1,
"filling_rate" : 85,
"stock" : 0
}
but also other parameters exist.
An example of such query to decide whether or not to include a product could be:
"$or" : [
{
"delivery.stock" : 1
},
{
"$or" : [
{
"$and" : [
{
"delivery.maximum_delivery_days" : {
"$lt" : 60
}
},
{
"delivery.filling_rate" : {
"$gt" : 90
}
}
]
},
{
"$and" : [
{
"delivery.maximum_delivery_days" : {
"$lt" : 40
}
},
{
"delivery.filling_rate" : {
"$gt" : 80
}
}
]
},
{
"$and" : [
{
"delivery.delivery_days" : {
"$lt" : 25
}
},
{
"delivery.filling_rate" : {
"$gt" : 70
}
}
]
}
]
}
]
Now to make this configurable, I need to be able to handle boolean logic, parameters and values.
So, I got the idea, since such query itself is JSON, to store it in Mongo and have my Java app retrieve it.
Next thing is using it in the filter (e.g. find, or whatever) and work on the corresponding selection of products.
The advantage of this approach is that I can actually analyse the data and the effectiveness of the query outside of my program.
I would store it by name in the database. E.g.
{
"name": "query1",
"query": { the thing printed above starting with "$or"... }
}
using:
db.queries.insert({
"name" : "query1",
"query": { the thing printed above starting with "$or"... }
})
Which results in:
2016-03-27T14:43:37.265+0200 E QUERY Error: field names cannot start with $ [$or]
at Error (<anonymous>)
at DBCollection._validateForStorage (src/mongo/shell/collection.js:161:19)
at DBCollection._validateForStorage (src/mongo/shell/collection.js:165:18)
at insert (src/mongo/shell/bulk_api.js:646:20)
at DBCollection.insert (src/mongo/shell/collection.js:243:18)
at (shell):1:12 at src/mongo/shell/collection.js:161
But I CAN STORE it using Robomongo, but not always. Obviously I am doing something wrong. But I have NO IDEA what it is.
If it fails, and I create a brand new collection and try again, it succeeds. Weird stuff that goes beyond what I can comprehend.
But when I try updating values in the "query", changes are not going through. Never. Not even sometimes.
I can however create a new object and discard the previous one. So, the workaround is there.
db.queries.update(
{"name": "query1"},
{"$set": {
... update goes here ...
}
}
)
doing this results in:
WriteResult({
"nMatched" : 0,
"nUpserted" : 0,
"nModified" : 0,
"writeError" : {
"code" : 52,
"errmsg" : "The dollar ($) prefixed field '$or' in 'action.$or' is not valid for storage."
}
})
seems pretty close to the other message above.
Needles to say, I am pretty clueless about what is going on here, so I hope some of the wizzards here are able to shed some light on the matter
I think the error message contains the important info you need to consider:
QUERY Error: field names cannot start with $
Since you are trying to store a query (or part of one) in a document, you'll end up with attribute names that contain mongo operator keywords (such as $or, $ne, $gt). The mongo documentation actually references this exact scenario - emphasis added
Field names cannot contain dots (i.e. .) or null characters, and they must not start with a dollar sign (i.e. $)...
I wouldn't trust 3rd party applications such as Robomongo in these instances. I suggest debugging/testing this issue directly in the mongo shell.
My suggestion would be to store an escaped version of the query in your document as to not interfere with reserved operator keywords. You can use the available JSON.stringify(my_obj); to encode your partial query into a string and then parse/decode it when you choose to retrieve it later on: JSON.parse(escaped_query_string_from_db)
Your approach of storing the query as a JSON object in MongoDB is not viable.
You could potentially store your query logic and fields in MongoDB, but you have to have an external app build the query with the proper MongoDB syntax.
MongoDB queries contain operators, and some of those have special characters in them.
There are rules for mongoDB filed names. These rules do not allow for special characters.
Look here: https://docs.mongodb.org/manual/reference/limits/#Restrictions-on-Field-Names
The probable reason you can sometimes successfully create the doc using Robomongo is because Robomongo is transforming your query into a string and properly escaping the special characters as it sends it to MongoDB.
This also explains why your attempt to update them never works. You tried to create a document, but instead created something that is a string object, so your update conditions are probably not retrieving any docs.
I see two problems with your approach.
In following query
db.queries.insert({
"name" : "query1",
"query": { the thing printed above starting with "$or"... }
})
a valid JSON expects key, value pair. here in "query" you are storing an object without a key. You have two options. either store query as text or create another key inside curly braces.
Second problem is, you are storing query values without wrapping in quotes. All string values must be wrapped in quotes.
so your final document should appear as
db.queries.insert({
"name" : "query1",
"query": 'the thing printed above starting with "$or"... '
})
Now try, it should work.
Obviously my attempt to store a query in mongo the way I did was foolish as became clear from the answers from both #bigdatakid and #lix. So what I finally did was this: I altered the naming of the fields to comply to the mongo requirements.
E.g. instead of $or I used _$or etc. and instead of using a . inside the name I used a #. Both of which I am replacing in my Java code.
This way I can still easily try and test the queries outside of my program. In my Java program I just change the names and use the query. Using just 2 lines of code. It simply works now. Thanks guys for the suggestions you made.
String documentAsString = query.toJson().replaceAll("_\\$", "\\$").replaceAll("#", ".");
Object q = JSON.parse(documentAsString);