As the title says, I need to retrieve the names of all the keys in my MongoDB collection, BUT I need them split up based on a key/value pair that each document has. Here's my clunky analogy: If you imagine the original collection is a zoo, I need a new collection that contains all the keys Zebras have, all the keys Lions have, and all the keys Giraffes have. The different animal types share many of the same keys, but those keys are meant to be specific to each type of animal (because the user needs to be able to (for example) search for Zebras taller than 3ft and giraffes shorter than 10ft).
Here's a bit of example code that I ran which worked well - it grabbed all the unique keys in my entire collection and threw them into their own collection:
db.runCommand({
"mapreduce" : "MyZoo",
"map" : function() {
for (var key in this) { emit(key, null); }
},
"reduce" : function(key, stuff) { return null; },
"out": "MyZoo" + "_keys"
})
I'd like a version of this command that would look through the MyZoo collection for animals with "type":"zebra", find all the unique keys, and place them in a new collection (MyZoo_keys) - then do the same thing for "type":"lion" & "type":"giraffe", giving each "type" its own array of keys.
Here's the collection I'm starting with:
{
"name": "Zebra1",
"height": "300",
"weight": "900",
"type": "zebra"
"zebraSpecific1": "somevalue"
},
{
"name": "Lion1",
"height": "325",
"weight": "1200",
"type": "lion",
},
{
"name": "Zebra2",
"height": "500",
"weight": "2100",
"type": "zebra",
"zebraSpecific2": "somevalue"
},
{
"name": "Giraffe",
"height": "4800",
"weight": "2400",
"type": "giraffe"
"giraffeSpecific1": "somevalue",
"giraffeSpecific2": "someothervalue"
}
And here's what I'd like the MyZoo_keys collection to look like:
{
"zebra": [
{
"name": null,
"height": null,
"weight": null,
"type": null,
"zebraSpecific1": null,
"zebraSpecific2": null
}
],
"lion": [
{
"name": null,
"height": null,
"weight": null,
"type": null
}
],
"giraffe": [
{
"name": null,
"height": null,
"weight": null,
"type": null,
"giraffeSpecific1": null,
"giraffeSpecific2": null
}
]
}
That's probably imperfect JSON, but you get the idea...
Thanks!
You can modify your code to dump the results in a more readable and organized format.
The map function:
Emit the type of animal as key, and an array of keys for
each animal(document). Leave out the _id field.
Code:
var map = function(){
var keys = [];
Object.keys(this).forEach(function(k){
if(k != "_id"){
keys.push(k);
}
})
emit(this.type,{"keys":keys});
}
The reduce function:
For each type of animal, consolidate and return the unique keys.
Use an Object(uniqueKeys) to check for duplicates, this increases the running
time even if it occupies some memory. The look up is O(1).
Code:
var reduce = function(key,values){
var uniqueKeys = {};
var result = [];
values.forEach(function(value){
value.keys.forEach(function(k){
if(!uniqueKeys[k]){
uniqueKeys[k] = 1;
result.push(k);
}
})
})
return {"keys":result};
}
Invoking Map-Reduce:
db.collection.mapReduce(map,reduce,{out:"t1"});
Aggregating the result:
db.t1.aggregate([
{$project:{"_id":0,"animal":"$_id","keys":"$value.keys"}}
])
Sample o/p:
{
"animal" : "lion",
"keys" : [
"name",
"height",
"weight",
"type"
]
}
{
"animal" : "zebra",
"keys" : [
"name",
"height",
"weight",
"type",
"zebraSpecific1",
"zebraSpecific2"
]
}
{
"animal" : "giraffe",
"keys" : [
"name",
"height",
"weight",
"type",
"giraffeSpecific1",
"giraffeSpecific2"
]
}
Related
Below is the structure of my Collection.
{
"_id": "61b70e9d7ba0e2a555e06a59",
"ProjId": 12,
"ArtifactAttributes": [
{
"_t": "ArtifactAttributes",
"AttrId": 1,
"AttributeName": "Description",
"AttributeValue": "Test Description"
},
{
"_t": "ArtifactAttributes",
"AttrId": 2,
"AttributeName": "Details",
"AttributeValue": "Test Details"
}
]
}
I wanted to add below mentioned array in the ArtifactAttributes array field present in the collection but while adding the same wanted to set AttrId by fetching max value + 1 of AttrId present in the existing array list.
AttrId is totally dependents on the max value of the AttrId present in the existing array it could 3,4,5 anything.
{
"_t": "ArtifactAttributes",
"AttrId": 3,
"AttributeName": "Owner",
"AttributeValue": "Test Owner"
}
Final document after adding above array will be as below
{
"_id": "61b70e9d7ba0e2a555e06a59",
"ProjId": 12,
"ArtifactAttributes": [
{
"_t": "ArtifactAttributes",
"AttrId": 1,
"AttributeName": "Description",
"AttributeValue": "Test Description"
},
{
"_t": "ArtifactAttributes",
"AttrId": 2,
"AttributeName": "Details",
"AttributeValue": "Test Details"
},
{
"_t": "ArtifactAttributes",
"AttrId": 3,
"AttributeName": "Owner",
"AttributeValue": "Test Owner"
}
]
}
I tried around a little bit, to use $push or $addField does not work, so I came up with concatinating 2 arrays where I define your object as an object of the 2nd array for concatination.
So simplified:
Set the Array you allready have, as a concatination of that array with a new array containing your object with calculated AttrId of the other arrays AttrId.
I directly added the function into the object you want to insert:
let toBeInserted = {
"_t": "ArtifactAttributes",
"AttributeName": "Owner",
"AttributeValue": "Test Owner"
}
// If you are using typescript make sure AttrId is previously unset
// because it will throw an error if it wants to insert this into a "number" field
toBeInserted["AttrId"] = {
'$add': [{ '$max': '$ArtifactAttributes.AttrId' }, 1]
}
await this.testSchema.updateOne({}, // Make sure the following is an array, array indicates aggregation, this does not work without
[{
'$set': {
'ArtifactAttributes': {
'$concatArrays': [
'$ArtifactAttributes', [
toBeInserted
]
]
}
}
}])
Here proof of it working:
Given a jsonb and set of keys how can I get a new jsonb with required keys.
I've tried extracting key-values and assigned to text[] and then using jsonb_object(text[]). It works well, but the problem comes when a key has a array of jsons.
create table my_jsonb_table
(
data_col jsonb
);
insert into my_jsonb_table (data_col) Values ('{
"schemaVersion": "1",
"Id": "20180601550002",
"Domains": [
{
"UID": "29aa2923",
"quantity": 1,
"item": "book",
"DepartmentDomain": {
"type": "paper",
"departId": "10"
},
"PriceDomain": {
"Price": 79.00,
"taxA": 6.500,
"discount": 0
}
},
{
"UID": "bbaa2923",
"quantity": 2,
"item": "pencil",
"DepartmentDomain": {
"type": "wood",
"departId": "11"
},
"PriceDomain": {
"Price": 7.00,
"taxA": 1.5175,
"discount": 1
}
}
],
"finalPrice": {
"totalTax": 13.50,
"total": 85.0
},
"MetaData": {
"shopId": "1405596346",
"locId": "95014",
"countryId": "USA",
"regId": "255",
"Date": "20180601"
}
}
')
This is what I am trying to achieve :
SELECT some_magic_fun(data_col,'Id,Domains.UID,Domains.DepartmentDomain.departId,finalPrice.total')::jsonb FROM my_jsonb_table;
I am trying to create that magic function which extracts the given keys in a jsonb format, as of now I am able to extract scalar items and put them in text[] and use jsonb_object. but don't know how can I extract all elements of array
expected output :
{
"Id": "20180601550002",
"Domains": [
{
"UID": "29aa2923",
"DepartmentDomain": {
"departId": "10"
}
},
{
"UID": "bbaa2923",
"DepartmentDomain": {
"departId": "11"
}
}
],
"finalPrice": {
"total": 85.0
}
}
I don't know of any magic. You have to rebuild it yourself.
select jsonb_build_object(
-- Straight forward
'Id', data_col->'Id',
'Domains', (
-- Aggregate all the "rows" back together into an array.
select jsonb_agg(
-- Turn each array element into a new object
jsonb_build_object(
'UID', domain->'UID',
'DepartmentDomain', jsonb_build_object(
'departId', domain#>'{DepartmentDomain,departId}'
)
)
)
-- Turn each element of the Domains array into a row
from jsonb_array_elements( data_col->'Domains' ) d(domain)
),
-- Also pretty straightforward
'finalPrice', jsonb_build_object(
'total', data_col#>'{finalPrice,total}'
)
) from my_jsonb_table;
This probably is not a good use of a JSON column. Your data is relational and would better fit traditional relational tables.
I have a user document that looks like this:
{
"userId": "249869823570",
"name": "john",
"country": "usa",
"active": true,
"serviceProviders": [
{
"serviceProvidersId": "897892893",
"serviceProvidersName": "AT&T",
"active": true,
"serviceProviderContactPerson": []
},
{
"serviceProvidersId": "82589628569",
"serviceProvidersName": "T-Mobile",
"active": true,
"serviceProviderContactPerson": []
}
]
}
and I want to create 3 methods to insert/update/delete a serviceProviderContactPerson. my dilema is, my main document have array of serviceProviders, and a serviceProvider have a list of serviceProviderContactPerson...what would be the best practice in mongo to work this?
serviceProviderContactPerson document will look like this:
{
"contactId": "873498798",
"name": "Mark",
"email": "mark#tmobile.com",
"phone": "917-475-4637"
}
You can use update to perform all operations in 3.6.
Insert a new array element where serviceProvidersId = "897892893"
db.colname.update(
{"serviceProviders.serviceProvidersId":"897892893"},
{$push: {"serviceProviders.$.serviceProviderContactPerson":serviceProviderContactPersonDoc}}
)
Update phone where serviceProvidersId = "897892893" and contactId = "873498798"
db.colname.update(
{},
{$set: {"serviceProviders.$[sp].serviceProviderContactPerson.$[spc].phone":phone value}},
{arrayFilters:[{"sp.serviceProvidersId":"897892893"}, {"spc.contactId":"873498798"}]}
)
Delete array element where serviceProvidersId = "897892893" and contactId = "873498798"
db.colname.update(
{},
{$pull: {"serviceProviders.$[sp].serviceProviderContactPerson":{"contactId":"873498798"}}},
{arrayFilters:[{"sp.serviceProvidersId":"897892893"}]}
)
I upgraded Wekan from 0.48 to 0.95. It looks like what happened in Mongo is that it took the checklist collection from one containing a nested list of items and split it out into a new checklistItems collection. It appears to have copied the data correctly- except that instead of copying each item's title, it copied the checklist title to each list.
I started with this in wekan.checklists:
{
"_id": "z329QEDfjsuQcxz7E",
"cardId": "TBgz6gMGCcn9XNPSW",
"title": "A list",
"sort": 0,
"createdAt": {
"$date": "2018-05-09T22:20:50.537Z"
},
"items": [
{
"_id": "z329QEDfjsuQcxz7E0",
"title": "Do some stuff",
"isFinished": false,
"sort": 0
},
{
"_id": "z329QEDfjsuQcxz7E1",
"title": "Do some other stuff",
"isFinished": false,
"sort": 1
}
],
"userId": "YndMrPQ5XhZTTKD2S"
}
and wound up with the following in wekan.checklistItems:
{
"_id": "RADPEu4nhr9PgwPHH",
"title": "A list",
"sort": 0,
"isFinished": false,
"checklistId": "z329QEDfjsuQcxz7E",
"cardId": "TBgz6gMGCcn9XNPSW"
}
{
"_id": "Guy3aaJL4WLJQjzRX",
"title": "A list",
"sort": 1,
"isFinished": false,
"checklistId": "z329QEDfjsuQcxz7E",
"cardId": "TBgz6gMGCcn9XNPSW"
}
and this in wekan.checklists:
{ "_id" : "z329QEDfjsuQcxz7E", "cardId" : "TBgz6gMGCcn9XNPSW", "title" : "MVP", "sort" : 0, "createdAt" : ISODate("2018-05-09T22:20:50.537Z"), "userId" : "YndMrPQ5XhZTTKD2S" }
Is there a quick query to go back through my original wekan.checklists and update the titles in wekan.checklistItems? I note that the checklistIDs stayed the same but the card id's are different- I can of course load the old wekan.checklists collection into my current (upgraded) db to query against.
Fix: load your old db.checklists into db.checklistsOld (I used mongoimport -d wekan -c checklistsOld ~/checklistsOld.bson, where checklistsOld.bson held my backup from before the upgrade. Use the following script in Robo3T:
db.checklistsOld.find({}, {"_id": 1, "items.title":1, "items.sort": 1 }).forEach( (list, i, lists) => {
var checklistId = list._id;
list.items.forEach( (item, j, items) => {
var sort = item.sort,
title = item.title;
db.checklistItems.update({"checklistId": checklistId, "sort":sort}, {$set: {"title": title}} );
});
});
Depending on how many items you have, you may need to adjust "shellTimeoutSec" in Robo3T (https://github.com/Studio3T/robomongo/wiki/Robomongo-Config-File-Guide)
I have collection with the following (sample) documents:
{
"label": "Tree",
"properties": {
"height": {
"type": "int",
"label": "Height",
"description": "In meters"
},
"coordinates": {
"type": "coords",
"label": "Coordinates"
},
"age": {
"type": "int",
"label": "Age"
}
}
}
Keys in the properties attribute are different for almost each of the documents in collection.
I want to find all documents that have at least one property of given type.
What I'm looking for is to query this for {"properties.*.type": "coords"}. But this is not working as it is only my invention of mongo query.
Every help I was able to find concerned the $elemMatch operator which I can not use here because properties is an object, not an array.
Hi as per my knowledge in mongodb not provide this kind of search. So for finding this first I separated out all keys using map-reduce and then find query form so below code will help you
var mapReduce = db.runCommand({
"mapreduce": "collectionName",
"map": function() {
for (var key in this.properties) {
emit(key, null);
}
},
"reduce": function(key, stuff) {
return null;
},
"out": "collectionName" + "_keys"
})
db[mapReduce.result].distinct("_id").forEach(function(data) {
findkey = [];
findkey.push("properties." + data + ".type");
var query = {};
query[findkey] = "coords";
var myCursor = db.collectionName.find(query);
while (myCursor.hasNext()) {
print(tojson(myCursor.next()));
}
})
MongoDB doesn't support searches on keys - things like properties.* to match all subkeys of properties, etc. You shouldn't have arbitrary keys or keys that you don't know about in your schema, unless they are just for display, generally, because you will not be able to interact with them very easily in MongoDB.
If you do want to store dynamic attributes, the best approach is usually an array like the following:
{
"properties" : [
{
"key" : "height",
"value" : {
"type" : "Int",
"label" : "Height",
"description" : "In meters"
}
},
...
]
}
Efficient querying for your use case
find all documents that have at least one property of given type
results from an index on { "key" : 1 }:
db.test.find({ "properties.key" : { "$in" : ["height", "coordinates", "age"] } })