I am new to MongoDB and would appreciate any suggestion I can get here. We have recently changed the data storage from MySQL to MongoDB but we didn't change the schema. Now we store data as follow: each document in collection has data from all tables for a given entry. Each table now a field in document with rows stored in array. All operations that were fast and easy in SQL are very slow now and require to write customer code.
My current issue is how to implement SQL join for fields (tables before). An example:
{
table1:[{id: 1, prop1: a}, {id: 2, prop1: b}],
table2:[{id: 1, prop2: c}]
table3:[{id: 2, prop3: d}]
}
I need to get list of objects merged as follow:
[
{id: 1, prop1: a, prop2:c},
{id: 2, prop1: b, prop3:d}
]
Could you recommend how to best achieve this transformation? Thank you!
Perform aggregation with $lookup on tables.
e.g.
db.table1.aggregate([
{
$lookup:{
from: "table2",
localField: "id",
foreignField: "id",
as: "table2"
}
}
])
It will return output as
{id: 1, prop1: a, table2: [{id: 1, prop2: c}]}
for better understanding follow https://docs.mongodb.com/manual/reference/operator/aggregation/lookup/
Related
I'm trying to $merge documents from Read-Only View (aka ROV) into On-Demand Materialized View (aka OMV). With Compass for now. I've created a pipeline in Aggregations section of Compass:
[{
$project: {
_id: 0,
RunDate: 1,
Manufacturer: 1,
Name: 1,
Workgroup: 1,
TotalPhysicalMemory: 1,
Model: 1,
PartOfDomain: 1,
Domain: 1,
NumberOfLogicalProcessors: 1
}
}, {
$merge: {
into: 'omv_WSCReportIndex',
on: 'Name',
whenMatched: 'replace',
whenNotMatched: 'insert'
}
}]
Basically I want OMV with unique 'Name' field where I will be inserting computer docs like this one (and overwriting ones, if needed):
RunDate: "2022-08-23 10:54:38Z"
Manufacturer: "VMware, Inc."
Name: "SRV010"
Workgroup: null
TotalPhysicalMemory: 16
Model: "VMware7,1"
PartOfDomain: true
Domain: "test.loc"
NumberOfLogicalProcessors: 4
Problem is I'm getting an error when I click RUN: Cannot find index to verify that join fields will be unique. I've tried creating index on 'omv_WSCReportIndex' for 'Name' field like "1-Type Index with Unique", "Text-Type Index with Unique", tried to create index with Collation object..
I can't just figure it out how to make this OMV. Please help!
I a model schema(created from mongoose)(say collection A) has a reference to another collection(say collection B), then how do you upsert to this referenced collection so that they respect the referenced relationship.
e.g.
A = [...{
name: "a",
b_id: "EXISTENT_ID"
}]
B = [..., {
_id: "EXISTENT_ID",
"subject": "science",
"age": 23
}]
I tried to bulk update like this:
A.bulkWrite([{
updateOne: {
filter: {_id: "EXISTENT_ID"},
update: {$set: {"A.b_id": {subject: "maths", "age": 22}}},
upsert: true
}
}])
and I get a write error that says: Updating the path 'a.b_id' would create a conflict at 'b_id', I was expecting the associated reference to be updated, since the schema of A is defined as:
Schema({
name: String,
b_id: {
type: mongoose.Types.ObjectId,
ref: 'B',
required: true,
}
})
The reason why I'd have to bulkWrite A is because the record is to be created if it doesn't exist and the reference linked.
For now I'm using javascript to do things manually making multiple round trips to the database, but, I'd like to use queries if possible. Is something I'm doing currently wrong or is there a mechanism to do this sort of a referenced update? I'd like a header to proceed. Thanks in advance.
I have two collections in MongoDB. Let's call one "objects":
{
_id: 11,
name: "Foo"
},
{
_id: 12,
name: "Bar"
},
{
_id: 13,
name: "Quux"
}
Another one is "values", that references "objects":
{
_id: 21,
value: "One",
object_id: 11
},
{
_id: 22,
value: "Two",
object_id: 11
},
{
_id: 21,
value: "Three",
object_id: 13
}
I want to query "objects" collection, showing the number of incoming links from "values" for each document in "objects" and sort by that number, in an descending order. I want to receive something like this:
{
_id: 11,
name: "Foo",
links: 2
},
{
_id: 13,
name: "Quux"
links: 1
},
{
_id: 12,
name: "Bar",
links: 0
}
In SQL, this is achievable using a simple outer join, GROUP BY and ORDER BY.
I can't use MapReduce (this must be an online query, not a scheduled job). I can't denormalize anything as "objects" and "values" in reality are collections of pretty large documents with lots of legacy code depending on their exact structure.
Any advice?
MongoDB has no JOINs. There is no way to get data from more than one collection with a single query. This means you need to program the JOIN on the application layer and then do your GROUP BY, also in the application. How to do this depends on the technology which your application is using.
Sorry, but this is what you get when you structure your data for MongoDB as if it were an SQL database.
I have collection called company with
{
_id: ObjectId("1111"),
name: 'ABc COrp',
address: {
Street: '124 Main Street'
city: 'San Diego'
state: 'California'
zip: "92115"
}
vendor: [{
vendor_id: ObjectId("1234")
}, {
vendor_id: ObjectId("4567"
}]
}
}
// vendor collection
{
_id: ObjectId("1234"),
name: 'foodCA company',
company: [company_id: ObjectId('11111')]
}
Now i have types of vendor like food, office supplies,etc for a company. Vendor and company has many-many relationship.
Can i use dbref (http://www.mongodb.org/display/DOCS/Database+References#DatabaseReferences-DBRef) for many-many when I am already using embedded document structure)
Thank you! I am a newbie to mongoDB. I apologize for my ignorance.
Based on official documentation are two options:
dbRef
manual references.
dbRef
Keep in mind that:
To resolve DBRefs, your application must perform additional queries to
return the referenced documents. Many drivers have helper methods that
form the query for the DBRef automatically. The drivers 1 do not
automatically resolve DBRefs into documents.
Because of these some people (1, 2, 3 and I by myself do not see a point of using them). Also dbRef allows you to reference not only another collection, but also another database. (I have never found it useful, but it does not means that other have not).
Manual referencing:
I prefer this method because it is super simple and easy to read. You have companies and vendors, so
{
companyID : 1,
vendors: [1, 3, 19]
}
Note that there is no reason of putting each vendor as an object as you have done. Have small amount of different vendors, than you can do:
{
companyID : 1,
vendors_food: [1, 3],
vendors_drinks: [17]
}
Many vendors with different types, than you can do what you have done:
{
companyID : 1,
vendors: [{
'type' : 'drink',
'id' : 18
},{
'type' : 'food',
'id' : 3
}],
}
And after you found vendors for your company [1, 6, 29], you can combine them and do only one db request to find info about all of them with $in:
db.vendors.find({
'_id': {
$in: [1, 6, 29]
}
});
Note, that this is only my preference of working with mongo.
I need query data from collection a first, then according to those data, query from collection b. Such as:
For each id queried from a
query data from b where "_id" == id
In SQL, this can be done by join table a & b in a single select. But in mongodb, it needs do multi query, it seems inefficient, doesn't it? Or it can be done by just 2 queries?(one for a, another for b, rather than 1 plus n) I know NoSQL doesn't support join, but is there a way to batch execute queries in for loop into a single query?
You'll need to do it as two steps.
Look into the $in operator (reference) which allows passing an array of _ids for example. Many would suggest you do those in batches of, say, 1000 _ids.
db.myCollection.find({ _id : { $in : [ 1, 2, 3, 4] }})
It's very simple, don't make so many DB calls for each id, it is very inefficient, it is possible to execute a single query which will return all documents relevant to each of the ids in a single pass using the $in operator in MongoDB, which is synonymous to in syntax in SQL so for example if you need to find out the documents for 5 ids in a single pass then
const ids = ['id1', 'id2', 'id3', 'id4', 'id5'];
const results = db.collectionName.find({ _id : { $in : ids }})
This will get you all the relevant documents in a single pass.
This is basically a join in SQL parlance and you can do it in mongo using an aggregate query called $lookup.
For example if you had some types in collections like these:
interface IFoo {
_id: ObjectId
name: string
}
interface IBar {
_id: ObjectId
fooId: ObjectId
title: string
}
Then query with an aggregate like this:
await db.foos.aggregate([
{
$match: { _id: { $in: ids } } // query criteria here
},
{
$lookup: {
from: 'bars',
localField: '_id',
foreignField: 'fooId',
as: 'bars'
}
}
])
May produce resulting objects like this:
{
"_id": "foo0",
"name": "example foo",
"bars": [
{ _id: "bar0", "fooId": "foo0", title: "crow bar" }
]
}