I've been trying to become familiar with using MongoDB, particularly GridFS (which is implemented through the client). In a project I'm playing around with, I have users with corresponding locations, followed by corresponding data, but it is currently just stored on a filesystem which will ultimately lead to problems.
So an example data structure I have now is:
./(userId)/images/img1.png
./(userId)/data/sensor1/sensorOutput1.data
./(userId)/data/sensor1/sensorOutput2.data
./(userId)/data/sensor2/sensorOutput1.data
./(userId)/data/sensor2/sensorOutput2.data
So, in looking at GridFS, I see that you can pass attribute/values to be associated with the meta-data for each file. Looking at this setup and after reading tutorials on MongoDB, I thought of this approach:
Have one database with multiple collections (such as data or images just to separate some of the data). Then in each of the collections, have attributes for each document such as:
Under image collection:
{
"userId": 5,
"path": "",
"filename":"img1.png"
...
}
or
Under data collection:
{
"userId": 5,
"path": "sensor1",
"filename": "sensorOutput1.data"
...
}
Alternatively, I could just have a collection, and for the sake of this example, I'll call "Everything"
{
"userId": 5,
"path": "images/",
"filename": "img1.png"
...
}
{
"userId": 5,
"path": "data/sensor1/",
"filename": "sensorOutput1.data"
...
}
{
"userId": 5,
"path": "data/sensor2/",
"filename": "sensorOutput1.data"
...
}
Do either of these solutions seem reasonable? Would I then create an index on the "path" attribute? I've seen examples for adding files to MongoDB just haven't found one with how to structure user files.
Thanks!
Related
In my angular app, I store items in groups(categories):
pendants, earrings, sets, etc (instead of a flat items array).
I would like the database structure to be grouped by category as well.
I use the following typescript data structure in Angular
items:Map<Category, Map<string, Item>>
So far I was able to create a JSON representation of it and load it to the google Realtime database. This has worked fine so far.
But now I would like to move to MongoDB which has mongoose Schemas
I'm not sure how to model this map of maps in Mongoose.
I understand I need to create a Schema of the single item.
But what about the items map? Do I need an Items Schema as well?
My collection in MongoDb is a collection of Item. But this is not a flat collection. It should be a "by-category" collection. How does MongoDB represent this structure and what is the appropriate Mongoose schema solution?
I've uploaded the items JSON to MongoDB via the mongoimport in the shell, the result in Mongo looks like this:
I think MongoDB sees my structure as a collection with a single huge, nested Document. This is not exactly what I want. I want the structure to look like on this image, but each item to be recognized as a Document in the same nested collection. And hopefully, to be able to query that structure accordingly, that is pull from the DB just the items from the pendants category, for example.
Also, should I go with typescript or javascript for mongoose?
I looked into typescript documentation and suspect this is not mature yet. I expected the mongoose schemas to actively use my typescript models and be based on them, but it still just creates schemas as hardcoded model copies. Kind of weird. Is there a robust typescript solution, which reuses frontend models and minimizes code duplication?
This is how the JSON representation of my items looks like
{
"pendants": {
"01001A": {
"catId": "01001A",
"name": "moon",
"category": "pendants",
"price": 90,
"inStockQuantity": 1,
"isOnSale": false,
"rating": 5,
"importance": 1,
"size": {
"diameter": 3.5,
"lanyardLength": 43
}
},
"01002A": {
"catId": "01002A",
"name": "fish",
"category": "pendants",
"price": 90,
"inStockQuantity": 1,
"isOnSale": false,
"rating": 5,
"importance": 1,
"size": {
"diameter": 2,
"lanyardLength": 43
}
}
},
"earrings": {
"02001A": {
"catId": "02001A",
"name": "winter",
"category": "earrings",
"price": 90,
"inStockQuantity": 1,
"isOnSale": false,
"rating": 5,
"importance": 1,
"size": {
"diameter": 1.3,
"chainLength": 43
}
},
"02002A": {
"catId": "02002A",
"name": "flower",
"category": "earrings",
"price": 160,
"inStockQuantity": 1,
"isOnSale": false,
"rating": 5,
"importance": 1,
"size": {
"height": 2.6,
"vavLength": 2
}
}
}
}
I am trying to refactor all the keys over a bunch of collections that match a certain regex or have the same name. The issue is that in different documents or collections, the keys in question may appear at different nesting levels in different locations.
For example, let's say we need to replace the key "sound" to "noise" in the following:
Animals collection:
{
"_id": "4ebb8fd68f7aaffc5d287383",
"animal": "cat",
"name": "fluffy",
"type": "long-haired",
"sound": "meow"
}
Events collection:
{
"_id": "4ebb8fd68f7abac75d287341",
"event": "thunder",
"description": {
"type": "natural",
"sound": "boom"
}
}
How would you go about doing it? Via raw mongo queries ideally, or pymongo if necessary
I am very new to coding, so my apologies if I use the wrong terms or am unclear. I have a list of 400+ words I need to be able to enter into my database. I found this collection to loop through my data and post it to the database. I am able to mostly get it to work, except for this one section (the meanings section) where it's an array of objects. If it makes a difference, this is for a React project and using MongoDB for the database.
Each word looks like this:
{
"text": "happy",
"traits": [
"wordlist100"
],
"meanings": [
{
"meaning": "happy",
"category": "default"
}
],
"approved": true,
"createdby": "60dcd7ba69b1e52ac8138051",
"approvedBy": "60dcd7ba69b1e52ac8138051",
"lastUserEdit": "60dcd7ba69b1e52ac8138051",
"blocked": false
}
Where each word can have multiple traits and multiple meanings.
I have all the words in a JSON file, and am running it through Postman using the collection linked above. This is the body of my request in Postman:
{
"text": "{{text}}",
"traits": [
"{{traits}}"
],
"meanings": [
{
"meaning": "{{meaning}}",
"category": "{{category}}"
}
],
"approved": true,
"createdby": "60dcd7ba69b1e52ac8138051",
"approvedBy": "60dcd7ba69b1e52ac8138051",
"lastUserEdit": "60dcd7ba69b1e52ac8138051",
"blocked": false
}
Using this, everything comes through correctly in my database except for the meaning and category of a word. This is what shows up in MongoDB:
{
"_id": {"$oid": "6121bd6addff936ba4eb84bf"},
"traits": [
"wordlist100"
],
"text": "happy",
"meanings": [
{
"_id": {"$oid": "6121bd6addff936ba4eb84c0"},
"meaning": "{{meaning}}",
"category": "{{category}}"
}
],
"approved": true,
"createdBy": {"$oid": "60dcd7ba69b1e52ac8138051"},
"approvedBy": {"$oid": "60dcd7ba69b1e52ac8138051"},
"lastUserEdit": {"$oid": "60dcd7ba69b1e52ac8138051"},
"blocked": false,
"createdAt": {"$date": "2021-08-22T02:58:50.679Z"},
"updatedAt": {"$date": "2021-08-22T02:58:50.679Z"},
"__v": 0
}
What am I doing wrong with the meanings section of the word? I tried playing around with several combinations of brackets and such for how to set up the meanings part.
I did try:
"meanings": [
"{{meanings}}"
]
...but that didn't work at all. Most of what I tried created an error that stopped the collection from running at all (it was probably bad syntax, as I said, I'm really new). This is the only arrangement that seems to get me close to right. Any help would be greatly appreciated.
Thanks!!
Attaching pics of the Postman code and created MongoDB document if it's easier for any of you to look at it with that formatting:
Image of Postman code
Image of MongoDB document created
So I feel foolish and this just shows how new I am to all of this. When I posted this question, all I had done was to try copying the style of what I saw in the body text of the example I found (linked in my question), but didn't do anything with actually defining variables or touching the pre-request script because I didn't realize that was a separate step. So basically, I just got lucky that it was smart enough to figure out the other values without me defining them as variables.
If anyone is reading this later, here's 3 pages of documentation I used to help me figure out my errors: Using Variables, Importing Data Files, and Scripting in Postman.
From reading those, I added the following code in the pre-request script:
let meanings = pm.iterationData.get("meanings");
pm.variables.set("meaning", meanings[0].meaning);
pm.variables.set("category", meanings[0].category);
I also set "meaning" and "category" as named variables with no initial value in the collection.
Image of adding variables to the Postman collection
One caveat I should note is that I'm not certain how well this will work once I start entering stuff with more than one trait or more than one meaning, because I haven't tested those parts yet. I suspect I still need to tweak things to make it work under those instances. But it is working in the examples I have with just one trait and one object in the meanings array.
I would like to remove all key/value pairs where the key is url from all documents inside a mongodb collection.
I've been agonising over this all weekend and have only managed to partially solve it using $unset, but I'm struggling to access all items particularly key/values inside objects in arrayKey and also do it in one query/function (if possible).
This is generally how each document looks. There sometimes may be more objects in the arrayKey, but other than that they're pretty similar:
{
"keyOne": "String",
"keyTwo": 7,
"keyThree": {
"subKeyOne": "String",
"url": "String"
},
"arrayKey": [
{
"arrayKeyOne":"String",
"url": "String"
},
{
"arrayKeyOne":"String",
"url": "String"
}
],
"url":"String"
}
Any help would be greatly appreciated. I'm still new to programming and mongodb so the documentation is a bit dense for a beginner.
Thanks :)
Let's assume we have the following collections:
Users
{
"id": MongoId,
"username": "jsloth",
"first_name": "John",
"last_name": "Sloth",
"display_name": "John Sloth"
}
Places
{
"id": MongoId,
"name": "Conference Room",
"description": "Some longer description of this place"
}
Meetings
{
"id": MongoId,
"name": "Very important meeting",
"place": <?>,
"timestamp": "1506493396",
"created_by": <?>
}
Later on, we want to return (e.g. from REST webservice) list of upcoming events like this:
[
{
"id": MongoId(Meetings),
"name": "Very important meeting",
"created_by": {
"id": MongoId(Users),
"display_name": "John Sloth",
},
"place": {
"id": MongoId(Places),
"name": "Conference Room",
}
},
...
]
It's important to return basic information that need to be displayed on the main page in web ui (so no additional calls are needed to render the table). That's why, each entry contains display_name of the user who created it and name of the place. I think that's a pretty common scenario.
Now my question is: how should I store this information in db (question mark values in Metting document)? I see 2 options:
1) Store references to other collections:
place: MongoId(Places)
(+) data is always consistent
(-) additional calls to db have to be made in order to construct the response
2) Denormalize data:
"place": {
"id": MongoId(Places),
"name": "Conference room",
}
(+) no need for additional calls (response can be constructed based on one document)
(-) data must be updated each time related documents are modified
What is the proper way of dealing with such scenario?
If I use option 1), how should I query other documents? Asking about each related document separately seems like an overkill. How about getting last 20 meetings, aggregate the list of related documents and then perform a query like db.users.find({_id: { $in: <id list> }})?
If I go for option 2), how should I keep the data in sync?
Thanks in advance for any advice!
You can keep the DB model you already have and still only do a single query as MongoDB introduced the $lookup aggregation in version 3.2. It is similar to join in RDBMS.
$lookup
Performs a left outer join to an unsharded collection in the same database to filter in documents from the “joined” collection for processing. The $lookup stage does an equality match between a field from the input documents with a field from the documents of the “joined” collection.
So instead of storing a reference to other collections, just store the document ID.