Writing a subquery for a relation's relation in Objection.js - objection.js

I'm using version 2.2.15 at the moment and I'm following the relation subqueries
recipe. I've outlined my models and their relationships towards the bottom.
Currently, I'm fetching an event as well as it's post count using this query:
const events = await Event.query().select([
'events.*',
Event.relatedQuery('posts')
.count()
.as('numberOfPosts'),
]);
Which works fine - but how can I include the count of each of the event's post's users, that is, the total amount of user's going to an event. I've tried using a refs so far without luck.
Models and their relationships:
Event:
posts: {
relation: Model.HasManyRelation,
modelClass: Post,
join: {
from: 'events.id',
to: 'posts.eventId',
},
},
Post:
event: {
relation: Model.BelongsToOneRelation,
modelClass: Event,
join: {
from: 'posts.eventId',
to: 'events.id',
},
},
users: {
relation: Model.ManyToManyRelation,
modelClass: User,
join: {
from: 'posts.id',
through: {
// users_posts is the join table.
from: 'users_posts.postId',
to: 'users_posts.userId',
},
to: 'users.id',
},
},
User:
posts: {
relation: Model.ManyToManyRelation,
modelClass: Post,
join: {
from: 'users.id',
through: {
// users_posts is the join table.
from: 'users_posts.userId',
to: 'users_posts.postId',
},
to: 'posts.id',
},
},

You should be able to do it like so, i haven't tried it in code so you might need to tweek a few things, what i'd do usually if i have a count that needs to be tracked is keep a field on the posts that increments users in the application logic and value is retrieved much faster, in your case you can just count the users_posts where the postId match and get your users count and save it in that field
const events = await Event.query().select([
'events.*',
Event.relatedQuery('posts')
.count()
.as('numberOfPosts'),
Event.relatedQuery('posts')
.join('users_posts', 'posts.id', '=', 'users_posts.postId')
.join('users', 'users.id', '=', 'users_posts.userId')
.countDistinct('users.id')
.as('numberOfPostUsers'),
]);

Related

Mongoose scheme design many-to-many

I'm trying to come up with a mongoose scheme architecture/design for a task management app.
Desired functionality and models:
Models:
User - has boards
Boards - has lists
Lists - has cards
Cards
Functionality:
User can create a board
User can create a list in the board
User can create a card in the list
User can add members to the board
User can add members to the card from the board members
User can delete board, deleting all the lists and cards, and removing members
User can delete list, deleting all the cards in the list
User can delete card, removing all the members from the card
User can update the list position (sort) in the board
User can update cards position (sort) in the list
Additional functionality (Optional)
User can add comments to the card
Card activity logging (On move, on edit, on comment)
I know its alot to ask, but how would go about designing a scheme for this kind of functionality?
Would child-parent references be the best solution?
Ex.
User:
{
name: 'Bob',
_id: '2626'
}
Board:
{
name: 'Board name',
_id: '123456',
members: [
{ type: ObjectID, ref: 'user' } // '2626'
]
}
List:
{
name: 'List name',
_id: '2525',
boardId: { type: ObjectID, ref: 'board' } // '123456'
}
Card:
{
name: 'Card name',
boardId: { type: ObjectID, ref: 'board' } // '123456',
listId: { type: ObjectID, ref: 'list' } // '2525'
}
How would I go about querying this type of structure?
1) Get a list of boards by users id
2) Get a list of lists by board id
3) Get a list of cards by board id & listid
So for the board view I would go and grab all the lists, but then I would have to go and get all the cards for the each list, seems not really efficient.
Perhaps when entering board view I should query just the cards by the board id, but then how do I get the lists, and put each card into its own list?
How do I handle deleting the card or moving a card from one list to another?
Don't be hard on me please, I'm really new to the mongodb world, but I'm really trying my best.
The schema you've defined is pretty good, here's how the flow would be.
Initially, when a user signs in, you'll need to show them the list of boards. This should be easy since you'll just do a find query with the user_id on the board collection.
Board.find({members: user_id}) // where user_id is the ID of the user
Now when a user clicks on a particular board, you can get the lists with the board_id, similar to the above query.
List.find({boardId: board_id}) // where board_id is the ID of the board
Similarly, you can get cards with the help of list_id and board_id.
Card.find({boardId: board_id, listId: list_id}) // where board_id is the ID of the board and listId is the Id of the list
Now, let's look at cases wherein you might need data from 2 or more collection at the same time.
For example, when a user clicks on board, you not only need the lists in the board but also the cards in that board.
In that case, you'll need to write an aggregation as such,
Board.aggregate([
// get boards which match a particular user_id
{
$match: {members: user_id}
},
// get lists that match the board_id
{
$lookup:
{
from: 'list',
localField: '_id',
foreignField: 'boardId',
as: 'lists'
}
}
])
This will return the boards, and in each board, there'll be an array of lists associated with that board. If a particular board doesn't have a list, then it'll have an empty array.
Similarly, if you want to add cards to the list and board, the aggregation query will be a bot more complex, as such,
Board.aggregate([
// get boards which match a particular user_id
{
$match: {members: user_id}
},
// get lists that match the board_id
{
$lookup:
{
from: 'list',
localField: '_id',
foreignField: 'boardId',
as: 'lists'
}
},
// get cards that match the board_id
{
$lookup:
{
from: 'card',
localField: '_id',
foreignField: 'boardId',
as: 'cards'
}
}
])
This will add an array of cards as well to the mix. Similarly, you can get cards of the lists as well.
Now, let's think about whether this is the best schema or not. I personally think the schema you suggested is pretty good because another way to go about it would be to store IDs in the parent collection, which will let you use populate to get the data instead of a lookup query.
For example, storing list ids in board collection. The downside to this is, whenever a new list is added, you need to add that list in the list collection and also update the board to which the list is connected to (add the list ID), which I think is too tedious.
Finally, some suggestion on the schema you've given,
I think you should add user_id (creator's ID) in every collection, cause there are many cases wherein you need to show the name of the user who created that particular board or list or anything else and also since you have the feature of adding users to a particular card, etc I think you should have two fields, one is creator_id and the other should be associated_users, which will be an array (obviously you can choose better names).
You should add position field in cards and other collections which you want to sort by position. This field should be a number.
Deleting a card or moving it from one list to another should be pretty easy and self-explanatory by now.
Edit 1: Based on the comment
You don't need to assign cards to the list 'after' the aggregation, you can do this in your aggregation itself, so it'll be something like this,
Board.aggregate([
// get boards which match a particular user_id
{
$match: { members: user_id }
},
// get lists that match the board_id
{
$lookup:
{
from: 'list',
localField: '_id',
foreignField: 'boardId',
as: 'lists'
}
},
// unwind lists (because it's an array at the moment)
{
$unwind: '$lists'
},
// Now you have object named lists in every board object
// get cards that match the list_id (note that the cards contain list_id)
{
$lookup:
{
from: 'card',
localField: '_id',
foreignField: 'listId',
as: 'cards'
}
},
// now merge back the objects and get a simple object for each boardID
{
$group: {
_id: "$_id",
members: { $addToSet: "$members" },
lists: { $addToSet: "$lists" }
}
}
])
This will give you something like this,
data = {
'_id': '123456',
'members': [
{
name: 'Bob',
_id: '2626'
},
{
name: 'Matthew',
_id: '2627'
}
],
'lists': [
{
name: 'List 1',
_id: '2525',
boardId: '123456',
cards: [
{
name: 'Card 1',
boardId: '123456',
listId: '2525'
},
{
name: 'Card 2',
boardId: '123456',
listId: '2525'
}
]
},
{
name: 'List 2',
_id: '2526',
boardId: '123456',
cards: [
{
name: 'Card 3',
boardId: '123456',
listId: '2525'
},
{
name: 'Card 4',
boardId: '123456',
listId: '2525'
}
]
}
]
}
So basically, you can get the list and the cards for those list in a single query itself and it is quite efficient.
Now coming to the two queries you asked for,
Card moved from one list to another, just edit the listId field in the card document to the new listID (it's quite simple actually).
Card moved up a position in a list
As I said, if you want the position you need to add a field called position in the documents and then whenever the card is moved you need to change the value of 'position' of those cards.
In the aggregation, you just need to add another stage called '$sort' and sort it according to the position value.
This is going to be a bit tedious since whenever you move a card up, you need to update the position of the card above as well.

Trying to design Facebook-like notification in mongoose. Need help aggregating

Like Facebook, I would like to aggregate the results. But I can't figure out how to go about it.
Example:
Let's say 10 users like my posts.
I don't want to get 10 notifications. 1 is of course enough.
This is my schema:
var eventLogSchema = mongoose.Schema({
//i.e. Somebody commented, sombody liked, etc.
event: String,
//to a comment, to a post, to a profile, etc.
toWhat: String,
//who is the user we need to notify?
toWho: {type: mongoose.Schema.Types.ObjectId, ref:'User'},
//post id, comment id, whatever..
refID: {type: mongoose.Schema.Types.ObjectId},
//who initiated the event.
whoDid: {type: mongoose.Schema.Types.ObjectId, ref: 'User'},
// when the event happened
date: {type:Date, default: Date.now()},
//whether the user already saw this notification or not.
seen: {type:Boolean, default: false}
})
so I need to count the times
Ex.1: event='liked' and toWhat="post" and refID=myPostID and seen=false
But at the same time, I would like to populate the last event with this parameters on the 'who' path so I could display "Michael and 9 other people liked your post(link to post)"
Every way I can think of doing this is clunky and requires multiple queries that feel like they would cost a lot of system resources and I am wondering if there's a simple way to do it.
Actually it gets more complicated then that.
I do not want to specify values like I did in Ex.1.
Instead I would like to say
aggregate all events with similar 'event', 'toWhat',
'refID' with value seen=false and populate the last one on the 'who' path.
Would love some reading materials, links, advice, or anything.
Thanks!
Managed to solve it like this.
Not sure if it's optimal, but it works.
//The name of my Schema
Notification.aggregate([
{
$match: {
//I only want to display new notifications
seen: {$ne: true}
//to query a specific user add
// toWho: UserID
}
},
{
$group: {
//groups when event, toWhat, and refID are similar
_id: {
event: '$event',
toWhat: '$toWhat',
refID: '$refID',
},
//gets the total number of this type of notification
howMany: {$sum: 1},
//gets the date of the last document in this query
date: {$max: '$date'},
//pulls the user ID of the last user in this query
user: {$last: '$whoDid'}
}
}
]).exec(function (err, results) {
if (err) throw err;
if (results) {
//after I get the results, I want to populate my user to get his name.
Notification.populate(results, {path: 'user', model: "User"}, function (err, notifications) {
if (err) throw err;
if (notifications) res.send(notifications);
})
}
})
I'm not sure whether it's possible to populate the aggregated result in one query, I assume that if it's possible it would be optimal, but so far, this seems acceptable for my needs.
Hope this helps.

Meteor: Speeding up MongoDB Join's for Big Data?

I have two collections: Data and Users. In the Data collection, there is an array of user IDs consisting of approximately 300 to 800 users.
I need to join together the countries of all of the users for each row in the Data collection, which hangs my web browser due to there being too much data being queried at once.
I query for about 16 rows of the Data collection at once, and also there are 18833 users so far in the Users collection.
So far I have tried to make both a Meteor method and a transform() JOIN for the Meteor collection which is what's hanging my app.
Mongo Collection:
UserInfo = new Mongo.Collection("userInfo")
GlyphInfo = new Mongo.Collection("GlyphAllinOne", {
transform: function(doc) {
doc.peopleInfo = doc.peopleInfo.forEach(function(person) {
person.code3 = UserInfo.findOne({userId: person.name}).code3;
return person;
})
return doc;
}
});
'code3' designates user's country.
Publication:
Meteor.publish("glyphInfo", function (courseId) {
this.unblock();
var query = {};
if (courseId) query.courseId = courseId;
return [GlyphInfo.find(query), UserInfo.find({})];
})
Tested Server Method:
Meteor.methods({
'glyph.countryDistribution': function(courseId) {
var query = {};
if (courseId) query.courseId = courseId;
var glyphs = _.map(_.pluck(GlyphInfo.find(query).fetch(), 'peopleInfo'), function(glyph) {
_.map(glyph, function(user) {
var data = Users.findOne({userId: user.name});
if (data) {
user.country = data ? data.code3 : null;
console.log(user.country)
return user;
}
});
return glyph;
});
return glyphs;
}
});
Collection Data:
There is an option of preprocessing my collection so that countries would already be included, however I'm not allowed to modify these collections. I presume that having this JOIN be done on startup of the server and thereafter exposed through an array as a Meteor method may stall startup time of the server for far too long; though I'm not sure.
Does anyone have any ideas on how to speed up this query?
EDIT: Tried out MongoDB aggregation commands as well and it appears to be extremely slow on Meteor's minimongo. Took 4 minutes to query in comparison to 1 second on a native MongoDB client.
var codes = GlyphInfo.aggregate([
{$unwind: "$peopleInfo"},
{$lookup: {
from: "users",
localField: "peopleInfo.name",
foreignField: "userId",
as: "details"
}
},
{$unwind: "$details"},
{$project: {"peopleInfo.Count": 1, "details.code3": 1}}
])
I would approach this problem slightly differently, using reywood:publish-composite
On the server I would publish the glyphinfo using publish-composite and then include the related user and their country field in the publication
I would join up the country on the client wherever I needed to display the country name along with the glyphinfo object
Publication:
Meteor.publishComposite('glyphInfo', function(courseId) {
this.unblock();
return {
find: function() {
var query = {};
if (courseId) query.courseId = courseId;
return GlyphInfo.find(query);
},
children: [
{
find: function(glyph) {
var nameArray = [];
glyph.person.forEach(function(person){
nameArray.push(person.name);
};
return UserInfo.find({ userId: {$in: nameArray }});
}
]
}
});
Solved the problem by creating a huge MongoDB aggregation call, with the biggest factor in solving latency being indexing unique columns in your database.
After carefully implementing indices into my database with over 4.6 million entries, it took 0.3 seconds on Robomongo and 1.4 seconds w/ sending data to the client on Meteor.
Here is the aggregation code for those who'd like to see it:
Meteor.methods({
'course.countryDistribution': function (courseId, videoId) {
var query = {};
if (courseId) query.courseId = courseId;
var data = GlyphInfo.aggregate([
{$unwind: "$peopleInfo"},
{$lookup: {
from: "users",
localField: "peopleInfo.name",
foreignField: "userId",
as: "details"
}
},
{$unwind: "$details"},
{$project: {"peopleInfo.Count": 1, "details.code3": 1}},
{$group: {_id: "$details.code3", count: {$sum: "$peopleInfo.Count"}}}
])
return data;
}
});
If anyone else is tackling similar issues, feel free to contact me. Thanks everyone for your support!

Publish cursor with simplified array data

I need to publish a simplified version of posts to users. Each post includes a 'likes' array which includes all the users who liked/disliked that post, e.g:
[
{
_id: user_who_liked,
liked: 1 // or -1 for disliked
},
..
]
I'm trying to send a simplified version to the user who subscribes an array which just includes his/her like(s):
Meteor.publish('posts', function (cat) {
var _self = this;
return Songs.find({ category: cat, postedAt: { $gte: Date.now() - 3600000 } }).forEach(function (post, index) {
if (_self.userId === post.likes[index]._id) {
// INCLUDE
} else
// REMOVE
return post;
})
});
I know I could change the structure, including the 'likes' data within each user, but the posts are usually designed to be short-lived, to it's better to keep that data within each post.
You need to use this particular syntax to find posts having a likes field containing an array that contains at least one embedded document that contains the field by with the value this.userId.
Meteor.publish("posts", function (cat) {
return Songs.find({
category: cat,
postedAt: { $gte: Date.now() - 3600000 },
"likes._id":this.userId
},{
fields:{
likes:0
}
});
});
http://docs.mongodb.org/manual/tutorial/query-documents/#match-an-array-element
EDIT : answer was previously using $elemMatch which is unnecessary because we only need to filter on one field.

StrongLoop loopback - REST example using filters on related models?

I found this example to use the Node API to apply filters to related models, but I was wondering if it was possible to achieve the same result using REST?
Node Example:
Post.find({
include: {
relation: 'owner', // include the owner object
scope: { // further filter the owner object
fields: ['username', 'email'], // only show two fields
include: { // include orders for the owner
relation: 'orders',
scope: {
where: {orderId: 5} // only select order with id 5
}
}
}
}
}, function() { ... });
The closest version of a REST url I can get to work is:
...?filter[include][owners][orders]
Is it possible to create a REST url that behaves the same way as the above Node example, by limiting the results based on a related model filter... in this case orders?
I have this functions so when I call the Hdates/coming REST API it shows the events with date greater than today and also includes the venues... Hope it helps.
Hdate.coming = function(cb) {
Hdate.find({
where : {
event_date :{gt: Date.now()}
},
include : {
relation: 'event',
scope : {
include: {
relation: 'venue'
}
}
}
}, cb);
};
Hdate.setup = function() {
Hdate.base.setup.apply(this, arguments);
this.remoteMethod('coming', {
description: 'Find Coming Events by Date',
returns: {arg: 'events', root: true},
http: { verb: 'GET' }
});
};
Hdate.setup();