Inconsistent results with Meteor's pub/sub feature - mongodb

I'm experiencing inconsistent results with Meteor's pub/sub feature, and I suspect it's a source of confusion for a lot of developers hitting the threshold of an MVP built in Meteor becoming a production app.
Maybe this is a limitation of MergeBox:
Let's say I have a collection called Events, in which I have document-oriented structures, ie, nested Array, Objects. An Events document might look like so:
// an Events document //
{
_id: 'abc',
name: 'Some Event',
participation: {
'userOneId': {
games: {
'gameOneId': {
score: 100,
bonus: 10
}
},
{
'gameTwoId': : {
score: 100,
bonus: 10
}
}
}
},
'userTwoId': {
games: {
'gameOneId': {
score: 70,
bonus: 15
}
},
contests: {
'contestOneId': [2, 3, 6, 7, 4],
'contestTwoId': [9, 3, 7, 2, 1],
}
}
},
}
}
So at these events, users can optionally participate in games of certain types and contests of certain types.
Now, I want to restrict subscriptions to the Events collection based on the user (show only this user's participation), and, sometimes I'm only interested in changes to one subset of the data (like, show only the user's scores on 'gameOneId').
So I've created a publication like so:
Meteor.publish("events.participant", function(eventId, userId) {
if(!Meteor.users.findOne({_id: this.userId})) return this.ready();
check(eventId, String);
check(userId, String);
const includeFields = {
name: 1,
[`participation.${userId}`]: 1
};
return Events.find({_id: eventId}, {fields: includeFields});
});
This publication seems to work fine on the client if I do:
// in onCreated of events template //
template.autorun(function() {
const
eventId = FlowRouter.getParam('id'),
userId = Meteor.userId(),
subscription = template.subscribe('events.participant', eventId, userId);
if (subscription.ready()) {
const event = Events.findOne({_id: eventId}, parseIncludeFields(['name', `participation.${userId}`]));
template.props.event.set(event);
}
});
Happily, I can use the Event document returned that includes only the name field and all of the user's participation data.
But, later, in another template if I do:
// in onCreated of games template //
template.autorun(function() {
const
eventId = FlowRouter.getParam('id')
gameId = FlowRouter.getParam('gameId'),
userId = Meteor.userId(),
subscription = template.subscribe('events.participant', eventId, userId);
if(subscription.ready()) {
const event = Events.findOne({_id: eventId}, {fields: {[`participation.${userId}.games.${gameId}`]: 1}});
template.props.event.set(event);
}
});
I sometimes get back the data at event.participation[userId].games[gameId], and sometimes I don't - the Object that's suppose to be at gameId is non-existent, even the it exists in the Mongo document, and the subscription should include it. Why?
The only difference is between the two calls to Events.findOne() is that in the latter, I'm not requesting the name field. But, if this is a problem, why?. If minimongo already has the document, who cares if I request parts of it?
The subscriptions in both templates are identical - I'm doing this because the games template is available at a route, so the user could go straight to the games url, by-passing the events template altogether, so I want to be sure the client has the document it needs to render correctly.
The only way I've gotten around this is to make a straight Meteor method call to the server in the games template to fetch the subset of interest, but this seems like a cop-out.
If you've read this far, you're a champ!

Related

Where to put version __v in redux state?

I have a model that is scattered all around the application. I have a redux state tree:
{
page: {
modelPart1: ...,
... : {
modelPart2: ...
}
}
I need to keep a reference to mongoDb __v in my state too. Where is the best place to place it?
I was thinking about a separate branch model_metadata that would keep the metadata about docs (_id, __v, ...).
{
model_metadata: { <------------------------ HERE
model: {
_id: id,
__v: 2
}
}
page: {
modelPart1: ...,
... : {
modelPart2: ...
}
}
Is it a valid approach or would you recommend a different one?
Every reducer only can access its own part of state, so when you do
combineReducers({
one,
another
});
and access state in one, it is equivalent to doing store.getState().one, and the same for another. So, you need to split the data in page property of state into two parts: actual data and metadata. Just like the object you retrieve from Mongo.
The point in having metadata and actual data being processed by the same reducer is that every time a reducer function is performed, you have everything you need about your object in state argument of that function. Splitting the data into two different reducers would make things way more complicated.
So, the new data representation in page would look like
{
model_metadata: { <------------------------ HERE
model: {
_id: id,
__v: 2
}
}
page: {
modelPart1: ...,
... : {
modelPart2: ...
}
}
while connecting to page would look like
connect(state => ({
page: state.page
})(...)

Meteorjs - What is the proper way to join collections on backend

I am very new to meteor.js and try to build an application with it. This time I wanted to try it over MEAN stack but at this point I am struggled to understand how to join two collection on server side...
I want very identical behaviour like mongodb populate to fetch some properties of inner document.
Let me tell you about my collection it is something like this
{
name: 'Name',
lastName: 'LastName',
anotherObject: '_id of another object'
}
and another object has some fields
{
neededField1: 'asd',
neededField2: 'zxc',
notNeededField: 'qwe'
}
So whenever I made a REST call to retrieve the first object I want it contains only neededFields of inner object so I need join them at backend but I cannot find a proper way to do it.
So far while searching it I saw some packages here is the list
Meteor Collections Helper
Publish with Relations
Reactive joins in Meteor (article)
Joins in Meteor.js (article)
Meteor Publish Composite
You will find the reywood:publish-composite useful for "joining" related collections even though SQL-like joins are not really practical in Mongo and Meteor. What you'll end up with is the appropriate documents and fields from each collection.
Using myCollection and otherCollection as pseudonyms for your two collections:
Meteor.publishComposite('pseudoJoin', {
find: function() {
return myCollection.find();
},
children: [
{
find: function(doc) {
return otherCollection.find(
{ _id: post.anotherObject },
{ fields: { neededField1: 1, neededField2: 1 } });
}
}
]
});
Note that the _id field of the otherCollection will be included automatically even though it isn't in the list of fields.
Update based on comments
Since you're only looking to return data to a REST call you don't have to worry about cursors or reactivity.
var myArray = myCollection.find().fetch();
var myOtherObject = {};
var joinedArray = myArray.map(function(el){
myOtherObject = otherCollection.findOne({ _id: el.anotherObject });
return {
_id: el._id,
name: el.name,
lastName: el.lastName,
neededField1: myOtherObject.neededField1,
neededField2: myOtherObject.neededField2
}
});
console.log(joinedArray); // These should be the droids you're looking for
This is based on a 1:1 relation. If there are many related objects then you have to repeat the parent object to the number of children.

Publish cursor with simplified array data

I need to publish a simplified version of posts to users. Each post includes a 'likes' array which includes all the users who liked/disliked that post, e.g:
[
{
_id: user_who_liked,
liked: 1 // or -1 for disliked
},
..
]
I'm trying to send a simplified version to the user who subscribes an array which just includes his/her like(s):
Meteor.publish('posts', function (cat) {
var _self = this;
return Songs.find({ category: cat, postedAt: { $gte: Date.now() - 3600000 } }).forEach(function (post, index) {
if (_self.userId === post.likes[index]._id) {
// INCLUDE
} else
// REMOVE
return post;
})
});
I know I could change the structure, including the 'likes' data within each user, but the posts are usually designed to be short-lived, to it's better to keep that data within each post.
You need to use this particular syntax to find posts having a likes field containing an array that contains at least one embedded document that contains the field by with the value this.userId.
Meteor.publish("posts", function (cat) {
return Songs.find({
category: cat,
postedAt: { $gte: Date.now() - 3600000 },
"likes._id":this.userId
},{
fields:{
likes:0
}
});
});
http://docs.mongodb.org/manual/tutorial/query-documents/#match-an-array-element
EDIT : answer was previously using $elemMatch which is unnecessary because we only need to filter on one field.

Meteor Publish Distinct Values of Field in Collection

I'm stuck on a pretty simple scenario in Meteor:
I have a huge collection of things with many fields, some of them containing quite a bit of text.
I want to create a page for searching that collection.
One of the fields that each item in the collection has is "category".
I'd like to give the user the ability to filter by that category.
For that, I need to publish just the distinct values of the category field in the collection.
I can't figure out a way to do that without publishing the whole collection which takes way too long. How can I publish just the distinct categories and use them to fill a dropdown?
Bonus question and somewhat related: How do I publish a count of all items in the collection without publishing the whole collection?
A good starting point to make this easier would be to normalize your categories into a separate database collection.
However assuming that is not possible or practical, the best (though imperfect) solution will be to publish two separate versions of your collection, one which returns only the categories field of the entire collection and another which returns all fields of the collection for the selected category only. That would look like the following:
// SERVER
Meteor.startup(function(){
Meteor.publish('allThings', function() {
// return only id and categories field for all your things
return Things.find({}, {fields: {categories: 1}});
});
Meteor.publish('thingsByCategory', function(category) {
// return all fields for things having the selected category
// you can then subscribe via something like a client-side Session variable
// e.g., Meteor.subscribe("thingsByCategory", Session.get("category"));
return Things.find({category: category});
});
});
Note that you will still need to assemble your array of categories client side from the Things cursor (for example, by using underscore's _.pluck and _.uniq methods to grab the categories and remove any dups). But the data set will be much smaller as you are only working with single-field documents now.
(Note that ideally, you would want to use Mongo's distinct() method in your publish function to publish only the distinct categories, but that is not possible directly as it returns an array which cannot be published).
You could use the internal this._documents.collectionName to only send new categories down to the client. Tracking which categories to remove becomes a bit ugly so you probably will still end up maintaining a separate 'categories' collection eventually.
Example:
Meteor.publish( 'categories', function(){
var self = this;
largeCollection.find({},{fields: {category: 1}).observeChanges({
added: function( id, doc ){
if( ! self._documents.categories[ doc.category ] )
self.added( 'categories', doc.category, {category: doc.category});
},
removed: function(){
_.keys( self._documents.categories ).forEach( category ){
if ( largeCollection.find({category: category},{limit: 1}).count() === 0 )
self.removed( 'categories', category );
}
}
});
self.ready();
};
Re: the bonus question, publishing counts: take a look at the meteorite package publish-counts. I think that does what you want.
These patterns might be helpful to you. Here is a publication that publishes counts:
/*****************************************************************************/
/* Counts Publish Function
/*****************************************************************************/
// server: publish the current size of a collection
Meteor.publish("countsByProject", function (arguments) {
var self = this;
if (this.userId) {
var roles = Meteor.users.findOne({_id : this.userId}).roles;
if ( _.contains(roles, arguments.projectId) ) {
//check(arguments.video_id, Integer);
// observeChanges only returns after the initial `added` callbacks
// have run. Until then, we don't want to send a lot of
// `self.changed()` messages - hence tracking the
// `initializing` state.
Videos.find({'projectId': arguments.projectId}).forEach(function (video) {
var count = 0;
var initializing = true;
var video_id = video.video_id;
var handle = Observations.find({video_id: video_id}).observeChanges({
added: function (id) {
//console.log(video._id);
count++;
if (!initializing)
self.changed("counts", video_id, {'video_id': video_id, 'observations': count});
},
removed: function (id) {
count--;
self.changed("counts", video_id, {'video_id': video_id, 'observations': count});
}
// don't care about changed
});
// Instead, we'll send one `self.added()` message right after
// observeChanges has returned, and mark the subscription as
// ready.
initializing = false;
self.added("counts", video_id, {'video_id': video_id, 'observations': count});
self.ready();
// Stop observing the cursor when client unsubs.
// Stopping a subscription automatically takes
// care of sending the client any removed messages.
self.onStop(function () {
handle.stop();
});
}); // Videos forEach
} //if _.contains
} // if userId
return this.ready();
});
And here is one that creates a new collection from a specific field:
/*****************************************************************************/
/* Tags Publish Functions
/*****************************************************************************/
// server: publish the current size of a collection
Meteor.publish("tags", function (arguments) {
var self = this;
if (this.userId) {
var roles = Meteor.users.findOne({_id : this.userId}).roles;
if ( _.contains(roles, arguments.projectId) ) {
var observations, tags, initializing, projectId;
initializing = true;
projectId = arguments.projectId;
observations = Observations.find({'projectId' : projectId}, {fields: {tags: 1}}).fetch();
tags = _.pluck(observations, 'tags');
tags = _.flatten(tags);
tags = _.uniq(tags);
var handle = Observations.find({'projectId': projectId}, {fields : {'tags' : 1}}).observeChanges({
added: function (id, fields) {
if (!initializing) {
tags = _.union(tags, fields.tags);
self.changed("tags", projectId, {'projectId': projectId, 'tags': tags});
}
},
removed: function (id) {
self.changed("tags", projectId, {'projectId': projectId, 'tags': tags});
}
});
initializing = false;
self.added("tags", projectId, {'projectId': projectId, 'tags': tags});
self.ready();
self.onStop(function () {
handle.stop();
});
} //if _.contains
} // if userId
return self.ready();
});
I have not tested it on Meteor, and according to the replies, I'm getting skeptical that it will work but using a mongoDB distinct would do the trick.
http://docs.mongodb.org/manual/reference/method/db.collection.distinct/

Auto increment in MongoDB to store sequence of Unique User ID

I am making a analytics system, the API call would provide a Unique User ID, but it's not in sequence and too sparse.
I need to give each Unique User ID an auto increment id to mark a analytics datapoint in a bitarray/bitset. So the first user encounters would corresponding to the first bit of the bitarray, second user would be the second bit in the bitarray, etc.
So is there a solid and fast way to generate incremental Unique User IDs in MongoDB?
As selected answer says you can use findAndModify to generate sequential IDs.
But I strongly disagree with opinion that you should not do that. It all depends on your business needs. Having 12-byte ID may be very resource consuming and cause significant scalability issues in future.
I have detailed answer here.
You can, but you should not
https://web.archive.org/web/20151009224806/http://docs.mongodb.org/manual/tutorial/create-an-auto-incrementing-field/
Each object in mongo already has an id, and they are sortable in insertion order. What is wrong with getting collection of user objects, iterating over it and use this as incremented ID? Er go for kind of map-reduce job entirely
I know this is an old question, but I shall post my answer for posterity...
It depends on the system that you are building and the particular business rules in place.
I am building a moderate to large scale CRM in MongoDb, C# (Backend API), and Angular (Frontend web app) and found ObjectId utterly terrible for use in Angular Routing for selecting particular entities. Same with API Controller routing.
The suggestion above worked perfectly for my project.
db.contacts.insert({
"id":db.contacts.find().Count()+1,
"name":"John Doe",
"emails":[
"john#doe.com",
"john.doe#business.com"
],
"phone":"555111322",
"status":"Active"
});
The reason it is perfect for my case, but not all cases is that as the above comment states, if you delete 3 records from the collection, you will get collisions.
My business rules state that due to our in house SLA's, we are not allowed to delete correspondence data or clients records for longer than the potential lifespan of the application I'm writing, and therefor, I simply mark records with an enum "Status" which is either "Active" or "Deleted". You can delete something from the UI, and it will say "Contact has been deleted" but all the application has done is change the status of the contact to "Deleted" and when the app calls the respository for a list of contacts, I filter out deleted records before pushing the data to the client app.
Therefore, db.collection.find().count() + 1 is a perfect solution for me...
It won't work for everyone, but if you will not be deleting data, it works fine.
Edit
latest versions of pymongo:
db.contacts.count() + 1
First Record should be add
"_id" = 1 in your db
$database = "demo";
$collections ="democollaction";
echo getnextid($database,$collections);
function getnextid($database,$collections){
$m = new MongoClient();
$db = $m->selectDB($database);
$cursor = $collection->find()->sort(array("_id" => -1))->limit(1);
$array = iterator_to_array($cursor);
foreach($array as $value){
return $value["_id"] + 1;
}
}
I had a similar issue, namely I was interested in generating unique numbers, which can be used as identifiers, but doesn't have to. I came up with the following solution. First to initialize the collection:
fun create(mongo: MongoTemplate) {
mongo.db.getCollection("sequence")
.insertOne(Document(mapOf("_id" to "globalCounter", "sequenceValue" to 0L)))
}
An then a service that return unique (and ascending) numbers:
#Service
class IdCounter(val mongoTemplate: MongoTemplate) {
companion object {
const val collection = "sequence"
}
private val idField = "_id"
private val idValue = "globalCounter"
private val sequence = "sequenceValue"
fun nextValue(): Long {
val filter = Document(mapOf(idField to idValue))
val update = Document("\$inc", Document(mapOf(sequence to 1)))
val updated: Document = mongoTemplate.db.getCollection(collection).findOneAndUpdate(filter, update)!!
return updated[sequence] as Long
}
}
I believe that id doesn't have the weaknesses related to concurrent environment that some of the other solutions may suffer from.
// await collection.insertOne({ autoIncrementId: 1 });
const { value: { autoIncrementId } } = await collection.findOneAndUpdate(
{ autoIncrementId: { $exists: true } },
{
$inc: { autoIncrementId: 1 },
},
);
return collection.insertOne({ id: autoIncrementId, ...data });
I used something like nested queries in MySQL to simulate auto increment, which worked for me. To get the latest id and increment one to it you can use:
lastContact = db.contacts.find().sort({$natural:-1}).limit(1)[0];
db.contacts.insert({
"id":lastContact ?lastContact ["id"] + 1 : 1,
"name":"John Doe",
"emails": ["john#doe.com", "john.doe#business.com"],
"phone":"555111322",
"status":"Active"
})
It solves the removal issue of Alex's answer. So no duplicate id will appear if any record is removed.
More explanation: I just get the id of the latest inserted document, add one to it, and then set it as the id of the new record. And ternary is for cases that we don't have any records yet or all of the records are removed.
this could be another approach
const mongoose = require("mongoose");
const contractSchema = mongoose.Schema(
{
account: {
type: mongoose.Schema.Types.ObjectId,
required: true,
},
idContract: {
type: Number,
default: 0,
},
},
{ timestamps: true }
);
contractSchema.pre("save", function (next) {
var docs = this;
mongoose
.model("contract", contractSchema)
.countDocuments({ account: docs.account }, function (error, counter) {
if (error) return next(error);
docs.idContract = counter + 1;
next();
});
});
module.exports = mongoose.model("contract", contractSchema);
// First check the table length
const data = await table.find()
if(data.length === 0){
const id = 1
// then post your query along with your id
}
else{
// find last item and then its id
const length = data.length
const lastItem = data[length-1]
const lastItemId = lastItem.id // or { id } = lastItem
const id = lastItemId + 1
// now apply new id to your new item
// even if you delete any item from middle also this work
}