Why are so many documents getting fetched in Meteor? - mongodb

My application is fetching a huge number of documents from the server in Meteor. I've tried optimizing the Live Query to reduce this, but I'm still getting several thousand documents per hour when the collection.find().count() is just 13.
Subscription:
Template.calendar.onCreated( function() {
var self = this;
self.ready = new ReactiveVar();
PostSubs = new SubsManager();
self.autorun(function() {
var handle = PostSubs.subscribe('meals');
self.ready.set(handle.ready());
});
});
Publication:
Meteor.publish('meals', function() {
return Meals.find({
userId : this.userId,
start : {
$gte : moment().subtract(2, 'w').format("YYYY-MM-DD"),
$lte : moment().add(2,'w').format("YYYY-MM-DD")
}
});
return this.ready();
});
Usage in context:
Template.calendar.onRendered(() => {
$( '#meals-calendar' ).fullCalendar({
defaultView: 'basicWeek',
firstDay: 1,
height: 200,
editable: true,
events( start, end, timezone, callback ) {
let data = Meals.find().fetch().map( ( event ) => {
return event;
});
if ( data ) {
callback( data );
}
},
eventRender( event, element ) {
element.find( '.fc-content' ).html(
`<strong>${ event.title }</strong>`
);
},
dayClick( date ) {
Session.set( 'eventModal', { type: 'add', date: date.format() } );
$( '#add-edit-event-modal' ).modal( 'show' );
$('#title').focus();
},
eventClick( event ) {
Session.set( 'eventModal', { type: 'edit', event: event._id } );
$( '#add-edit-event-modal' ).modal( 'show' );
},
eventDrop( event, delta, revert ) {
let date = event.start.format();
let update = {
_id: event._id,
start: date
};
Meteor.call( 'editEvent', update, ( error ) => {
if ( error ) {
Bert.alert( error.reason, 'danger' );
}
});
},
});
Tracker.autorun( () => {
Meals.find().fetch();
$( '#meals-calendar' ).fullCalendar( 'refetchEvents' );
});
});

Related

React-Query: useInfiniteQuery

So, I have looked through the docs and answers on here and I'm still needing some help:
index.tsx
const getInfiniteArticles = ({ pageParams = 0 }) => {
const res = await axios.get('/api/articles', { params: { page: pageParams } });
return res.data;
}
api/articles.ts
const getArticles = async (req: NextApiRequest, res: NextApiResponse) => {
try {
const { page } = req.query;
const pageNum = Number(page);
const data = await NewsService.getArticles(getRange(pageNum));
return res.status(200).json({
data,
previousPage: pageNum > 0 ? (pageNum - 1) : null,
nextPage: pageNum + 1,
});
} catch (err) {
res.json(err);
res.status(405).end();
}
};
export default getArticles;
index.tsx
const { data: articlePages, fetchNextPage } = useInfiniteQuery(
'infinite-articles',
getInfiniteArticles,
{
getNextPageParam: (lastPage, allGroups) => {
console.log('lastPage: ', lastPage);
console.log('allGroups: ', allGroups);
return lastPage.nextPage;
}
});
const handleLoadMore = () => {
fetchNextPage();
};
console after clicking next page:
lastPage: { data: Array(50), previousPage: null, nextPage: 1}
allGroups: [
{ data: Array(50), previousPage: null, nextPage: 1},
{ data: Array(50), previousPage: null, nextPage: 1},
]
Any help on why I'm getting the same groups is appreciated! :)
So, it turns out my structure wasn't correct
const {
fetchNextPage,
fetchPreviousPage,
hasNextPage,
hasPreviousPage,
isFetchingNextPage,
isFetchingPreviousPage,
...result
} = useInfiniteQuery(queryKey, ({ pageParam = 1 }) => fetchPage(pageParam), {
...options,
getNextPageParam: (lastPage, allPages) => lastPage.nextCursor,
getPreviousPageParam: (firstPage, allPages) => firstPage.prevCursor,
})
queryFn: (context: QueryFunctionContext) => Promise<TData>
The queryFn is supposed to be a synchronous function that returns a Promise
I was either passing an async function or I was returning the TData not a promise.
updated and working:
const getInfiniteArticles = ({ pageParam = 0 }) => axios.get('/api/articles', { params: { page: pageParam } });
const { data: articlePages, fetchNextPage } = useInfiniteQuery('articles', getInfiniteArticles, {
getNextPageParam: (lastPage, pages) => {
// the returned axios response
return lastPage.data.nextPage;
}
});
Reference Page

I am learning express and mongoose and I cannot get a schema to save, but can see it with console.log()

EDIT: I think weekCount is saving before the records are loaded into it. Perhaps incorrect use of the find()?
I have started a website to record animals in camps (paddocks). I am using express, mongo, mongoose, and pug.
I have a schema (see below) that has an array of type: Schema.Types.ObjectId that I can not save to mongo. The array is empty.
I have a form that fetches the camp names, and the animal types (cows, bulls, calves) and creates a form.
The form is displayed, and the POST can read the form data. The form data is read into schema and is displayed ok with console.log, but does not save the entire schema in mongo.
I have read a lot of posts, and tried many things such as markModified, used a full Schema.
Its been hours now, and I would appreciate some help.
Express mongoose Models:
var WeekCountSchema = new Schema({
dateOfCount: { type: Date },
campCounts: [{type: Schema.Types.ObjectId, ref: 'CampCount'}] < ----- problem
});
var CampCountSchema = new Schema({
campName: String,
campCountDate: Date,
count: {
ox: Number,
cow: Number,
bull: Number,
heifer: Number,
calf: Number,
weaner: Number
}
});
weekCountController:
Scroll Down to the <----
exports.weekCount_create_post = [
validator
.body("dateOfCount", "Week Count Data Required")
.toDate()
.isLength({ min: 1 })
.trim(),
validator.sanitizeBody("*").escape(),
(req, res, next) => {
var weekCount = new WeekCount({
dateOfCount: req.body.dateOfCount
});
const errors = validator.validationResult(req);
if (!errors.isEmpty()) {
// There are errors. Render the form again with sanitized values/error messages.
console.log("ERRORS!s");
async.parallel(
{
camps: function(callback) {
Camp.find()
.sort("name")
.exec(callback);
},
cattleClasses: function(callback) {
CattleClass.find(callback);
}
},
function(err, results) {
if (err) {
return next(err);
}
res.render("weekCount_form", {
title: "There were Errors! New Week Count",
camps: results.camps,
cattleClasses: results.cattleClasses
});
}
);
return;
} else {
// Data from form is valid.
Camp.find({}, "name").exec(function(err, list_camps) {
if (err) {
return next(err);
}
CattleClass.find({}, "name").exec(function(err, list_cattleClasses) {
if (err) {
return next(err);
}
var campCountArray = [];
list_camps.forEach(function(campName) {
var campCount = new CampCount({
campName: campName.name
});
var emptyCount = true;
list_cattleClasses.forEach(function(cattleClassName) {
var tempVar = campName.name + "." + cattleClassName.name;
var tempNum = parseInt(req.body[tempVar]);
// console.log(tempNum);
if (tempNum) {
// console.log(req.body[tempVar]);
campCount.count[cattleClassName.name] = req.body[tempVar];
emptyCount = false;
} else {
campCount.count[cattleClassName.name] = 0;
}
});
if (!emptyCount) {
campCount.save(function(err) {
if (err) {
return next(err);
}
});
campCountArray.push(campCount);
}
});
console.log("CampCountArray");
console.log(campCountArray);
weekCount.campCounts = campCountArray;
});
});
// ****************************************************************
// Check if Week Count with same date already exists.
WeekCount.findOne({ dateOfCount: req.body.dateOfCount }).exec(function(
err,
found_weekCount
) {
if (err) {
console.log("ERROR findone " + err);
return next(err);
}
if (found_weekCount) {
// Week count exists, redirect to its detail page.
console.log("FOUND");
res.redirect(found_weekCount.url);
} else {
console.log("NOT FOUND");
// weekCount.markModified('campCounts');
weekCount.save(function(err) { // <---- does not save
if (err) {
console.log("ERROR SAVING: " + err);
return next(err);
}
console.log("weekCount saved");
console.log(weekCount);
// output below
// Week Count saved. Redirect to week count detail page.
// console.log(weekCount.campCounts);
res.redirect(weekCount.url);
});
}
});
}
}
];
Output from console.log:
GET /catalog/WeekCount/create 200 219.085 ms - 3782
NOT FOUND <------ count not a duplicate (OK)
CampCountArray
[ { count: { calf: 1, bull: 0, cow: 0, weaner: 0, ox: 0, heifer: 0 },
_id: 5d83720e2279011e90a1614b,
campName: 'Bloekom' },
{ count: { calf: 1, bull: 0, cow: 0, weaner: 0, ox: 0, heifer: 0 },
_id: 5d83720e2279011e90a1614c,
campName: 'Davel' },
{ count: { calf: 1, bull: 0, cow: 0, weaner: 0, ox: 0, heifer: 0 },
_id: 5d83720e2279011e90a1614d,
campName: 'Elfas' },
{ count: { calf: 1, bull: 0, cow: 0, weaner: 0, ox: 0, heifer: 0 },
_id: 5d83720e2279011e90a1614e,
campName: 'Groot kamp' } ]
weekCount saved
{ campCounts:
[ { count: [Object],
_id: 5d83720e2279011e90a1614b,
campName: 'Bloekom',
__v: 0 },
{ count: [Object],
_id: 5d83720e2279011e90a1614c,
campName: 'Davel',
__v: 0 },
{ count: [Object],
_id: 5d83720e2279011e90a1614d,
campName: 'Elfas',
__v: 0 },
{ count: [Object],
_id: 5d83720e2279011e90a1614e,
campName: 'Groot kamp',
__v: 0 } ],
_id: 5d83720e2279011e90a1614a,
dateOfCount: 2019-09-06T00:00:00.000Z,
__v: 0 }
from mongo:
{
"_id" : ObjectId("5d83720e2279011e90a1614a"),
"campCounts" : [ ], <---------------------- nothing here!
"dateOfCount" : ISODate("2019-09-06T00:00:00Z"),
"__v" : 0
}
the campCounts ARE in mongo (sample):
{
"_id" : ObjectId("5d83720e2279011e90a1614d"),
"count" : {
"calf" : 1,
"bull" : 0,
"cow" : 0,
"weaner" : 0,
"ox" : 0,
"heifer" : 0
},
"campName" : "Elfas",
"__v" : 0
}
but weekCount does not update. This is the problem.
I think your suspicion is correct that weekCount is saving before the CattleClass.find() callback has finished executing.
You could fix this with .then syntax:
CattleClass.find({}, "name").exec(function(err, list_cattleClasses) {
...
}).then( function () {
WeekCount.findOne({ dateOfCount: req.body.dateOfCount }).exec(function(
err, found_weekCount) {
...
}
})
you could also use ES6 async/await syntax, but your code would take quite a bit of rewriting because you can't use .exec() with await.
Both methods will ensure CattleClass.find() finishes executing before running WeekCount.findOne()
If you need more info there are lots of great posts on Stack Overflow addressing asynchronous code.
The below code works properly now.
Use of .then() instead of mixing callbacks and promises.
Thanks for the help!
exports.weekCount_create_post = [
validator
.body("dateOfCount", "Week Count Data Required")
.toDate()
.isLength({ min: 1 })
.trim(),
// Sanitize (escape) the name field.
validator.sanitizeBody("*").escape(),
// Process request after validation and sanitization.
(req, res, next) => {
var weekCountDetail = {
dateOfCount: req.body.dateOfCount
};
const errors = validator.validationResult(req);
if (!errors.isEmpty()) {
// There are errors. Render the form again with sanitized values/error messages.
console.log("ERRORS!s");
async.parallel(
{
camps: function(callback) {
Camp.find()
.sort("name")
.exec(callback);
},
cattleClasses: function(callback) {
CattleClass.find(callback);
}
},
function(err, results) {
if (err) {
return next(err);
}
res.render("weekCount_form", {
title: "There were Errors! New Week Count",
camps: results.camps,
cattleClasses: results.cattleClasses
});
}
);
return;
} else {
Camp.find({}, "name")
.exec()
.then(list_camps => {
return CattleClass.find({}, "name")
.exec()
.then(list_cattleClasses => {
return [list_camps, list_cattleClasses];
});
})
.then(qResult => {
list_camps = qResult[0];
list_cattleClasses = qResult[1];
var campCountArray = [];
list_camps.forEach(function(campName) {
var campCount = new CampCount({
campName: campName.name
});
var emptyCount = true;
list_cattleClasses.forEach(function(cattleClassName) {
var tempVar = campName.name + "." + cattleClassName.name;
var tempNum = parseInt(req.body[tempVar]);
if (tempNum) {
campCount.count[cattleClassName.name] = req.body[tempVar];
emptyCount = false;
} else {
campCount.count[cattleClassName.name] = 0;
}
});
if (!emptyCount) {
campCount.save(function(err) {
if (err) {
return next(err);
}
});
campCountArray.push(campCount);
}
});
weekCountDetail.campCounts = campCountArray;
return weekCountDetail;
})
.then(weekCountDetail => {
WeekCount.findOne({ dateOfCount: req.body.dateOfCount })
.exec()
.then(found_weekCount => {
if (found_weekCount) {
res.redirect(found_weekCount.url);
} else {
console.log("Not FOUND");
var weekCount = new WeekCount(weekCountDetail);
console.log("WEEKCOUNT3");
console.log(weekCount);
weekCount.save(err => {
if (err) {
return next(err);
}
res.redirect(weekCount.url);
});
}
})
.catch(err => {
console.log("error findOne " + err);
return next(err);
});
});
}
}
];

Can yadcf be used to filter rows on a class

Im looking for an easy way to filter 3 cols which certain rows have a specific class.
I have a datatable for live football games, this shows me inplay stats and if stats get above a certain level I add a class to the TD, for example
if(Number(chars[0])+Number(chars[1])>15)
{
$(row).find('td:eq(8)').css('background-color', 'lime');
$(row).find('td:eq(8)').addClass('lime');
}
On a busy day I can have over 100 live games at a time and I would like to filter just showing the games which have a class of 'lime'. There are 3 cols I shade using this so ideally I want a single filter to cover all 3
Ive tried using the search plugin for datatables but I can only get it to work on the first page of results.
const table = $('#example').DataTable( {
responsive: {
details: {
}
},
"ajax": "console.json",
"search": {
"regex": true
},
"columns": [
{
"class": 'details-control',
"orderable": false,
"data": null,
"defaultContent": ''
},
{ "data": "time" },
{ "data": "firstgoal" },
{ "data": "score" },
{ "data": "league" },
{ "data": "teams", "defaultContent": "0" },
{ "data": function (data, type, dataToSet) {
return data.nextover + "% (" + data.sno + ")";}
},
{ "data": "pi1",
"render": function (data, type, row) {
return '<span class="show-modalp1" data-modal-id="'+ row.gameid + '">' + data + '</span>';
}},
{ "data": "pi2",
"render": function (data, type, row) {
return '<span class="show-modalp2" data-modal-id="'+ row.gameid + '">' + data + '</span>';
}},
{ "render": function (data, type, JsonResultRow, meta) {
return '<img src="'+JsonResultRow.rc+'" class="center">';
}}
],
"order": [[1, 'asc']],
rowCallback: function(row, data, index){
if((data["nextover"] >85 && data["sno"]>14) || (data["nextover"] >70 && data["sno"]>10 && data["time"]>=70) || (data["nextover"] >55 && data["sno"]>5 && data["time"]>=81))
{
$(row).find('td:eq(6)').css('background-color', 'lime');
$(row).find('td:eq(7)').addClass('lime');
}
else
{
$(row).find('td:eq(6)').css('background-color', 'mistyrose');
}
var chars = data["pi2"].split(':');
if(Number(chars[0])+Number(chars[1])>15)
{
$(row).find('td:eq(8)').css('background-color', 'lime');
$(row).find('td:eq(8)').addClass('lime');
}
else
{
$(row).find('td:eq(8)').css('background-color', 'mistyrose');
}
var chars2 = data["pi1"].split(':');
if(Number(chars2[0])+Number(chars2[1])>70)
{
$(row).find('td:eq(7)').css('background-color', 'lime');
$(row).find('td:eq(7)').addClass('lime');
}
else
{
$(row).find('td:eq(7)').css('background-color', 'mistyrose');
}
},
drawCallback: function(settings) {
$(".show-modalp1").each(function() {
var id = $(this).attr('data-modal-id');
$(this).popover({
content: function() {
return '<div style="width: 440px; height=220px"><iframe src="../pressure/showgraph.php?gameid=' + id + '&pi=pi1" width=440 height=220></iframe></div>';
},
trigger: 'hover',
placement: 'left',
html: true,
title: 'Pressure Index 1 = (10*SOT)+(10*SOFT)+1 per 5%poss'
}
)
});
$(".show-modalp2").each(function() {
var id = $(this).attr('data-modal-id');
$(this).popover({
content: function() {
return '<div style="width: 440px; height=220px"><iframe src="../pressure/showgraph.php?gameid=' + id + '&pi=pi2" width=440 height=220></iframe></div>';
},
trigger: 'hover',
placement: 'left',
html: true,
title: 'Pressure Index 2 = 1 *dangerous attack - 1 point if no corners in last 8'
}
)
});
},
} );
$.fn.DataTable.ext.search.push(function(_,__,rowIdx){
return $(table.row(rowIdx).node()).has('td.lime').length || !$('#showLime').hasClass('limeOnly');
});
$('#example').on('draw.dt', function () {
$('[data-toggle="tooltip"]').tooltip();
});
// Add event listener for opening and closing details
$('#example tbody').on('click', 'td.details-control', function () {
var tr = $(this).parents('tr');
var row = table.row( tr );
if ( row.child.isShown() ) {
// This row is already open - close it
row.child.hide();
tr.removeClass('shown');
}
else {
// Open this row
row.child( format(row.data()) ).show();
tr.addClass('shown');
}
} );
$('#reset').on('click', function () {
table.columns(1).search("").draw();
table.ajax.reload(null, false);
})
$('#secondhalf').on('click', function () {
regExSearch = '(4[6-9]|[5-8][0-9]|90)';
table.column(1).search(regExSearch, true, false).draw();
});
$('#firsthalf').on('click', function () {
regExSearch = '([1-3][0-9]|4[0-5])';
table.column(1).search(regExSearch, true, false).draw();
});
$('#halftime').on('click', function () {
table.columns(1).search("HT").draw();
});
;
$('#showLime').on('change', function () {
$(this).toggleClass('limeOnly');
table.draw();
});
setInterval( function () {
table.ajax.reload( null, false ); // user paging is not reset on reload
}, 30000 )
}
);

mongodb model contain changed on calling a function levelQuestion

Here is the code for schema
schema stores questions for a particular course and which contains chapter and there question
questions: [
{
Chapter: String,
chques: [
{
description: String,
questions: [
{
question: String,
options: [String],
answer: Number,
Explanation: String,
code: { type: String, default: null },
images: [{ type: String, default: null }],
level: String
}
]
}
]
}
],
Here is the code for route
Router.get(
"/queformock/:course_id",
passport.authenticate("jwt", { session: false }),
(req, res) => {
Courses.findOne({ _id: req.params.course_id })
.lean()
.exec()
.then(course => {
if (!course) res.status(404).json({ course: "course not found" });
else {
let coursepass = Object.assign({}, course);
console.log("coursepass1: ", coursepass); //before calling levelQuestion it output as expected
let level2 = levelQuestion(coursepass, "medium");
console.log("coursepass2: ", coursepass);
let level3 = levelQuestion(coursepass, "hard");
console.log("coursepass3: ", coursepass);
res.json(level2);
}
});
}
);
Here is the levelQuestion function code
function levelQuestion(coursed, type) {
let arr = [];
coursed.questions.forEach(item => {
item.chques.forEach(i => arr.unshift(i));
});
arr.forEach(item => {
_.remove(item.questions, i => {
return i.level !== type;
});
});
_.remove(arr, item => {
return item.questions == false;
});
return arr;
}
now the problem is on every function call of levelQuestion coursepass is changed...why??

Loopback: remote method - multiple update

So, I want to have a function, that would reorder my records with orderId
I am passing an array like this:
[
{id: "5b1007aeb39c961a40540db9", orderId: 0},
{id: "5b150352184eb8471c34cf7c", orderId: 1}
]
and what I want is to multiple update all records with that ids with the orderId
so how can I do that?
I am trying something like this.... but it's not working, I imagine it is not chaining the promises at all ...
'use strict';
module.exports = function(Matchtimelineevents) {
Matchtimelineevents.reorder = function(items, cb) {
let count = 0;
if (typeof items !== 'undefined' && items.constructor === Array) {
items.forEach(item => {
Matchtimelineevents.update({'id': item.id, 'orderId': item.orderId}, function() {
count++;
console.log('UPDATING: ' + item.id, item.orderId, count);
});
});
// Matchtimelineevents.bulkUpdate(items, null, function() {
// console.log(items);
// });
}
cb(null, count);
};
Matchtimelineevents.remoteMethod('reorder', {
accepts: {
arg: 'items',
type: 'array',
},
returns: {
arg: 'count',
type: 'number',
},
http: {'verb': 'patch', 'path': '/reorder'},
description: 'Reorder the items by orderId',
});
};
What is the best way to do that?
Try to use updateAll with where like this:
const updateAllToPromise = item => new Promise((resolve, reject) => {
Matchtimelineevents.updateAll({
where: { id: item.id },
}, {
orderId: item.orderId,
}, function (err) {
if (err) resolve(false);
else resolve(true);
});
});
Matchtimelineevents.reorder = (items, cb) => {
if (!Array.isArray(items)) cb(new Error('Items not is a Array object'));
else {
Promise.all(items.map(item => updateAllToPromise(item)))
.then(items => cb(null, items.filter(item => item).length))
.catch(cb)
}
};