The mongoDB data I am displaying is information i scraped from a website using cheerio. I scrape the data, save it into a mongo collection (called Store), and then display that information in an HTML table using ejs.
Problem:
The first table displays information just fine, with 26 rows of data. The second table, however, displays 26 blank rows and then 25 rows of the data i scraped. Sometimes the roles will be reversed, just depending on which scrape occurs first, so I know it has something to do with when it gets put into the db.
Can someone please explain how I can fix this?
Code:
scrape function (function is exactly the same for second set of data but instead of SUP, it says QA):
var scrapeSUP = function() {
var SUPStoreInfo = [];
return Axios.get("http://example.com").then(function(response){
var $ = cheerio.load(response.data);
$('.row > tbody > tr:contains("SUP")').each(function(i, element){
let SUPResult = {
"environment.SUP": $(element).children().next(),
"store.SUP": $(element).children().next().next(),
"version.SUP": $(element).children().next().next().next()
};
SUPStoreInfo.push(SUPResult);
//make sure Store collection is empty
Store.count({}, function(err, count){
if (count === 0) {
//if empty, add newly scraped data
Store.create(SUPResult, function(err, new){
if (err){
console.log(err);
}
});
} else {
//if Store collection is not empty (has old data), remove all contents
Store.remove({}, function(err, doc) {
if (err) {
console.log(err);
} else {
//once all contents removed, add newly scraped data
Store.create(SUPResult, function(err, new){
if (err){
console.log(err);
}
});
}
})
}
});
});
return SUPStoreInfo;
});
}
please let me know if you need to see other code (html tables, mongoose schema, etc)
Related
I'm trying to return create a paginated list. I used graphql to query the data. With my query, I pass the number of records I need (In a variable named first) and the ID of the last fetched record (In a varible called after). Now I managed to write a query (Note that I used mongoose) to fetch the records. Now what I need to do is get the relavant information to perform the pagination like hasNextPage, hasPreviousPage, currentPage and totalPages.
To get most of these information I need to get the total number of records in the database. To do that I need to send another db request.
I also need to know the position of the record in the table. No idea how.
Here's the query:
new Promise((resolve, reject) =>
Company.where('_id')
.gt(after)
.limit(first)
.lean()
.exec((error, doc) => {
if (error) {
reject(error);
}
resolve({
edges: doc,
pageInfo: {
hasNextPage: '...',
hasPreviousPage: '...',
currentPage: '...',
totalPages: '...'
}
});
}))
Any idea how to do this efficiently?
you can try this module mongoose-paginate
here what i uses, for pagination,
var current = req.query.filter.current;
var limit = req.query.filter.limit;
console.log('params.query.filter.current',current);
var skip = Number(limit)*Number(current)-Number(limit);
console.log('skip::',skip);
Cours.find({'attributes.version.status': true}).skip(skip).limit(limit).sort({_id:'asc'}).exec(function (err, resulta) {
if (err) {
console.log('erreur trouverCours');
res.json({
protecteds: err
});
}
console.log('cours ::', resulta);
res.json({
"data": resulta
});
});
I have a collection Files which contain items with a userId. I want a unique array with all userIds in the collection.
Here my first approach (code from here: LINK)
Server Method
'uniqUser': function(){
const distinct = Meteor.wrapAsync(Files.rawCollection().distinct,Files.rawCollection());
const result = distinct('userId');
console.log(result)
return result
}
Client Call
uniqUser(){
Meteor.call('uniqUser', function(err, data) {
console.log(data);
return data;
});
}
the console.log looks good:
but my view show nothing!? also no errors...
uniqUser:
{{#each uniqUser}}
<p>{{this}}</p>
{{/each}}
I have to use meteor-reactive-method LINK:
uniqUser() {
return ReactiveMethod.call("uniqUser");
}
and eveything is fine!
router.get('/wiki/:topicname', function(req, res, next) {
var topicname = req.params.topicname;
console.log(topicname);
summary.wikitext(topicname, function(err, result) {
if (err) {
return res.send(err);
}
if (!result) {
return res.send('No article found');
}
$ = cheerio.load(result);
var db = req.db;
var collection = db.get('try1');
collection.insert({ "topicname" : topicname, "content": result }, function (err, doc){
if (err) {
// If it failed, return error
res.send("There was a problem adding the information to the database.");
}
else {
// And forward to success page
res.send("Added succesfully");
}
});
});
Using this code, I am trying to add the fetched content from Wikipedia in to the collection try1. The message "Added succesfully" is displayed. But the collection seems to be empty. The data is not inserted in the database
The data must be there, mongodb has { w: 1, j: true } write concern options by default so its only returns without an error if the document is truly inserted if there were any document to insert.
Things you should consider:
-Do NOT use insert function, its depricated use insertOne, insertMany or bulkWrite. ref.: http://mongodb.github.io/node-mongodb-native/2.1/api/Collection.html#insert
-The insert methods callback has two parameters. Error if there was an error, and result. The result object has several properties with could be used for after insert result testing like: result.insertedCount will return the number of inserted documents.
So according to these in your code you only test for error but you can insert zero documents without an error.
Also its not clear to me where do you get your database name from. Is the following correct in your code? Are you sure you are connected to the database you want to use?
var db = req.db;
Also you don't have to enclose your property names with " in your insert method. The insert should look something like this:
col.insertOne({topicname : topicname, content: result}, function(err, r) {
if (err){
console.log(err);
} else {
console.log(r.insertedCount);
}
});
Start your mongod server in a correct path,i.e, same path as that of what you are using to check the contents of collection.
sudo mongod --dbpath <actual-path>
When I findOne based on deliverables.steps._id it's returning my entire document rather than just my step with the particular id. Can I not have it return my individual step rather than the whole document? The reason I ask is so when I need to update I just update this step rather than updating the whole document each time.
exports.findStep = function(req, res) {
Project.findOne({'deliverables.steps._id': req.params.stepId}).sort('-created').exec(function(err, step) {
if (err) {
return res.status(400).send({
message: errorHandler.getErrorMessage(err)
});
} else {
res.jsonp(step);
}
});
};
I am running into a question when to use which one, the following is update function for mongoose, it works fine.
// Updates an existing form in the DB.
exports.update = function(req, res) {
if(req.body._id) { delete req.body._id; }
Form.findById(req.params.id, function (err, form) {
if (err) { return handleError(res, err); }
if(!form) { return res.send(404); }
var updated = _.assign(form, req.body);
updated.formContent = req.body.formContent;
updated.save(function (err) {
if (err) { return handleError(res, err); }
return res.json(200, form);
});
});
};
Tried the following to replace the form data.
_.assign(form, req.body); // Works (update database)
_.merge(form, req.body); // Not Work (database not updating, remain the same)
_.extend(form, req.body); // Works (update database)
The above result show merge doesn't work when there is object within the post data.
Could some please explain why one is not working the others is ok. I have read the following question
Lodash - difference between .extend() / .assign() and .merge()
but i am curious to understanding which one won't update the database, but when applied with assign and extend it's working.