How to create a simple retrieve API after the connection has been done? - mongodb

I am very new to MongoDB and Nodejs and I would like to know how to create a very simple find all the records from a collection.
What I have done so far and it is listening to port 5000 in the console log:
const express = require('express')
const BodyParser = require("body-parser");
const {MongoClient} = require('mongodb')
const port = 5000;
const app = express()
app.use(BodyParser.json());
app.use(BodyParser.urlencoded({ extended: true }));
var db;
const uri = "mongodb+srv://user1:mypassword#cluster0.2sgiu.mongodb.net/ecommerce?retryWrites=true&w=majority";
app.listen(port,() => {
MongoClient.connect(uri, { useNewUrlParser: true, useUnifiedTopology: true}, (err, database) => {
if (err) {
console.log("Error occurred connecting to MongoDB...")
}
console.log(`Listening to part ${port}...`)
});
})
How do I proceed from here to create an API to retrieve all records from a Books collection after the connection has been done? Thanks

You should first save the database object that is returned to you by the callback function of connect (let's called that db).
Then, you can use that object to access your collections like "Books" in this example:
db.Books
To get all the documents inside a collection you can use the find method with an empty object as the argument.
const books = db.Books.find({});
If you want to tie MongoDB queries to your Express app, you should a create routes in which, you can respond the request with data that you fetched. Considering that this data is going to be publicly available.
app.get('/books', (req, res) => {
const books = db.Books.find({});
res.status(200).send({ books });
});
You can now make a GET request to localhost:5000/books and it should respond to you back with the contents of the Books collection. Either use your browser or curl to make the request.
$ curl localhost:5000/books

Related

mongoose model has no access to connections in-progress transactions

In my current express application I want to use the new ability of mongodb multi-doc transactions.
First of all it is important to point out how I connect and handle the models
My app.js (server) firstly connects to the db by using db.connect().
I require all models in my db.index file. Since the models will be initiated with the same mongoose reference, I assume that future requires of the models in different routes point to the connected and same connection. Please correct me if I'm wrong with any of these assumptions.
I save the connection reference inside the state object and returning it when needed for my transaction later
./db/index.ts
const fs = require('fs');
const path = require('path');
const mongoose = require('mongoose');
const state = {
connection = null,
}
// require all models
const modelFiles = fs.readdirSync(path.join(__dirname, 'models'));
modelFiles
.filter(fn => fn.endsWith('.js') && fn !== 'index.js')
.forEach(fn => require(path.join(__dirname, 'models', fn)));
const connect = async () => {
state.connection = await mongoose.connect(.....);
return;
}
const get = () => state.connection;
module.exports = {
connect,
get,
}
my model files are containing my required schemas
./db/models/example.model.ts
const mongoose = require('mongoose');
const Schema = mongoose.Schema;
const ExampleSchema = new Schema({...);
const ExampleModel = mongoose.model('Example', ExampleSchema);
module.exports = ExampleModel;
Now the route where I try to do a basic transaction. F
./routes/item.route.ts
const ExampleModel = require('../db/models/example.model');
router.post('/changeQty', async (req,res,next) => {
const connection = db.get().connection;
const session = await connection.startSession(); // works fine
// start a transaction
session.startTransaction(); // also fine
const {someData} = req.body.data;
try{
// jsut looping that data and preparing the promises
let promiseArr = [];
someData.forEach(data => {
// !!! THIS TRHOWS ERROR !!!
let p = ExampleModel.findOneAndUpdate(
{_id : data.id},
{$incr : {qty : data.qty}},
{new : true, runValidators : true}
).session(session).exec();
promiseArr.push(p);
})
// running the promises parallel
await Promise.all(promiseArr);
await session.commitTransaction();
return res.status(..)....;
}catch(err){
await session.abortTransaction();
// MongoError : Given transaction number 1 does not match any in-progress transactions.
return res.status(500).json({err : err});
}finally{
session.endSession();
}
})
But I always get the following error, which probably has to do sth with the connection reference of my models. I assume, that they don't have access to the connection which started the session, so they are not aware of the session.
MongoError: Given transaction number 1 does not match any in-progress
transactions.
Maybe I somehow need to initiate the models inside db.connect with the direct connection reference ?
There is a big mistake somewhere and I hope you can lead me to the correct path. I appreciate Any help, Thanks in advance
This is because you're doing operations in parallel:
So you've got a bunch of race conditions. Just use async/await
and make your life easier.
let p = await ExampleModel.findOneAndUpdate(
{_id : data.id},
{$incr : {qty : data.qty}},
{new : true, runValidators : true}
).session(session).exec();
Reference : https://github.com/Automattic/mongoose/issues/7311
If that does not work try to execute promises one by one rather than promise.all().

insertMany drop down mongodb service

I have API, where I get datas. I use mongoose to save it into my local MongoDB. Each time when I save, I create dynamically new model and use insertMany on it:
const data = result.Data;
const newCollectionName = `tab_${name.toLowerCase()}`;
const Schema = new mongoose.Schema({}, { strict: false });
const TAB = mongoose.model(newCollectionName, Schema);
return TAB.collection.drop(() => {
// eslint-disable-next-line
const clearJSONs = Object.values(data)
.filter(x => typeof x === 'object');
return TAB.insertMany(clearJSONs, (err, result2) => {
// console.log(result2);
res.json({ success: true });
});
});
But... later, when all almost complete, my mongoose service falls down ... And... I even dont know what to do.
upd. mongo log:
2018-06-17T13:43:09.883+0300 E STORAGE [conn58] WiredTiger error (24) [1529232189:883394][4799:0x7f9fe1d30700], WT_SESSION.create: /var/lib/mongodb/: directory-sync: open: Too many open files
How to solve this?
The problem was in ulimits of the system. The best solution was described by this guy

Receiving Data from Mongodb using Mongoose [duplicate]

I just started learning MongoDB and mongoose. Currently I have the following structure:
database -> skeletonDatabase
collection -> adminLogin
When I run db.adminLogin.find() from the command line I get:
{ "_id" : ObjectId("52lhafkjasfadsfea"), "username" : "xxxx", "password" : "xxxx" }
My connection (this works, just adding it FYI)
module.exports = function(mongoose)
{
mongoose.connect('mongodb://localhost/skeletonDatabase');
var db = mongoose.connection;
db.on('error', console.error.bind(console, 'connection error:'));
db.once('open', function callback () {
console.log('Conntected To Mongo Database');
});
}
My -js-
module.exports = function(mongoose)
{
var Schema = mongoose.Schema;
// login schema
var adminLogin = new Schema({
username: String,
password: String
});
var adminLoginModel = mongoose.model('adminLogin', adminLogin);
var adminLogin = mongoose.model("adminLogin");
adminLogin.find({}, function(err, data){
console.log(">>>> " + data );
});
}
My console.log() returns as >>>>
So what am I doing wrong here? Why do I not get any data in my console log? Thanks in advance for any help.
mongoose by default takes singular model names and pairs them with a collection named with the plural of that, so mongoose is looking in the db for a collection called "adminLogins" which doesn't exist. You can specify your collection name as the 2nd argument when defining your schema:
var adminLogin = new Schema({
username: String,
password: String
}, {collection: 'adminLogin'});
Had a problem with injecting it within an express route for my api so I changed it thanks to #elkhrz by first defining the schema and then compiling that one model I want to then pull like so:
app.get('/lists/stored-api', (req, res) => {
Apis.find(function(err, apis) {
if (err) return console.error(err);
res.send(apis);
});
});
I wouldn't send it to the body, I would actually do something else with it especially if you plan on making your API a production based application.
Run through this problem and read up on possible proper ways of rendering your data:
How to Pass Data Between Routes in Express
Always a good idea to practice safe procedures when handling data.
first compile just one model with the schema as an argument
var adminLogin = mongoose.model('adminLogin', adminLogin);
in your code adminLogin does not exist, adminLoginModel does;
after that ,instead to
adminLogin.find({}, function(err, data){
console.log(">>>> " + data );
});
try this
adminLogin.find(function (err, adminLogins) {
if (err) return console.error(err);
console.log(adminLogins);
is important the "s" because mongo use the plural of the model to name the collection, sorry for my english...

Dynamically retrieve records in mongodb using sails.js and waterline

I have created an action in sails.js that passes in a mongo collection as a url parameter and retrieves the records.
'formRecords': function(req, res, next){
var orm = new Waterline();
var config = {
// Setup Adapters
// Creates named adapters that have been required
adapters: {
'default': 'mongo',
mongo: require('sails-mongo')
},
// Build Connections Config
// Setup connections using the named adapter configs
connections: {
'default': {
adapter: 'mongo',
url: 'mongodb://localhost:27017/db'
}
}
};
var record = Waterline.Collection.extend({
identity: req.param('collection'),
connection: 'default'
});
orm.loadCollection(record);
orm.initialize(config, function(err, models) {
var mongoCollection = models.collections[req.param('collection')];
//May need to create a whole new page to re-direct to for form records so we can orm.teardown() like in the create action
mongoCollection.find()
.exec(function(err, result){
console.log(result);
res.json(result);
/*res.view('forms/formRecords', {
data: result
});*/
});
//Must have orm.teardown() to close the connection then when adding a new collection I do not get the Connection is already registered error.
//orm.teardown();
});
}
};
The url looks like http://localhost:1337/forms/formRecords?collection=quotes it returns the records in a json object. If I try to use the same action again with a different collection like so http://localhost:1337/forms/formRecords?collection=users Sails errors out TypeError: Cannot read property 'collections' of undefined I tried adding the orm.teardown() function but it returns a blank view (undefined). Any idea how to to re-initialize waterline with and load a new collection?
I managed to figure it out. I call the action like so
localhost:1337/forms/formRecords?collection=collectionName
Then in my formRecords action looks like so
'formRecords': function(req, res, cb){
var findRecords = function(db, callback) {
// Get the collection records
var collection = db.collection(req.param('collection'));
// Find some records
collection.find({}).toArray(function(err, records) {
assert.equal(err, null);
//Returns the records found for the specified collection
res.json(records);
callback(records);
});
};
var MongoClient = require('mongodb').MongoClient
, assert = require('assert');
// Connection URL
var url = 'mongodb://localhost:27017/databaseName';
// Use connect method to connect to the Server
MongoClient.connect(url, function(err, db) {
assert.equal(null, err);
console.log("Connected correctly to server");
findRecords(db, function() {
db.close();
});
});
}
I pass in the argument req.param('collection') and it retrieves all records for any collection in the mongo database.

How to access a preexisting collection with Mongoose?

I have a large collection of 300 question objects in a database test. I can interact with this collection easily through MongoDB's interactive shell; however, when I try to get the collection through Mongoose in an express.js application I get an empty array.
My question is, how can I access this already existing dataset instead of recreating it in express? Here's some code:
var mongoose = require('mongoose');
var Schema = mongoose.Schema;
mongoose.connect('mongodb://localhost/test');
mongoose.model('question', new Schema({ url: String, text: String, id: Number }));
var questions = mongoose.model('question');
questions.find({}, function(err, data) { console.log(err, data, data.length); });
This outputs:
null [] 0
Mongoose added the ability to specify the collection name under the schema, or as the third argument when declaring the model. Otherwise it will use the pluralized version given by the name you map to the model.
Try something like the following, either schema-mapped:
new Schema({ url: String, text: String, id: Number},
{ collection : 'question' }); // collection name
or model mapped:
mongoose.model('Question',
new Schema({ url: String, text: String, id: Number}),
'question'); // collection name
Here's an abstraction of Will Nathan's answer if anyone just wants an easy copy-paste add-in function:
function find (name, query, cb) {
mongoose.connection.db.collection(name, function (err, collection) {
collection.find(query).toArray(cb);
});
}
simply do find(collection_name, query, callback); to be given the result.
for example, if I have a document { a : 1 } in a collection 'foo' and I want to list its properties, I do this:
find('foo', {a : 1}, function (err, docs) {
console.dir(docs);
});
//output: [ { _id: 4e22118fb83406f66a159da5, a: 1 } ]
You can do something like this, than you you'll access the native mongodb functions inside mongoose:
var mongoose = require("mongoose");
mongoose.connect('mongodb://localhost/local');
var connection = mongoose.connection;
connection.on('error', console.error.bind(console, 'connection error:'));
connection.once('open', function () {
connection.db.collection("YourCollectionName", function(err, collection){
collection.find({}).toArray(function(err, data){
console.log(data); // it will print your collection data
})
});
});
Update 2022
If you get an MongoInvalidArgumentError: The callback form of this helper has been removed. error message, here's the new syntax using async/await:
const mongoose = require("mongoose");
mongoose.connect('mongodb://localhost/productsDB');
const connection = mongoose.connection;
connection.on('error', console.error.bind(console, 'connection error:'));
connection.once('open', async function () {
const collection = connection.db.collection("Products");
collection.find({}).toArray(function(err, data){
console.log(data); // it will print your collection data
});
});
I had the same problem and was able to run a schema-less query using an existing Mongoose connection with the code below. I've added a simple constraint 'a=b' to show where you would add such a constraint:
var action = function (err, collection) {
// Locate all the entries using find
collection.find({'a':'b'}).toArray(function(err, results) {
/* whatever you want to do with the results in node such as the following
res.render('home', {
'title': 'MyTitle',
'data': results
});
*/
});
};
mongoose.connection.db.collection('question', action);
Are you sure you've connected to the db? (I ask because I don't see a port specified)
try:
mongoose.connection.on("open", function(){
console.log("mongodb is connected!!");
});
Also, you can do a "show collections" in mongo shell to see the collections within your db - maybe try adding a record via mongoose and see where it ends up?
From the look of your connection string, you should see the record in the "test" db.
Hope it helps!
Something else that was not obvious, to me at least, was that the when using Mongoose's third parameter to avoid replacing the actual collection with a new one with the same name, the new Schema(...) is actually only a placeholder, and doesn't interfere with the exisitng schema so
var User = mongoose.model('User', new Schema({ url: String, text: String, id: Number}, { collection : 'users' })); // collection name;
User.find({}, function(err, data) { console.log(err, data, data.length);});
works fine and returns all fields - even if the actual (remote) Schema contains none of these fields. Mongoose will still want it as new Schema(...), and a variable almost certainly won't hack it.
Go to MongoDB website, Login > Connect > Connect Application > Copy > Paste in 'database_url' > Collections > Copy/Paste in 'collection' .
var mongoose = require("mongoose");
mongoose.connect(' database_url ');
var conn = mongoose.connection;
conn.on('error', console.error.bind(console, 'connection error:'));
conn.once('open', function () {
conn.db.collection(" collection ", function(err, collection){
collection.find({}).toArray(function(err, data){
console.log(data); // data printed in console
})
});
});
I tried all the answers but nothing worked out, finally got the answer hoe to do it.
var mongoose = require('mongoose');
mongoose.connect('mongodb://0.0.0.0:27017/local');
// let model = require('./test1');
setTimeout(async () => {
let coll = mongoose.connection.db.collection(<Your collection name in plural form>);
// let data = await coll.find({}, {limit:2}).toArray();
// let data = await coll.find({name:"Vishal"}, {limit:2}).toArray();
// let data = await coll.find({name:"Vishal"}, {projection:{player:1, _id:0}}).toArray();
let data = await coll.find({}, {limit:3, sort:{name:-1}}).toArray();
console.log(data);
}, 2000);
I have also mentioned some of the criteria to filter out. Delete and update can also be done by this.
Thanks.
Make sure you're connecting to the right database as well as the right collection within the database.
You can include the name of the database in the connection string.
notice databasename in the following connection string:
var mongoose = require('mongoose');
const connectionString = 'mongodb+srv://username:password#hosturl.net/databasename';
mongoose.connect(connectionString);