I need to make a call to a MySQL database whose access is restricted by IP from the MongoDB Atlas Function.
For this to work, I need to allow the MongoDB Atlas Function IP ranges on the MySQL server.
I can't find anything about this in the documentation. Do you know the IP ranges to authorize?
You can find MongoDB Atlas Function IP with this snippet of code:
const https = require('https');
https.get('https://api.ipify.org?format=json', (res) => {
console.log('statusCode:', res.statusCode);
console.log('headers:', res.headers);
let data = '';
res.on('data', (chunk) => {
data += chunk;
});
res.on('end', () => {
console.log(data);
});
}).on('error', (e) => {
console.error(e);
});
Once the function is deployed, the IP doesn't often change even when you update the function's code.
Related
I am very new to MongoDB and Nodejs and I would like to know how to create a very simple find all the records from a collection.
What I have done so far and it is listening to port 5000 in the console log:
const express = require('express')
const BodyParser = require("body-parser");
const {MongoClient} = require('mongodb')
const port = 5000;
const app = express()
app.use(BodyParser.json());
app.use(BodyParser.urlencoded({ extended: true }));
var db;
const uri = "mongodb+srv://user1:mypassword#cluster0.2sgiu.mongodb.net/ecommerce?retryWrites=true&w=majority";
app.listen(port,() => {
MongoClient.connect(uri, { useNewUrlParser: true, useUnifiedTopology: true}, (err, database) => {
if (err) {
console.log("Error occurred connecting to MongoDB...")
}
console.log(`Listening to part ${port}...`)
});
})
How do I proceed from here to create an API to retrieve all records from a Books collection after the connection has been done? Thanks
You should first save the database object that is returned to you by the callback function of connect (let's called that db).
Then, you can use that object to access your collections like "Books" in this example:
db.Books
To get all the documents inside a collection you can use the find method with an empty object as the argument.
const books = db.Books.find({});
If you want to tie MongoDB queries to your Express app, you should a create routes in which, you can respond the request with data that you fetched. Considering that this data is going to be publicly available.
app.get('/books', (req, res) => {
const books = db.Books.find({});
res.status(200).send({ books });
});
You can now make a GET request to localhost:5000/books and it should respond to you back with the contents of the Books collection. Either use your browser or curl to make the request.
$ curl localhost:5000/books
I have created an action in sails.js that passes in a mongo collection as a url parameter and retrieves the records.
'formRecords': function(req, res, next){
var orm = new Waterline();
var config = {
// Setup Adapters
// Creates named adapters that have been required
adapters: {
'default': 'mongo',
mongo: require('sails-mongo')
},
// Build Connections Config
// Setup connections using the named adapter configs
connections: {
'default': {
adapter: 'mongo',
url: 'mongodb://localhost:27017/db'
}
}
};
var record = Waterline.Collection.extend({
identity: req.param('collection'),
connection: 'default'
});
orm.loadCollection(record);
orm.initialize(config, function(err, models) {
var mongoCollection = models.collections[req.param('collection')];
//May need to create a whole new page to re-direct to for form records so we can orm.teardown() like in the create action
mongoCollection.find()
.exec(function(err, result){
console.log(result);
res.json(result);
/*res.view('forms/formRecords', {
data: result
});*/
});
//Must have orm.teardown() to close the connection then when adding a new collection I do not get the Connection is already registered error.
//orm.teardown();
});
}
};
The url looks like http://localhost:1337/forms/formRecords?collection=quotes it returns the records in a json object. If I try to use the same action again with a different collection like so http://localhost:1337/forms/formRecords?collection=users Sails errors out TypeError: Cannot read property 'collections' of undefined I tried adding the orm.teardown() function but it returns a blank view (undefined). Any idea how to to re-initialize waterline with and load a new collection?
I managed to figure it out. I call the action like so
localhost:1337/forms/formRecords?collection=collectionName
Then in my formRecords action looks like so
'formRecords': function(req, res, cb){
var findRecords = function(db, callback) {
// Get the collection records
var collection = db.collection(req.param('collection'));
// Find some records
collection.find({}).toArray(function(err, records) {
assert.equal(err, null);
//Returns the records found for the specified collection
res.json(records);
callback(records);
});
};
var MongoClient = require('mongodb').MongoClient
, assert = require('assert');
// Connection URL
var url = 'mongodb://localhost:27017/databaseName';
// Use connect method to connect to the Server
MongoClient.connect(url, function(err, db) {
assert.equal(null, err);
console.log("Connected correctly to server");
findRecords(db, function() {
db.close();
});
});
}
I pass in the argument req.param('collection') and it retrieves all records for any collection in the mongo database.
When I am inserting/updating a document in a collection, is the lock applied on the database or the collection. Suppose I have two collections and they are independant of each other in the same database and wants to do write operations on them concurrently. Is this possible?
Here is the code I am using to test this:
var assert = require('assert'),
MongoClient = require('mongodb').MongoClient,
async = require('async');
var station_list = require('./station_list.json'),
trains_list = require('./trains_list.json');
var stationList = [],
trainsList = [];
var MONGO_URL = 'mongodb://localhost:27017/test';
for(var i=0; i<station_list.stations.length; i++)
stationList.push(station_list.stations[i].station_code);
for(var i=0; i<trains_list.trains.length; i++)
trainsList.push(trains_list.trains[i].code);
console.log('trains : ' + trainsList.length + ' stations : ' + stationList.length);
populateTrains();
populateStations();
function populateTrains() {
async.eachSeries(trainsList, populateTrainDb, function (err) {
assert.equal(null, err);
});
}
function populateTrainDb(code, callback) {
MongoClient.connect(MONGO_URL, function (err, db) {
assert.equal(null, err);
var jsonData = {};
jsonData.code = code;
db.collection('trainsCon').replaceOne(
{'code' : code}, jsonData, {upsert: true, w:1}, function (err, res) {
assert.equal(null, err);
db.close();
callback();
});
});
}
function populateStations() {
async.eachSeries(stationList, populateStationDb, function (err) {
assert.equal(null, err);
});
}
function populateStationDb(code, callback) {
MongoClient.connect(MONGO_URL, function (err, db) {
assert.equal(null, err);
var jsonData = {};
jsonData.code = code;
db.collection('stationsCon').replaceOne(
{'code' : code}, jsonData, {upsert:true, w:1}, function (err, res) {
assert.equal(null, err);
db.close();
callback();
});
});
}
The two json files : station_list.json and trains_list.json have around 5000 entries. So after running the given program I get this error after a while :
C:\Users\Adnaan\Desktop\hopSmart\node_modules\mongodb\lib\server.js:242
process.nextTick(function() { throw err; })
^
AssertionError: null == { [MongoError: connect EADDRINUSE 127.0.0.1:27017]
name: 'MongoError',
message: 'connect EADDRINUSE 127.0.0.1:27017' }
at C:\Users\Adnaan\Desktop\hopSmart\testing.js:52:10
at C:\Users\Adnaan\Desktop\hopSmart\node_modules\mongodb\lib\mongo_client.js:276:20
at C:\Users\Adnaan\Desktop\hopSmart\node_modules\mongodb\lib\db.js:224:14
at null.<anonymous> (C:\Users\Adnaan\Desktop\hopSmart\node_modules\mongodb\lib\server.js:240:9)
at g (events.js:273:16)
at emitTwo (events.js:100:13)
at emit (events.js:185:7)
at null.<anonymous> (C:\Users\Adnaan\Desktop\hopSmart\node_modules\mongodb-core\lib\topologies\server.js:301:68)
at emitTwo (events.js:100:13)
at emit (events.js:185:7)
When I check the number of entries entered the database, around 4000 entries had already been entered in both the collections. So what I get from the above experiment was that an error might have occured when one write was being attempted while inside other collection a document must have been getting written.
So how should I proceed to have this concurrency without conflicting locks.
The answer to this question can be quite long and depends on various factors (MongoDB version, storage engine, type of operations you are doing, sharding, etc.). I can only recommend you to read carefully the Concurrency section of the MongoDB documentation, and in particular the lock granularity part.
Make sure to choose the right version of MongoDB first as the behaviour varies greatly from one version to another (e.g. database locking in pre-3.0 vs. collection locking for most operations in post-3.0 using NMAPv1).
I don't think it's concurrency issue with MongoDB, but I could be driver or even with test itself.
I have created a sample application couple of weeks ago to stress test MongoDB while working on a nasty bug. I used C# and MongoDB 3.0 on Windows 10. I have inserted million of documents in multithreaded environment but couldn't crash MongoDB.
Parallel.For(0, 10000, (x =>
{
var lstDocs = new List<BsonDocument>();
for (var i = 0; i < 100; i++)
{
lstDocs.Add(new BsonDocument(doc));
}
collection.InsertMany(lstDocs);
lstDocs.Clear();
}));
You can find code in gist here.
You should not be calling MongoClient.connect every time. That's causing a ton of connections to open and close all the time which is overloading mongo. You should let the MongoClient manage the connection pool. Change it so that you store the db object from MongoClient.connect. Something like this:
var db
MongoClient.connect(url, function(err, database){
db = database;
}
I am new to mongodb and node.js. I have a simple node.js app that in order to run I have to run the command
mongod
to make sure mongodb is running.
locally this works fine.
now I am moving to heroku, I createrd a Procfile where I declare :
web: node http.js
so this take care of starting node.js.
Now how can I call the command
mongod
on heroku?
Update
before the heroku requirement this is how I used to initialize my mongodb :
var mongo = require('mongodb');
var Server = mongo.Server,
Db = mongo.Db,
BSON = mongo.BSONPure,
ObjectID = mongo.ObjectID;
var server = new Server('localhost', 27017, {auto_reconnect: true});
db = new Db('sampledb', server);
db.open(function(err, db) {
if(!err) {
console.log("Connected to 'sampledb' database");
db.collection('sample', {strict:true}, function(err, collection) {
if (err) {
console.log("The 'sample' collection doesn't exist. Creating it with sample data...");
populateDB();
}
});
}
});
in order to accomodate heroku here is how I initialize mongodb :
var mongo = require('mongodb');
var Server = mongo.Server,
Db = mongo.Db,
BSON = mongo.BSONPure,
ObjectID = mongo.ObjectID;
var mongoUri = process.env.MONGOLAB_URI ||
process.env.MONGOHQ_URL ||
'mongodb://localhost/sampledb';
mongo.Db.connect(mongoUri, function (err, db) {
db.collection('sample', function(er, collection) {
if (err) {
console.log("The 'sample' collection doesn't exist. Creating it with
sample data...");
populateDB();
}
});
});
Now this throws an error because db is not recognized. I think I am missing a conceptual issue here, I am new to this world all together.
any help is appreciated.
You don't call the command mongod on heroku.
You'd add one of the mongo add-ons from addons.heroku.com to your app, then have your code connect to the credentials in the config variable exported to your app by the add-on provider.
I have a number of records stored in a MongoDB I'm trying to output them to the browser window by way of a Node.JS http server. I think I'm a good portion of the way along but I'm missing a few little things that are keeping it from actually working.
The code below uses node-mongo-native to connect to the database.
If there is anyone around who can help me make those last few connections with working in node I'd really appreciate it. To be fair, I'm sure this is just the start.
var sys = require("sys");
var test = require("assert");
var http = require('http');
var Db = require('../lib/mongodb').Db,
Connection = require('../lib/mongodb').Connection,
Server = require('../lib/mongodb').Server,
//BSON = require('../lib/mongodb').BSONPure;
BSON = require('../lib/mongodb').BSONNative;
var host = process.env['MONGO_NODE_DRIVER_HOST'] != null ? process.env['MONGO_NODE_DRIVER_HOST'] : 'localhost';
var port = process.env['MONGO_NODE_DRIVER_PORT'] != null ? process.env['MONGO_NODE_DRIVER_PORT'] : Connection.DEFAULT_PORT;
sys.puts("Connecting to " + host + ":" + port);
function PutItem(err, item){
var result = "";
if(item != null) {
for (key in item) {
result += key + '=' + item[key];
}
}
// sys.puts(sys.inspect(item)) // debug output
return result;
}
function ReadTest(){
var db = new Db('mydb', new Server(host, port, {}), {native_parser:true});
var result = "";
db.open(function (err, db) {
db.collection('test', function(err, collection) {
collection.find(function (err, cursor){
cursor.each( function (err, item) {
result += PutItem(err, item);
});
});
});
});
return result;
}
http.createServer(function (req, res) {
res.writeHead(200, {'Content-Type': 'text/plain'});
res.end("foo"+ReadTest());
}).listen(8124);
console.log('Server running on 8124');
Sources:
- mongo connectivity code:
https://github.com/christkv/node-mongodb-native/blob/master/examples/simple.js
- node. http code: nodejs.org
EDIT CORRECTED CODE
Thanks to Mic below who got me rolling in the right direction. For anyone interested, the corrected solution is here:
function ReadTest(res){
var db = new Db('mydb', new Server(host, port, {}), {native_parser:true});
var result = "";
res.write("in readtest\n");
db.open(function (err, db) {
res.write("now open\n");
db.collection('test', function(err, collection) {
res.write("in collection\n");
collection.find(function (err, cursor){
res.write("found\n");
cursor.each( function (err, item) {
res.write("now open\n");
var x = PutItem(err, item);
sys.puts(x);
res.write(x);
if (item == null) {
res.end('foo');
}
});
});
});
});
}
http.createServer(function (req, res) {
res.writeHead(200, {'Content-Type': 'text/plain'});
res.write("start\n");
ReadTest(res);
}).listen(8124);
console.log('Server running on 8124');
My guess is that you are returning result, writing the response, and closing the connection before anything is fetched from the db.
One solution would be to pass the response object to where you actually need it, something like:
function readTest(res) {
db.open(function (err, db) {
db.collection('test', function(err, collection) {
collection.find(function (err, cursor) {
res.writeHead(200, {'Content-type' : 'text/plain'});
cursor.each( function (err, item) { res.write(item); });
res.end();
...
Of course, you should also handle errors and try to avoid nesting too many levels, but that's a different discussion.
Instead of writing all the low-level Mongodb access code, you might want to try a simple library like mongous so that you can focus on your data, not on MongoDB quirks.
You might want to try mongoskin too.
Reading documents
To apply specific value filters, we can pass specific values to the find() command. Here is a SQL query:
SELECT * FROM Table1 WHERE name = 'ABC'
which is equivalent to the following in MongoDB (notice Collection1 for Table1):
db.Collection1.find({name: 'ABC'})
We can chain count() to get the number of results, pretty() to get a readable result. The results can be further narrowed by adding additional parameters:
db.Collection1.find({name: 'ABC', rollNo: 5})
It's important to notice that these filters are ANDed together, by default. To apply an OR filter, we need to use $or. These filters will be specified depending upon the structure of the document. Ex: for object attribute name for an object school, we need to specify filter like "school.name" = 'AUHS'
We're using here the DOT notation, by trying to access a nested field name of a field school. Also notice that the filters are quoted, without which we'll get syntax errors.
Equality matches on arrays can be performed:
on the entire arrays
based on any element
based on a specific element
more complex matches using operators
In the below query:
db.Collection1.find({name: ['ABC','XYZ']})
MongoDB is going to identify documents by an exact match to an array of one or more values. Now for these types of queries, the order of elements matters, meaning that we will only match documents that have ABC followed by XYZ and those are the only 2 elements of the array name
{name:["ABC","GHI","XYZ"]},
{name:["DEF","ABC","XYZ"]}
In the above document, let's say that we need to get all the documnts where ABC is the first element. So, we'll use the below filter:
db.Schools.find({'name.0': 'ABC' })