How to connect to mongodb database using Nextjs? - mongodb

Trying to connect to my mongodb database in the latest version of Nextjs. Things have changed so much, so I don't longer know what to do.
There's an example of how to set up the connection here: https://github.com/vercel/next.js/tree/canary/examples/with-mongodb
They use this file:
//The mongodb.js file from the example
import { MongoClient } from 'mongodb'
const uri = process.env.MONGODB_URI
const options = {}
let client
let clientPromise
if (!process.env.MONGODB_URI) {
throw new Error('Please add your Mongo URI to .env.local')
}
if (process.env.NODE_ENV === 'development') {
// In development mode, use a global variable so that the value
// is preserved across module reloads caused by HMR (Hot Module Replacement).
if (!global._mongoClientPromise) {
client = new MongoClient(uri, options)
global._mongoClientPromise = client.connect()
}
clientPromise = global._mongoClientPromise
} else {
// In production mode, it's best to not use a global variable.
client = new MongoClient(uri, options)
clientPromise = client.connect()
}
// Export a module-scoped MongoClient promise. By doing this in a
// separate module, the client can be shared across functions.
export default clientPromise
However, they forgot to add how to actually use it. I can't even begin to figure it out.
//pages/api/user.js
import client from '/lib/mongodb.js'
export default async function handler(req, res) {
//How do I connect here?
}
And two bonus questions:
I used to do caching on my database connection. Is it not needed anymore?
What happened to the utils folder? It used to be special, in that it didn't send anything there to the client. Now everyone seem to use lib but I don't think there's anything special with it?

You can do like this:
const dbClient = await client;
const db = dbClient.db('db-name');
const collection = db.collection('collection-name');
// example to get a doc in collection
const doc = await collection.findOne({query:""}, {...options})

Related

Connecting to MongoDB from Vercel

I have a SvelteKit application deployed on vercel.app that uses a MongoDB (Atlas). In most cases the database connection works, but sometimes I get a connection error (connection timed out).
If this error occurs, and I try again to do something that uses the database, it immeadiately logs the same error again. This problem persists for some time, and then suddendly the database connection works again.
(When running the app locally with "npm run dev", using the same database, I've never experienced this error.)
To connect to the database, I defined:
mongodb-client.ts:
import { MongoClient } from 'mongodb';
const uri = process.env.DB_URI;
const dbClient = new MongoClient(uri).connect();
export default dbClient;
and use it like this (in several places):
import dbClient from '$lib/server/mongodb-client';
const user = await (await dbClient).db().collection('users').findOne({username: username});
I guess that, when the dbClient Promise is rejected (for whatever reason), it stays rejected and any subsequent await will immediately result in "rejected" (and therefore it will not try to reconnect; except that at some point it will ...?). Is my understanding correct? How should this be implemented correctly? (E.g. Do I need to add some options to the connection URI when this connection is create from a serverless function? Do I need to add some options when creating/connecting the MongoClient? Do I need to do this manually and add a loop, check if the promise is rejected and try again? Or should this be implemented in a completely different way?)
As you probably have guessed I'm new to JavaScript/TypeScript, MongoDB, Serverless and everything ... Thanks for any help and advice!
You can declare a function handling the connection to the database.
You will handle connection errors there and also check if a connection is already established:
import { MongoClient } from 'mongodb';
const uri = process.env.DB_URI;
const dbClient = new MongoClient(uri);
export const connectDb = async () => {
try {
if (!dbClient.isConnected()) {
await dbClient.connect();
}
return await dbClient.db();
} catch (e) {
console.log(e);
process.exit(1); // Or do something else...
}
};
Usage:
import { connectDb } from '$lib/server/mongodb-client';
const db = await connectDb();
const user = await db.collection('users').findOne({username: username});

Why does Mongoose setClient return MongooseError: Must call `setClient()` with an instance of MongoClient?

I am having trouble getting setClient() to work. My understanding of this is, that I can make a connection to MongoDB with the native MongoDB MongoClient and then in another piece of code I should be able to use this instance of the MongoClient to connect to Mongoose. So set the client to Mongoose while using the same identical connection, not a separate one.
https://mongoosejs.com/docs/api/connection.html#connection_Connection-setClient
Set the MongoDB driver MongoClient instance that this connection uses to talk to MongoDB. This is useful if you already have a MongoClient instance, and want to reuse it.
Case A
import mongoose from 'mongoose';
import { MongoClient } from 'mongodb';
async function run() {
try {
const uri = 'mongodb://localhost:27017';
// Create a new MongoClient
const client = new MongoClient(uri);
const conn = mongoose.createConnection().setClient(client);
conn.getClient(); // MongoClient { ... }
conn.readyState; // 1, means 'CONNECTED'
} catch (error) {
console.log(error);
}
}
run();
This returns
MongooseError: Must call setClient() with an instance of MongoClient at NativeConnection.setClient (.../node_modules/mongoose/lib/connection.js:1391:11).
Why am I getting this error ? What is the correct code for setClient()?
Case B
Vercel shows how to used MongoDB and Mongoose in their environment exporting a clientPromise here with MongoDB https://github.com/vercel/next.js/blob/canary/examples/with-mongodb/lib/mongodb.js and a dbConnect here with Mongoose https://github.com/vercel/next.js/blob/canary/examples/with-mongodb-mongoose/lib/dbConnect.js.
So given in db.js I have this exemplary code with the native MongoDB driver
import { MongoClient } from 'mongodb'
const uri = process.env.MONGODB_URI
const options = {}
let client
let clientPromise
if (!process.env.MONGODB_URI) {
throw new Error('Please add your Mongo URI to .env.local')
}
if (process.env.NODE_ENV === 'development') {
// In development mode, use a global variable so that the value
// is preserved across module reloads caused by HMR (Hot Module Replacement).
if (!global._mongoClientPromise) {
client = new MongoClient(uri, options)
global._mongoClientPromise = client.connect()
}
clientPromise = global._mongoClientPromise
} else {
// In production mode, it's best to not use a global variable.
client = new MongoClient(uri, options)
clientPromise = client.connect()
}
// Export a module-scoped MongoClient promise. By doing this in a
// separate module, the client can be shared across functions.
export default clientPromise
and in another file I like to use this clientPromise with Mongoose what do I do ?
Is my understanding correct here that I in fact can use the exported MongoDB clientPromise with Mongoose at all through the use of setClient() ? Or is setClient() used for something different ?
In another file, if I try this
import mongoose from 'mongoose';
import clientPromise from '$lib/mongodb';
export async function get() {
try {
const client = await clientPromise;
console.log(client); // logs client just fine
const conn = mongoose.createConnection().setClient(client);
// starts complaining here with
// MongooseError: Must call `setClient()` with an instance of MongoClient
conn.getClient(); // MongoClient { ... }
conn.readyState; // 1, means 'CONNECTED'
console.log(conn);
return {
status: 200,
body: {
message: 'ok'
}
};
} catch (error) {
console.log(error);
}
}
the same error is shown.
How do I get setClient() to work on either, the MongoClient instance or the clientPromise?
edit:
I also get this error.
Argument of type 'import(".../node_modules/mongodb/mongodb").MongoClient' is not assignable to parameter of type 'import(".../node_modules/mongoose/node_modules/mongodb/mongodb").MongoClient'.
The types of 'options.credentials' are incompatible between these types.
Type 'import(".../node_modules/mongodb/mongodb").MongoCredentials' is not assignable to type 'import(".../node_modules/mongoose/node_modules/mongodb/mongodb").MongoCredentials'.

SvelteKit With MongoDB ReferenceError: global is not defined

I'm trying to setup MongoDB connection library function. I know this function is solid, its used in a whole lot of places (search for Global is used here to maintain a cached connection across hot reloads) and you'll find a whole lot of uses including next.js releases. Note, the purpose of global storage for the database connection is to reduce the overall # of db connections in use at any one time.
What I'm not understanding is the error I'm getting when I import this library via import { connectToDatabase } from '$lib/database';
database.js
// https://github.com/mongodb-developer/mongodb-next-todo/blob/main/util/mongodb.js
import { ENV_OBJ } from "$lib/env";
import { MongoClient } from "mongodb";
const uri = ENV_OBJ.MONGODB_URI;
if (!uri) {
throw new Error("Please define the Mongodb uri environment variable inside .env");
}
/**
* Global is used here to maintain a cached connection across hot reloads
* in development. This prevents connections growing exponentially
* during API Route usage.
*/
let cached = global.mongo
if (!cached) {
cached = global.mongo = { conn: null, promise: null }
}
export const connectToDatabase = async() => {
if (cached.conn) {
return cached.conn;
}
if (!cached.promise) {
const options = {
useNewUrlParser: true,
useUnifiedTopology: true
};
cached.promise = MongoClient.connect(MONGODB_URI, opts).then((client) => {
return {
client,
db: client.db(MONGODB_DB),
}
})
}
cached.conn = await cached.promise;
return cached.conn;
}
The errors:
global is not defined
ReferenceError: global is not defined
at node_modules/mongodb/lib/promise_provider.js (http://localhost:3000/node_modules/.vite/mongodb.js?v=3885e04e:548:25)
at __require2 (http://localhost:3000/node_modules/.vite/chunk-6ODJH7E3.js?v=3885e04e:10:44)
at node_modules/mongodb/lib/utils.js (http://localhost:3000/node_modules/.vite/mongodb.js?v=3885e04e:6524:30)
at __require2 (http://localhost:3000/node_modules/.vite/chunk-6ODJH7E3.js?v=3885e04e:10:44)
at node_modules/mongodb/lib/cursor/abstract_cursor.js (http://localhost:3000/node_modules/.vite/mongodb.js?v=3885e04e:10873:19)
at __require2 (http://localhost:3000/node_modules/.vite/chunk-6ODJH7E3.js?v=3885e04e:10:44)
at node_modules/mongodb/lib/index.js (http://localhost:3000/node_modules/.vite/mongodb.js?v=3885e04e:25281:29)
at __require2 (http://localhost:3000/node_modules/.vite/chunk-6ODJH7E3.js?v=3885e04e:10:44)
at http://localhost:3000/node_modules/.vite/mongodb.js?v=3885e04e:25616:23
Note, I do see a file in my generated minimal sveltekit repo called global.d.ts I'm not sure of its purpose. It contains only:
/// <reference types="#sveltejs/kit" />
Any ideas on what's causing the error?
Reference: "#sveltejs/kit": "version": "1.0.0-next.118",
Edit: After spending a whole lot of time on this issue, the global not defined error seems to come from import { MongoClient } from "mongodb"; If I add appropriate console.logs, I can see that the MongoClient function works fine on the server, but then I get the global error on the client. The server indicates no errors at all.
So it turns out I was calling import { connectToDatabase } from '$lib/database' not in a .js helper file or api style (.js) endpoints. I was attempting to use that import and make a database call directly from the <script> portion of a xxx.svelte file.
Definite no go. That generates an immediate global not defined error.

Module export of pg-promise object derived from promise chain

We're using HashiCorp's Vault to store database connection credentials, then constructing the connection string for pg-promise from those. The 'catch' is that the Vault details are provided from a Promise wrapper, due to request callbacks to the Vault API.
Example database.js module:
const pgp = require('pg-promise')(/* options obj */);
const getDbo = () => {
return new Promise( (resolve, reject) => {
vault.init().then(secrets => {
let credentials = secrets.dbUser + ':' + secrets.dbPass
let connStr = 'postgres://' + credentials + '<#endpoint/db>'
let dbo = pgp(connStr, (err) => {
reject(err)
})
resolve(dbo);
})
}
module.exports = { get: getDbo }
This is being imported in multiple routes. With this we are seeing the warning "WARNING: Creating a duplicate database object for the same connection." Is there a better way to resolve this so there is only one object per connection details?
Creating and initializing a connection for pg-promise is a completely synchronous operation, as per the API, so there is no point using promises for that.
For initializing the library see Where should I initialize pg-promise.
See also:
Verify database connection with pg-promise when starting an app.

How to use GridFS to store images using Node.js and Mongoose

I am new to Node.js. Can anyone provide me an example of how to use GridFS for storing and retrieving binary data, such as images, using Node.js and Mongoose? Do I need to directly access GridFS?
I was not satisfied with the highest rated answer here and so I'm providing a new one:
I ended up using the node module 'gridfs-stream' (great documentation there!) which can be installed via npm.
With it, and in combination with mongoose, it could look like this:
var fs = require('fs');
var mongoose = require("mongoose");
var Grid = require('gridfs-stream');
var GridFS = Grid(mongoose.connection.db, mongoose.mongo);
function putFile(path, name, callback) {
var writestream = GridFS.createWriteStream({
filename: name
});
writestream.on('close', function (file) {
callback(null, file);
});
fs.createReadStream(path).pipe(writestream);
}
Note that path is the path of the file on the local system.
As for my read function of the file, for my case I just need to stream the file to the browser (using express):
try {
var readstream = GridFS.createReadStream({_id: id});
readstream.pipe(res);
} catch (err) {
log.error(err);
return next(errors.create(404, "File not found."));
}
Answers so far are good, however, I believe it would be beneficial to document here how to do this using the official mongodb nodejs driver instead of relying on further abstractions such as "gridfs-stream".
One previous answer has indeed utilized the official mongodb driver, however they use the Gridstore API; which has since been deprecated, see here. My example will be using the new GridFSBucket API.
The question is quite broad as such my answer will be an entire nodejs program. This will include setting up the express server, mongodb driver, defining the routes and handling the GET and POST routes.
Npm Packages Used
express (nodejs web application framework to simplify this snippet)
multer (for handling multipart/form-data requests)
mongodb (official mongodb nodejs driver)
The GET photo route takes a Mongo ObjectID as a parameter to retrieve the image.
I configure multer to keep the uploaded file in memory. This means the photo file will not be written to the file system at anytime, and instead be streamed straight from memory into GridFS.
/**
* NPM Module dependencies.
*/
const express = require('express');
const photoRoute = express.Router();
const multer = require('multer');
var storage = multer.memoryStorage()
var upload = multer({ storage: storage, limits: { fields: 1, fileSize: 6000000, files: 1, parts: 2 }});
const mongodb = require('mongodb');
const MongoClient = require('mongodb').MongoClient;
const ObjectID = require('mongodb').ObjectID;
let db;
/**
* NodeJS Module dependencies.
*/
const { Readable } = require('stream');
/**
* Create Express server && Routes configuration.
*/
const app = express();
app.use('/photos', photoRoute);
/**
* Connect Mongo Driver to MongoDB.
*/
MongoClient.connect('mongodb://localhost/photoDB', (err, database) => {
if (err) {
console.log('MongoDB Connection Error. Please make sure that MongoDB is running.');
process.exit(1);
}
db = database;
});
/**
* GET photo by ID Route
*/
photoRoute.get('/:photoID', (req, res) => {
try {
var photoID = new ObjectID(req.params.photoID);
} catch(err) {
return res.status(400).json({ message: "Invalid PhotoID in URL parameter. Must be a single String of 12 bytes or a string of 24 hex characters" });
}
let bucket = new mongodb.GridFSBucket(db, {
bucketName: 'photos'
});
let downloadStream = bucket.openDownloadStream(photoID);
downloadStream.on('data', (chunk) => {
res.write(chunk);
});
downloadStream.on('error', () => {
res.sendStatus(404);
});
downloadStream.on('end', () => {
res.end();
});
});
/**
* POST photo Route
*/
photoRoute.post('/', (req, res) => {
upload.single('photo')(req, res, (err) => {
if (err) {
return res.status(400).json({ message: "Upload Request Validation Failed" });
} else if(!req.body.name) {
return res.status(400).json({ message: "No photo name in request body" });
}
let photoName = req.body.name;
// Covert buffer to Readable Stream
const readablePhotoStream = new Readable();
readablePhotoStream.push(req.file.buffer);
readablePhotoStream.push(null);
let bucket = new mongodb.GridFSBucket(db, {
bucketName: 'photos'
});
let uploadStream = bucket.openUploadStream(photoName);
let id = uploadStream.id;
readablePhotoStream.pipe(uploadStream);
uploadStream.on('error', () => {
return res.status(500).json({ message: "Error uploading file" });
});
uploadStream.on('finish', () => {
return res.status(201).json({ message: "File uploaded successfully, stored under Mongo ObjectID: " + id });
});
});
});
app.listen(3005, () => {
console.log("App listening on port 3005!");
});
I wrote a blog post on this subject; is is an elaboration of my answer. Available here
Further Reading/Inspiration:
NodeJs Streams: Everything you need to know
Multer NPM docs
Nodejs MongoDB Driver
I suggest taking a look at this question: Problem with MongoDB GridFS Saving Files with Node.JS
Copied example from the answer (credit goes to christkv):
// You can use an object id as well as filename now
var gs = new mongodb.GridStore(this.db, filename, "w", {
"chunk_size": 1024*4,
metadata: {
hashpath:gridfs_name,
hash:hash,
name: name
}
});
gs.open(function(err,store) {
// Write data and automatically close on finished write
gs.writeBuffer(data, true, function(err,chunk) {
// Each file has an md5 in the file structure
cb(err,hash,chunk);
});
});
It looks like the writeBuffer has since been deprecated.
/Users/kmandrup/private/repos/node-mongodb-native/HISTORY:
82 * Fixed dereference method on Db class to correctly dereference Db reference objects.
83 * Moved connect object onto Db class(Db.connect) as well as keeping backward compatibility.
84: * Removed writeBuffer method from gridstore, write handles switching automatically now.
85 * Changed readBuffer to read on Gridstore, Gridstore now only supports Binary Buffers no Strings anymore.
remove the fileupload library
and if it is giving some multi-part header related error than remove the content-type from the headers