Mongoose save() does not work when putting documents in collection and uploading files using multer-gridfs-storage - mongodb

I am trying to build a music app and while working on the back end for the app (using express), I am facing this weird issue of documents not saving in mongo collections.
I made a post route to which user submits form data, which contains the song's mp3 file and the name of the song (it will have more data later on).
I am using multer to parse multipart form data.
I am able to save the mp3 file to mongoDB using multer-gridfs-storage. I want to save the song info such as name, artists etc in a different collection and here is the schema for the collection:
import mongoose from 'mongoose';
const Schema = mongoose.Schema;
const SongsInfo = new Schema({
name: {
type: String,
required: true,
},
});
const Song = mongoose.model('Song', SongsInfo);
export default Song;
index.js file:
import Grid from 'gridfs-stream';
import GridFsStorage from 'multer-gridfs-storage';
const app = express();
const conn = mongoose.createConnection(mongoURI);
let gfs;
conn.once('open', () => {
console.log('Connected to mongodb');
gfs = Grid(conn.db, mongoose.mongo);
gfs.collection('songFiles');
});
// storage engine
const storage = new GridFsStorage({
url: mongoURI,
file: (req, file) => new Promise((resolve, reject) => {
crypto.randomBytes(16, (err, buf) => {
if (err) {
return reject(err);
}
const filename = buf.toString('hex') +
path.extname(file.originalname);
let fileInfo;
fileInfo = {
filename,
bucketName: 'songFiles',
};
resolve(fileInfo);
});
}),
});
let upload;
middleWare(app);
app.post('/api/uploadSong', async (req, res) => {
upload = multer({ storage }).any();
upload(req, res, async (err) => {
console.log('in');
if (err) {
// console.log(err);
return res.end('Error uploading file.');
}
const { name } = req.body;
// push a Song into songs collection
const songInfo = new Song({
name,
});
const si = await songInfo.save(); // (*)
console.log(songInfo);
res.json({
songInfo: si,
file: req.file,
});
});
});
On line (*) the server just freezes until the request gets timed out.
No errors shown on console. Don't know what to do :(

I solved the issue finally!
So what i did was bring the models in index.js file and changed up some stuff here and there..
index.js
const app = express();
mongoose.connect(mongoURI); //(*)
const conn = mongoose.connection; // (*)
let gfs;
conn.once('open', () => {
console.log('Connected to mongodb');
gfs = Grid(conn.db, mongoose.mongo);
gfs.collection('songFiles');
});
// models
const Schema = mongoose.Schema; //(***)
const SongsInfo = new Schema({
name: {
type: String,
required: true,
},
});
const Song = mongoose.model('Song', SongsInfo);
// storage engine
const storage = new GridFsStorage({
url: mongoURI,
file: (req, file) => new Promise((resolve, reject) => {
crypto.randomBytes(16, (err, buf) => {
if (err) {
return reject(err);
}
const filename = buf.toString('hex') + path.extname(file.originalname);
let fileInfo;
if (file.mimetype === 'image/jpeg' || file.mimetype === 'image/png') {
fileInfo = {
filename,
bucketName: 'imageFiles',
};
} else {
fileInfo = {
filename,
bucketName: 'songFiles',
};
}
resolve(fileInfo);
});
}),
});
let upload;
middleWare(app);
app.post('/api/uploadSong', async (req, res) => {
upload = multer({ storage }).any();
upload(req, res, async (err) => {
console.log('in');
if (err) {
return res.end('Error uploading file.');
}
const { name } = req.body;
// push a Song into songs collection
const songInfo = new Song({
name,
});
songInfo.save((er, song) => {
if (!er) {
res.send('err');
} else {
console.log(err, song);
res.send('err');
}
});
});
});
At line (***) I used the same instance of mongoose that was initialized.. in the previous file I imported it again from the package...

Kia ora!
For those stumbling here three years on. I came across this issue as well. Abhishek Mehandiratta has the answer hidden in their code snippet.
The Fix
I went from instantiating mongoose with:
connect(connStr)
to doing:
const conn = createConnection(connStr)
This is the breaking change. So for an easy fix, change your code back to using connect(...).
I too, followed the documentation and Youtube tutorials to alter my code in such a way. It's an unfortunate misleading for developers who have not encountered a need to understand the difference.
I changed it back, and now it's working again (even with 'multer-gridfs-storage'). You can reference the connection with:
import {connection} from "mongoose";
connection.once('open', ...)
Why is this happening?
BenSower writes up the differences between connect and createConnection here. So from my basic understanding of BenSower's write up, you're referencing the wrong connection pool when you create your 'Song' schema, and thus, referencing the wrong pool when saving. I guess this results in a time out, have not looked further, but I'm sure a more in-depth answer exists somewhere.
import mongoose from 'mongoose';
const Song = mongoose.model('Song', SongsInfo);
As BenSower states in their answer "For me, the big disadvantage of creating multiple connections in the same module, is the fact that you can not directly access your models through mongoose.model, if they were defined "in" different connections". They're spot on with this one. Having finally overcome this obstacle, I shall have a look into createConnection on another project. For now, good ol' connect() will fix this issue, and do just fine :D

Related

trying to get uploads saving in MongoDB

I currently have the following code, which saves the temp file to public/files I have tried to understand the MongoDB GridFS documentation but with no success.
I am wondering how do I get the files to save inside MongoDB GridFS instead of my public/file directory
I am aware I am missing the part where I need to send the uploaded file to mongodb - this is the part I don't know how to do.
In mongodb example they say to do something like:
fs.createReadStream('./myFile').pipe(
bucket.openUploadStream('myFile', {
chunkSizeBytes: 1048576,
metadata: { field: 'myField', value: 'myValue' },
})
);
however I am not using FS or do I need to upload the file to the temp and then do the fs
import formidable from 'formidable';
import { MongoClient, ObjectId } from 'mongodb';
var Grid = require('gridfs-stream');
export const config = {
api: {
bodyParser: false,
},
};
export default async (req, res) => {
const uri = process.env.MONGODB_URI;
let client;
let clientPromise;
const options = {};
client = new MongoClient(uri, options);
clientPromise = client.connect();
const clients = await clientPromise;
const database = clients.db('AdStitchr');
var gfs = Grid(database, client);
gfs.collection('uploads');
const form = new formidable.IncomingForm();
form.uploadDir = 'public/files';
form.keepExtensions = true;
form.parse(req, (err, fields, files) => {
var file = files.file;
console.log(JSON.stringify(file));
try {
const newFile = File.create({
name: `files\${file.newFilename}.mp3`,
});
res.status(200).json({ status: 'success' });
} catch (error) {
res.send(error);
}
});
};

Setup error handlers in express/mongoose/mongoDB

I'm currently part of a web dev Bootcamp and my current project is requesting I create error handlers in a specific manner that I do not understand. Below is a screenshot of the directions . . .
Here are my current files in hopes that it makes sense . . .
/* app.js */
const express = require('express');
const mongoose = require('mongoose');
const userRouter = require('./routes/users');
const cardRouter = require('./routes/cards');
const { PORT = 3000 } = process.env;
const app = express();
mongoose.connect('mongodb://localhost:27017/aroundb', {
useNewUrlParser: true,
useCreateIndex: true,
useFindAndModify: false,
useUnifiedTopology: true,
});
app.use(express.json());
app.use(express.urlencoded({ extended: false }));
app.use((req, res, next) => {
req.user = { _id: '60c4e0e2a80be4c8c2de5474' };
next();
});
app.use('/users', userRouter);
app.use('/cards', cardRouter);
app.listen(PORT, () => logMsg(`listening on port ${PORT} . . .`));
/* routes/users.js */
const express = require('express');
const { getUsers, getUser, createUser } = require('../controllers/users');
const router = express.Router();
router.get('/', getUsers);
router.get('/:id', getUser);
router.post('/', createUser);
module.exports = router;
/* controllers/users.js */
const User = require('../models/user');
module.exports.getUsers = (req, res) => {
User.find({})
.then((users) => res.status(200).send({ data: users }))
.catch((err) => res.status(500).send({ message: err }));
};
module.exports.getUser = (req, res, next) => {
User.findById(req.params.id)
.then((user) => res.send({ data: user }))
.catch((err) => res.status(404).send({ message: err }));
};
module.exports.createUser = (req, res) => {
const { name, about, avatar } = req.body;
User.create({ name, about, avatar })
.then((user) => res.status(201).send({ data: user }))
.catch((err) => res.status(400).send({ message: err }));
};
My questions are:
Where should the code example provided (in the screenshot) go? Am I creating a separate controller or middleware? Or maybe it goes in the already coded controller?
Would I be creating my own errors and using a conditional to read the message?
I already thought I was handling errors, as seen in controllers/users.js, is that not the case?
NOTE: My apologies, I know that since it's from a course it might not make sense outside the context of the lesson(s). I also know there are various ways projects can be coded/solved. Unfortunately, my Bootcamp does not have live instruction, just a slack channel where 97% of responses come from Alumni. Please do not hesitate to ask questions that may help clarify things.
It seems you're directly sending an error in the last two cases, without knowing which type of it is, however it looks fine for fetching all users (1st case).
The workaround that might help you is,
Get user :
User.findById(req.params.id), function(err, user) {
if(err) {
return res.status(500).send({ message: "Default error" });
} else if (!user) {
return res.status(404).send({ message: "User not found" });
}
}
For creating a user you need to manually verify all the fields that are required in schema for ex.,
createUsers : {
const { name, about, avatar } = req.body;
if (name === null || about === null || avatar === null) {
return res.status(400).send({
message : "Required data missing in request"
})
}
... // create user
}

Upload image to mongoDB and get back the url of the image: Multer

I'm trying to upload an image to MongoDB, but when I visualize it in mongoCompass it only shows its "_id". So I don't know if it went through or not.
I need to:
Store the image in the DB
Get the image's URL after it's stored there.
I'm using Express, Multer, Body-parser, fs, mongoose
Here's my app.js
const express = require("express"),
app = express(),
bodyParser = require("body-parser"),
fs = require("fs"),
multer = require("multer"),
mongoose = require("mongoose");
mongoose.connect("mongodb://localhost/Images");
app.use(bodyParser.urlencoded(
{ extended:true }
))
app.set("view engine","ejs");
//Schema
var imgSchema = mongoose.Schema({
img:{data:Buffer,contentType: String,nom:String}
});
var image = mongoose.model("image",imgSchema);
// SET STORAGE
var storage = multer.diskStorage({
destination: function (req, file, cb) {
cb(null, './uploads')
},
filename: function (req, file, cb) {
cb(null, 'uploaded-' + Date.now()+".png")
}
})
var upload = multer({ storage: storage })
app.get("/",(req,res)=>{
res.render("index");
})
app.get("/show",(req,res)=>{
image.find().toArray(function (err,result){
const imgArray = result.map(element =>element._id);
console.log(imgArray);
if(err){
return console.error(err);
}
res.send(imgArray)
})
});
app.post("/uploadphoto",upload.single('myImage'),(req,res)=>{
var img = fs.readFileSync(req.file.path);
var encode_img = img.toString('base64');
var final_img = {
contentType:req.file.mimetype,
image:new Buffer(encode_img,'base64'),
nom:"hi"
};
image.create(final_img,function(err,result){
if(err){
console.log(err);
}else{
console.log(result.img.Buffer);
console.log("Saved To database");
res.contentType(final_img.contentType);
res.send(final_img.image);
}
})
})
//Code to start server
app.listen(2000,function () {
console.log("Server Started at PORT 2000");
})
this is what mongo shows me
This is the image upload folder
Thank you in advance
Everything is here
This Github repo has the answer detailed.
https://github.com/AnasGara/Upload-Image-Express-Mutler

Puppeteer and express can not load new data using REST API

I'm using puppeteer to scrape page that has contents that change periodically and use express to present data in rest api.
If I turn on headless chrome to see what is being shown in the browser, the new data is there, but the data is not showing up in get() and http://localhost:3005/api-weather. The normal browser only shows the original data.
const express = require('express');
const server = new express();
const cors = require('cors');
const morgan = require('morgan');
const puppeteer = require('puppeteer');
server.use(morgan('combined'));
server.use(
cors({
allowHeaders: ['sessionId', 'Content-Type'],
exposedHeaders: ['sessionId'],
origin: '*',
methods: 'GET, HEAD, PUT, PATCH, POST, DELETE',
preflightContinue: false
})
);
const WEATHER_URL = 'https://forecast.weather.gov/MapClick.php?lat=40.793588904953985&lon=-73.95738513173298';
const hazard_url2 = `file://C:/Users/xdevtran/Documents/vshome/wc_api/weather-forecast-nohazard.html`;
(async () => {
try {
const browser = await puppeteer.launch({ headless: true });
const page = await browser.newPage();
await page.setRequestInterception(true);
page.on("request", request => {
console.log(request.url());
request.continue();
});
await page.goto(hazard_url2, { timeout: 0, waitUntil: 'networkidle0' });
hazard = {
"HazardTitle": "stub",
"Hazardhref": "stub"
}
let forecast = await page.evaluate(() => {
try {
let forecasts = document.querySelectorAll("#detailed-forecast-body.panel-body")[0].children;
let weather = [];
for (var i = 0, element; element = forecasts[i]; i++) {
period = element.querySelector("div.forecast-label").textContent;
forecast = element.querySelector("div.forecast-text").textContent;
weather.push(
{
period,
forecast
}
)
}
return weather;
} catch (err) {
console.log('error in evaluate: ', err);
res.end();
}
}).catch(err => {
console.log('err.message :', err.message);
});
weather = forecast;
server.get('/api-weather', (req, res) => {
try {
res.end(JSON.stringify(weather, null, ' '));
console.log(weather);
} catch (err) {
console.log('failure: ', err);
res.sendStatus(500);
res.end();
return;
}
});
} catch (err) {
console.log('caught error :', err);
}
browser.close();
})();
server.listen(3005, () => {
console.log('http://localhost:3005/api-weather');
});
I've tried several solutions WaitUntil, WaitFor, .then and sleep but nothing seems to work.
I wonder if it has something to do with express get()? I'm using res.end() instead of res.send() is because the json looks better when I use res.end(). I don't really know the distinction.
I'm also open to using this reload solution. But I received errors and didn't use it.
I also tried waitForNavigation(), but I don't know how it works, either.
Maybe I'm using the wrong search term to find the solution. Could anyone point me in the right direction? Thank you.

Mongoose is returning an empty array from my database when doing a Model.find() query

I looked at this popular question, but it didn't seem to fix my issue, so I'm going to post this.
I currently have an express.js server file using mongoose, that keeps returning an empty array. I have no idea if it might by an async issue, and I don't know what I can use to indicate that I'm connected to my database.
const express = require('express');
const app = express();
const bodyParser = require('body-parser');
const mongoose = require('mongoose');
const PORT = process.env.PORT || 8080;
//Mongoose stuff
mongoose.connect('mongodb+srv://excelsiorAdmin:Mysecretpassword#excelsiorcluster-zakfd.mongodb.net/test?retryWrites=true', { useNewUrlParser: true, dbName: 'excelsiorDB'});
const dbConnection = mongoose.connection;
dbConnection.on('error', console.error.bind(console, 'connection error:'));
dbConnection.once('open', function() {
console.log('connected to the database');
let charSchema = new mongoose.Schema({
imageURL: String,
company: String,
name: String,
civName: String,
alignment: String,
firstDebut: String,
abilities: Array,
teams: Array,
desc: String
});
let Char = mongoose.model('Char', charSchema, 'chars');
//root
app.get('/', (req, res, next) => res.send('Welcome to the API!'));
//get all characters
app.get('/chars', (req, res, next) => {
console.log('getting all characters');
Char.find(function (err, chars) {
if (err) {
res.status(404).send(err);
console.log('there was an error');
};
console.log(chars);
res.send(chars);
});
});
//get heroes
app.get('/chars/heroes', (req, res, next) => {
Char.find({alignment: "Hero"}, function (err, chars) {
if (err) {
res.status(404).send(err);
};
res.send(chars);
});
});
});
app.listen(PORT, () => console.log(`This API is listening on port ${PORT}!`));
The mongoose.model will set the collection it's looking for equal to the lowercase, pluralized form of the name of the model.
let Char = mongoose.model('Char', charSchema);
This will look for the "chars" collection. However, if the database you're connecting to doesn't have a collection with the same name as the mongoose default, it will return results from a collection that doesn't exist. To make sure it hits the right collection if they don't match, you'll have to manually enter the collection's name as a third parameter:
let Char = mongoose.model('Char', charSchema, "excelsiorCollection");