Related
I'm new to AWS so I apologize for any newbie stuff.
I'm trying to connect a MongoDB Atlas M0 cluster with our AWS EC2 instance, which is running a nodejs / react stack. The problem is that I can't make these two instances connect - AWS and MongoDB that is. When trying to use the backend sign in function (our nodejs api), it just gives this error:
Operation `user_profile.findOne()` buffering timed out after 10000ms
This is our index / connection:
import config from './config';
import app from './app';
import { connect } from 'mongoose'; // MongoDB
import { ServerApiVersion } from 'mongodb';
import https from 'https';
import AWS from 'aws-sdk';
const makeLogger = (bucket: string) => {
const s3 = new AWS.S3({
accessKeyId: <ACCESS_KEY_ID>,
secretAccessKey: <SECRET_ACCESS_KEY>
});
return (logData: any, filename: string) => {
s3.upload({
Bucket: bucket, // pass your bucket name
Key: filename, // file will be saved as testBucket/contacts.csv
Body: JSON.stringify(logData, null, 2)
}, function (s3Err: any, data: any) {
if (s3Err) throw s3Err
console.log(`File uploaded successfully at ${data.Location}`)
});
console.log(`log (${filename}): ${logData}`);
};
};
const log = makeLogger('xxx-xxxx');
log(config.MONGO_DB_ADDRESS, 'mongo_db_address.txt');
const credentials = <CREDENTIALS>
connect(config.MONGO_DB_ADDRESS, {
sslKey: credentials,
sslCert: credentials,
serverApi: ServerApiVersion.v1
}) //, { useNewUrlParser: true })
.then(() => console.log('Connected to MongoDB'))
.catch((err) => console.error('Failed connection to MongoDB', err));
app.on('error', error => {
console.error('app error: ' + error);
});
app.listen(config.WEB_PORT, () => {
console.log(`Example app listening on port ${config.WEB_PORT}`);
});
One of the endpoints giving the timeout error:
router.post('/signin', async (req, res) => {
var form_validation = signin_schema.validate({
email: req.body.email,
password: req.body.password,
});
if (form_validation.error) {
console.log('form validation sent');
//return res.status(400).send(form_validation);
return res.status(400).send({
kind: 'ERROR',
message: 'Sorry - something didn\'t go well. Please try again.'
});
}
var User = model('model', UserSchema, 'user_profile');
User.findOne({ email: req.body.email }, (err: any, the_user: any) => {
if (err) {
return res.status(400).send({
kind: 'ERROR',
message: err.message
});
}
if (!the_user) {
return res.status(400).send({
kind: 'ERROR',
message: 'the_user undefined',
});
}
compare(req.body.password, the_user.password)
.then((result) => {
if (result == true) {
const user_payload = { name: the_user.name, email: the_user.email };
const access_token = sign(user_payload, config.SECRET_TOKEN);
res.cookie('authorization', access_token, {
httpOnly: true,
secure: false,
maxAge: 3600000,
});
return res.send({ kind: "LOADING" });
// return res.send(access_token);
} else {
return res.status(400).send({
kind: 'ERROR',
message: 'Sorry - wrong password or email used.'
});
}
})
})
});
The strange thing is that I can connect from my local developer machine, when running our frontend. Just as I can connect from wsl2 ubuntu cli.
On the Mongo side, I have whitelisted every possible ip address. On the AWS side, I have created the outbound security group policy required. Regarding the inbound, I think it is correct. I've allowed access on the ports 27000 - 28018.
Again - I'm new to AWS, so if anyone can tell me what it is I'm simply not understanding here, I would be very grateful
Thanks
open mongodb atlas Network Access
open 0.0.0.0/0 (includes your current IP address)
I am building a content management system for an art portfolio app, with React. The client will POST to the API which uses Mongoose to insert into a MongoDB. The API then queries the DB for the newly inserted image, and returns it to the client.
Here's my code to connect to MongoDB using Mongoose:
mongoose.connect('mongodb://localhost/test').then(() =>
console.log('connected to db')).catch(err => console.log(err))
mongoose.Promise = global.Promise
const db = mongoose.connection
db.on('error', console.error.bind(console, 'MongoDB connection error:'))
const Schema = mongoose.Schema;
const ImgSchema = new Schema({
img: { data: Buffer, contentType: String }
})
const Img = mongoose.model('Img', ImgSchema)
I am using multer and fs to handle the image file. My POST endpoint looks like this:
router.post('/', upload.single('image'), (req, res) => {
if (!req.file) {
res.send('no file')
} else {
const imgItem = new Img()
imgItem.img.data = fs.readFileSync(req.file.path)
imgItem.contentType = 'image/png'
imgItem
.save()
.then(data =>
Img.findById(data, (err, findImg) => {
console.log(findImg.img)
fs.writeFileSync('api/uploads/image.png', findImg.img.data)
res.sendFile(__dirname + '/uploads/image.png')
}))
}
})
I can see in the file structure that writeFileSync is writing the image to the disk. res.sendFile grabs it and sends it down to the client.
Client side code looks like this:
handleSubmit = e => {
e.preventDefault()
const img = new FormData()
img.append('image', this.state.file, this.state.file.name)
axios
.post('http://localhost:8000/api/gallery', img, {
onUploadProgress: progressEvent => {
console.log(progressEvent.loaded / progressEvent.total)
}
})
.then(res => {
console.log('responsed')
console.log(res)
const returnedFile = new File([res.data], 'image.png', { type: 'image/png' })
const reader = new FileReader()
reader.onloadend = () => {
this.setState({ returnedFile, returned: reader.result })
}
reader.readAsDataURL(returnedFile)
})
.catch(err => console.log(err))
}
This does successfully place both the returned file and the img data url on state. However, in my application, the image always displays broken.
Here's some screenshots:
How to fix this?
Avoid sending back base64 encoded images (multiple images + large files + large encoded strings = very slow performance). I'd highly recommend creating a microservice that only handles image uploads and any other image related get/post/put/delete requests. Separate it from your main application.
For example:
I use multer to create an image buffer
Then use sharp or fs to save the image (depending upon file type)
Then I send the filepath to my controller to be saved to my DB
Then, the front-end does a GET request when it tries to access: http://localhost:4000/uploads/timestamp-randomstring-originalname.fileext
In simple terms, my microservice acts like a CDN solely for images.
For example, a user sends a post request to http://localhost:4000/api/avatar/create with some FormData:
It first passes through some Express middlewares:
libs/middlewares.js
...
app.use(cors({credentials: true, origin: "http://localhost:3000" })) // allows receiving of cookies from front-end
app.use(morgan(`tiny`)); // logging framework
app.use(multer({
limits: {
fileSize: 10240000,
files: 1,
fields: 1
},
fileFilter: (req, file, next) => {
if (!/\.(jpe?g|png|gif|bmp)$/i.test(file.originalname)) {
req.err = `That file extension is not accepted!`
next(null, false)
}
next(null, true);
}
}).single(`file`))
app.use(bodyParser.json()); // parses header requests (req.body)
app.use(bodyParser.urlencoded({ limit: `10mb`, extended: true })); // allows objects and arrays to be URL-encoded
...etc
Then, hits the avatars route:
routes/avatars.js
app.post(`/api/avatar/create`, requireAuth, saveImage, create);
It then passes through some user authentication, then goes through my saveImage middleware:
services/saveImage.js
const createRandomString = require('../shared/helpers');
const fs = require("fs");
const sharp = require("sharp");
const randomString = createRandomString();
if (req.err || !req.file) {
return res.status(500).json({ err: req.err || `Unable to locate the requested file to be saved` })
next();
}
const filename = `${Date.now()}-${randomString}-${req.file.originalname}`;
const filepath = `uploads/${filename}`;
const setFilePath = () => { req.file.path = filepath; return next();}
(/\.(gif|bmp)$/i.test(req.file.originalname))
? fs.writeFile(filepath, req.file.buffer, (err) => {
if (err) {
return res.status(500).json({ err: `There was a problem saving the image.`});
next();
}
setFilePath();
})
: sharp(req.file.buffer).resize(256, 256).max().withoutEnlargement().toFile(filepath).then(() => setFilePath())
If the file is saved, it then sends a req.file.path to my create controller. This gets saved to my DB as a file path and as an image path (the avatarFilePath or /uploads/imagefile.ext is saved for removal purposes and the avatarURL or [http://localhost:4000]/uploads/imagefile.ext is saved and used for the front-end GET request):
controllers/avatars.js (I'm using Postgres, but you can substitute for Mongo)
create: async (req, res, done) => {
try {
const avatarurl = `${apiURL}/${req.file.path}`;
await db.result("INSERT INTO avatars(userid, avatarURL, avatarFilePath) VALUES ($1, $2, $3)", [req.session.id, avatarurl, req.file.path]);
res.status(201).json({ avatarurl });
} catch (err) { return res.status(500).json({ err: err.toString() }); done();
}
Then when the front-end tries to access the uploads folder via <img src={avatarURL} alt="image" /> or <img src="[http://localhost:4000]/uploads/imagefile.ext" alt="image" />, it gets served up by the microservice:
libs/server.js
const express = require("express");
const path = app.get("path");
const PORT = 4000;
//============================================================//
// EXPRESS SERVE AVATAR IMAGES
//============================================================//
app.use(`/uploads`, express.static(`uploads`));
//============================================================//
/* CREATE EXPRESS SERVER */
//============================================================//
app.listen(PORT);
What it looks when logging requests:
19:17:54 INSERT INTO avatars(userid, avatarURL, avatarFilePath) VALUES ('08861626-b6d0-11e8-9047-672b670fe126', 'http://localhost:4000/uploads/1536891474536-k9c7OdimjEWYXbjTIs9J4S3lh2ldrzV8-android.png', 'uploads/1536891474536-k9c7OdimjEWYXbjTIs9J4S3lh2ldrzV8-android.png')
POST /api/avatar/create 201 109 - 61.614 ms
GET /uploads/1536891474536-k9c7OdimjEWYXbjTIs9J4S3lh2ldrzV8-android.png 200 3027 - 3.877 ms
What the user sees upon successful GET request:
I have a code block in my Mongoose controller which attempts to find both Projects and Levels:
exports.landing = (req, res, next) => {
console.log(req.params.projectid);
Project.findById(req.params.projectid, (err, project) => {
if (err) return res.status(500).send(err);
//find the level based on the projectid
Level.find({'projectid': req.params.projectid}, (err, level) => {
if (err) return res.status(500).send(err);
//find the level based on the projectid
res.json({
success: true,
message: 'got',
level: level.leveltempnodes
});
//res.render(path + 'project', {project: project, moment: moment, level: level});
});
});
};
Within the res.json section, If I just use 'level' without the dot notation, all the results come back as expected. When I try and get the 'levelnodes' entry, nothing comes back. The only thing I see differently with the level document compared to the other documents is that the JSON result includes a '[':
{"success":true,"message":"got","level":{"_id":"5b4205ea5b44e146b5978175" ...
The above works fine. But I am not able to use dot syntax on the below result:
{"success":true,"message":"got","level":[{"_id":"5b4202fc94855d56204c8bb7"
I am saving the level document like this:
var data = {
levelname: levelname,
leveltempnodes: leveltempnodes,
projectid: projectid};
var level = new Level(data);
level.save(function (err) {
if (err) return handleError(err);
})
My error is nothing is coming back at all:
{"success":true,"message":"got"}
Schema:
const mongoose = require('mongoose');
const LevelSchema = mongoose.Schema({
levelname: String,
leveltempnodes: String,
projectid: String
});
module.exports = mongoose.model('Level', LevelSchema);
Data is being stored on the DB without issue. I am adding it via Ajax:
var p = {
projectname : $("#projectname").val(),
levelname : 'Root',
leveltempnodes : '{"class":"go.GraphLinksModel","nodeKeyProperty":"id","nodeDataArray":[{"id":1,"loc":"226 226","text":"sensor"},{"text":"perception","loc":"426 225.99999999999997","id":-2},{"text":"planning","loc":"626 225.99999999999997","id":-3},{"text":"gate","loc":"826 225.99999999999997","id":-4}],"linkDataArray":[{"from":1,"to":-2,"text":"msg","points":[296.7874157629703,237.73538061447854,340.03133208792605,227.76937481449303,383.33478829426565,227.0952320784595,426.7981545990892,236.1401244399739]},{"from":-2,"to":-3,"text":"msg","points":[523.225709890083,236.1861908341044,558.0349502392196,229.00680324793404,592.1479459982006,228.54232080927673,626.6289592123036,236.76409981273324]},{"from":-3,"to":-4,"text":"msg","points":[709.6483081744094,237.23795381070627,748.7663709980919,229.48139598538538,787.383185499046,229.48139598538538,826.1210439041331,238.64104211943584]}]}',
}
if(p.projectname == ''){
console.log('e');
}else{
$.ajax({
type: 'POST',
contentType : "application/json",
url: 'api/project/save',
data : JSON.stringify(p),
success: function(res) {
window.location.replace("/project/"+res.id);
}
});
This is quite a broad question, however I currently have a Sails API server and a React Front-end (Standalone).
Note: The React Front-End is NOT part of Sails
I'm trying to get to grips with sockets, so I figured I would start simple. I want to achieve the following:
User visits my website (React)
React opens a socket and connects to Sails
Sails streams the data from within a function/model
React updates when new data is added to the model
I semi understand how this works using Express and React, however I cannot get my head around how Sails implements their version of WebSockets on top of Sockets.io.
What I've done is install the sockets.io-client within React, and then trying to use sails.sockets inside Sails.
This is what I currently have:
React Component NB: I don't think this is correct at all
componentDidMount =()=> {
this.getSessionData();
UserStore.listen(this.getSessionData);
Socket.emit('/listSessions', function(data){
console.log(data);
})
}
Sails Function (listSessions)
listSessions: function(req, res) {
Session.find({ where: {visible: true}, sort: 'createdAt DESC'},
function(err, sessions){
if(req.isSocket){
Session.watch(req.socket);
console.log('User subscribed to ' + req.socket.id);
}
if(err) return res.json(500, {
error: err,
message: 'Something went wrong when finding trades'
});
return res.json(200, {
sessions: sessions,
});
})
},
Sails Function (createSession) Trying to use publishCreate to use in conjunction with Session.watch in the above function
createSession: function(req, res){
var token = jwt.sign({
expiresIn: 30,
}, 'overwatch');
Session.create({
username: req.body.username,
platform: req.body.platform,
lookingFor: req.body.lookingFor,
microphone: req.body.microphone,
gameMode: req.body.gameMode,
comments: req.body.comments,
avatar: null,
level: null,
hash: token,
competitiveRank: null,
region: req.body.region,
visible: true,
}).exec(function(err, created){
Session.publishCreate(created);
if(err) {
console.log(err);
return res.send({
error: err,
message: 'Something went wrong when adding a session',
code: 91
})
}
if(req.isSocket){
Session.watch(req.socket);
console.log('User subscribed to ' + req.socket.id);
}
return res.send({
session: created,
code: 00,
})
});
},
Both of the Sails functions are called using POST/GET.
I'm completely stumped as where to go with this, and it seems to documentation or explanation on how to get this working is limited. All the Sails documentation on Sockets seems to relate to using Sails as a front-end and server
OK so I managed to solve this:
Simply put:
Within React, I had to include https://github.com/balderdashy/sails.io.js/tree/master
Then within my React component I did:
componentDidMount =()=> {
io.socket.get('/listSessions',(resData, jwres) => {
console.log('test');
this.setState({
sessions: resData.sessions,
loaded: true,
})
})
io.socket.on('session', (event) => {
if(event.verb == 'created') {
let sessions = this.state.sessions;
sessions.push(event.data);
this.setState({
sessions: sessions
})
} else {
console.log('nah');
}
});
}
This makes a virtual get request to Sails using Socket.io, and sets the response in state. It also watches for updates to the 'session' connection and updates the state with these updates meaning I can update a list in real time
Within my Sails controller I have:
listSessions: function(req, res) {
if(req.isSocket){
Session.find({ where: {visible: true}, sort: 'createdAt DESC'},
function(err, sessions){
Session.watch(req.socket);
if(err) return res.json(500, {
error: err,
message: 'Something went wrong when finding trades'
});
return res.json(200, {
sessions: sessions,
});
})
}
},
The Session.watch line listens for updates via publishCreate on the model which is found in my model as follows:
afterCreate: function(message, next) {
Session.publishCreate(message);
next();
},
Adding to answer by #K20GH , add the following to my "index.js" in React to help get sails.io.js from the CDN :
const fetchJsFromCDN = (src, externals = []) => {
return new Promise((resolve, reject) => {
const script = document.createElement('script');
script.setAttribute('src', src);
script.addEventListener('load', () => {
resolve(
externals.map(key => {
const ext = window[key];
typeof ext === 'undefined' &&
console.warn(`No external named '${key}' in window`);
return ext;
})
);
});
script.addEventListener('error', reject);
document.body.appendChild(script);
});
};
fetchJsFromCDN(
'https://cdnjs.cloudflare.com/ajax/libs/sails.io.js/1.0.1/sails.io.min.js',
['io']
).then(([io]) => {
if (process.env.NODE_ENV === 'development') {
io.sails.url = 'http://localhost:1337';
}
});
Once you have this, you'll be able to use the HTTP type GET, PUT, POST and DELETE methods. So here you can do:
componentDidMount =()=> {
io.socket.get('/listSessions',(resData, jwres) => {
console.log('test');
this.setState({
sessions: resData.sessions,
loaded: true,
})
})
io.socket.on('session', (event) => {
if(event.verb == 'created') {
let sessions = this.state.sessions;
sessions.push(event.data);
this.setState({
sessions: sessions
})
} else {
console.log('Not created session');
}
});
}
And you can do the required setup in sails for the models of sessions as suggested above
I've been all over SO and Sailsjs.org trying to figure out what's going wrong, and to no avail. Just trying to learn the basics of SailsJS. I have a UserController, whose create() method gets called when a POST request is sent to /user.
create: function (req, res) {
var params = req.params.all();
User.create({
name: params.FirstName + ' ' + params.LastName,
email: params.Email,
password: params.Password,
jobTitle: params.JobTitle
}).exec(function createCB(err,created)
{
created.save(function(err)
{
// No error . . . still nothing in db
});
return res.json({name: created.name, jobTitle: created.jobTitle, email: created.email, password: created.password});
});
}
No errors here. All the request params are coming in fine and going back to the client without trouble. But nothing is actually being written to the database.
In development.js:
connections: {
mongo: {
adapter: 'sails-mongo',
host: 'localhost',
port: 27017,
// user: 'username',
// password: 'password',
database: 'sails_test'
}
},
models: {
connection: 'mongo'
}
I've tried this with the above both there in development.js, as well as separately in connections.js and models.js, respectively. No difference.
In User.js:
attributes: {
FirstName : { type: 'string' },
LastName : { type: 'string' },
Email : { type: 'string' },
Password : { type: 'string' },
JobTitle : { type: 'string' }
}
My front end request:
$.ajax({
method: 'post',
url: '/user',
data: {
FirstName: 'Yo',
LastName: 'Momma',
Email: 'yourmom#yourdadshouse.com',
Password: 'YouWish123',
JobTitle: 'Home Maker Extraordinaire'
},
success: function (sailsResponse)
{
$('#result').html(sailsResponse).fadeIn();
},
error: function()
{
console.log('error');
}
});
Again, none of this is producing an explicit error. There is just nothing being inserted into the database. Or if there is, I don't know how to find it. I've confirmed the existence of this db in the mongo shell, thusly:
show dbs
My db, sails_test shows up in the list. And I've confirmed that there isn't anything in it like so:
db.sails_test.find()
I would very much appreciate some guidance here :)
Update:
Turns out the data is being written just fine. I'm just unable to query the database from the command line. I confirmed this by first creating a sample user, and then using Waterline's findOne() method:
User.findOne({FirstName: params.FirstName}).exec(function (err, user) {
if (err) {
res.send(400);
} else if (user) {
return res.json({firstName: user.FirstName, lastName: user.LastName, jobTitle: user.JobTitle, email: user.Email, password: user.Password});
} else {
return res.send('no users match those criteria');
}
});
The above works as expected. So my problem now is simply that I cannot interact with the database from the command line. db.<collectionName>.find({}) produces nothing.
This was simply a failure to understand the MongoDb docs. I read db.collection.find({}) as DatabaseName.CollectionName.find({}), when you literally need to use db. So if my database is Test, and my collection is Users, the query is use Test, and then db.Users.find({}).
Also of note, 3T Mongo Chef is a pretty rockin' GUI (graphical user interface) for nosql databases, and it's free for non-commercial use.