SailsJS Mongodb timeout - mongodb

I have this controller code in my sails app:
let userId = req.body.userId;
User.findOne({ id: userId })
.then((user) => {
console.log('User found:', user);
return res.ok('It worked!');
}).catch((err) => {
sails.log.error('indexes - error', err);
return res.badRequest(err);
});
When I start my server it works, but then after some times (~5min) it stops working and I end up having the following error message:
web_1 | Sending 400 ("Bad Request") response:
web_1 | Error (E_UNKNOWN) :: Encountered an unexpected error
web_1 | MongoError: server 13.81.244.244:27017 received an error {"name":"MongoError","message":"read ETIMEDOUT"}
web_1 | at null.<anonymous> (/myapp/node_modules/sails-mongo/node_modules/mongodb/node_modules/mongodb-core/lib/topologies/server.js:213:40)
web_1 | at g (events.js:260:16)
web_1 | at emitTwo (events.js:87:13)
web_1 | at emit (events.js:172:7)
web_1 | at null.<anonymous> (/myapp/node_modules/sails-mongo/node_modules/mongodb/node_modules/mongodb-core/lib/connection/pool.js:119:12)
web_1 | at g (events.js:260:16)
web_1 | at emitTwo (events.js:87:13)
web_1 | at emit (events.js:172:7)
web_1 | at Socket.<anonymous> (/myapp/node_modules/sails-mongo/node_modules/mongodb/node_modules/mongodb-core/lib/connection/connection.js:154:93)
web_1 | at Socket.g (events.js:260:16)
web_1 | at emitOne (events.js:77:13)
web_1 | at Socket.emit (events.js:169:7)
web_1 | at emitErrorNT (net.js:1269:8)
web_1 | at nextTickCallbackWith2Args (node.js:511:9)
web_1 | at process._tickDomainCallback (node.js:466:17)
web_1 |
web_1 | Details: MongoError: server 13.81.244.244:27017 received an error {"name":"MongoError","message":"read ETIMEDOUT"}
The DB is still up at this point, and looking at the logs, everything seems fine on this side:
2017-03-23T16:45:51.664+0000 I NETWORK [thread1] connection accepted from 13.81.243.59:51558 #7811 (39 connections now open)
2017-03-23T16:45:51.664+0000 I NETWORK [conn7811] received client metadata from 13.81.253.59:51558 conn7811: { driver: { name: "nodejs", version: "2.2.25" }, os: { type: "Linux", name: "linux", architecture: "x64", version: "4.4.0-62-generic" }, platform: "Node.js v4.7.3, LE, mongodb-core: 2.1.9" }
2017-03-23T16:45:51.723+0000 I ACCESS [conn7811] Successfully authenticated as principal username on dbname
My connections.js hook looks like this:
module.exports.connections = {
sailsmongo: {
adapter : 'sails-mongo',
host : process.env.MONGODB_HOST,
port : 27017,
user : process.env.MONGODB_USERNAME,
password : process.env.MONGODB_PASSWORD,
database : process.env.MONGODB_DBNAME
},
}
and in package.json:
"sails": "~0.12.4",
"sails-mongo": "^0.12.1",
Notes:
Among the unconfirmed possible sources of misbehavior, I see:
the app is dockerized
I have a query that takes quite some time (~1/2min) and call several child processes, so I'd suspect some memory leak out there, though this is still unconfirmed!
Any idea on this?
EDIT:
After some digging, I have the impression, looking at the DB logs that sails/waterline opens a new connection on each query while it should be plugged once and kept alive. This would be the cause of the issue.
From this, I decided to try along Mongoose, and bingo, in Mongoose it works like a charm.
I'm guessing this is a Sails/Waterline bug occurring then, though I'm not clear on how to reproduce this correctly.
Anyway, I'm now moving my app from Waterline to Mongoose.

Related

Strapi v4 PostgreSQL connection from local project to azure db

Trying to connect to an Azure PostgreSQL server from my local Strapi project (eventually deployed in a docker container).
I have the connection configured according to the strapi docs, including using the cert downloaded from the azure portal:
https://docs.strapi.io/developer-docs/latest/setup-deployment-guides/configurations/required/databases.html#configuration-structure
my database.js file:
// PostgreSQL
////////////////////////////////////////
const fs = require("#strapi/strapi/lib/services/fs");
const parse = require("pg-connection-string").parse;
const db = parse("azure-conn-string");
module.exports = ({ env }) => ({
connection: {
client: "postgres",
connection: {
host: db.host,
port: db.port,
database: db.database,
user: db.user,
password: db.password,
ssl: {
ca: fs.readFileSync(`${__dirname}/db.crt.pem`).toString(),
},
},
},
});
when starting the server i get this in the terminal:
error: no pg_hba.conf entry for host "ip-address", user "username", database "db_name", SSL off
at Parser.parseErrorMessage (/Users/x/x/dockertest/node_modules/pg-protocol/dist/parser.js:287:98)
at Parser.handlePacket (/Users/x/x/dockertest/node_modules/pg-protocol/dist/parser.js:126:29)
at Parser.parse (/Users/x/x/dockertest/node_modules/pg-protocol/dist/parser.js:39:38)
at Socket.<anonymous> (/Users/x/x/dockertest/node_modules/pg-protocol/dist/index.js:11:42)
at Socket.emit (events.js:400:28)
at Socket.emit (domain.js:475:12)
at addChunk (internal/streams/readable.js:293:12)
at readableAddChunk (internal/streams/readable.js:267:9)
at Socket.Readable.push (internal/streams/readable.js:206:10)
at TCP.onStreamRead (internal/stream_base_commons.js:188:23)
It seems like its trying to connect with ssl off, I'm not sure why since it should be configured to be enabled. If I remove the ssl rule from the azure database, the connection will go through so it seems like the problem is something with the ssl config.
Can anyone help?
module.exports = ({ env }) => ({
defaultConnection: "default",
connection: {
client: "postgres",
connection: {
host: env("DATABASE_HOST", "localhost"),
port: env.int("DATABASE_PORT", 5432),
database: env("DATABASE_NAME", "postgres"),
user: env("DATABASE_USER", "postgres"),
password: env("DATABASE_PASSWORD", "0000"),
schema: env("DATABASE_SCHEMA", "public"),
},
}
});

Can a Mongo-Express container connect to MongoDB with TLS?

I have a mongo container, started with the requireTLS TLS mode, and a mongo-express container. Mongo-express does not seem to manage to connect to mongo using TLS.
My docker-compose.yml:
version: '3.1'
services:
mongodb1:
image : "mongo:4.2"
container_name : "mongodb-001"
ports:
- '27017:27017'
environment:
MONGO_INITDB_ROOT_USERNAME : "admin"
MONGO_INITDB_ROOT_PASSWORD : "adminpasswd"
volumes:
- "./mongo-data:/data/db"
- "./etc_mongod.conf:/etc/mongod.conf"
- "./certificates:/etc/certificates:ro"
command:
- "--tlsMode"
- "preferTLS"
- "--tlsDisabledProtocols"
- "none"
- "--tlsCertificateKeyFile"
- "/etc/certificates/certificateKey.pem"
- "--tlsCAFile"
- "/etc/certificates/CA.crt"
- "--tlsAllowConnectionsWithoutCertificates"
mongo-express:
image : "mongo-express:latest"
container_name : "mongo-express-001"
ports:
- '8081:8081'
depends_on:
- mongodb1
volumes:
- "./certificates/CA.crt:/etc/certificates/CA.crt:ro"
environment:
ME_CONFIG_MONGODB_SERVER: "mongodb-001"
ME_CONFIG_MONGODB_PORT: "27017"
ME_CONFIG_MONGODB_ENABLE_ADMIN: "false"
ME_CONFIG_MONGODB_AUTH_DATABASE: "admin"
ME_CONFIG_MONGODB_AUTH_USERNAME: "admin"
ME_CONFIG_MONGODB_AUTH_PASSWORD: "adminpasswd"
ME_CONFIG_MONGODB_ADMINUSERNAME: "admin"
ME_CONFIG_MONGODB_ADMINPASSWORD: "adminpasswd"
ME_CONFIG_SITE_SSL_ENABLED: "true"
ME_CONFIG_MONGODB_CA_FILE: "/etc/certificates/CA.crt"
...and the error message I get:
mongodb-001 | 2020-10-09T14:16:13.299+0000 I NETWORK [listener] connection accepted from 172.31.0.3:44774 #2 (1 connection now open)
mongodb-001 | 2020-10-09T14:16:13.305+0000 I NETWORK [conn2] Error receiving request from client: SSLHandshakeFailed: The server is configured to only allow SSL connections. Ending connection from 172.31.0.3:44774 (connection id: 2)
mongodb-001 | 2020-10-09T14:16:13.305+0000 I NETWORK [conn2] end connection 172.31.0.3:44774 (0 connections now open)
mongo-express-001 |
mongo-express-001 | /node_modules/mongodb/lib/server.js:265
mongo-express-001 | process.nextTick(function() { throw err; })
mongo-express-001 | ^
mongo-express-001 | Error [MongoError]: connection 0 to mongodb-001:27017 closed
mongo-express-001 | at Function.MongoError.create (/node_modules/mongodb-core/lib/error.js:29:11)
mongo-express-001 | at Socket.<anonymous> (/node_modules/mongodb-core/lib/connection/connection.js:200:22)
mongo-express-001 | at Object.onceWrapper (events.js:422:26)
mongo-express-001 | at Socket.emit (events.js:315:20)
mongo-express-001 | at TCP.<anonymous> (net.js:674:12)
mongo-express-001 exited with code 1
Note that:
I can connect to MongoDB using a mongo shell with the same parameters I pass to mongo-express:
mongo "mongodb://admin:adminpasswd#mongodb-001:27017/admin?authSource=admin" --tls --tlsCAFile certificates/CA.crt
If I start MongoDB in preferTLS mode, the mongo-express connection works
tl;dr
Create a new config.js file with the following code
module.exports = {
mongodb: {
connectionOptions: {
ssl: true,
}
}
};
and mount that file in your docker compose file at /node_modules/mongo-express/config.js
Explanation
It appears to be an issue with their config.default.js file. In it, they have this
module.exports = {
mongodb: {
// if a connection string options such as server/port/etc are ignored
connectionString: mongo.connectionString || getConnectionStringFromEnvVariables(),
connectionOptions: {
// ssl: connect to the server using secure SSL
ssl: process.env.ME_CONFIG_MONGODB_SSL || mongo.ssl,
// sslValidate: validate mongod server certificate against CA
sslValidate: process.env.ME_CONFIG_MONGODB_SSLVALIDATE || true,
// sslCA: array of valid CA certificates
sslCA: sslCAFromEnv ? [sslCAFromEnv] : [],
// autoReconnect: automatically reconnect if connection is lost
autoReconnect: true,
// poolSize: size of connection pool (number of connections to use)
poolSize: 4,
}
}
You'll notice the existence of an env var that's not listed (with reason) in their documentation. ME_CONFIG_MONGODB_SSL
This is required to enable tls support. Setting the env var yourself however does nothing but throw an error and break express due to it not being cast to a Boolean. So it just reads as a string 'true' or 'false'.
This code is fixed in their npm package code, but they haven't updated their docker image since late 2021. So the only "fix" I've found for this is to create a new config.js file with the following code
module.exports = {
mongodb: {
connectionOptions: {
ssl: true,
sslValidate: true,
}
}
};
Then mount this file at /node_modules/mongo-express/config.js in your docker-compose file. It'll read these and overwrite the defaults.
Note: I added the sslValidate key:value pair as well due to the fact that it suffers from the same lack of type casting. So if you omit the ME_CONFIG_MONGODB_SSLVALIDATE env var entirely, it'll be set as true, but if you include the env var as either true or false, it'll just simply break (undefined behaviour).

Connection to Mongodb fails from Strapi on Heroku

I deployed Strapi CMS to Heroku, but I get this error
Error connecting to the Mongo database. Server selection timed out after 30000 ms
Log:
2020-05-27T17:43:55.398256+00:00 heroku[web.1]: Starting process with command `npm start`
2020-05-27T17:43:58.724121+00:00 app[web.1]:
2020-05-27T17:43:58.724143+00:00 app[web.1]: > strapi-oskogen-mongodb#0.1.0 start /app
2020-05-27T17:43:58.724143+00:00 app[web.1]: > strapi start
2020-05-27T17:43:58.724143+00:00 app[web.1]:
2020-05-27T17:44:02.234160+00:00 app[web.1]: (node:23) Warning: Accessing non-existent property 'count' of module exports inside circular dependency
2020-05-27T17:44:02.234179+00:00 app[web.1]: (Use `node --trace-warnings ...` to show where the warning was created)
2020-05-27T17:44:02.234732+00:00 app[web.1]: (node:23) Warning: Accessing non-existent property 'findOne' of module exports inside circular dependency
2020-05-27T17:44:02.234879+00:00 app[web.1]: (node:23) Warning: Accessing non-existent property 'remove' of module exports inside circular dependency
2020-05-27T17:44:02.235021+00:00 app[web.1]: (node:23) Warning: Accessing non-existent property 'updateOne' of module exports inside circular dependency
2020-05-27T17:44:32.238852+00:00 app[web.1]: [2020-05-27T17:44:32.238Z] debug ⛔️ Server wasn't able to start properly.
2020-05-27T17:44:32.253150+00:00 app[web.1]: [2020-05-27T17:44:32.253Z] error Error connecting to the Mongo database. Server selection timed out after 30000 ms
My environments settings:
database.js
** server.js **
** response.js **
** config vars **
Site works well on localhost with both dev and prod environment. So it connects to MongoDB on Atlas and no problem with that.
I do not have any addons installed on Heroku.
** packages.json **
On Atlas side I opened all IPs in white list.
Any idea? Thank you! :)
My configuration for deployment of Strapi 3.0.1 on Heroku, both for develop and production environments:
module.exports = ({ env }) => ({
defaultConnection: "default",
connections: {
default: {
connector: "mongoose",
settings: {
uri: env("DATABASE_URI"),
ssl: { rejectUnauthorized: false }
},
options: {
ssl: true,
authenticationDatabase: "",
useUnifiedTopology: true,
pool: {
min: 0,
max: 10,
idleTimeoutMillis: 30000,
createTimeoutMillis: 30000,
acquireTimeoutMillis: 30000
}
},
},
},
});
server.js
module.exports = ({ env }) => ({
host: env('HOST', '0.0.0.0'),
port: env.int('PORT', 443), <-- 443 was critical to make it work on production
});
.env - local file
DATABASE_URI="mongodb+srv://OdegXXXXXUser:OXXXX20#odXXXXcluster-h9o2e.mongodb.net/odeXXXXXndb?retryWrites=true&w=majority"
HOST="0.0.0.0"
PORT="1337"
Vars on Heroku:
DATABASE_URI er identical as on localhost, the same database.
I hope it will help to anybody :-)
make this your database.json
{
"defaultConnection": "default",
"connections": {
"default": {
"connector": "mongoose",
"settings": {
"client": "mongo",
"host": "${process.env.DATABASE_HOST}",
"port": "${process.env.DATABASE_PORT}",
"database": "${process.env.DATABASE_NAME}",
"username": "${process.env.DATABASE_USERNAME}",
"password": "${process.env.DATABASE_PASSWORD}",
},
"options": {}
}
}
}
Also make sure to add package-lock.json to your git.ignore file.
then do
git add .
git commit -m "(message) "
git push heroku master
then
heroku open
Start your app servers too

Difficulty connecting to PostgreSQL#localhost using node-postgress (Error:28000)

I'm currently running openSuse on an rPi3B+ (aarch64) and have hit a wall running a NodeJS connection script.
I went through the standard install of PostgreSQL (v10 is what is offered on this version of openSuse) then created a new role with
CREATE ROLE new_role LOGIN PASSWORD 'passwd';
and then a db with
CREATE DATABASE new_db OWNER new_role;
Both the \l & \du return the expected outputs show that both the role and db have been created successfully with the correct owner.
So I then quickly created a node project directory and copied the test script from the docs: https://node-postgres.com/features/connecting
const { Pool, Client } = require('pg')
const connectionString = 'postgresql://new_role:passwd#localhost:5432/new_db'
const pool = new Pool({
connectionString: connectionString,
})
pool.query('SELECT NOW()', (err, res) => {
console.log(err, res)
pool.end()
})
const client = new Client({
connectionString: connectionString,
})
client.connect()
client.query('SELECT NOW()', (err, res) => {
console.log(err, res)
client.end()
})
This returns a few broken promise errors that I haven't caught(.cath()) correctly yet, and an error code of 28000 that looks like this:
{ error: Ident authentication failed for user "new_role"
at Connection.parseE (/home/eru/postgresDB/node_modules/pg/lib/connection.js:554:11)
at Connection.parseMessage (/home/eru/postgresDB/node_modules/pg/lib/connection.js:379:19)
at Socket.<anonymous> (/home/eru/postgresDB/node_modules/pg/lib/connection.js:119:22)
at Socket.emit (events.js:182:13)
at addChunk (_stream_readable.js:283:12)
at readableAddChunk (_stream_readable.js:264:11)
at Socket.Readable.push (_stream_readable.js:219:10)
at TCP.onStreamRead [as onread] (internal/stream_base_commons.js:94:17)
name: 'error',
length: 99,
severity: 'FATAL',
code: '28000',
detail: undefined,
hint: undefined,
position: undefined,
internalPosition: undefined,
internalQuery: undefined,
where: undefined,
schema: undefined,
table: undefined,
column: undefined,
dataType: undefined,
constraint: undefined,
file: 'auth.c',
line: '328',
routine: 'auth_failed' } undefined
So I'm pretty sure the attempt made it to the intended port otherwise I wouldn't have received the detailed error in terminal. The error code = invalid_authorization_specification
Is the there something I need to do on the server ,psql interface, that will fulfill the authorization specification?
When I've looked into that specific one I can't seem to find useful search results relevant to my situation.
Fairly new to postgres here so I'm sure this is a pretty noob mistake that I'm missing but any help or input is very appreciated!
Found an answer after a little more digging here: error: Ident authentication failed for user
Ended up editing my pg_hba.conf from the ident method to md5
This is rather crude because I don't really understand what I changed aside from telling postgreSQL to check the md5 encrypted password instead of checking if my username matched the roles created on the server.
If anyone has a proper explanation for what's changed and why I'm all ears.

Google Cloud Functions GitHub auto-deployer says: The request has errors

I have installed the GitHub auto-deployer for Google Cloud Functions, but when I now push my function into my GitHub repository, I receive an abstract error message "The request has errors" with the following rather non-descript details. What could specifically be going wrong here?
E githubAutoDeployer [CODE] 2017-12-30 19:19:37.362
Failed to create function projects/[MY_BUCKET]/locations/us-central/functions/[MY_FUNCTION] { Error: The request has errors
at Request._callback (/user_code/node_modules/googleapis/node_modules/google-auth-library/lib/transporters.js:85:15)
at Request.self.callback (/user_code/node_modules/googleapis/node_modules/request/request.js:186:22)
at emitTwo (events.js:106:13)
at Request.emit (events.js:191:7)
at Request.<anonymous> (/user_code/node_modules/googleapis/node_modules/request/request.js:1163:10)
at emitOne (events.js:96:13)
at Request.emit (events.js:188:7)
at IncomingMessage.<anonymous> (/user_code/node_modules/googleapis/node_modules/request/request.js:1085:12)
at IncomingMessage.g (events.js:292:16)
at emitNone (events.js:91:20)
code: 400,
errors:
[ { message: 'The request has errors',
domain: 'global',
reason: 'badRequest' } ] }
E githubAutoDeployer [CODE] 2017-12-30 19:19:37.363 Error: The request has errors
at Request._callback (/user_code/node_modules/googleapis/node_modules/google-auth-library/lib/transporters.js:85:15)
at Request.self.callback (/user_code/node_modules/googleapis/node_modules/request/request.js:186:22)
at emitTwo (events.js:106:13)
at Request.emit (events.js:191:7)
at Request.<anonymous> (/user_code/node_modules/googleapis/node_modules/request/request.js:1163:10)
at emitOne (events.js:96:13)
at Request.emit (events.js:188:7)
at IncomingMessage.<anonymous> (/user_code/node_modules/googleapis/node_modules/request/request.js:1085:12)
at IncomingMessage.g (events.js:292:16)
at emitNone (events.js:91:20)
D githubAutoDeployer [CODE] 2017-12-30 19:19:37.365
Function execution took 2319 ms, finished with status code: 500
UPDATE The mentioning of google-auth-library in the stack trace made me think that something may be wrong with my credentials. But the output from gcloud auth list appears alright:
Credentialed Accounts
ACTIVE ACCOUNT
* [MY_ID]#gmail.com
UPDATE What is perhaps slightly unconventional is that I have "path":"", in my config.json. But then my index.js resides directly at the top of my repository, so there is no path to specify.
UPDATE This is where the error from Google Cloud Functions is passed on by githubAutoDeployer (unfortunately source code for the upstream server is apparently not available):
gcf.projects.locations.functions.create({ resource, location }, (err, operation) => {
if (err && err.errors && err.errors[0] && err.errors[0].reason === 'alreadyExists') {
// ...
} else if (err) {
console.error(`Failed to create function ${resource.name}`, err);
reject(err);
}
The trouble was that I was specifying "location" : "us-central" instead of us-central1 (which supports Google Cloud Functions) in my config.json.
I found out by sending a raw POST request to the Google Cloud Functions API (after obtaining a service account and access token, etc.). At this level the API returns a clear error indication:
"fieldViolations": [
{
"field": "region",
"description": "region us-central is not supported."
}
]
Apparently and unfortunately this does not enter any of the log files when githubAutoDeployer attempts the same call.