Exceeded timeout of 5000 ms for a test with Jest and MongoDB - mongodb

I'm trying to implement database population by using a migration function. The code works perfectly, it saves all the data into the database, but the test for the function is failing, and now I would like to know why?
I'm getting the "Exceeded timeout of 5000 ms" error for this particular test. I've written 166 tests for this app and all of them are passing.
Here is the function I want to test:
const doMigration = async ({ model, data }) => {
await model.collection.insertMany(data)
}
And here is the test:
const { Amodel } = require('../../../models/Amodel')
const { doMigration } = require('../../../database/migrations')
describe('Database Population', () => {
it ('Should populate the database using migrations', async () => {
const data = [{ name: 'A' }, { name: 'B' }]
const model = Amodel
const migration = { name: 'Amodel', model, data }
await doMigration(migration)
const countAfter = await Amodel.count()
expect(countAfter).toBe(2)
})
})
In this test I simply import the function, the model and create a migration object that then is passed to the function.
What did I try?
Tried using just the countAfter without using the doMigration function, and it still generates the same timeout error.
Tried increasing the time for this test to 30000, failed with error saying that the mongodb time exceeded the 10000 ms.
Here is the github repository: https://github.com/Elvissamir/Fullrvmovies
What is happening, how can I solve this error?

The problem was the way the mongodb connection was handled. When testing, the app created a connection to the db on startup, and then the jest tests used that connection, that caused some issues.
The solution was to connect to the database on startup only if the environment is set to testing, otherwise the connection will be handled by each set of tests.
In each set I added a beforeAll and afterAll to open and close the connection to the database.
Hope it helps anyone that finds the same problem or has similar issues.

The orientation is that the message reflect the actual reason, So i recommand to follow the following steps:
use the following code to check mongo state:
const { MongoMemoryServer } = require("mongodb-memory-server");
const mongoose = require("mongoose");
(async () => {
mongod = await MongoMemoryServer.create();
const mongoUri = mongod.getUri();
await mongoose.connect(mongoUri, {
useNewUrlParser: true,
useUnifiedTopology: true,
}).then((result) => {
console.log(result.connection.readyState)
console.log(result.connection.host)
}).catch((err) => {
});;
})();
if you are using mongodb-memory-server add "testTimeout" attribute:
"jest": {
"preset": "ts-jest",
"testEnvironment": "node",
"setupFilesAfterEnv": [
"./src/test/setup.ts"
],
"testTimeout": 15000
},
If all above still huppens check the time-out of all inter-test operation

Related

Minimal Sveltekit + pg integration fails with "status" error

I'm trying to get Postgres working with sveltekit and a very minimal example is giving me issues. This is probably a configuration thing but the error I'm getting back from sveltekit makes no sense to me.
I start by installing a new project:
npm create svelte#latest my-testapp
Then I install "pg" to get Postgres pooling:
npm i pg
Then I add a page under src/lib/db.js:
import { Client, Pool } from 'pg';
const pool = new Pool({
user: 'xxx',
host: 'xxx',
database: 'xxx',
password: 'xxx',
port: 5432,
})
export const connectToDB = async () => await pool.connect();
Finally I add src/hooks.server.js to give me access to the pool within routes:
import { connectToDB } from '$lib/db';
export const handle = async ({event, resolve}) => {
const dbconn = await connectToDB();
event.locals = { dbconn };
const response = await resolve(event);
dbconn.release();
}
The server fails to compile with a couple of these errors:
Cannot read properties of undefined (reading 'status')
TypeError: Cannot read properties of undefined (reading 'status')
at respond (file:///C:/Users/user/code/svelte/my-testapp/node_modules/#sveltejs/kit/src/runtime/server/index.js:314:16)
at async file:///C:/Users/user/code/svelte/my-testapp/node_modules/#sveltejs/kit/src/exports/vite/dev/index.js:406:22
Not sure where "status" is coming from, seems to be part of the initial scaffolding. Any help appreciated.
Also - if there is a more straightforward way to integrate pg with sveltekit then I'm happy to hear about it. Thanks
My bad - the hooks function wasn't returning the response.
Hooks.server.js should read:
import { connectToDB } from '$lib/db';
export const handle = async ({event, resolve}) => {
const dbconn = await connectToDB();
event.locals = { dbconn };
const response = await resolve(event);
dbconn.release();
return response
}

How to solve Vercel 500 Internal Server Error?

I have created a project that uses MongoDB to store user info and Next-Auth to authenticate users. On local host this is all working seamlessly. Previously I had a couple errors with my next-auth config, but that seems to be working fine now on Vercel live site. Once the user logs in they are redirected to "my-project/suggestions". On this page I am using getServerSideProps to identify if there is a valid session token. If so, data is pulled from a local json file.
On the live site, when the user logs in, the page is redirected to "/suggestions", yet I am receiving an 500 Internal Server Error page. On the function logs I am getting this error message:
[GET] /_next/data/KpsnuV9k44lUAhQ-0rK-B/suggestions.json
10:10:57:12
2022-05-05T14:10:59.270Z 5b7a7375-045f-4518-864b-7968c3c9385f ERROR [Error: ENOENT: no such file or directory, open '/var/task/public/data/data.json'] {
errno: -2,
syscall: 'open',
path: '/var/task/public/data/data.json',
page: '/suggestions'
}
RequestId: 5b7a7375-045f-4518-864b-7968c3c9385f Error: Runtime exited with error: exit status 1
Runtime.ExitError
This is my first project using MongoDB and Next-Auth.. not so sure what the issue is in this case. In my .env.local file I only have these two variables:
NEXTAUTH_SECRET="MUNKNATION"
NEXTAUTH_URL=http://localhost:3000
How I am pulling the data on local host:
export const getServerSideProps = async (context) => {
const session = await getSession({ req: context.req });
if (!session) {
return {
redirect: {
destination: "/",
permanent: false,
},
};
} else {
let filePath = path.join(process.cwd(), "public", "data", "data.json");
let jsonData = await fs.readFile(filePath);
const data = JSON.parse(jsonData);
const inProgressStatusData = data.productRequests.filter(
(item) => item.status == "in-progress"
);
const liveStatusData = data.productRequests.filter(
(item) => item.status == "live"
);
const plannedStatusData = data.productRequests.filter(
(item) => item.status == "planned"
);
let filterData = filteredData(data, "suggestion");
let feedbackData = {
suggestions: filterData,
progress: inProgressStatusData,
planned: plannedStatusData,
live: liveStatusData,
};
return {
props: { session, feedbackData },
};
}
};
Folder structure:
A simple solution to this problem would be to, inside of your getServerSideProps, instead of calling readFile use readFileSync as follows:
export const getServerSideProps = async (context) => {
...
const file = readFileSync(
join(process.cwd(), "public", "data", "data.json"),
"utf8"
);
const data = JSON.parse(fileData);
...
I have tested this solution with Vercel and it works correctly, in development and production mode.

How to enforce SLONIK to close open pools

THE PROBLEM
I have a NodeJs server where I'm using jest for testing. In case of integration tests i get a message after all tests where passing:
"Jest has detected the following X (15-20) open handles potentially keeping Jest from exiting"
I knew what this means, have already seen it when i was using Sequelize as ORM, but now I'm using Slonik.
I found this topic what was really useful:
https://github.com/gajus/slonik/issues/63
so when i set the idleTimeOut as advised it is solved.
test("foo", async () => {
const slonik = createPool(
`postgres://postgres:password#127.0.0.1:7002/postgres`,
{
maximumPoolSize: 1,
minimumPoolSize: 1,
idleTimeout: 1 // milliseconds!
}
);
await slonik.many(
sql`SELECT table_name FROM information_schema.tables WHERE table_schema='public';`
);
});
I tried to solve this problem from an other perspective. My ide was to close the connection in the afterAll block of jest. The test setup is:
const db = container.resolve(DbConnectionPool).connection;
let appServer;
let api;
beforeEach(() => {
appServer = app.listen(config.port);
api = supertest(appServer);
});
afterEach(async () => {
appServer.close();
await db.query(sql`TRUNCATE reports`);
});
What i have tried in the afterAll block:
afterAll(async () => {
await db.end()
});
It does not solves my problem as the documentation tells:
'Note: pool.end() does not terminate active connections/ transactions.'
I do not found anything about how to enforce the close of a pool until now.
So thought i can be tricky and i will close the connection using SQL:
afterAll(async () => {
await db.query(sql`DISCONNECT ALL`);
});
Does not work as well.
I still had an idea to play with. Slonik documentation tells, that the default idleTimeOut for a connection is 5000ms in default.
So i have tried to set a timeout in the afterAll block with 6000 ms, but i still get the warning from Jest.
So does anyone have any idea how to force-close the connection for my tests?

ConnectionError: Connection lost - read ECONNRESET in protractor

I am using protractor 52.2 and cucumber 3.2.2. I am using selenium grid(selenium-server-standalone-3.14.0.jar) with the protractor and running my script in 4 browsers of 4 different nodes. I have a table of 600 rows in the DB. Initially, I am accessing data from this table and entering the data of each row through my protractor script and updating the DB column after successful entering of each row. But after entering some rows successfully, protractor script abruptly ends with error "ConnectionError: Connection lost - read ECONNRESET in protractor".And I am getting an error message in update SQL query, that "RequestError: Resource ID: 1. The request limit for the database is 60 and has been reached. See 'http://go.microsoft.com/fwlink/?LinkId=267637' for assistance." The update query which I am using is given below(i am using azure sql). I am not getting a clear idea of how to solve this. Thanks in advance.
var Connection = require('tedious').Connection;
var Request = require('tedious').Request;
var config =
{
userName: 'xxx',
password: 'xxxxx',
server: 'xxxxxx',
options:
{
database: 'xxx' ,
encrypt: true,
rowCollectionOnRequestCompletion: true
}
}
var connection = new Connection(config);
defineSupportCode(function ({ setDefaultTimeout, Given, When, Then }) {
setDefaultTimeout(30000 * 1000);
function updatedb(LPAID){
request = new Request("UPDATE COM_Location_Post with (rowlock) SET IsPublished = 1 WHERE Id ="+LPAID,function(err,rowCount, rows) {
if(err){
console.log(err)
}
});
connection.execSql(request);
}
});
You did not use connection close in your script.
By your question its clear after max instance you face this problem.
Try closing your connection every time for each transaction.
(async () => {
const config = {
user: 'User',
password: 'iPg$',
server: 'cp-sql',
database: 'DBI',
options: {
encrypt: true // Use this if you're on Windows Azure
}
}
try {
let pool = await sql.connect(config)
var envcode, testcode;
let result1 = await pool.request()
.query(`query 1 goes here`)
// console.dir(result1)
pool.close();
sql.close();
let pool1 = await sql.connect(config)
let result2 = await pool1.request()
.query(`query 2 goes here`)
// console.dir(result2)
pool1.close();
sql.close();
resolve(result2);
} catch (err) {
console.log(err)
}
})()

Reliably reconnect to MongoDB

UPDATE: I am using the 2.1 version on the driver, against 3.2
I have a node application that uses MongoDB. The problem I have is that if the MongoDB server goes down for any reason, the application doesn't reconnect.
To get this right, I based my tests on the code in this official tutorial.
var MongoClient = require('mongodb').MongoClient
, f = require('util').format;
MongoClient.connect('mongodb://localhost:27017/test',
// Optional: uncomment if necessary
// { db: { bufferMaxEntries: 3 } },
function(err, db) {
var col = db.collection('t');
setInterval(function() {
col.insert({a:1}, function(err, r) {
console.log("insert")
console.log(err)
col.findOne({}, function(err, doc) {
console.log("findOne")
console.log(err)
});
})
}, 1000)
});
The idea is to run this script, and then stop mongod, and then restart it.
So, here we go:
TEST 1: stopping mongod for 10 seconds
Stopping MongoDb for 10 seconds does the desired result: it will stop running the queries for those 10 seconds, and then will run all of them once the server is back ip
TEST 2: stopping mongod for 30 seconds
After exactly 30 seconds, I start getting:
{ [MongoError: topology was destroyed] name: 'MongoError', message: 'topology was destroyed' }
insert
{ [MongoError: topology was destroyed] name: 'MongoError', message: 'topology was destroyed' }
The trouble is that from this on, when I restart mongod, the connection is not re-establised.
Solutions?
Does this problem have a solution? If so, do you know what it is?
Once my app starts puking "topology was destroyed", the only way to get everything to work again is by restarting the whole app...
There are 2 connection options that control how mongo nodejs driver reconnects after connection fails
reconnectTries: attempt to reconnect #times (default 30 times)
reconnectInterval: Server will wait # milliseconds between retries
(default 1000 ms)
reference on mongo driver docs
Which means that mongo will keep trying to connect 30 times by default and wait 1 second before every retry. Which is why you start seeing errors after 30 seconds.
You should tweak these 2 parameters based on you needs like this sample.
var MongoClient = require('mongodb').MongoClient,
f = require('util').format;
MongoClient.connect('mongodb://localhost:27017/test',
{
// retry to connect for 60 times
reconnectTries: 60,
// wait 1 second before retrying
reconnectInterval: 1000
},
function(err, db) {
var col = db.collection('t');
setInterval(function() {
col.insert({
a: 1
}, function(err, r) {
console.log("insert")
console.log(err)
col.findOne({}, function(err, doc) {
console.log("findOne")
console.log(err)
});
})
}, 1000)
});
This will try 60 times instead of the default 30, which means that you'll start seeing errors after 60 seconds when it stops trying to reconnect.
Sidenote: if you want to prevent the app/request from waiting until the expiration of the reconnection period you have to pass the option bufferMaxEntries: 0. The price for this is that requests are also aborted during short network interruptions.
package.json: "mongodb": "3.1.3"
Reconnect existing connections
To fine-tune the reconnect configuration for pre-established connections, you can modify the reconnectTries/reconnectInterval options (default values and further documentation here).
Reconnect initial connection
For the initial connection, the mongo client does not reconnect if it encounters an error (see below). I believe it should, but in the meantime, I've created the following workaround using the promise-retry library (which uses an exponential backoff strategy).
const promiseRetry = require('promise-retry')
const MongoClient = require('mongodb').MongoClient
const options = {
useNewUrlParser: true,
reconnectTries: 60,
reconnectInterval: 1000,
poolSize: 10,
bufferMaxEntries: 0
}
const promiseRetryOptions = {
retries: options.reconnectTries,
factor: 1.5,
minTimeout: options.reconnectInterval,
maxTimeout: 5000
}
const connect = (url) => {
return promiseRetry((retry, number) => {
console.log(`MongoClient connecting to ${url} - retry number: ${number}`)
return MongoClient.connect(url, options).catch(retry)
}, promiseRetryOptions)
}
module.exports = { connect }
Mongo Initial Connect Error: failed to connect to server [db:27017] on first connect
By default the Mongo driver will try to reconnect 30 times, one every second. After that it will not try to reconnect again.
You can set the number of retries to Number.MAX_VALUE to keep it reconnecting "almost forever":
var connection = "mongodb://127.0.0.1:27017/db";
MongoClient.connect(connection, {
server : {
reconnectTries : Number.MAX_VALUE,
autoReconnect : true
}
}, function (err, db) {
});
With mongodb driver 3.1.10, you can set up your connection as
MongoClient.connect(connectionUrl, {
reconnectInterval: 10000, // wait for 10 seconds before retry
reconnectTries: Number.MAX_VALUE, // retry forever
}, function(err, res) {
console.log('connected')
})
You do not have to specify autoReconnect: true as that's the default.
It's happening because it might have crossed the retry connection limit. After number of retries it destroy the TCP connection and become idle. So for it increase the number of retries and it would be better if you increase the gap between connection retry.
Use below options:
retryMiliSeconds {Number, default:5000}, number of milliseconds between retries.
numberOfRetries {Number, default:5}, number of retries off connection.
For more details refer to this link https://mongodb.github.io/node-mongodb-native/driver-articles/mongoclient.html
Solution:
MongoClient.connect("mongodb://localhost:27017/integration_test_?", {
db: {
native_parser: false,
retryMiliSeconds: 100000,
numberOfRetries: 100
},
server: {
socketOptions: {
connectTimeoutMS: 500
}
}
}, callback)
Behavior may differ with different versions of driver. You should mention your driver version.
driver version : 2.2.10 (latest)
mongo db version : 3.0.7
Below code will extend the time mongod can take to come back up.
var MongoClient = require('mongodb').MongoClient
, f = require('util').format;
function connectCallback(err, db) {
var col = db.collection('t');
setInterval(function() {
col.insert({a:1}, function(err, r) {
console.log("insert")
console.log(err)
col.findOne({}, function(err, doc) {
console.log("findOne")
console.log(err)
});
})
}, 1000)
}
var options = { server: { reconnectTries: 2000,reconnectInterval: 1000 }}
MongoClient.connect('mongodb://localhost:27017/test',options,connectCallback);
2nd argument can be used to pass server options.
If you was using Mongoose for your Schemas, it would be worth considering my option below since mongoose was never retrying to reconnect to mongoDB implicitly after first attempt failed.
Kindly note I am connecting to Azure CosmosDB for MongoDB API. On yours maybe on the local machine.
Below is my code.
const mongoose = require('mongoose');
// set the global useNewUrlParser option to turn on useNewUrlParser for every connection by default.
mongoose.set('useNewUrlParser', true);
// In order to use `findOneAndUpdate()` and `findOneAndDelete()`
mongoose.set('useFindAndModify', false);
async function mongoDbPool() {
// Closure.
return function connectWithRetry() {
// All the variables and functions in here will Persist in Scope.
const COSMODDBUSER = process.env.COSMODDBUSER;
const COSMOSDBPASSWORD = process.env.COSMOSDBPASSWORD;
const COSMOSDBCONNSTR = process.env.COSMOSDBCONNSTR;
var dbAuth = {
auth: {
user: COSMODDBUSER,
password: COSMOSDBPASSWORD
}
};
const mongoUrl = COSMOSDBCONNSTR + '?ssl=true&replicaSet=globaldb';
return mongoose.connect(mongoUrl, dbAuth, (err) => {
if (err) {
console.error('Failed to connect to mongo - retrying in 5 sec');
console.error(err);
setTimeout(connectWithRetry, 5000);
} else {
console.log(`Connected to Azure CosmosDB for MongoDB API.`);
}
});
};}
You may decide to export and reuse this module everywhere you need to connect to db via Dependency Injection. But instead I will only show how to access the database connection for now.
(async () => {
var dbPools = await Promise.all([mongoDbPool()]);
var mongoDbInstance = await dbPools[0]();
// Now use "mongoDbInstance" to do what you need.
})();