THE PROBLEM
I have a NodeJs server where I'm using jest for testing. In case of integration tests i get a message after all tests where passing:
"Jest has detected the following X (15-20) open handles potentially keeping Jest from exiting"
I knew what this means, have already seen it when i was using Sequelize as ORM, but now I'm using Slonik.
I found this topic what was really useful:
https://github.com/gajus/slonik/issues/63
so when i set the idleTimeOut as advised it is solved.
test("foo", async () => {
const slonik = createPool(
`postgres://postgres:password#127.0.0.1:7002/postgres`,
{
maximumPoolSize: 1,
minimumPoolSize: 1,
idleTimeout: 1 // milliseconds!
}
);
await slonik.many(
sql`SELECT table_name FROM information_schema.tables WHERE table_schema='public';`
);
});
I tried to solve this problem from an other perspective. My ide was to close the connection in the afterAll block of jest. The test setup is:
const db = container.resolve(DbConnectionPool).connection;
let appServer;
let api;
beforeEach(() => {
appServer = app.listen(config.port);
api = supertest(appServer);
});
afterEach(async () => {
appServer.close();
await db.query(sql`TRUNCATE reports`);
});
What i have tried in the afterAll block:
afterAll(async () => {
await db.end()
});
It does not solves my problem as the documentation tells:
'Note: pool.end() does not terminate active connections/ transactions.'
I do not found anything about how to enforce the close of a pool until now.
So thought i can be tricky and i will close the connection using SQL:
afterAll(async () => {
await db.query(sql`DISCONNECT ALL`);
});
Does not work as well.
I still had an idea to play with. Slonik documentation tells, that the default idleTimeOut for a connection is 5000ms in default.
So i have tried to set a timeout in the afterAll block with 6000 ms, but i still get the warning from Jest.
So does anyone have any idea how to force-close the connection for my tests?
Related
I'm trying to implement database population by using a migration function. The code works perfectly, it saves all the data into the database, but the test for the function is failing, and now I would like to know why?
I'm getting the "Exceeded timeout of 5000 ms" error for this particular test. I've written 166 tests for this app and all of them are passing.
Here is the function I want to test:
const doMigration = async ({ model, data }) => {
await model.collection.insertMany(data)
}
And here is the test:
const { Amodel } = require('../../../models/Amodel')
const { doMigration } = require('../../../database/migrations')
describe('Database Population', () => {
it ('Should populate the database using migrations', async () => {
const data = [{ name: 'A' }, { name: 'B' }]
const model = Amodel
const migration = { name: 'Amodel', model, data }
await doMigration(migration)
const countAfter = await Amodel.count()
expect(countAfter).toBe(2)
})
})
In this test I simply import the function, the model and create a migration object that then is passed to the function.
What did I try?
Tried using just the countAfter without using the doMigration function, and it still generates the same timeout error.
Tried increasing the time for this test to 30000, failed with error saying that the mongodb time exceeded the 10000 ms.
Here is the github repository: https://github.com/Elvissamir/Fullrvmovies
What is happening, how can I solve this error?
The problem was the way the mongodb connection was handled. When testing, the app created a connection to the db on startup, and then the jest tests used that connection, that caused some issues.
The solution was to connect to the database on startup only if the environment is set to testing, otherwise the connection will be handled by each set of tests.
In each set I added a beforeAll and afterAll to open and close the connection to the database.
Hope it helps anyone that finds the same problem or has similar issues.
The orientation is that the message reflect the actual reason, So i recommand to follow the following steps:
use the following code to check mongo state:
const { MongoMemoryServer } = require("mongodb-memory-server");
const mongoose = require("mongoose");
(async () => {
mongod = await MongoMemoryServer.create();
const mongoUri = mongod.getUri();
await mongoose.connect(mongoUri, {
useNewUrlParser: true,
useUnifiedTopology: true,
}).then((result) => {
console.log(result.connection.readyState)
console.log(result.connection.host)
}).catch((err) => {
});;
})();
if you are using mongodb-memory-server add "testTimeout" attribute:
"jest": {
"preset": "ts-jest",
"testEnvironment": "node",
"setupFilesAfterEnv": [
"./src/test/setup.ts"
],
"testTimeout": 15000
},
If all above still huppens check the time-out of all inter-test operation
I am testing an app with react-testing-library and I am using a mock service worker.
All my tests pass until the last one which gives this error in the title.
When testing only the part which gives error isolated (test.only), then it doesn't throw error.
The error points to localhost:5000 which is my data server (my app is running on 3000)
This is my last test which only works when running alone:
import { findByRole, getByRole, render, screen } from '#testing-library/react'
import userEvent from '#testing-library/user-event'
import Layout from '../../layout'
describe('tests for headers and content of the table', () => {
test('to check headers in inital state', async () => {
render(<Layout />)
const headerFirstCell = await screen.findByRole('columnheader', {name: /name/i, })
expect(headerFirstCell).toHaveTextContent('Name')
const headerSecondCell = await screen.findByRole('columnheader', {name: /courses/i,}) //second
expect(headerSecondCell).toHaveTextContent('Courses')
})
It is also strange that when I run only one part of the last code + all the rest, fex:
const headerFirstCell = await screen.findByRole('columnheader', {name: /name/i, })
expect(headerFirstCell).toHaveTextContent('Name')
Or the other part + all the rest:
const headerSecondCell = await screen.findByRole('columnheader', {name: /courses/i,}) //second
expect(headerSecondCell).toHaveTextContent('Courses')
Then all tests pass. It seems like any line of code I add after that point will make the error throw.
I know it might not be easy to see from this info...but I'm lost...Any clue??
Just in case this is my setupTests.js:
import '#testing-library/jest-dom'
import { server } from './mocks/server.js'
beforeAll(() => server.listen())
afterEach(() => server.resetHandlers())
afterAll(() => server.close())
Just was a typo in one of the server url addresses in my mock service worker handlers.
So when testing a part which gets requests to that url it threw an error sometimes, in case I place that function at the end, the tests finished before doing the get request to the wrong url. That's why it had that strange behaviour.
Launch npm test, but it never end. It happens when there is a connection with a MongoDB involve. It is rare because the test works, but it never ends.
before('setupApplication', async () => {
({ app, client } = await setupApplication());
//await prepareTestData()
});
after(async () => {
//await cleanupTestData();
await app.stop();
});
it('login with a test user', async () => {
const res = await client.post('/user-accounts/login').set('urlLogin', TEST_TENANT_URL)
.send({
email: TEST_EMAIL,
password: TEST_PASS
}).expect(200);
token = res.body.token;
expect(res.body).to.ownProperty("token").and.length(EXPECTED_TOKEN_LENGTH);
console.log("Logged in with token ", res.body.token);
});
The problem: The terminal never end. Is like a thread still working. I don´t know if it is a problem with loopback or what.
Any idea? I hope somebody could help me.
Thanks.
Cross-posting my comment from the discussion in https://github.com/strongloop/loopback-next/issues/3672:
Please use Mocha option exit, it will end the process after all tests (and after/afterEach hooks) have finished. See https://mochajs.org/#configuring-mocha-nodejs to learn more. Alternatively, you can add --exit option to mocha CLI, e.g. mocha --exit src/__tests__/**/*.js.
I have been doing some research and I'm not able to find a good answer about using Knex JS within a Lambda function:
How do I use Knex with AWS Lambda? #1875
Serverless URL Shortener with Apex and AWS Lambda
Use Promise.all() in AWS lambda
Here is what I have in my index.js:
const knex = require('knex')({
client: 'pg',
connection: {...},
});
exports.handler = (event, context, callback) => {
console.log('event received: ', event);
console.log('knex connection: ', knex);
knex('goals')
.then((goals) => {
console.log('received goals: ', goals);
knex.client.destroy();
return callback(null, goals);
})
.catch((err) => {
console.log('error occurred: ', err);
knex.client.destroy();
return callback(err);
});
};
I am able to connect and execute my code fine locally, but I'm running into an interesting error when it's deployed to AWS - the first call is always successful, but anything after fails. I think this is related to the knex client being destroyed, but then trying to be used again on the next call. If I re-upload my index.js, it goes back to working for one call, and then failing.
I believe this can be resolved somehow using promises but this my first time working with Lambda so I'm not familiar with how it's managing the connection to RDS on subsequent calls. Thanks in advance for any suggestions!
For me, it worked on my local machine but not after deploying. I was kind of be mislead.
It turns out the RDS inbound source is not open to my Lambda function. Found solution at AWS Lambda can't connect to RDS instance, but I can locally?: either changing RDS inbound source to 0.0.0.0/0 or use VPC.
After updating RDS inbound source, I can use Lambda with Knex successfully.
The Lambda runtime I am using is Node.js 8.10 with packages:
knex: 0.17.0
pg: 7.11.0
The code below using async also just works for me
const Knex = require('knex');
const pg = Knex({ ... });
module.exports.submitForm = async (event) => {
const {
fields,
} = event['body-json'] || {};
return pg('surveys')
.insert(fields)
.then(() => {
return {
status: 200
};
})
.catch(err => {
return {
status: 500
};
});
};
Hopefully it will help people who might meet same issue in future.
The most reliable way of handling database connections in AWS Lambda is to connect and disconnect from the database within the invocation process itself.
In your code above, since you disconnected already after the first invocation, the second one does not have a connection anymore.
To fix it, just move your instantiation of knex.
exports.handler = (event, context, callback) => {
console.log('event received: ', event);
// Connect
const knex = require('knex')({
client: 'pg',
connection: {...},
});
console.log('knex connection: ', knex);
knex('goals')
.then((goals) => {
console.log('received goals: ', goals);
knex.client.destroy();
return callback(null, goals);
})
.catch((err) => {
console.log('error occurred: ', err);
// Disconnect
knex.client.destroy();
return callback(err);
});
};
There are ways to reuse an existing connection but success rates for that varies widely depending on database server configuration and production load.
I got the exact same issue as you said: Used destroy() in an AWS lambda function (like this: await knex.destroy() at the bottom) and suddenly all my AWS lambdas were in error.
Because I did not suspect it, I searched for hours what was causing the issue and even started to investigate using lambda + vpc + nat etc.. Turns out it's just that AWS freezes lambda in a way that if you destroy the connection, on the next handler invocation it will try to reuse the connection.
Solution: do not use .destroy() at the end of lambda and redeploy.
I have been using MLab MongoDB and mongoose library to create a db connection inside a serverless (Lambda) handler. It works smoothly on local machine. But sometimes it doesn't work after deployment.The request returns an Internal server error. The weird thing is sometimes it works. But If I remove the database connection code, the handler works. The serverless log just says Process exited before completing request. No real errors so no idea what to do.
The db connection looks like this:
handler.js
// Connect to database
mongoose.connect(process.env.DATABASE_URL, {
useMongoClient: false
}).then((ee) => {
console.log('------------------------invoke db ', ee);
})
.catch(err => console.error('-----------error db ', err));
No error in here too. Any idea what's happening?
When you get Process exited before completing request, it means that the node process has crashed before Lambda was able to call callback. If you go to Cloudwatch logs, there would be an error and stack trace of what happened.
You should connect to the MongoDB instance inside your handler and before you call callback(), disconnect first.
It would be like this...
exports.handler = (event, context, callback) => {
let response;
return mongoose.connect(process.env.DATABASE_URL, {
useMongoClient: false
}).then((ee) => {
// prepare your response
response = { hello: 'world' }
}).then(() => {
mongoose.disconnect()
}).then(() => {
// Success
callback(null, response)
}).catch((err) => {
console.error(err);
callback(err);
})
};
Here is an article explaining with details how lambda work with node and an example of how to implement DB connection.
Differently of #dashmug suggested, you should NOT disconnect your DB since connecting every time will decrease your performance.