Sequelize version: 6.13.0
Database version: PostgreSQL 13.3
Whenever there's an issue with a query, Sequelize doesn't tell me the actual issue. For example, this old Stack Overflow post shows Sequelize displaying an error that a column is unknown. In my case, I receive a generic error message and need to manually inspect the queries and my database tables to find out what's wrong. Here's an example:
const user = await User.findByPk(params.id);
const settings = await Setting.findAll({ attributes: ['id', 'key', 'defaultValue'] });
const userSettings = await user.getUserSettings({ attributes: ['id', 'key', 'value'] });
This will produce the following SQL:
Executing (default): SELECT "id", "name", "email", "password", "referralCode", "referredByUserId", "bio", "twitterUrl", "facebookUrl", "instagramUrl", "createdAt", "updatedAt" FROM "Users" AS "User" WHERE "User"."id" = '1';
Executing (default): SELECT "id", "key", "defaultValue" FROM "Settings" AS "Setting";
Executing (default): SELECT "id", "key", "value" FROM "UserSettings" AS "UserSetting" WHERE "UserSetting"."userId" = 1;
And here's the error message:
/home/user/app/backend/node_modules/sequelize/lib/dialects/postgres/query.js:76
const errForStack = new Error();
^
Error:
at Query.run (/home/user/app/backend/node_modules/sequelize/lib/dialects/postgres/query.js:76:25)
at /home/user/app/backend/node_modules/sequelize/lib/sequelize.js:642:28
at processTicksAndRejections (node:internal/process/task_queues:96:5)
at PostgresQueryInterface.select (/home/user/app/backend/node_modules/sequelize/lib/dialects/abstract/query-interface.js:979:12)
at Function.findAll (/home/user/app/backend/node_modules/sequelize/lib/model.js:1789:21)
at HasMany.get (/home/user/app/backend/node_modules/sequelize/lib/associations/has-many.js:228:21)
I had to manually check my database tables, and I found out that the key column is not present on the UserSettings table.
Why doesn't Sequelize tell me this on any of my models? Is there some configuration I'm missing?
As #Anatoly mentioned in the comments, I just needed to catch the errors in try-catch block, like this for example:
try {
const user = await User.findByPk(params.id);
const settings = await Setting.findAll({ attributes: ['id', 'key', 'defaultValue'] });
const userSettings = await user.getUserSettings({ attributes: ['id', 'key', 'value'] });
} catch (e) {
console.log(e);
next(e);
}
Coming from a Ruby on Rails background, I had expected the errors to be shown in the output of my terminal running the Express server, but that doesn't seem to be the case.
Thanks #Anatoly, debugging will be a lot easier now!
Related
I'm trying to implement database population by using a migration function. The code works perfectly, it saves all the data into the database, but the test for the function is failing, and now I would like to know why?
I'm getting the "Exceeded timeout of 5000 ms" error for this particular test. I've written 166 tests for this app and all of them are passing.
Here is the function I want to test:
const doMigration = async ({ model, data }) => {
await model.collection.insertMany(data)
}
And here is the test:
const { Amodel } = require('../../../models/Amodel')
const { doMigration } = require('../../../database/migrations')
describe('Database Population', () => {
it ('Should populate the database using migrations', async () => {
const data = [{ name: 'A' }, { name: 'B' }]
const model = Amodel
const migration = { name: 'Amodel', model, data }
await doMigration(migration)
const countAfter = await Amodel.count()
expect(countAfter).toBe(2)
})
})
In this test I simply import the function, the model and create a migration object that then is passed to the function.
What did I try?
Tried using just the countAfter without using the doMigration function, and it still generates the same timeout error.
Tried increasing the time for this test to 30000, failed with error saying that the mongodb time exceeded the 10000 ms.
Here is the github repository: https://github.com/Elvissamir/Fullrvmovies
What is happening, how can I solve this error?
The problem was the way the mongodb connection was handled. When testing, the app created a connection to the db on startup, and then the jest tests used that connection, that caused some issues.
The solution was to connect to the database on startup only if the environment is set to testing, otherwise the connection will be handled by each set of tests.
In each set I added a beforeAll and afterAll to open and close the connection to the database.
Hope it helps anyone that finds the same problem or has similar issues.
The orientation is that the message reflect the actual reason, So i recommand to follow the following steps:
use the following code to check mongo state:
const { MongoMemoryServer } = require("mongodb-memory-server");
const mongoose = require("mongoose");
(async () => {
mongod = await MongoMemoryServer.create();
const mongoUri = mongod.getUri();
await mongoose.connect(mongoUri, {
useNewUrlParser: true,
useUnifiedTopology: true,
}).then((result) => {
console.log(result.connection.readyState)
console.log(result.connection.host)
}).catch((err) => {
});;
})();
if you are using mongodb-memory-server add "testTimeout" attribute:
"jest": {
"preset": "ts-jest",
"testEnvironment": "node",
"setupFilesAfterEnv": [
"./src/test/setup.ts"
],
"testTimeout": 15000
},
If all above still huppens check the time-out of all inter-test operation
GraphQL has mutations, Postgres has INSERT; GraphQL has queries, Postgres has SELECT's; etc., etc.. I haven't found an example showing how you could use both in a project, for example passing all the queries from front end (React, Relay) in GraphQL, but to a actually store the data in Postgres.
Does anyone know what Facebook is using as DB and how it's connected with GraphQL?
Is the only option of storing data in Postgres right now to build custom "adapters" that take the GraphQL query and convert it into SQL?
GraphQL is database agnostic, so you can use whatever you normally use to interact with the database, and use the query or mutation's resolve method to call a function you've defined that will get/add something to the database.
Without Relay
Here is an example of a mutation using the promise-based Knex SQL query builder, first without Relay to get a feel for the concept. I'm going to assume that you have created a userType in your GraphQL schema that has three fields: id, username, and created: all required, and that you have a getUser function already defined which queries the database and returns a user object. In the database I also have a password column, but since I don't want that queried I leave it out of my userType.
// db.js
// take a user object and use knex to add it to the database, then return the newly
// created user from the db.
const addUser = (user) => (
knex('users')
.returning('id') // returns [id]
.insert({
username: user.username,
password: yourPasswordHashFunction(user.password),
created: Math.floor(Date.now() / 1000), // Unix time in seconds
})
.then((id) => (getUser(id[0])))
.catch((error) => (
console.log(error)
))
);
// schema.js
// the resolve function receives the query inputs as args, then you can call
// your addUser function using them
const mutationType = new GraphQLObjectType({
name: 'Mutation',
description: 'Functions to add things to the database.',
fields: () => ({
addUser: {
type: userType,
args: {
username: {
type: new GraphQLNonNull(GraphQLString),
},
password: {
type: new GraphQLNonNull(GraphQLString),
},
},
resolve: (_, args) => (
addUser({
username: args.username,
password: args.password,
})
),
},
}),
});
Since Postgres creates the id for me and I calculate the created timestamp, I don't need them in my mutation query.
The Relay Way
Using the helpers in graphql-relay and sticking pretty close to the Relay Starter Kit helped me, because it was a lot to take in all at once. Relay requires you to set up your schema in a specific way so that it can work properly, but the idea is the same: use your functions to fetch from or add to the database in the resolve methods.
One important caveat is that the Relay way expects that the object returned from getUser is an instance of a class User, so you'll have to modify getUser to accommodate that.
The final example using Relay (fromGlobalId, globalIdField, mutationWithClientMutationId, and nodeDefinitions are all from graphql-relay):
/**
* We get the node interface and field from the Relay library.
*
* The first method defines the way we resolve an ID to its object.
* The second defines the way we resolve an object to its GraphQL type.
*
* All your types will implement this nodeInterface
*/
const { nodeInterface, nodeField } = nodeDefinitions(
(globalId) => {
const { type, id } = fromGlobalId(globalId);
if (type === 'User') {
return getUser(id);
}
return null;
},
(obj) => {
if (obj instanceof User) {
return userType;
}
return null;
}
);
// a globalId is just a base64 encoding of the database id and the type
const userType = new GraphQLObjectType({
name: 'User',
description: 'A user.',
fields: () => ({
id: globalIdField('User'),
username: {
type: new GraphQLNonNull(GraphQLString),
description: 'The username the user has selected.',
},
created: {
type: GraphQLInt,
description: 'The Unix timestamp in seconds of when the user was created.',
},
}),
interfaces: [nodeInterface],
});
// The "payload" is the data that will be returned from the mutation
const userMutation = mutationWithClientMutationId({
name: 'AddUser',
inputFields: {
username: {
type: GraphQLString,
},
password: {
type: new GraphQLNonNull(GraphQLString),
},
},
outputFields: {
user: {
type: userType,
resolve: (payload) => getUser(payload.userId),
},
},
mutateAndGetPayload: ({ username, password }) =>
addUser(
{ username, password }
).then((user) => ({ userId: user.id })), // passed to resolve in outputFields
});
const mutationType = new GraphQLObjectType({
name: 'Mutation',
description: 'Functions to add things to the database.',
fields: () => ({
addUser: userMutation,
}),
});
const queryType = new GraphQLObjectType({
name: 'Query',
fields: () => ({
node: nodeField,
user: {
type: userType,
args: {
id: {
description: 'ID number of the user.',
type: new GraphQLNonNull(GraphQLID),
},
},
resolve: (root, args) => getUser(args.id),
},
}),
});
We address this problem in Join Monster, a library we recently open-sourced to automatically translate GraphQL queries to SQL based on your schema definitions.
This GraphQL Starter Kit can be used for experimenting with GraphQL.js and PostgreSQL:
https://github.com/kriasoft/graphql-starter-kit - Node.js, GraphQL.js, PostgreSQL, Babel, Flow
(disclaimer: I'm the author)
Have a look at graphql-sequelize for how to work with Postgres.
For mutations (create/update/delete) you can look at the examples in the relay repo for instance.
Postgraphile https://www.graphile.org/postgraphile/ is Open Source
Rapidly build highly customisable, lightning-fast GraphQL APIs
PostGraphile is an open-source tool to help you rapidly design and
serve a high-performance, secure, client-facing GraphQL API backed
primarily by your PostgreSQL database. Delight your customers with
incredible performance whilst maintaining full control over your data
and your database. Use our powerful plugin system to customise every
facet of your GraphQL API to your liking.
You can use an ORM like sequelize if you're using Javascript or Typeorm if you're using Typescript
Probably FB using mongodb or nosql in backend. I've recently read a blog entry which explain how to connect to mongodb. Basically, you need to build a graph model to match the data you already have in your DB. Then write resolve, reject function to tell GQL how to behave when posting a query request.
See https://www.compose.io/articles/using-graphql-with-mongodb/
Have a look at SequelizeJS which is a promise based ORM that can work with a number of dialects; PostgreSQL, MySQL, SQLite and MSSQL
The below code is pulled right from its example
const Sequelize = require('sequelize');
const sequelize = new Sequelize('database', 'username', 'password', {
host: 'localhost',
dialect: 'mysql'|'sqlite'|'postgres'|'mssql',
pool: {
max: 5,
min: 0,
acquire: 30000,
idle: 10000
},
// SQLite only
storage: 'path/to/database.sqlite',
// http://docs.sequelizejs.com/manual/tutorial/querying.html#operators
operatorsAliases: false
});
const User = sequelize.define('user', {
username: Sequelize.STRING,
birthday: Sequelize.DATE
});
sequelize.sync()
.then(() => User.create({
username: 'janedoe',
birthday: new Date(1980, 6, 20)
}))
.then(jane => {
console.log(jane.toJSON());
});
I am looking for a way to insert an object into postgreSQL db using knex.raw() method.
Here is my try, it works, but I don't know if it is a best way.
const newLink = {
url: 'http://github.com',
name: 'GitHub',
description: 'GitHub is awesome',
rel: '',
last_update: new Date()
};
const columns = Object.keys(newLink);
const values = Object.values(newLink);
console.log('columns: ', columns);
console.log('values: ', values);
const sql = `
INSERT INTO link (${columns.map(col => col).join(',')})
VALUES (${values.map(() => '?').join(',')})
`;
console.log('sql: ', sql);
try {
const insertObjResp = await knex.raw(sql, values);
console.log('insertObjResp: ', insertObjResp);
} catch (error) {
console.error(error);
}
knex give me debug information like this:
{ method: 'raw',
sql: '\n INSERT INTO link (url,name,description,rel,last_update)\n VALUES (?,?,?,?,?)\n ',
bindings:
[ 'http://github.com',
'GitHub',
'GitHub is awesome',
'',
2018-11-20T04:47:21.085Z ],
options: {},
__knexQueryUid: '304fb5e5-9864-4ec6-9a63-1051e8e3c9ad' }
Is there an knex.js official way to do this work? thanks.
Is there an knex.js official way to do this work? thanks.
Official way is to use knex.insert().
Your code seems correct and there is no better way. Arrays of values and knex.raw is a bit inconvenient to use. Of course you should wrap that monster inside a helper function at least.
Per the docs, StrongLoop doesn't support running custom sql statements.
https://docs.strongloop.com/display/public/LB/Executing+native+SQL
How anyone thinks you can build an enterprise app with just simple joins is beyond me, but I did find this post which says you can do it:
Execute raw query on MySQL Loopback Connector
But this is for MySql. When I try it with Postgres I get the error: "Invalid value for argument 'byId' of type 'object': 0. Received type was converted to number." And it returns no data. Here is my code:
module.exports = function(account) {
account.byId = function(byId, cb){
var ds=account.dataSource;
var sql = "SELECT * FROM account where id > ?";
ds.connector.execute(sql, [Number(byId)], function(err, accounts) {
if (err) console.error(err);
console.info(accounts);
cb(err, accounts);
});
};
account.remoteMethod(
'byId',
{
http: {verb: 'get'},
description: "Get accounts greater than id",
accepts: {arg: 'byId', type: 'integer'},
returns: {arg: 'data', type: ['account'], root: true}
}
);
};
For the part [Number(byId)], I've also tried [byId] and just byId. Nothing works.
Any ideas? So far I really like StrongLoop, but it looks like the Postgresql connector is not ready for production. I'll be doing a prototype with Sails next if this doesn't work. :-(
Here's the thing arg is of type 'integer' which is not a valid Loopback Type. Use `Number instead. Check the corrected code below:
module.exports = function(account) {
account.byId = function(byId, cb){
var ds = account.dataSource;
var sql = "SELECT * FROM account WHERE id > $1";
ds.connector.execute(sql, byId, function(err, accounts) {
if (err) console.error(err);
console.info(accounts);
cb(err, accounts);
});
};
account.remoteMethod(
'byId',
{
http: {verb: 'get'},
description: "Get accounts greater than id",
accepts: {arg: 'byId', type: 'Number'},
returns: {arg: 'data', type: ['account'], root: true} //here 'account' will be treated as 'Object'.
}
);
};
Note: MySQL's prepared statements natively use ? as the parameter placeholder, but PostgreSQL uses $1, $2 etc.
Hope this works for you. Else try with [byId] instead of byId as per the docs.
i am building a web service using express.js and Sequilize with a Postgres DB.
Database holds a table 'country' under schema 'schema1'. Table 'country' has fields 'name', 'isoCode'.
Created a seed file to insert a list of countries inside table 'country'.
Seed file looks like :
'use strict';
module.exports = {
up: function (queryInterface, Sequelize) {
return queryInterface.bulkInsert(
'country',
[
{
"name":"Afghanistan",
"isoCode":"AF"
},
{
"name":"Åland Islands",
"isoCode":"AX"
},
{
"name":"Albania",
"isoCode":"AL"
},
{
"name":"Algeria",
"isoCode":"DZ"
},
{
"name":"American Samoa",
"isoCode":"AS"
},
{
"name":"Andorra",
"isoCode":"AD"
}
],
{
schema : 'schema1'
}
);
},
down: function (queryInterface, Sequelize) {
}
};
While running seed i get this error :
node_modules/sequelize-cli/bin/sequelize --url postgres://user:password#localhost:5432/database db:seed
Sequelize [Node: 0.12.6, CLI: 2.0.0, ORM: 3.11.0, pg: ^4.4.2]
Parsed url postgres://user:*****#localhost:5432/database
Starting 'db:seed'...
Finished 'db:seed' after 165 ms
== 20151029161319-Countries: migrating =======
Unhandled rejection SequelizeDatabaseError: relation "country" does not exist
at Query.formatError (node_modules/sequelize/lib/dialects/postgres/query.js:437:14)
at null.<anonymous> (node_modules/sequelize/lib/dialects/postgres/query.js:112:19)
at emit (events.js:107:17)
at Query.handleError (node_modules/pg/lib/query.js:108:8)
at null.<anonymous> (node_modules/pg/lib/client.js:171:26)
at emit (events.js:107:17)
at Socket.<anonymous> (node_modules/pg/lib/connection.js:109:12)
at Socket.emit (events.js:107:17)
at readableAddChunk (_stream_readable.js:163:16)
at Socket.Readable.push (_stream_readable.js:126:10)
at TCP.onread (net.js:538:20)
I think i am stuck on this. I would appreciate any provided help / guidance etc.
Thank you for your time.
I executed SQL query on Postgres :
ALTER ROLE <username> SET search_path TO schema1,public;
as noted here : Permanently Set Postgresql Schema Path
Then, executed seeder again succesfully :
node_modules/sequelize-cli/bin/sequelize --url postgres://user:password#localhost:5432/database db:seed
Sequelize [Node: 0.12.6, CLI: 2.0.0, ORM: 3.11.0, pg: ^4.4.2]
Parsed url postgres://user:*****#localhost:5432/database
Using gulpfile node_modules/sequelize-cli/lib/gulpfile.js
Starting 'db:seed'...
Finished 'db:seed' after 558 ms
== 20151029161319-Countries: migrating =======
== 20151029161319-Countries: migrated (0.294s)
Thanks #a_horse_with_no_name for the information about search_path. I wish the sequelize library could handle this situation, or maybe i misuse it.
update :
Opened a ticket on Github (https://github.com/sequelize/sequelize/issues/4778#issuecomment-152566806) and the solution is quite simple :
instead of setting only the table as the first argument, set
{tableName: 'country', schema : 'schema1'}
You can actually specify the schema and table name via object like is explained in this Github issue:
'use strict';
module.exports = {
up: function (queryInterface, Sequelize) {
return queryInterface.bulkInsert(
{ tableName: 'account', schema: 'crm' },
{
name: 'Michael'
},
{}
);
},
down: function (queryInterface, Sequelize) {
return queryInterface.bulkDelete({ tableName: 'account', schema: 'crm' }, null, {});
}
};