For migration management, I've decided to use Prisma Migrate instead of node-pg-migrate (PostgreSQL). I followed the instructions in this link and everything works fine. However, the main challenge is that my integration tests fail when trying to run migrations on the test database (not the develop database). How can I override the configurations of my test database?
In node-pg-migrate I could simply provide configurations before running the integration test:
import migrationRunner from 'node-pg-migrate';
// --- Integration test setup:
beforeAll(async () => {
await migrationRunner({
migrationsTable: dbConfig['migrations-table'],
dir: dbConfig['migrations-dir'],
schema: dbConfig.schema,
databaseUrl: databaseURL,
// --- other configs
});
}, config.get('test').timeout);
You can override the entire connection string directly in your test setup as follows. Prisma reads the DB string from environment variables so overriding in the following manner will work fine.
Related
Problem
running out of database connections in prod leading to errors like this;
Error:
Invalid `prisma.queryRaw()` invocation:
Timed out fetching a new connection from the connection pool. More info: http://pris.ly/d/connection-pool (Current connection pool timeout: 10, connection limit: 5)
at Object.request (/usr/src/app/node_modules/#prisma/client/runtime/index.js:45629:15)
at async Proxy._request (/usr/src/app/node_modules/#prisma/client/runtime/index.js:46456:18)
Situation
multiple API containers running in Google Cloud Run running node/express/prisma API
using Supabase's hosted postgres. In supabase Settings > Database, Connection Pooling is enabled.
in db connection string, using :6543/postgres?pgbouncer=true
Attempt to diagnose
in supabase, Database > Roles, I can see a list of roles and the number of connections for each. pgBouncer has 0 and the role which my application uses has several.
If I query pg_stat_activity, I can see connections for the usename which is used by my application, and client_addr values representing ip addresses for a couple of different container instances. Are these "forwarded on" from pgBouncer? or have they bypassed pgBouncer entirely?
I am not familiar with what either of these should look like if I were using pgBouncer correctly so it's hard for me to tell what's going on.
I assume this means that I either haven't configured pgBouncer correctly, or I'm not connecting to it properly, or both. I'd be really grateful if someone could point out how I could either check or fix my connection to pgBouncer and clarify what I should see in pg_stat_activity if I was correctly connected to pgBouncer. Thanks.
Figured out what's going wrong here, so writing out how I fixed it in case anyone else runs into this issue.
Better understanding of the problem
in my prisma schema file I'm getting my database url from the env
datasource db {
provider = "postgresql"
url = env("SUPABASE_POSTGRES_URL")
}
and when I'm instantiating the prisma client I'm using the same variable
export const prisma = new PrismaClient({
datasources: {
db: {
url: process.env.SUPABASE_POSTGRES_URL,
},
},
});
I have a build trigger in Google Cloud Build that builds containers when branches that are merged into certain other branches, eg when PRs are merged in to master, build new containers and deploy them to prod.
In the build trigger, the SUPABASE_POSTGRES_URL value is set in the env, using :5432 which connects directly to postgres, bypassing pgBouncer. This is a requirement for prisma migrations which can't be run through pgBouncer.
The Google Cloud Run container env vars specify a different value for SUPABASE_POSTGRES_URL however it looks like the this not being used, and instead the direct-to-postgres :5432 value is used while the app is running, to connect to the db and run queries - so pgBouncer was permanently bypassed.
Solution
where the prisma client is instantiated, I'm using a second env var. It turns out that prisma uses the env var referenced in the schema file for the DB URL for migrations and the db url in the client instantiation for queries when the app is running, and you can happily have two completely separate values for these two URLs.
export const prisma = new PrismaClient({
datasources: {
db: {
url: process.env.SUPABASE_PGBOUNCER_URL,
},
},
});
Now, SUPABASE_POSTGRES_URL is still populated from the build trigger, but it doesn't get used at runtime; instead I set SUPABASE_PGBOUNCER_URL in the Google Cloud Run env vars and that gets used during the prisma client instantiation, so queries a run through pgBouncer.
Result
Effective Prisma migrations direct to postgres
Effective connection pooling by running queries through pgBouncer
I'm planning to create a secondary database for testing purposes. As of now I'm using Prisma's Client to perform CRUD to my secondary database. Good thing it worked fine.
My problem is the migration part. I still need to do it manually. Like editing the schema.prisma file, change the database url, and run prisma migrate dev manually.
I already browsed to Prisma's github, but I can't find an exact solution. I would like to ask here, maybe someone has an idea on how to achieve it.
Working code below.
import { PrismaClient } from '#prisma/client'
const prisma = new PrismaClient({
datasources: {
db: {
url: 'postgresql://capstone:capstone#postgres:5432/blogpost_test?schema=public'
}
}
})
export default prisma
I expect something like below, in which we can just migrate the database programmatically??
prisma.$migrate or prisma.migrate()
This is not directly supported by Prisma. There is an open feature request that you could +1 here.
Best workaround for now would be to writes scripts that use the CLI commands.
I am trying to alter my Postgres database using Knex migrations but they are not working.
My knexfile.js looks like this:
module.exports = {
development: {
client: 'pg',
connection: {
database: 'cms_dev',
},
},
};
Then I have a db.js that looks like this:
const config = require('../knexfile.js');
const env = 'development';
const knex = require('knex')(config[env]);
module.exports = knex;
knex.migrate.latest([config]);
If I am starting with a fresh database my first migration works but if I try to update the database with a new (2nd) migration it does nothing.
When I try running knex migrate:latest --env development it says:
Using environment: development
Already up to date
*****EDIT*****
I ended up generating another new (3rd) migration and pasted in the exact same code from the previous one that was being ignored and it worked. No clue why the 2nd migration silently failed and the 3rd one worked.
I figured out the issue. I had forgotten to configure nodemon to ignore my migrations folder.
Because I have this line in db.js:
knex.migrate.latest([config]);
If I saved at all while creating the migration, my server would restart and cause the latest and incomplete migration to fire off. Since migrations only run once, Knex would think the database was already up to date.
If you're just looking to re-run any migrations from the start, delete the migrations tables in the schema/database of your choice.
I am brand-new to Sails and I'm looking to build my first application in javascript land! I am currently trying to connect my sails application to a postgresql database and I want to make sure that I am doing this correctly.
I started by creating a postgresql db with dbName, userName, password. I have added all this information into my connections.js file:
somePostgresqlServer: {
adapter: 'sails-postgresql',
host: 'localhost',
user: '<username>', // optional
password: '<password>', // optional
database: '<databasename>' //optional
}
I want to be sure I can migrate and perform all operations on my own so my models.js is set to migrate: safe. I ran npm install sails-postgresql. Now, to my understanding if I have migrate set to safe I will need the sails-db-migrate module (https://github.com/building5/sails-db-migrate). I followed this module step by step. I generated a User model by running sails generate api user. After this I ran grunt db:migrate. After all this, I check my psql database and no User table has been created. I know there is something I am missing, or maybe there is a more simple way to see if my postgresql db is connected.
Any advice or suggestions on how I should approach this would be greatly appreciated.
I was able to connect my PSQL DB to my application by simply:
creating db in psql
configuring it in sails app/connections.js
running npm install sails-postgresql
Adding to config/env/development.js:
models: {
connection: 'somePostgresqlServer'
}
running sails generate api user
sails lift
When asked for migration mode I chose options 2 which is alter
Opened up psequal and sure enough there was my new user table.
I'm not sure which of these actually triggered it to working from adding my db name to development.js, leaving off a PW on my DB, or changing migration mode to alter. But these steps helped it to connect. Will look further into.
I'm learning how to set up a server on OpenShift that uses node, express, GraphQL and Postgres and I need some help.
When trying to set up the GraphQL Server I've been following this and it works on my local machine:
import Express from 'express';
import GraphQL from 'express-graphql';
import Schema from './db/GQLschema';
const app = Express();
app.use('/graphql', GraphQL({
schema: Schema,
pretty: true,
graphiql: true
}));
...
I'm using the server.js template provided by OpenShift so the above changes to:
...
self.app = express();
self.app.configure(function() {
self.app.use('/public', express.static(__dirname+'/public'));
self.app.use('/graphql', GraphQL({
schema: Schema,
pretty: true,
graphiql: true
}));
});
...
But this doesn't work when pushed up to OpenShift. I get 'Internal Server Error' instead of the graphiql interface.
I'm new to this, but here are some guesses as to what it might have to do with:
Connecting to database
Missing dependencies
Connecting to database: I have 2 json config files. One for my local machine which is connected using a port-forward SSH tunnel to OpenShift. That works fine. The other is one for production. In it I have all the strings for the database name, user, port, host, and password, that OpenShift provided me. I'm using the config library which looks at NODE_ENV to decide which config json file to use. I set the NODE_ENV variable on OpenShift to production. When I query it, it shows all the correct strings. Is there something else I need to set?
Missing dependencies: To keep it neat, I am keeping my back-end code separate from my front-end code. So the back-end code has only the express, graphql, sequelize, config, and pg stuff installed. Do I need something else to make the graphiql page work? Also, I put all the /node_modules/ in the .gitignore file -- I assumed that OpenShift installs them post push. Was that the right thing to do?
Would appreciate any help!
OK - I solved it by doing the following:
in the openshift cli tools turn on error logs
rhc tail -a <yourappname>
This will give you some clues. In my case it was a message from express-graphql "unresolved promise". After researching I found that if you install and save es-promise and add this line to the top of the server.js file:
require ('es6-promise').polyfill();
Hope that helps someone else!