In our dev environment, each dev has their own schema to replicate the stage and prod databases. Need to be able to pass user schema as part of the client config. Here's the config...
"client": {
"username": "user_login",
"password": "user_password",
"host": "db_host",
"port": 5432,
"database": "db_name",
"dialect": "pg",
"schema": "user_schema"
}
...and here's the instantiation...
var Knex = require('knex')({
client: client.dialect,
connection: {
host: client.host,
user: client.username,
password: client.password,
database: client.database,
searchPath: client.schema
});
var Bookshelf = require('bookshelf')(Knex);
Inspecting the Bookshelf and Knex objects, I can't determine anywhere which schema is being used. I had to dig into the SchemaCompiler_PG.prototype.hasTable method, where I happened to find it in an array on a query response object, which includes a property of table_schema, happily set to public. I tried to set it to my schema, but nothing succeeded.
So we set default search paths for each user login.
ALTER ROLE user_login SET search_path TO 'user_schema'
Inspecting the response object, table_schema was set to my search path. But when another dev tried to run the code on her machine (local instances of Node and PG), that response object was set to MY schema. We can find no way to manually set it.
No idea what kind of black magic that would be, but need to know how to set the user schema for Bookshelf/Knex/PG.
While I haven't been able to find an answer for the original question, how to set the PG schema, I was able to resolve the issue at hand. Turned out to be a permissions issue in PG. Ensure that all users had r/w perms in their respective search_path and all was well.
No error messages that I could find anywhere within Bookshelf, Knex or PG client pointed to this as a possible culprit, so hopefully this helps somebody.
Related
Problem
running out of database connections in prod leading to errors like this;
Error:
Invalid `prisma.queryRaw()` invocation:
Timed out fetching a new connection from the connection pool. More info: http://pris.ly/d/connection-pool (Current connection pool timeout: 10, connection limit: 5)
at Object.request (/usr/src/app/node_modules/#prisma/client/runtime/index.js:45629:15)
at async Proxy._request (/usr/src/app/node_modules/#prisma/client/runtime/index.js:46456:18)
Situation
multiple API containers running in Google Cloud Run running node/express/prisma API
using Supabase's hosted postgres. In supabase Settings > Database, Connection Pooling is enabled.
in db connection string, using :6543/postgres?pgbouncer=true
Attempt to diagnose
in supabase, Database > Roles, I can see a list of roles and the number of connections for each. pgBouncer has 0 and the role which my application uses has several.
If I query pg_stat_activity, I can see connections for the usename which is used by my application, and client_addr values representing ip addresses for a couple of different container instances. Are these "forwarded on" from pgBouncer? or have they bypassed pgBouncer entirely?
I am not familiar with what either of these should look like if I were using pgBouncer correctly so it's hard for me to tell what's going on.
I assume this means that I either haven't configured pgBouncer correctly, or I'm not connecting to it properly, or both. I'd be really grateful if someone could point out how I could either check or fix my connection to pgBouncer and clarify what I should see in pg_stat_activity if I was correctly connected to pgBouncer. Thanks.
Figured out what's going wrong here, so writing out how I fixed it in case anyone else runs into this issue.
Better understanding of the problem
in my prisma schema file I'm getting my database url from the env
datasource db {
provider = "postgresql"
url = env("SUPABASE_POSTGRES_URL")
}
and when I'm instantiating the prisma client I'm using the same variable
export const prisma = new PrismaClient({
datasources: {
db: {
url: process.env.SUPABASE_POSTGRES_URL,
},
},
});
I have a build trigger in Google Cloud Build that builds containers when branches that are merged into certain other branches, eg when PRs are merged in to master, build new containers and deploy them to prod.
In the build trigger, the SUPABASE_POSTGRES_URL value is set in the env, using :5432 which connects directly to postgres, bypassing pgBouncer. This is a requirement for prisma migrations which can't be run through pgBouncer.
The Google Cloud Run container env vars specify a different value for SUPABASE_POSTGRES_URL however it looks like the this not being used, and instead the direct-to-postgres :5432 value is used while the app is running, to connect to the db and run queries - so pgBouncer was permanently bypassed.
Solution
where the prisma client is instantiated, I'm using a second env var. It turns out that prisma uses the env var referenced in the schema file for the DB URL for migrations and the db url in the client instantiation for queries when the app is running, and you can happily have two completely separate values for these two URLs.
export const prisma = new PrismaClient({
datasources: {
db: {
url: process.env.SUPABASE_PGBOUNCER_URL,
},
},
});
Now, SUPABASE_POSTGRES_URL is still populated from the build trigger, but it doesn't get used at runtime; instead I set SUPABASE_PGBOUNCER_URL in the Google Cloud Run env vars and that gets used during the prisma client instantiation, so queries a run through pgBouncer.
Result
Effective Prisma migrations direct to postgres
Effective connection pooling by running queries through pgBouncer
I am new to sails and its connection to database. I plan to use Heroku to host my app so I would like to use Postgresql.
I have changed config.datastore file with the following code:
default: {
adapter: 'sails-postgresql',
url: 'postgresql://postgres:postgres#localhost:5432/postgres',
max: 1
}
On Sails documentation it sais that url is url: 'postgresql://user:password#host:port/database', but I dont understand where should I get user and password from. Do I need to initiate the postgresql and set it up and define a new database?
I am just trying to understand the mechanics behind this setup.
What you need to do is create a new user for your local PostgreSQL instead of using the default user and database. What I mean is, create a user-specific to your particular project in SQL following this should help https://www.postgresql.org/docs/8.0/sql-createuser.html.(This should also help you set the password)
Then create the database you want to connect to your Sails app to be used by Waterline by following this: https://www.postgresql.org/docs/9.0/sql-createdatabase.html (make sure you grant the user you created permissions for this database as well https://www.postgresql.org/docs/9.0/sql-grant.html)
Then all you need is replace the url property value in config/datastore.js with the value following the format
postgres://USERNAME:PASSWORD#localhost:5432/DATABASE_NAME
N.B: Replace USERNAME, PASSWORD, and DATABASE_NAME with the actual username, password, and database name respectively.
When you are deploying to Heroku, Heroku would provide you with an environment variable called DATABASE_URL if you install the Heroku Postgres add-on. You could just pass that to url like so
url: process.env.DATABASE_URL
I can't seem to connect to my DB instance in AWS. I'm using the pg package and following the examples from the website is not working.
A search for "aws postgres database does not exist" really isn't returning anything helpful. Going through the open/closed issues on the PG github isnt helpful either.
Running $nc <RDS endpoint> <port number> returns a success message so it's definitely there. Every value placed in the Client config is copy/pasted from my DB instance.
I'm starting to wonder if the databases have a different name than what it shows in the "Instances" section of RDS on AWS?
const client = new Client({
host : '<<RDS ENDPOINT>>',
database : '<<RDS NAME>>', // maybe this isnt the real name?
user : '<<username>>',
password : '<<password>>',
port : <<port>>
});
client.connect()
.then(data => {
console.log('connected');
})
.catch(err => {
console.log(err);
})
I ran into this issue as well. It looks like the DB Instance Name and the actual DB name are two different things and even when you add a name when you create your DB, it defaults to 'postgres'. When I put in the name of my DB it gave me the same error. However, when I just put in 'postgres' it worked fine. Try that and see if it works for you.
The initial configuration of RDS instances is quite messy, since the parameter "database name" is only the name of the instance, not the proper name of the database. If you want AWS to create a database at the moment you create the db instance, you have to select "Additional configuration" and explicitly add a parameter called "Initial database name". Check the screenshot I attach here.
Try adding postgres as dbname. It worked for me!
After connecting with postgres as db name, you can type \l to list all database on that PSQL cluster, that will return a bunch of default dbs and also the one you created (the name) so you can connect to it
I ran into the same issue after creating a DB instance on AWS RDS. I wanted to test the connection of my database using PostBird, and I used my actual DB instance name but it could not work.
But I used "postgres in field of DB_name and it worked. That means that my default username was posgres and db_name was also "posgres.
I hope it will help you too.
Try this if the above answer does not work.
Remove the:5439/lab ending so that the Host value ends with: .com
I need to get the execution plan of a query that execute from Apache Drill using the PostgreSQL storage plugin (not the Drill execution plan, but the PostGIS one).
So I enable the explain plan logs with following commands;
SET auto_explain.log_min_duration = 0;
SET auto_explain.log_analyze = true;
And if I execute a query from pgAdmin, it shows the statement and the plan. But if I execute the same query from Drill, it does not log anything.
Do you know why this happens and how can be solved this situation?
Note: I checked the connection and it's ok, they are the same in pgAdmin and Drill, also in Drill I execute queries and I get results, so I assume that there is no connection problems.
I suspect you are executing the SET commands from a postgres command line, so those settings would only apply to Drill's postgres session. In order to apply those settings to Drill's postgres session, try adding those properties to Drill's storage plugin configuration. Here is an example configuration for those properties:
{
"type": "jdbc",
"driver": "org.postgresql.Driver",
"url": "jdbc:postgresql://localhost:5432;auto_explain.log_min_duration=0;auto_explain.log_analyze=true",
"username": "postgres",
"enabled": true
}
I am brand-new to Sails and I'm looking to build my first application in javascript land! I am currently trying to connect my sails application to a postgresql database and I want to make sure that I am doing this correctly.
I started by creating a postgresql db with dbName, userName, password. I have added all this information into my connections.js file:
somePostgresqlServer: {
adapter: 'sails-postgresql',
host: 'localhost',
user: '<username>', // optional
password: '<password>', // optional
database: '<databasename>' //optional
}
I want to be sure I can migrate and perform all operations on my own so my models.js is set to migrate: safe. I ran npm install sails-postgresql. Now, to my understanding if I have migrate set to safe I will need the sails-db-migrate module (https://github.com/building5/sails-db-migrate). I followed this module step by step. I generated a User model by running sails generate api user. After this I ran grunt db:migrate. After all this, I check my psql database and no User table has been created. I know there is something I am missing, or maybe there is a more simple way to see if my postgresql db is connected.
Any advice or suggestions on how I should approach this would be greatly appreciated.
I was able to connect my PSQL DB to my application by simply:
creating db in psql
configuring it in sails app/connections.js
running npm install sails-postgresql
Adding to config/env/development.js:
models: {
connection: 'somePostgresqlServer'
}
running sails generate api user
sails lift
When asked for migration mode I chose options 2 which is alter
Opened up psequal and sure enough there was my new user table.
I'm not sure which of these actually triggered it to working from adding my db name to development.js, leaving off a PW on my DB, or changing migration mode to alter. But these steps helped it to connect. Will look further into.