I've been trying to connect to my Cloud SQL instance using the pg module but haven't been successful so far.
I've looked around a lot online but couldn't understand much on the topic. I also would like to deploy my Express app on Cloud Run at some point and have it connect to my Cloud SQL instance but I don't know how to go about doing that.
Here's a list of things I don't understand and would like a brief explanation on:
What are Unix socket connections and why should I use them over normal connections?
What is a Cloud SQL Proxy? Do I need to use it? If so, why?
Would I need to do any extra work to connect to my Cloud SQL instance from Cloud Run?
Here are all the connection objects and connection strings I have tried with the pg.Client object:
First connection string: postgresql+psycopg2://postgres:password#/cloudsql/myapp:us-central1:mydb?host=/var/lib/postgresql
Second connection string: postgresql://postgres:password#hostip:5432/myapp:us-central1:mydb
Third connection string: postgresql://postgres:password#hostip:5432/sarcdb
Connection object: { host: "/cloudsql/myapp:us-central1:mydb", username: "postgres", password: "password", database: "mydb" }
All of these give me a Connection terminated unexpectedly error.
The Cloud Functions documentation for Node.js & Cloud SQL (scroll down to PostgreSQL) has applicable information on structuring the connection string and the additional configuration needed for credentials.
Once that's in place for your app, you'll need to add the Cloud SQL instance to your Cloud Run service before it will be able to use that connection string to reach the database.
Here's directly copying the code sample from the docs, with Cloud Run the max configuration of 1 might not keep pace with other concurrency settings.
const pg = require('pg');
/**
* TODO(developer): specify SQL connection details
*/
const connectionName =
process.env.INSTANCE_CONNECTION_NAME || '<YOUR INSTANCE CONNECTION NAME>';
const dbUser = process.env.SQL_USER || '<YOUR DB USER>';
const dbPassword = process.env.SQL_PASSWORD || '<YOUR DB PASSWORD>';
const dbName = process.env.SQL_NAME || '<YOUR DB NAME>';
const pgConfig = {
max: 1,
user: dbUser,
password: dbPassword,
database: dbName,
};
if (process.env.NODE_ENV === 'production') {
pgConfig.host = `/cloudsql/${connectionName}`;
}
// Connection pools reuse connections between invocations,
// and handle dropped or expired connections automatically.
let pgPool;
exports.postgresDemo = (req, res) => {
// Initialize the pool lazily, in case SQL access isn't needed for this
// GCF instance. Doing so minimizes the number of active SQL connections,
// which helps keep your GCF instances under SQL connection limits.
if (!pgPool) {
pgPool = new pg.Pool(pgConfig);
}
pgPool.query('SELECT NOW() as now', (err, results) => {
if (err) {
console.error(err);
res.status(500).send(err);
} else {
res.send(JSON.stringify(results));
}
});
// Close any SQL resources that were declared inside this function.
// Keep any declared in global scope (e.g. mysqlPool) for later reuse.
};
What are Unix socket connections and why should I use them over normal connections?
A Unix domain socket is a socket for interprocess communication. If you have the choice between communication between a TCP connection and a Unix domain socket, the Unix domain socket is likely faster.
What is a Cloud SQL Proxy? Do I need to use it? If so, why?
The Cloud SQL proxy allows you to authenticate a connection to connect to your database using IAM permissions of a service account.
Since Cloud SQL is a cloud database, it requires (by default) some form of authentication to help it remain secure. The proxy is a more secure method of connecting compared to a self-managed SSL Certificate or a whitelisted IP address.
Would I need to do any extra work to connect to my Cloud SQL instance from Cloud Run?
Cloud Run takes care of running the proxy for you, but you need to do the following:
Enable the Cloud SQL Admin API
Add the Cloud SQL instance to your Run deployment(follow these steps).
Ensure that the service account running your code has the Cloud SQL Client IAM permissions (this is done for the default service account by step 2)
Configure your application to connect with /cloudsql/INSTANCE_CONNECTION_NAME
Related
I'm trying to connect to a public IP CloudSQL Postgres database from a cloud function with Prisma.
The database has SSL enforced. I assume I can use the Cloud Auth Proxy, and it works locally, but when I deploy it gives me an error.
I've tried both:
Option 1:
datasource db {
provider = "postgresql"
url = "postgresql://USER:PASSWORD#localhost:3307/DATABASE_NAME?host=CONNECTION_URL"
}
Got error:
Can't reach database server at `CONNECTION_URL`:`3307`
Option 2:
datasource db {
provider = "postgresql"
url = "postgresql://USER:PASSWORD#localhost/DATABASE_NAME?host=CONNECTION_URL"
}
Got error:
Can't reach database server at `IP_ADDRESS`:`5432`
Where IP_ADDRESS is the correct public IP address for the database that I can see in the console
CONNECTION_URL is /cloudsql/PROJ:REGION:INSTANCE
You application is trying to connect on "CONNECTION_URL" and "3307". To use public IP on Cloud Functions you should by connecting by unix socket: https://cloud.google.com/sql/docs/mysql/connect-functions#connect_to
Based on this issue, it looks like your URL should be something like:
postgresql://username#localhost/databasename?host=/cloudsql/CONNECTION_NAME
Where your instance connection name is PROJ:REGION:INSTANCE.
Figure out I had the vpc connector enabled to all traffic, so all traffic was trying to go through the private VPC instead of the public IP. Disabling it fixed it.
Problem
running out of database connections in prod leading to errors like this;
Error:
Invalid `prisma.queryRaw()` invocation:
Timed out fetching a new connection from the connection pool. More info: http://pris.ly/d/connection-pool (Current connection pool timeout: 10, connection limit: 5)
at Object.request (/usr/src/app/node_modules/#prisma/client/runtime/index.js:45629:15)
at async Proxy._request (/usr/src/app/node_modules/#prisma/client/runtime/index.js:46456:18)
Situation
multiple API containers running in Google Cloud Run running node/express/prisma API
using Supabase's hosted postgres. In supabase Settings > Database, Connection Pooling is enabled.
in db connection string, using :6543/postgres?pgbouncer=true
Attempt to diagnose
in supabase, Database > Roles, I can see a list of roles and the number of connections for each. pgBouncer has 0 and the role which my application uses has several.
If I query pg_stat_activity, I can see connections for the usename which is used by my application, and client_addr values representing ip addresses for a couple of different container instances. Are these "forwarded on" from pgBouncer? or have they bypassed pgBouncer entirely?
I am not familiar with what either of these should look like if I were using pgBouncer correctly so it's hard for me to tell what's going on.
I assume this means that I either haven't configured pgBouncer correctly, or I'm not connecting to it properly, or both. I'd be really grateful if someone could point out how I could either check or fix my connection to pgBouncer and clarify what I should see in pg_stat_activity if I was correctly connected to pgBouncer. Thanks.
Figured out what's going wrong here, so writing out how I fixed it in case anyone else runs into this issue.
Better understanding of the problem
in my prisma schema file I'm getting my database url from the env
datasource db {
provider = "postgresql"
url = env("SUPABASE_POSTGRES_URL")
}
and when I'm instantiating the prisma client I'm using the same variable
export const prisma = new PrismaClient({
datasources: {
db: {
url: process.env.SUPABASE_POSTGRES_URL,
},
},
});
I have a build trigger in Google Cloud Build that builds containers when branches that are merged into certain other branches, eg when PRs are merged in to master, build new containers and deploy them to prod.
In the build trigger, the SUPABASE_POSTGRES_URL value is set in the env, using :5432 which connects directly to postgres, bypassing pgBouncer. This is a requirement for prisma migrations which can't be run through pgBouncer.
The Google Cloud Run container env vars specify a different value for SUPABASE_POSTGRES_URL however it looks like the this not being used, and instead the direct-to-postgres :5432 value is used while the app is running, to connect to the db and run queries - so pgBouncer was permanently bypassed.
Solution
where the prisma client is instantiated, I'm using a second env var. It turns out that prisma uses the env var referenced in the schema file for the DB URL for migrations and the db url in the client instantiation for queries when the app is running, and you can happily have two completely separate values for these two URLs.
export const prisma = new PrismaClient({
datasources: {
db: {
url: process.env.SUPABASE_PGBOUNCER_URL,
},
},
});
Now, SUPABASE_POSTGRES_URL is still populated from the build trigger, but it doesn't get used at runtime; instead I set SUPABASE_PGBOUNCER_URL in the Google Cloud Run env vars and that gets used during the prisma client instantiation, so queries a run through pgBouncer.
Result
Effective Prisma migrations direct to postgres
Effective connection pooling by running queries through pgBouncer
Background
I have a NestJS project with Prisma ORM, and I am continually receiving the error:
PrismaClientInitializationError: Can't reach database server at `localhost`:`5432`
This is happening during the Cloud Build Deploy step.
Since this is a containerized application (attempting to) run in a Cloud Run instance, I'm supposed to use a socket connection. Here's the documentation from Prisma on connecting to a Postgres DB through a socket connection: https://www.prisma.io/docs/concepts/database-connectors/postgresql#connecting-via-sockets
Connecting via sockets
To connect to your PostgreSQL database via sockets, you must add a host field as a query parameter to the connection URL (instead of setting it as the host part of the URI). The value of this parameter then must point to the directory that contains the socket, e.g.: postgresql://USER:PASSWORD#localhost/database?host=/var/run/postgresql/
Note that localhost is required, the value itself is ignored and can be anything.
I've done this to the letter, as described in the Cloud SQL documentation, with the exception that I percent-encoded my path to the directory containing the socket. I've included and excluded the trailing slash.
So my host var looks like this, mapped from the percent-encoded values:
/cloudsql/<MY CLOUD SQL CONNECTION NAME>/<DB>
I've read over the Cloud Run documentation, and in my mind, I should expect a different error if the instance itself can't connect to the Cloud SQL instance. I've followed the "Make sure you have the appropriate permissions and connection" from the documentation a few times now.
Is there anything obvious that I'm missing? Am I wrong about an error related to Cloud Run instance just not connecting with Cloud SQL instance?
Things I've tried & things I know
I CAN connect directly to the Cloud SQL instance locally through psql
I CAN run a local server with the Cloud SQL instance public IP and establish a client connection & interact with the database
I CAN successfully create an image and run a container from that image locally
My big concern
It doesn't make sense to me in which order things should connect to the Cloud SQL instance. To me, the Cloud Run - Cloud SQL connection MUST be established before the application run inside the Cloud Run instance can establish its connection through the socket to the Cloud SQL instance. -- Am I thinking through that correctly?
Currently, our api (deployed on cloudRun) connects to our Postgres database by passing in a pgConfig with a db configuration and a db user and password.
For example:
const configObject = {
host: cloudRunHost,
user: dbUser,
password: dbPassword,
database: dbName
}
async function connect() {
if(!client) {
const pgPool = new pg.Pool(configObject);
await pgPool.connect()
.then((result) => {
logger.info('Connected to DB')
client = result;
}).catch((err) => {
logger.error(err);
});
}
}
We want the app itself to connect to the database by using Cloud SQL IAM database authentication
So far:
The api cloudRun instance has a service account
The database and CloudSQL has been configured for IAM-based access (we can access with our machine service accounts)
The api service account has access to the DB via IAM, and permissions granted to said user on the Postgres db itself
When the above code runs, it logs error: empty password returned by client
We've tried removing the password line entirely from the configObject but it hasn't helped. Any thoughts on why my service account can access the db directly but the api's can't I suspect we need to indicate to pgPool that we're trying to connect via IAM instead of via user/password.
Unfortunately, there isn't a good way to use "automatic" IAM DB AuthN on Cloud Run - while Cloud Run does use the proxy, there is no way for users to flip on the "-enable-iam-login" flag.
Go, Python, and Java users can use the language-specific connectors, but there isn't one for Node.js.
It looks like node-postgres does have support for dynamic passwords, so you might be able to do something like this:
const {GoogleAuth} = require('google-auth-library');
async function main() {
const auth = new GoogleAuth();
const pool = new pg.Pool({
connectionString: 'postgresql://user#db.example:5432/my-db',
password: async () => auth.getAccessToken(),
})
}
From your question I believe you are using NodeJS. But currently IAM database authentication is supported by Python Connector, Java Connector and Cloud SQL proxy which is mentioned here. Also to use Automatic IAM database authentication it is required to use a Cloud SQL connector.
IAM database authentication is currently supported by the Cloud SQL Auth proxy, the Java connector, and the Python connector.
Automatic IAM database authentication requires the use of a Cloud SQL connector.
As you are using NodeJS which is not supported, that seems to be the reason why you are getting an error: empty password returned by client.
I have a PostgreSQL database deployed in Google Cloud that I am trying to connect to from a Cloud Run instance. I have tried the following two packages, both of them eventually leading to the same exception:
https://pub.dev/packages/postgres
https://pub.dev/packages/database_adapter_postgre
The exception I am getting is:
SocketException: Failed host lookup: '/cloudsql/{INSTANCE_CONNECTION_NAME}' (OS Error: Name or service not known, errno = -2)
I get here both times when trying to establish the connection, so in the case of the first package:
connection = new PostgreSQLConnection(
'/cloudsql/{INSTANCE_CONNECTION_NAME}',
5432,
'postgres',
username: 'username',
password: 'password');
await connection.open(); // <-- exception thrown here
I have tried changing the host string to /cloudsql/INSTANCE_CONNECTION_NAME}/.s.PGSQL.5432, but that did not work. My first thought were permissions, the service account the Cloud Run instance is using (xxx-compute#developer.gserviceaccount.com) has the Cloud SQL Editor role (tried Client and Admin too).
Running the same database code locally from a dart console app, I can connect to my database via its public IP address as the host with both packages, so the database itself is up and running.
Can someone point me in the right direction with this exception/have an example code for any of the packages above to show how to connect it to a Cloud SQL instance from a Cloud Run?
Edit:
I tried setting up a proxy locally to test out if the connection is wrong like so:
.\cloud_sql_proxy.exe -instances={INSTANCE_CONNECTION_NAME}=tcp:5433 psql
Then changing the connection host value in the code to localhost, and the port to 5433.
To my surprise it works - so from locally I am seemingly able to connect to the DB using that connection string. It still doesn't work when I use it from a Cloud Run instance though. Any help is appreciated!
It seems dart doesn't support connection through unix socket, you need to configure a IP (public or private, as you need).
Alternatively you can use pg which support unix socket connection
Hope this helps.
Just for those who come across this question in the future:
as it stands right now, I had to resort to the suggestion posted by Daniele Ricci and use the public IP for the database. The one thing to point out here was that since Cloud Runs don't have a static IPv4 address to run from, the DB had to be set to allow connections from anywhere (had to add an authorized connection from 0.0.0.0/0), which is unsafe. Until the kind development team of dart figures out how to use UNIX sockets, this seems to be the only way of getting it to work.
Not having actually tested this myself, according to the source code of the postgres package, you have to specify that you want a Unix socket:
connection = PostgreSQLConnection(
...
isUnixSocket: true, // <-- here
);
The default is false.
The host you pass is must also be valid. The docs say:
[host] must be a hostname, e.g. "foobar.com" or IP address. Do not include scheme or port.
I was struggling with the same issue.
The solution is to create a connection as follows:
PostgreSQLConnection getProdConnection() {
final String connectionName = Platform.environment['CLOUD_SQL_CONNECTION_NAME']!;
final String databaseName = Platform.environment['DB_NAME']!;
final String user = Platform.environment['DB_USER']!;
final String password = Platform.environment['DB_PASS']!;
final String socketPath = '/cloudsql/$connectionName/.s.PGSQL.5432';
return PostgreSQLConnection(
socketPath,
5432,
databaseName,
username: user,
password: password,
isUnixSocket: true,
);
}
Then when you create a Cloud Run service, you need to define 'Enviroment variables' as follows.
You also need to select your sql instance in the 'connections' tab.
Then the last thing needed is to configure a Cloud Run service account.
Then the connection to instance should be successful and there should no longer be a need for a 0.0.0.0/0 connection.
However, if you try to run this connection locally on a Windows device during development the connection will not be allowed and you will be presented with this error message: 'Unix domain sockets are not available on this operating system.'
Therefore, I recommend that you open Google SQL networking to your public address and define a local environment using the 'Public IP address' of your SQL instance.
For more information on this topic, I can recommend these resources that have guided me to the right solution:
https://cloud.google.com/sql/docs/postgres/connect-instance-cloud-run#console_5
https://github.com/dart-lang/sdk/issues/47899