I have a PostgreSQL database deployed in Google Cloud that I am trying to connect to from a Cloud Run instance. I have tried the following two packages, both of them eventually leading to the same exception:
https://pub.dev/packages/postgres
https://pub.dev/packages/database_adapter_postgre
The exception I am getting is:
SocketException: Failed host lookup: '/cloudsql/{INSTANCE_CONNECTION_NAME}' (OS Error: Name or service not known, errno = -2)
I get here both times when trying to establish the connection, so in the case of the first package:
connection = new PostgreSQLConnection(
'/cloudsql/{INSTANCE_CONNECTION_NAME}',
5432,
'postgres',
username: 'username',
password: 'password');
await connection.open(); // <-- exception thrown here
I have tried changing the host string to /cloudsql/INSTANCE_CONNECTION_NAME}/.s.PGSQL.5432, but that did not work. My first thought were permissions, the service account the Cloud Run instance is using (xxx-compute#developer.gserviceaccount.com) has the Cloud SQL Editor role (tried Client and Admin too).
Running the same database code locally from a dart console app, I can connect to my database via its public IP address as the host with both packages, so the database itself is up and running.
Can someone point me in the right direction with this exception/have an example code for any of the packages above to show how to connect it to a Cloud SQL instance from a Cloud Run?
Edit:
I tried setting up a proxy locally to test out if the connection is wrong like so:
.\cloud_sql_proxy.exe -instances={INSTANCE_CONNECTION_NAME}=tcp:5433 psql
Then changing the connection host value in the code to localhost, and the port to 5433.
To my surprise it works - so from locally I am seemingly able to connect to the DB using that connection string. It still doesn't work when I use it from a Cloud Run instance though. Any help is appreciated!
It seems dart doesn't support connection through unix socket, you need to configure a IP (public or private, as you need).
Alternatively you can use pg which support unix socket connection
Hope this helps.
Just for those who come across this question in the future:
as it stands right now, I had to resort to the suggestion posted by Daniele Ricci and use the public IP for the database. The one thing to point out here was that since Cloud Runs don't have a static IPv4 address to run from, the DB had to be set to allow connections from anywhere (had to add an authorized connection from 0.0.0.0/0), which is unsafe. Until the kind development team of dart figures out how to use UNIX sockets, this seems to be the only way of getting it to work.
Not having actually tested this myself, according to the source code of the postgres package, you have to specify that you want a Unix socket:
connection = PostgreSQLConnection(
...
isUnixSocket: true, // <-- here
);
The default is false.
The host you pass is must also be valid. The docs say:
[host] must be a hostname, e.g. "foobar.com" or IP address. Do not include scheme or port.
I was struggling with the same issue.
The solution is to create a connection as follows:
PostgreSQLConnection getProdConnection() {
final String connectionName = Platform.environment['CLOUD_SQL_CONNECTION_NAME']!;
final String databaseName = Platform.environment['DB_NAME']!;
final String user = Platform.environment['DB_USER']!;
final String password = Platform.environment['DB_PASS']!;
final String socketPath = '/cloudsql/$connectionName/.s.PGSQL.5432';
return PostgreSQLConnection(
socketPath,
5432,
databaseName,
username: user,
password: password,
isUnixSocket: true,
);
}
Then when you create a Cloud Run service, you need to define 'Enviroment variables' as follows.
You also need to select your sql instance in the 'connections' tab.
Then the last thing needed is to configure a Cloud Run service account.
Then the connection to instance should be successful and there should no longer be a need for a 0.0.0.0/0 connection.
However, if you try to run this connection locally on a Windows device during development the connection will not be allowed and you will be presented with this error message: 'Unix domain sockets are not available on this operating system.'
Therefore, I recommend that you open Google SQL networking to your public address and define a local environment using the 'Public IP address' of your SQL instance.
For more information on this topic, I can recommend these resources that have guided me to the right solution:
https://cloud.google.com/sql/docs/postgres/connect-instance-cloud-run#console_5
https://github.com/dart-lang/sdk/issues/47899
Related
Background
I have a NestJS project with Prisma ORM, and I am continually receiving the error:
PrismaClientInitializationError: Can't reach database server at `localhost`:`5432`
This is happening during the Cloud Build Deploy step.
Since this is a containerized application (attempting to) run in a Cloud Run instance, I'm supposed to use a socket connection. Here's the documentation from Prisma on connecting to a Postgres DB through a socket connection: https://www.prisma.io/docs/concepts/database-connectors/postgresql#connecting-via-sockets
Connecting via sockets
To connect to your PostgreSQL database via sockets, you must add a host field as a query parameter to the connection URL (instead of setting it as the host part of the URI). The value of this parameter then must point to the directory that contains the socket, e.g.: postgresql://USER:PASSWORD#localhost/database?host=/var/run/postgresql/
Note that localhost is required, the value itself is ignored and can be anything.
I've done this to the letter, as described in the Cloud SQL documentation, with the exception that I percent-encoded my path to the directory containing the socket. I've included and excluded the trailing slash.
So my host var looks like this, mapped from the percent-encoded values:
/cloudsql/<MY CLOUD SQL CONNECTION NAME>/<DB>
I've read over the Cloud Run documentation, and in my mind, I should expect a different error if the instance itself can't connect to the Cloud SQL instance. I've followed the "Make sure you have the appropriate permissions and connection" from the documentation a few times now.
Is there anything obvious that I'm missing? Am I wrong about an error related to Cloud Run instance just not connecting with Cloud SQL instance?
Things I've tried & things I know
I CAN connect directly to the Cloud SQL instance locally through psql
I CAN run a local server with the Cloud SQL instance public IP and establish a client connection & interact with the database
I CAN successfully create an image and run a container from that image locally
My big concern
It doesn't make sense to me in which order things should connect to the Cloud SQL instance. To me, the Cloud Run - Cloud SQL connection MUST be established before the application run inside the Cloud Run instance can establish its connection through the socket to the Cloud SQL instance. -- Am I thinking through that correctly?
I am not able to connect through pgAdmin4 to Google Cloud SQL.
Following are the credentials I am providing and getting the error messege:
If you are configuring access for public IP connections, follow the next steps to connect with PgAdmin:
1) Create your PostgreSQL instance in Google Cloud Platform.
2) While creating the instance,
In Instance ID tab, write a name for your instance
In Default user password tab, write a password.
Choose the region.
Click on configuration options, go to Connectivity, enable Public IP, under Authorized networks, add network and their write the IP address of your PC. You can check the IP address in the link .
Click on create.
Then in PgAdmin:
1) In Host name/address put Public IP address of the instance.
To connect with Google Cloud SQL using pgadmin client, you can use cloud_sql_proxy
These steps need to follow:
Establish connection to Postgres.
cloud_sql_proxy -instances=<connection string>=tcp:5433
You will get connection string from GCP here
After executing cloud_sql_proxy, you will see something like this
Now create a new server connection using pgadmin client.
If you followed those steps properly it will connect and can use like local pgadmin.
Nibrass H has perfectly answered this question.
But I would like to add that if you have already created the Instance, then you can click on 'Edit Instance' and Add your IP in 'Connections'.
Adding some Images to help you.
I've been trying to connect to my Cloud SQL instance using the pg module but haven't been successful so far.
I've looked around a lot online but couldn't understand much on the topic. I also would like to deploy my Express app on Cloud Run at some point and have it connect to my Cloud SQL instance but I don't know how to go about doing that.
Here's a list of things I don't understand and would like a brief explanation on:
What are Unix socket connections and why should I use them over normal connections?
What is a Cloud SQL Proxy? Do I need to use it? If so, why?
Would I need to do any extra work to connect to my Cloud SQL instance from Cloud Run?
Here are all the connection objects and connection strings I have tried with the pg.Client object:
First connection string: postgresql+psycopg2://postgres:password#/cloudsql/myapp:us-central1:mydb?host=/var/lib/postgresql
Second connection string: postgresql://postgres:password#hostip:5432/myapp:us-central1:mydb
Third connection string: postgresql://postgres:password#hostip:5432/sarcdb
Connection object: { host: "/cloudsql/myapp:us-central1:mydb", username: "postgres", password: "password", database: "mydb" }
All of these give me a Connection terminated unexpectedly error.
The Cloud Functions documentation for Node.js & Cloud SQL (scroll down to PostgreSQL) has applicable information on structuring the connection string and the additional configuration needed for credentials.
Once that's in place for your app, you'll need to add the Cloud SQL instance to your Cloud Run service before it will be able to use that connection string to reach the database.
Here's directly copying the code sample from the docs, with Cloud Run the max configuration of 1 might not keep pace with other concurrency settings.
const pg = require('pg');
/**
* TODO(developer): specify SQL connection details
*/
const connectionName =
process.env.INSTANCE_CONNECTION_NAME || '<YOUR INSTANCE CONNECTION NAME>';
const dbUser = process.env.SQL_USER || '<YOUR DB USER>';
const dbPassword = process.env.SQL_PASSWORD || '<YOUR DB PASSWORD>';
const dbName = process.env.SQL_NAME || '<YOUR DB NAME>';
const pgConfig = {
max: 1,
user: dbUser,
password: dbPassword,
database: dbName,
};
if (process.env.NODE_ENV === 'production') {
pgConfig.host = `/cloudsql/${connectionName}`;
}
// Connection pools reuse connections between invocations,
// and handle dropped or expired connections automatically.
let pgPool;
exports.postgresDemo = (req, res) => {
// Initialize the pool lazily, in case SQL access isn't needed for this
// GCF instance. Doing so minimizes the number of active SQL connections,
// which helps keep your GCF instances under SQL connection limits.
if (!pgPool) {
pgPool = new pg.Pool(pgConfig);
}
pgPool.query('SELECT NOW() as now', (err, results) => {
if (err) {
console.error(err);
res.status(500).send(err);
} else {
res.send(JSON.stringify(results));
}
});
// Close any SQL resources that were declared inside this function.
// Keep any declared in global scope (e.g. mysqlPool) for later reuse.
};
What are Unix socket connections and why should I use them over normal connections?
A Unix domain socket is a socket for interprocess communication. If you have the choice between communication between a TCP connection and a Unix domain socket, the Unix domain socket is likely faster.
What is a Cloud SQL Proxy? Do I need to use it? If so, why?
The Cloud SQL proxy allows you to authenticate a connection to connect to your database using IAM permissions of a service account.
Since Cloud SQL is a cloud database, it requires (by default) some form of authentication to help it remain secure. The proxy is a more secure method of connecting compared to a self-managed SSL Certificate or a whitelisted IP address.
Would I need to do any extra work to connect to my Cloud SQL instance from Cloud Run?
Cloud Run takes care of running the proxy for you, but you need to do the following:
Enable the Cloud SQL Admin API
Add the Cloud SQL instance to your Run deployment(follow these steps).
Ensure that the service account running your code has the Cloud SQL Client IAM permissions (this is done for the default service account by step 2)
Configure your application to connect with /cloudsql/INSTANCE_CONNECTION_NAME
I need to use MySQL to send queries to a Google Cloud SQL database set up. I already have an instance created and a user, and I am able to access the database through the Cloud Shell. I can't seem to find the credentials to log into the database (host name, username, password, port and socket), and I'm not sure how to access them through the shell.
You can find the available methods to connect to your Cloud SQL instance here.
Connecting from an IP address without SSL is probably the easiest one:
In the Cloud Console, go to the cloud SQL instances screen and click on your instance’s name.
In the overview tab, take note of the Primary IP Address, you’ll use it instead of a hostname.
In the users tab, you can create a new user or reset the password of an existente one, including the root user.
In the authorization tab, add the ip or ip range where you are attempting the connection from, so Cloud SQL accept connections from your client (more on this here).
Start your mysql client as follows (note the port is not necessary as the default one is used):
mysql --host=[INSTANCE_IP_ADDR] --user=[USER_NAME] --password
I can't seem to connect to Cloud SQL with Workbench. I keep getting this error.
Failed to Connect to MySQL at CLOUD-SQL-IPv4:3306 with user root
Access denied for use 'root'#'WHITE-LISTED-IP-ADDRESS' (using password: YES)
I have white listed my IP.
I have set an IP for the SQL instance.
I have checked the username and Password several times.
Any idea why this is happening?
It seems you have to create a new user in the Google Cloud Console with the host name set to %(any host). You can't seem to connect using the root user.
Is it possible to connect with the #root user but after you finish the setup you have to restart the sql server.
What is important to set up an SSL certificate if you connect to you production database, but if you only try it out you can allow unsecured connection. Another important thing is to add your IP to the Authorised Networks in the Connection tab.