How do I connect to a GCP PostgreSQL 11 DB in Project A from a go1.12 Flex Instance in Project B using gorm - postgresql

Some preface:
I do not have access to the project that hosts the postgres DB.
The person who has access to the project and who set up the DB is unable to work with me directly on this. I have been told that they've followed the GCP documentation and that things are set up correctly on their end.
I am able to connect from my local deploy of the app because the person who set up the DB instance allowed my IP address; this doesn't work for an app engine flex deploy.
Problem:
I have an app engine flex instance built on golang 1.12. It connects to a postgreSQL 11 DB Cloud SQL instance in another project using github.com/jinzhu/gorm with the following code:
dbUri := fmt.Sprintf("host=%s user=%s dbname=%s password=%s", dbHost, dbUsername, dbName, dbPassword)
conn, err := gorm.Open("postgres", dbUri)
if err != nil {
log.Printf("Database error: %+v\n", err)
}
The dbhost is currently in the format of the DB's ip address.
I have tried /cloudsql/project:region:instance
I have tried /cloudsql/project:region:instance/.s.pgsql.5432
The relevant settings from the app.yaml I'm using:
beta_settings:
cloud_sql_instances: [project:region:instance]
I have tried adding =tcp:5432 with all variations of the above dbhost
Errors:
In the logs from the app engine in the cloud console, connecting provides only the following information:
sql: database is closed
As I understand, this means that no connection could be created to the database.
This error matches the error I get when running the code locally from any machine that hasn't had its IP address added as an authorized network from the postgreSQL instance.
Formalized Question:
Given the above, what code, options, settings, etc. do I need to add or change in order to successfully connect from the app engine flex deployment to the cloud SQL db?
If I had access to the project where the postgres Cloud SQL instance is, is there anything in specific I could do to ensure the connection?
Thank you.

my apologies for answering my own question.
Over the weekend I was able to solve this problem. After badgering for a double check, it turned out that the Cloud SQL instance was not set up with a private IP Address (so the assurances to me about the instance being set up correctly, and my own assurances in the question, were false).
A new Cloud SQL Instance had to be created and assigned a private IP address, as the documentation is very clear in stating that an existing Cloud SQL instance cannot be assigned a private IP address once created without one.
The App Engine Flex deploy was able to connect using the private IP address without incident.
Sorry again!

Related

What could I be missing with Prisma client, Cloud Run, and Cloud SQL - my Prisma client can't socket-connect to my Cloud SQL instance DB?

Background
I have a NestJS project with Prisma ORM, and I am continually receiving the error:
PrismaClientInitializationError: Can't reach database server at `localhost`:`5432`
This is happening during the Cloud Build Deploy step.
Since this is a containerized application (attempting to) run in a Cloud Run instance, I'm supposed to use a socket connection. Here's the documentation from Prisma on connecting to a Postgres DB through a socket connection: https://www.prisma.io/docs/concepts/database-connectors/postgresql#connecting-via-sockets
Connecting via sockets
To connect to your PostgreSQL database via sockets, you must add a host field as a query parameter to the connection URL (instead of setting it as the host part of the URI). The value of this parameter then must point to the directory that contains the socket, e.g.: postgresql://USER:PASSWORD#localhost/database?host=/var/run/postgresql/
Note that localhost is required, the value itself is ignored and can be anything.
I've done this to the letter, as described in the Cloud SQL documentation, with the exception that I percent-encoded my path to the directory containing the socket. I've included and excluded the trailing slash.
So my host var looks like this, mapped from the percent-encoded values:
/cloudsql/<MY CLOUD SQL CONNECTION NAME>/<DB>
I've read over the Cloud Run documentation, and in my mind, I should expect a different error if the instance itself can't connect to the Cloud SQL instance. I've followed the "Make sure you have the appropriate permissions and connection" from the documentation a few times now.
Is there anything obvious that I'm missing? Am I wrong about an error related to Cloud Run instance just not connecting with Cloud SQL instance?
Things I've tried & things I know
I CAN connect directly to the Cloud SQL instance locally through psql
I CAN run a local server with the Cloud SQL instance public IP and establish a client connection & interact with the database
I CAN successfully create an image and run a container from that image locally
My big concern
It doesn't make sense to me in which order things should connect to the Cloud SQL instance. To me, the Cloud Run - Cloud SQL connection MUST be established before the application run inside the Cloud Run instance can establish its connection through the socket to the Cloud SQL instance. -- Am I thinking through that correctly?

Error connection to Google Cloud Postgres with GORM v2 after upgrade of GORM

After upgrading to Gorm versions:
gorm.io/driver/postgres v1.0.2
gorm.io/gorm v1.20.2
Not able to connect to Google Cloud SQL PostgreSQL v12
The connection worked without issues on older GORM versions with:
db, err := gorm.Open("postgres", "host=/cloudsql/project_id:us-central1:sql_instance_name port=5432 user=... dbname=... password=... sslmode=disable")
New GORM version works on local install (mac) of localhost_golang / localhost_postgres
CODE on Google Cloud:
dsn := ""
dsn += "user=postgres "
dsn += "password=admin_password_here "
dsn += "host=/cloudsql/project_id:us-central1:sql_instance_name "
dsn += "dbname=db_name "
dsn += "port=5432 "
dsn += "sslmode=disable "
fmt.Println("dsn:%v", dsn)
db, err := gorm.Open(postgres.Open(dsn), &gorm.Config{
NamingStrategy: schema.NamingStrategy{
SingularTable: true,
},
})
ERROR on Google Cloud:
[0m[31m[error] [0mfailed to initialize database, got error failed to connect to host=/cloudsql/project_id:us-central1:sql_instance_name user=postgres database=db_name: dial error (dial unix /cloudsql/project_id:us-central1:sql_instance_name/.s.PGSQL.5432: connect: connection refused)
Thank you!
App Engine default service account needs proper Roles, even though the CLoud SQL Connections page indicates "App Engine authorization", All apps in this project are authorized by default.
https://cloud.google.com/sql/docs/postgres/connect-app-engine-standard?_ga=2.121100104.-1170262708.1601472976
To configure App Engine standard environment to enable connections to a Cloud SQL instance using public IP:
Make sure that the instance created above has a public IP address. You can verify this on the Overview page for your instance in the Google Cloud Console. If you need to add one, see the Configuring public IP page for instructions.
Get the INSTANCE_CONNECTION_NAME for your instance. This can be found on the Overview page for your instance in the Google Cloud Console. or by running the following command: gcloud sql instances describe [INSTANCE_NAME].
Ensure that the service account your app is using to authenticate calls to Cloud SQL has the appropriate Cloud SQL role and permissions.
The service account for your service needs one of the following IAM roles:
- Cloud SQL Client (preferred)
- Cloud SQL Editor
- Cloud SQL Admin
Or, you can manually assign the following IAM permissions:
- cloudsql.instances.connect
- cloudsql.instances.get
For detailed instructions on adding IAM roles to a service account, see Granting Roles to Service Accounts.
By default, your app will authorize your connections using an App Engine service account. The service account identity is in the format PROJECT_ID#appspot.gserviceaccount.com.
If the authorizing service account belongs to a different project than the Cloud SQL instance, the Cloud SQL Admin API and IAM permissions will need to be added for both projects.

How can I connect to a Cloud PostgreSQL database from dart code?

I have a PostgreSQL database deployed in Google Cloud that I am trying to connect to from a Cloud Run instance. I have tried the following two packages, both of them eventually leading to the same exception:
https://pub.dev/packages/postgres
https://pub.dev/packages/database_adapter_postgre
The exception I am getting is:
SocketException: Failed host lookup: '/cloudsql/{INSTANCE_CONNECTION_NAME}' (OS Error: Name or service not known, errno = -2)
I get here both times when trying to establish the connection, so in the case of the first package:
connection = new PostgreSQLConnection(
'/cloudsql/{INSTANCE_CONNECTION_NAME}',
5432,
'postgres',
username: 'username',
password: 'password');
await connection.open(); // <-- exception thrown here
I have tried changing the host string to /cloudsql/INSTANCE_CONNECTION_NAME}/.s.PGSQL.5432, but that did not work. My first thought were permissions, the service account the Cloud Run instance is using (xxx-compute#developer.gserviceaccount.com) has the Cloud SQL Editor role (tried Client and Admin too).
Running the same database code locally from a dart console app, I can connect to my database via its public IP address as the host with both packages, so the database itself is up and running.
Can someone point me in the right direction with this exception/have an example code for any of the packages above to show how to connect it to a Cloud SQL instance from a Cloud Run?
Edit:
I tried setting up a proxy locally to test out if the connection is wrong like so:
.\cloud_sql_proxy.exe -instances={INSTANCE_CONNECTION_NAME}=tcp:5433 psql
Then changing the connection host value in the code to localhost, and the port to 5433.
To my surprise it works - so from locally I am seemingly able to connect to the DB using that connection string. It still doesn't work when I use it from a Cloud Run instance though. Any help is appreciated!
It seems dart doesn't support connection through unix socket, you need to configure a IP (public or private, as you need).
Alternatively you can use pg which support unix socket connection
Hope this helps.
Just for those who come across this question in the future:
as it stands right now, I had to resort to the suggestion posted by Daniele Ricci and use the public IP for the database. The one thing to point out here was that since Cloud Runs don't have a static IPv4 address to run from, the DB had to be set to allow connections from anywhere (had to add an authorized connection from 0.0.0.0/0), which is unsafe. Until the kind development team of dart figures out how to use UNIX sockets, this seems to be the only way of getting it to work.
Not having actually tested this myself, according to the source code of the postgres package, you have to specify that you want a Unix socket:
connection = PostgreSQLConnection(
...
isUnixSocket: true, // <-- here
);
The default is false.
The host you pass is must also be valid. The docs say:
[host] must be a hostname, e.g. "foobar.com" or IP address. Do not include scheme or port.
I was struggling with the same issue.
The solution is to create a connection as follows:
PostgreSQLConnection getProdConnection() {
final String connectionName = Platform.environment['CLOUD_SQL_CONNECTION_NAME']!;
final String databaseName = Platform.environment['DB_NAME']!;
final String user = Platform.environment['DB_USER']!;
final String password = Platform.environment['DB_PASS']!;
final String socketPath = '/cloudsql/$connectionName/.s.PGSQL.5432';
return PostgreSQLConnection(
socketPath,
5432,
databaseName,
username: user,
password: password,
isUnixSocket: true,
);
}
Then when you create a Cloud Run service, you need to define 'Enviroment variables' as follows.
You also need to select your sql instance in the 'connections' tab.
Then the last thing needed is to configure a Cloud Run service account.
Then the connection to instance should be successful and there should no longer be a need for a 0.0.0.0/0 connection.
However, if you try to run this connection locally on a Windows device during development the connection will not be allowed and you will be presented with this error message: 'Unix domain sockets are not available on this operating system.'
Therefore, I recommend that you open Google SQL networking to your public address and define a local environment using the 'Public IP address' of your SQL instance.
For more information on this topic, I can recommend these resources that have guided me to the right solution:
https://cloud.google.com/sql/docs/postgres/connect-instance-cloud-run#console_5
https://github.com/dart-lang/sdk/issues/47899

How to set up MySQLi connection to Google Cloud SQL

I need to use MySQL to send queries to a Google Cloud SQL database set up. I already have an instance created and a user, and I am able to access the database through the Cloud Shell. I can't seem to find the credentials to log into the database (host name, username, password, port and socket), and I'm not sure how to access them through the shell.
You can find the available methods to connect to your Cloud SQL instance here.
Connecting from an IP address without SSL is probably the easiest one:
In the Cloud Console, go to the cloud SQL instances screen and click on your instance’s name.
In the overview tab, take note of the Primary IP Address, you’ll use it instead of a hostname.
In the users tab, you can create a new user or reset the password of an existente one, including the root user.
In the authorization tab, add the ip or ip range where you are attempting the connection from, so Cloud SQL accept connections from your client (more on this here).
Start your mysql client as follows (note the port is not necessary as the default one is used):
mysql --host=[INSTANCE_IP_ADDR] --user=[USER_NAME] --password

Google Cloud SQL: SQLSTATE[HY000] [2013] Lost connection to MySQL server at 'reading initial communication packet', system error: 0

I'm desperate since my Google Cloud SQL instance went down. I could connect to it yesterday without problem but since this morning i'm unable to connect to it in any way, it produces the following error: The database server returned this error: SQLSTATE[HY000] [2013] Lost connection to MySQL server at 'reading initial communication packet', system error: 0
This is what I did to try to fix this:
restart instance
added authorized ip-addresses in CIDR notation
reset root password
restored backup
pinged the ip-address and I get response
All these actions completed but i'm still unable to connect through:
PHP
MySQL workbench
Ubuntu MySQL command line
All without luck. What could I do to repair my Cloud SQL instance. Is anyone else having this problem?
I'm from the Cloud SQL team. We are looking into this issue, it should be resolved soon. See https://groups.google.com/forum/#!topic/google-cloud-sql-announce/SwomB2zuRDo. Updates will be posted on that thread (and if there's anything particularly important I'll edit this post).
The problem seems to only affect connections from outside Google Cloud. Clients connecting from App Engine and Compute Engine should work fine.
Our company has same problem.
We are unable to connect through both MySQL workbench and MySQL command line.
Our Google Appengine application has no problems to connect since its not using external IP.
there.I encountered the same problem.You need to find out your public ip address,for that type "my public ip" in Google.Now click on your Cloud SQL instance that you created,under that click on ACCESS CONTROL tab and then click on Authorization tab under that.Under Authorized network,give any name you want to the network and copy your public ip address in the network.Now save changes and try to run the command from console.It should work fine.