After upgrading to Gorm versions:
gorm.io/driver/postgres v1.0.2
gorm.io/gorm v1.20.2
Not able to connect to Google Cloud SQL PostgreSQL v12
The connection worked without issues on older GORM versions with:
db, err := gorm.Open("postgres", "host=/cloudsql/project_id:us-central1:sql_instance_name port=5432 user=... dbname=... password=... sslmode=disable")
New GORM version works on local install (mac) of localhost_golang / localhost_postgres
CODE on Google Cloud:
dsn := ""
dsn += "user=postgres "
dsn += "password=admin_password_here "
dsn += "host=/cloudsql/project_id:us-central1:sql_instance_name "
dsn += "dbname=db_name "
dsn += "port=5432 "
dsn += "sslmode=disable "
fmt.Println("dsn:%v", dsn)
db, err := gorm.Open(postgres.Open(dsn), &gorm.Config{
NamingStrategy: schema.NamingStrategy{
SingularTable: true,
},
})
ERROR on Google Cloud:
[0m[31m[error] [0mfailed to initialize database, got error failed to connect to host=/cloudsql/project_id:us-central1:sql_instance_name user=postgres database=db_name: dial error (dial unix /cloudsql/project_id:us-central1:sql_instance_name/.s.PGSQL.5432: connect: connection refused)
Thank you!
App Engine default service account needs proper Roles, even though the CLoud SQL Connections page indicates "App Engine authorization", All apps in this project are authorized by default.
https://cloud.google.com/sql/docs/postgres/connect-app-engine-standard?_ga=2.121100104.-1170262708.1601472976
To configure App Engine standard environment to enable connections to a Cloud SQL instance using public IP:
Make sure that the instance created above has a public IP address. You can verify this on the Overview page for your instance in the Google Cloud Console. If you need to add one, see the Configuring public IP page for instructions.
Get the INSTANCE_CONNECTION_NAME for your instance. This can be found on the Overview page for your instance in the Google Cloud Console. or by running the following command: gcloud sql instances describe [INSTANCE_NAME].
Ensure that the service account your app is using to authenticate calls to Cloud SQL has the appropriate Cloud SQL role and permissions.
The service account for your service needs one of the following IAM roles:
- Cloud SQL Client (preferred)
- Cloud SQL Editor
- Cloud SQL Admin
Or, you can manually assign the following IAM permissions:
- cloudsql.instances.connect
- cloudsql.instances.get
For detailed instructions on adding IAM roles to a service account, see Granting Roles to Service Accounts.
By default, your app will authorize your connections using an App Engine service account. The service account identity is in the format PROJECT_ID#appspot.gserviceaccount.com.
If the authorizing service account belongs to a different project than the Cloud SQL instance, the Cloud SQL Admin API and IAM permissions will need to be added for both projects.
Related
Currently, our api (deployed on cloudRun) connects to our Postgres database by passing in a pgConfig with a db configuration and a db user and password.
For example:
const configObject = {
host: cloudRunHost,
user: dbUser,
password: dbPassword,
database: dbName
}
async function connect() {
if(!client) {
const pgPool = new pg.Pool(configObject);
await pgPool.connect()
.then((result) => {
logger.info('Connected to DB')
client = result;
}).catch((err) => {
logger.error(err);
});
}
}
We want the app itself to connect to the database by using Cloud SQL IAM database authentication
So far:
The api cloudRun instance has a service account
The database and CloudSQL has been configured for IAM-based access (we can access with our machine service accounts)
The api service account has access to the DB via IAM, and permissions granted to said user on the Postgres db itself
When the above code runs, it logs error: empty password returned by client
We've tried removing the password line entirely from the configObject but it hasn't helped. Any thoughts on why my service account can access the db directly but the api's can't I suspect we need to indicate to pgPool that we're trying to connect via IAM instead of via user/password.
Unfortunately, there isn't a good way to use "automatic" IAM DB AuthN on Cloud Run - while Cloud Run does use the proxy, there is no way for users to flip on the "-enable-iam-login" flag.
Go, Python, and Java users can use the language-specific connectors, but there isn't one for Node.js.
It looks like node-postgres does have support for dynamic passwords, so you might be able to do something like this:
const {GoogleAuth} = require('google-auth-library');
async function main() {
const auth = new GoogleAuth();
const pool = new pg.Pool({
connectionString: 'postgresql://user#db.example:5432/my-db',
password: async () => auth.getAccessToken(),
})
}
From your question I believe you are using NodeJS. But currently IAM database authentication is supported by Python Connector, Java Connector and Cloud SQL proxy which is mentioned here. Also to use Automatic IAM database authentication it is required to use a Cloud SQL connector.
IAM database authentication is currently supported by the Cloud SQL Auth proxy, the Java connector, and the Python connector.
Automatic IAM database authentication requires the use of a Cloud SQL connector.
As you are using NodeJS which is not supported, that seems to be the reason why you are getting an error: empty password returned by client.
Im trying to create a connection for my SQL instance in GCP following their guide:
https://cloud.google.com/sql/docs/mysql/connect-admin-proxy
I set up the proxy running but I cant connect to my server.
I use MySQL workbench and the connection just timeout.
I went trough the trouble shoot guide and could not find the issue.
No errors in the cloud logs.
I try to connect using the owner google account of the project (I have all the permissions).
Cloud SQL Admin API is enabled.
I entered the password in the menu.
I saw another google guide telling to white list your IP.
I did this and its the same error.
It seems like there is a firewall or something is blocking from GCP to connect to the server but Im not sure what.
The solution for me was:
Use Cloud SQL authorized network as JM Gelilio suggested and to use pgAdmin 4 for Postgres connections.
Following the instructions here, I'm having problem connecting to the DB from Azure Data Studio using the token I generate. It connects to the DB successfully, but as soon as I want to run a simple query ( I already gave my user read access there), it gives me this connection error, and I need to connect using the token again and the disconnection happens again randomly after a short while:
FATAL: Cloud SQL IAM user authentication failed for user
"user#company.com" FATAL: pg_hba.conf rejects connection for host
"...", user "user#company.com", database "db-name",
SSL off
I did some search and found there is also a way of logging in with IAM database authentication using the Cloud SQL Auth proxy but the documentation is limited to Postgress command line and not a GUI database tool like Azure Data Studio. Can anyone shed some light on this about what's needed if you want to connect with a GUI tool in this case?
And about changing the pg_hba.conf file, since I work with a cloud SQL instance, I'm not sure how to turn sslmode off on the cloud instance. I checked the connection tab of my instance and SSL encryption wasn't checked there (not sure if that's the same),and I changed the sslmode to disable on my Azure Data Studio for the connection but it won't allow me to connect after this change:
FATAL: pg_hba.conf rejects connection for host "*.*.*.*", user "user#company.com", database "database", SSL off
Help, anyone?
I've found the answer: we can connect using IAM database authentication using the Cloud SQL Auth proxy. The only step after to be done from the GUI DB tool (mine is Azure Data Studio) would be, to connect to the IP (127.0.0.1 in my case)the Cloud SQL Auth proxy listens on(127.0.0.1 is the default) after starting the Cloud SQL Auth proxy using:
./cloud_sql_proxy -instances=<GCPproject:Region:DBname>=tcp:127.0.0.1:5432
Some preface:
I do not have access to the project that hosts the postgres DB.
The person who has access to the project and who set up the DB is unable to work with me directly on this. I have been told that they've followed the GCP documentation and that things are set up correctly on their end.
I am able to connect from my local deploy of the app because the person who set up the DB instance allowed my IP address; this doesn't work for an app engine flex deploy.
Problem:
I have an app engine flex instance built on golang 1.12. It connects to a postgreSQL 11 DB Cloud SQL instance in another project using github.com/jinzhu/gorm with the following code:
dbUri := fmt.Sprintf("host=%s user=%s dbname=%s password=%s", dbHost, dbUsername, dbName, dbPassword)
conn, err := gorm.Open("postgres", dbUri)
if err != nil {
log.Printf("Database error: %+v\n", err)
}
The dbhost is currently in the format of the DB's ip address.
I have tried /cloudsql/project:region:instance
I have tried /cloudsql/project:region:instance/.s.pgsql.5432
The relevant settings from the app.yaml I'm using:
beta_settings:
cloud_sql_instances: [project:region:instance]
I have tried adding =tcp:5432 with all variations of the above dbhost
Errors:
In the logs from the app engine in the cloud console, connecting provides only the following information:
sql: database is closed
As I understand, this means that no connection could be created to the database.
This error matches the error I get when running the code locally from any machine that hasn't had its IP address added as an authorized network from the postgreSQL instance.
Formalized Question:
Given the above, what code, options, settings, etc. do I need to add or change in order to successfully connect from the app engine flex deployment to the cloud SQL db?
If I had access to the project where the postgres Cloud SQL instance is, is there anything in specific I could do to ensure the connection?
Thank you.
my apologies for answering my own question.
Over the weekend I was able to solve this problem. After badgering for a double check, it turned out that the Cloud SQL instance was not set up with a private IP Address (so the assurances to me about the instance being set up correctly, and my own assurances in the question, were false).
A new Cloud SQL Instance had to be created and assigned a private IP address, as the documentation is very clear in stating that an existing Cloud SQL instance cannot be assigned a private IP address once created without one.
The App Engine Flex deploy was able to connect using the private IP address without incident.
Sorry again!
I can't seem to connect to Cloud SQL with Workbench. I keep getting this error.
Failed to Connect to MySQL at CLOUD-SQL-IPv4:3306 with user root
Access denied for use 'root'#'WHITE-LISTED-IP-ADDRESS' (using password: YES)
I have white listed my IP.
I have set an IP for the SQL instance.
I have checked the username and Password several times.
Any idea why this is happening?
It seems you have to create a new user in the Google Cloud Console with the host name set to %(any host). You can't seem to connect using the root user.
Is it possible to connect with the #root user but after you finish the setup you have to restart the sql server.
What is important to set up an SSL certificate if you connect to you production database, but if you only try it out you can allow unsecured connection. Another important thing is to add your IP to the Authorised Networks in the Connection tab.