Dataproc trying to connect to Postgres through JDBC, missing permissions - postgresql

Question

Thanks in advance...
I want to connect/write using JDBC APIs to a Postgres SQL instance running using Cloud SQL programmatically. I have used following jars:
postgresql
postgres-socket-factory
postgres-socket-factory-1.0.11-jar-with-dependencies.jar
I am running Dataproc which will try to connect using step #1, but I get following exception:
2019-04-01 11:05:03.998 IST
Something unusual has occurred to cause the driver to fail. Please report this exception.
at org.postgresql.Driver.connect(Driver.java:277)
at java.sql.DriverManager.getConnection(DriverManager.java:664)
at java.sql.DriverManager.getConnection(DriverManager.java:270)
at rdsConnector$.getConnection(rdsConnector.scala:33)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$4.run(ApplicationMaster.scala:721)
Caused by: java.lang.RuntimeException: Unable to retrieve information about Cloud SQL instance [projectID:us-east1:dB]
at com.google.cloud.sql.core.SslSocketFactory.obtainInstanceMetadata(SslSocketFactory.java:459)
at com.google.cloud.sql.core.SslSocketFactory.fetchInstanceSslInfo(SslSocketFactory.java:333)
at com.google.cloud.sql.core.SslSocketFactory.getInstanceSslInfo(SslSocketFactory.java:313)
at com.google.cloud.sql.core.SslSocketFactory.createAndConfigureSocket(SslSocketFactory.java:194)
at com.google.cloud.sql.core.SslSocketFactory.create(SslSocketFactory.java:160)
at com.google.cloud.sql.postgres.SocketFactory.createSocket(SocketFactory.java:96)
at org.postgresql.core.PGStream.<init>(PGStream.java:62)
at org.postgresql.core.v3.ConnectionFactoryImpl.tryConnect(ConnectionFactoryImpl.java:91)
at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:192)
at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:49)
at org.postgresql.jdbc.PgConnection.<init>(PgConnection.java:195)
at org.postgresql.Driver.makeConnection(Driver.java:454)
at org.postgresql.Driver.connect(Driver.java:256)
... 11 more
Caused by: com.google.api.client.googleapis.json.GoogleJsonResponseException: 403 Forbidden
2019-04-01 11:05:03.000 IST
User class threw exception: org.postgresql.util.PSQLException: Something unusual has occurred to cause the driver to fail. Please report this exception.
I understand this as a permission issue, but since I am using Dataproc to connect to Postgres what permission is missing? if I was running from a local laptop machine then I have to set GOOGLE_APPLICATION_CREDENTIALS to a json file. But what is the process in case of Dataproc?

Answer 1

The JDBC SocketFactory uses the Application Default Credentials strategy for accessing account credentials.
For Cloud Dataproc, a default service account is provided for you at [project-number]-compute#developer.gserviceaccount.com. You grant this account the "Cloud SQL Client" IAM role, the JDBC SocketFactory will use it to authenticate, and thus you will be able to connect to your application.

Answer 2

Just leave further notes for future visitors:
If the error you're getting is like:
{
"code" : 403,
"errors" : [ {
"domain" : "global",
"message" : "Insufficient Permission",
"reason" : "insufficientPermissions"
} ],
"message" : "Request had insufficient authentication scopes.",
"status" : "PERMISSION_DENIED"
}
then it could be caused by issues with the scopes (Cloud API access scopes) of the VM instances. One way to fix this is adding the scope of sql-admin when creating the Dataproc cluster. For example:
gcloud dataproc clusters create <your-cluster-name> \
--region=<your-region> \
--zone=<your-zone> \
--scopes=https://www.googleapis.com/auth/sqlservice.admin \
...
Another way is edited that in the VM instances.
See the gcloud documentation for details