Connecting with pg.Pool and GCP IAM-based database authentication - postgresql

Currently, our api (deployed on cloudRun) connects to our Postgres database by passing in a pgConfig with a db configuration and a db user and password.
For example:
const configObject = {
host: cloudRunHost,
user: dbUser,
password: dbPassword,
database: dbName
}
async function connect() {
if(!client) {
const pgPool = new pg.Pool(configObject);
await pgPool.connect()
.then((result) => {
logger.info('Connected to DB')
client = result;
}).catch((err) => {
logger.error(err);
});
}
}
We want the app itself to connect to the database by using Cloud SQL IAM database authentication
So far:
The api cloudRun instance has a service account
The database and CloudSQL has been configured for IAM-based access (we can access with our machine service accounts)
The api service account has access to the DB via IAM, and permissions granted to said user on the Postgres db itself
When the above code runs, it logs error: empty password returned by client
We've tried removing the password line entirely from the configObject but it hasn't helped. Any thoughts on why my service account can access the db directly but the api's can't I suspect we need to indicate to pgPool that we're trying to connect via IAM instead of via user/password.

Unfortunately, there isn't a good way to use "automatic" IAM DB AuthN on Cloud Run - while Cloud Run does use the proxy, there is no way for users to flip on the "-enable-iam-login" flag.
Go, Python, and Java users can use the language-specific connectors, but there isn't one for Node.js.
It looks like node-postgres does have support for dynamic passwords, so you might be able to do something like this:
const {GoogleAuth} = require('google-auth-library');
async function main() {
const auth = new GoogleAuth();
const pool = new pg.Pool({
connectionString: 'postgresql://user#db.example:5432/my-db',
password: async () => auth.getAccessToken(),
})
}

From your question I believe you are using NodeJS. But currently IAM database authentication is supported by Python Connector, Java Connector and Cloud SQL proxy which is mentioned here. Also to use Automatic IAM database authentication it is required to use a Cloud SQL connector.
IAM database authentication is currently supported by the Cloud SQL Auth proxy, the Java connector, and the Python connector.
Automatic IAM database authentication requires the use of a Cloud SQL connector.
As you are using NodeJS which is not supported, that seems to be the reason why you are getting an error: empty password returned by client.

Related

AWS EC2 connection to MongoDB Atlas failing, could not find user

I’m trying to connect to atlas cluster from an ec2, but either if I try by code (nodejs) or by cli, I get this error:
MongoError: Could not find user "arn:aws:sts::***:assumed-role/designspecs-staging-design-Api-1U4X5W-InstanceRole-1TTX7XR8B1D7N/*" for db "$external"
It is the right role, the problem is that the registered arn on atlas is the one of the role:
arn:aws:iam::***:role/designspecs-staging-design-Api-1U4X5W-InstanceRole-1TTX7XR8B1D7N
And I cannot register the STS one because atlas says it is an invalid arn.
This is the instance role which mongodb should retrieve. If I put in a .env file the keys of a iam user and I make that user a database user for Atlas it works (because the retrieved arn is correct).
Am I missing something? How can I connect the EC2 to atlas without using passwords?
For completeness I should say that I am not assuming any role explicitly, this is the connection code:
const remoteDb = `${MONGO_DATABASE_HOST}/${MONGO_DATABASE_NAME}?authSource=%24external&authMechanism=MONGODB-AWS&retryWrites=true&w=majority`;
const localDb = `mongodb://${MONGO_DATABASE_USERNAME}:${MONGO_DATABASE_PASSWORD}#${MONGO_DATABASE_HOST}:27017/${MONGO_INITDB_DATABASE}`;
const mongoURL = process.env.END !== 'dev' ? remoteDb : localDb;
const connect = () =>
mongoose
.connect(mongoURL, config)
.then(() => {
console.log('[MongoDB] CONNECTED!');
})
.catch(err => {
console.error(err);
console.error(`[MongoDB] ERRROR: NON CONNECTED! -> ${mongoURL}`);
});
connect();
module.exports = mongoose.connection;
Where MONGO_DATABASE_HOST is the srv connection string when I am on remote.
All the infrastructure is built with AWS Cloudformation, the role is associated to the instance trough the AWS::IAM::InstanceProfile.
I found that it was a problem given by the atals user scope. The user used to exist, the problem was that he didn't have the right to see the specific cluster I wanted to connect to.
This mistake was given by the fact that I used the aws quickstart template to deploy mongodb, which limits the scope of the user to some resources, removing that part now it works.

Pony ORM and IAM based connection to AWS PostgreSQL RDS

I am building a service using Python, Pony ORM and PostgreSQL. I plan hosting it in AWS. I will use AWS RDS PostgreSQL.
AWS supports passwordless connection to RDS (https://aws.amazon.com/premiumsupport/knowledge-center/rds-postgresql-connect-using-iam/, https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/UsingWithRDS.IAMDBAuth.Connecting.Python.html).
Essentially, I have to generate a short-living password before connecting to my database:
session = boto3.Session(profile_name='RDSCreds')
client = session.client('rds')
token = client.generate_db_auth_token(DBHostname=ENDPOINT, Port=PORT, DBUsername=USER, Region=REGION)
conn = psycopg2.connect(host=ENDPOINT, port=PORT, database=DBNAME, user=USER, password=token, sslrootcert="SSLCERTIFICATE")
Is there a way to integrate this approach with Pony ORM? Under the hood Pony just delegates connection to psycopg2. psycopg2 accepts username and password. But how can I make sure that when Pony reconnects to the database, there's a token that it can use?

Connecting to Cloud SQL from Azure Data Studio using an IAM user

Following the instructions here, I'm having problem connecting to the DB from Azure Data Studio using the token I generate. It connects to the DB successfully, but as soon as I want to run a simple query ( I already gave my user read access there), it gives me this connection error, and I need to connect using the token again and the disconnection happens again randomly after a short while:
FATAL: Cloud SQL IAM user authentication failed for user
"user#company.com" FATAL: pg_hba.conf rejects connection for host
"...", user "user#company.com", database "db-name",
SSL off
I did some search and found there is also a way of logging in with IAM database authentication using the Cloud SQL Auth proxy but the documentation is limited to Postgress command line and not a GUI database tool like Azure Data Studio. Can anyone shed some light on this about what's needed if you want to connect with a GUI tool in this case?
And about changing the pg_hba.conf file, since I work with a cloud SQL instance, I'm not sure how to turn sslmode off on the cloud instance. I checked the connection tab of my instance and SSL encryption wasn't checked there (not sure if that's the same),and I changed the sslmode to disable on my Azure Data Studio for the connection but it won't allow me to connect after this change:
FATAL: pg_hba.conf rejects connection for host "*.*.*.*", user "user#company.com", database "database", SSL off
Help, anyone?
I've found the answer: we can connect using IAM database authentication using the Cloud SQL Auth proxy. The only step after to be done from the GUI DB tool (mine is Azure Data Studio) would be, to connect to the IP (127.0.0.1 in my case)the Cloud SQL Auth proxy listens on(127.0.0.1 is the default) after starting the Cloud SQL Auth proxy using:
./cloud_sql_proxy -instances=<GCPproject:Region:DBname>=tcp:127.0.0.1:5432

Can't connect to Cloud SQL using node-postgres

I've been trying to connect to my Cloud SQL instance using the pg module but haven't been successful so far.
I've looked around a lot online but couldn't understand much on the topic. I also would like to deploy my Express app on Cloud Run at some point and have it connect to my Cloud SQL instance but I don't know how to go about doing that.
Here's a list of things I don't understand and would like a brief explanation on:
What are Unix socket connections and why should I use them over normal connections?
What is a Cloud SQL Proxy? Do I need to use it? If so, why?
Would I need to do any extra work to connect to my Cloud SQL instance from Cloud Run?
Here are all the connection objects and connection strings I have tried with the pg.Client object:
First connection string: postgresql+psycopg2://postgres:password#/cloudsql/myapp:us-central1:mydb?host=/var/lib/postgresql
Second connection string: postgresql://postgres:password#hostip:5432/myapp:us-central1:mydb
Third connection string: postgresql://postgres:password#hostip:5432/sarcdb
Connection object: { host: "/cloudsql/myapp:us-central1:mydb", username: "postgres", password: "password", database: "mydb" }
All of these give me a Connection terminated unexpectedly error.
The Cloud Functions documentation for Node.js & Cloud SQL (scroll down to PostgreSQL) has applicable information on structuring the connection string and the additional configuration needed for credentials.
Once that's in place for your app, you'll need to add the Cloud SQL instance to your Cloud Run service before it will be able to use that connection string to reach the database.
Here's directly copying the code sample from the docs, with Cloud Run the max configuration of 1 might not keep pace with other concurrency settings.
const pg = require('pg');
/**
* TODO(developer): specify SQL connection details
*/
const connectionName =
process.env.INSTANCE_CONNECTION_NAME || '<YOUR INSTANCE CONNECTION NAME>';
const dbUser = process.env.SQL_USER || '<YOUR DB USER>';
const dbPassword = process.env.SQL_PASSWORD || '<YOUR DB PASSWORD>';
const dbName = process.env.SQL_NAME || '<YOUR DB NAME>';
const pgConfig = {
max: 1,
user: dbUser,
password: dbPassword,
database: dbName,
};
if (process.env.NODE_ENV === 'production') {
pgConfig.host = `/cloudsql/${connectionName}`;
}
// Connection pools reuse connections between invocations,
// and handle dropped or expired connections automatically.
let pgPool;
exports.postgresDemo = (req, res) => {
// Initialize the pool lazily, in case SQL access isn't needed for this
// GCF instance. Doing so minimizes the number of active SQL connections,
// which helps keep your GCF instances under SQL connection limits.
if (!pgPool) {
pgPool = new pg.Pool(pgConfig);
}
pgPool.query('SELECT NOW() as now', (err, results) => {
if (err) {
console.error(err);
res.status(500).send(err);
} else {
res.send(JSON.stringify(results));
}
});
// Close any SQL resources that were declared inside this function.
// Keep any declared in global scope (e.g. mysqlPool) for later reuse.
};
What are Unix socket connections and why should I use them over normal connections?
A Unix domain socket is a socket for interprocess communication. If you have the choice between communication between a TCP connection and a Unix domain socket, the Unix domain socket is likely faster.
What is a Cloud SQL Proxy? Do I need to use it? If so, why?
The Cloud SQL proxy allows you to authenticate a connection to connect to your database using IAM permissions of a service account.
Since Cloud SQL is a cloud database, it requires (by default) some form of authentication to help it remain secure. The proxy is a more secure method of connecting compared to a self-managed SSL Certificate or a whitelisted IP address.
Would I need to do any extra work to connect to my Cloud SQL instance from Cloud Run?
Cloud Run takes care of running the proxy for you, but you need to do the following:
Enable the Cloud SQL Admin API
Add the Cloud SQL instance to your Run deployment(follow these steps).
Ensure that the service account running your code has the Cloud SQL Client IAM permissions (this is done for the default service account by step 2)
Configure your application to connect with /cloudsql/INSTANCE_CONNECTION_NAME

Mongo admin database credentials don't work in other databases on server

The documentation for Mongo states that when authentication is enabled, and for users added to the admin database, these users should be able to access the other databases in Mongo, with the rights granted at the admin database level.
"The admin database is unique. Users with normal access to the admin database have read and write access to all databases. Users with read only access to the admin database have read only access to all databases." http://docs.mongodb.org/manual/administration/security/
But in testing with the C# library version 1.7.0.4714, this is not the case.
Only accounts created in a specific database have access to that database.
I have tested with credentials on the connection string
and by setting credentials explicitly at the database level in C#
server.GetDatabase(...
new MongoClient(a connectionString ...
Does anyone know if this expected behavior? or can suggest a resolution.
The answer was already posted here on stackoverflow :)
Mongodb C# driver - can't use admin authentication to access other databases
the username used should have (admin) after it to use that account.
This is not a mongodb problem. I am sure that you could authenticate from mongodb shell like this:
use admin
db.auth(user, pass)
This is a mongodb c# driver trick. A long time ago i spent some time to read c# driver code in order to understand this.
So connection string should be like this:
mongodb://admin(admin):1#localhost:27020/myDb
The trick in (admin) in order to tell the driver that you are going to authenticate via admin user.
There is another way to connect with authentication against the admin database.
The downside is that you have to setup the whole connection object instead of packing all info solely on a connection string.
Instead of instantiating the MongoClient with a connection string like
var connectionString = "mongodb://localhost";
var client = new MongoClient(connectionString);
you can create a MongoClientSettings object, set the credentials (along with any other connection settings) and instantiate the client passing that object
string authenticationDB = "admin"
string authenticationUsername = "user"
string authenticationPassword = "pass"
MongoClientSettings settings = new MongoClientSettings();
settings.Credentials = new[] { MongoCredential.CreateMongoCRCredential(authenticationDB, authenticationUsername, authenticationPassword) };
settings.Servers = new[] { new MongoServerAddress("host_1"), new MongoServerAddress("host_2"), new MongoServerAddress("host_3") };
settings.ConnectionMode = ConnectionMode.ReplicaSet;
var client = new MongoClient(settings);
var db = client.GetServer().GetDatabase(database);
http://docs.mongodb.org/ecosystem/tutorial/authenticate-with-csharp-driver/