I’m trying to connect to atlas cluster from an ec2, but either if I try by code (nodejs) or by cli, I get this error:
MongoError: Could not find user "arn:aws:sts::***:assumed-role/designspecs-staging-design-Api-1U4X5W-InstanceRole-1TTX7XR8B1D7N/*" for db "$external"
It is the right role, the problem is that the registered arn on atlas is the one of the role:
arn:aws:iam::***:role/designspecs-staging-design-Api-1U4X5W-InstanceRole-1TTX7XR8B1D7N
And I cannot register the STS one because atlas says it is an invalid arn.
This is the instance role which mongodb should retrieve. If I put in a .env file the keys of a iam user and I make that user a database user for Atlas it works (because the retrieved arn is correct).
Am I missing something? How can I connect the EC2 to atlas without using passwords?
For completeness I should say that I am not assuming any role explicitly, this is the connection code:
const remoteDb = `${MONGO_DATABASE_HOST}/${MONGO_DATABASE_NAME}?authSource=%24external&authMechanism=MONGODB-AWS&retryWrites=true&w=majority`;
const localDb = `mongodb://${MONGO_DATABASE_USERNAME}:${MONGO_DATABASE_PASSWORD}#${MONGO_DATABASE_HOST}:27017/${MONGO_INITDB_DATABASE}`;
const mongoURL = process.env.END !== 'dev' ? remoteDb : localDb;
const connect = () =>
mongoose
.connect(mongoURL, config)
.then(() => {
console.log('[MongoDB] CONNECTED!');
})
.catch(err => {
console.error(err);
console.error(`[MongoDB] ERRROR: NON CONNECTED! -> ${mongoURL}`);
});
connect();
module.exports = mongoose.connection;
Where MONGO_DATABASE_HOST is the srv connection string when I am on remote.
All the infrastructure is built with AWS Cloudformation, the role is associated to the instance trough the AWS::IAM::InstanceProfile.
I found that it was a problem given by the atals user scope. The user used to exist, the problem was that he didn't have the right to see the specific cluster I wanted to connect to.
This mistake was given by the fact that I used the aws quickstart template to deploy mongodb, which limits the scope of the user to some resources, removing that part now it works.
Related
I have a nextjs project that's using prismaDB for the ORM. I'm able to connect just fine to my local postgres db but I'm getting this error when running npx prisma migrate.
Error: P1001: Can't reach database server at db-name.*.us-west-2.rds.amazonaws.com:5432.
schema.prisma:
datasource db {
provider = "postgresql"
url = env("DATABASE_URL")
//url = "postgresql://master_username:master_password#aws_host:5432/db_name"
}
The RDS db is currently public and I'm positive that I've copied over the RDS credentials correctly. There doesn't seem to be anything I should be including for the connection to work but I'm not getting any other info as to why I can't reach the db server.
Seems like you have to replace db-name.*.us-west-2.rds.amazonaws.com with the name of your actual database, unless you replaced it for the purpose of asking this question. Specifically the part where it says db-name.*.
Docs: https://www.prisma.io/docs/reference/api-reference/error-reference#common
P1001 indicates that it couldn't find the database given the connection string, NOT necessarily that the credentials you provided were wrong. Make sure you're specifying the correct database name/host and whatever else you need to make it work for AWS.
Somehow I was able to connect to RDS after deleting and creating a new DB for the third time. I confirmed connection through pgAdmin then tried it again my app deployed to vercel.
Currently, our api (deployed on cloudRun) connects to our Postgres database by passing in a pgConfig with a db configuration and a db user and password.
For example:
const configObject = {
host: cloudRunHost,
user: dbUser,
password: dbPassword,
database: dbName
}
async function connect() {
if(!client) {
const pgPool = new pg.Pool(configObject);
await pgPool.connect()
.then((result) => {
logger.info('Connected to DB')
client = result;
}).catch((err) => {
logger.error(err);
});
}
}
We want the app itself to connect to the database by using Cloud SQL IAM database authentication
So far:
The api cloudRun instance has a service account
The database and CloudSQL has been configured for IAM-based access (we can access with our machine service accounts)
The api service account has access to the DB via IAM, and permissions granted to said user on the Postgres db itself
When the above code runs, it logs error: empty password returned by client
We've tried removing the password line entirely from the configObject but it hasn't helped. Any thoughts on why my service account can access the db directly but the api's can't I suspect we need to indicate to pgPool that we're trying to connect via IAM instead of via user/password.
Unfortunately, there isn't a good way to use "automatic" IAM DB AuthN on Cloud Run - while Cloud Run does use the proxy, there is no way for users to flip on the "-enable-iam-login" flag.
Go, Python, and Java users can use the language-specific connectors, but there isn't one for Node.js.
It looks like node-postgres does have support for dynamic passwords, so you might be able to do something like this:
const {GoogleAuth} = require('google-auth-library');
async function main() {
const auth = new GoogleAuth();
const pool = new pg.Pool({
connectionString: 'postgresql://user#db.example:5432/my-db',
password: async () => auth.getAccessToken(),
})
}
From your question I believe you are using NodeJS. But currently IAM database authentication is supported by Python Connector, Java Connector and Cloud SQL proxy which is mentioned here. Also to use Automatic IAM database authentication it is required to use a Cloud SQL connector.
IAM database authentication is currently supported by the Cloud SQL Auth proxy, the Java connector, and the Python connector.
Automatic IAM database authentication requires the use of a Cloud SQL connector.
As you are using NodeJS which is not supported, that seems to be the reason why you are getting an error: empty password returned by client.
I've been trying to connect to my Cloud SQL instance using the pg module but haven't been successful so far.
I've looked around a lot online but couldn't understand much on the topic. I also would like to deploy my Express app on Cloud Run at some point and have it connect to my Cloud SQL instance but I don't know how to go about doing that.
Here's a list of things I don't understand and would like a brief explanation on:
What are Unix socket connections and why should I use them over normal connections?
What is a Cloud SQL Proxy? Do I need to use it? If so, why?
Would I need to do any extra work to connect to my Cloud SQL instance from Cloud Run?
Here are all the connection objects and connection strings I have tried with the pg.Client object:
First connection string: postgresql+psycopg2://postgres:password#/cloudsql/myapp:us-central1:mydb?host=/var/lib/postgresql
Second connection string: postgresql://postgres:password#hostip:5432/myapp:us-central1:mydb
Third connection string: postgresql://postgres:password#hostip:5432/sarcdb
Connection object: { host: "/cloudsql/myapp:us-central1:mydb", username: "postgres", password: "password", database: "mydb" }
All of these give me a Connection terminated unexpectedly error.
The Cloud Functions documentation for Node.js & Cloud SQL (scroll down to PostgreSQL) has applicable information on structuring the connection string and the additional configuration needed for credentials.
Once that's in place for your app, you'll need to add the Cloud SQL instance to your Cloud Run service before it will be able to use that connection string to reach the database.
Here's directly copying the code sample from the docs, with Cloud Run the max configuration of 1 might not keep pace with other concurrency settings.
const pg = require('pg');
/**
* TODO(developer): specify SQL connection details
*/
const connectionName =
process.env.INSTANCE_CONNECTION_NAME || '<YOUR INSTANCE CONNECTION NAME>';
const dbUser = process.env.SQL_USER || '<YOUR DB USER>';
const dbPassword = process.env.SQL_PASSWORD || '<YOUR DB PASSWORD>';
const dbName = process.env.SQL_NAME || '<YOUR DB NAME>';
const pgConfig = {
max: 1,
user: dbUser,
password: dbPassword,
database: dbName,
};
if (process.env.NODE_ENV === 'production') {
pgConfig.host = `/cloudsql/${connectionName}`;
}
// Connection pools reuse connections between invocations,
// and handle dropped or expired connections automatically.
let pgPool;
exports.postgresDemo = (req, res) => {
// Initialize the pool lazily, in case SQL access isn't needed for this
// GCF instance. Doing so minimizes the number of active SQL connections,
// which helps keep your GCF instances under SQL connection limits.
if (!pgPool) {
pgPool = new pg.Pool(pgConfig);
}
pgPool.query('SELECT NOW() as now', (err, results) => {
if (err) {
console.error(err);
res.status(500).send(err);
} else {
res.send(JSON.stringify(results));
}
});
// Close any SQL resources that were declared inside this function.
// Keep any declared in global scope (e.g. mysqlPool) for later reuse.
};
What are Unix socket connections and why should I use them over normal connections?
A Unix domain socket is a socket for interprocess communication. If you have the choice between communication between a TCP connection and a Unix domain socket, the Unix domain socket is likely faster.
What is a Cloud SQL Proxy? Do I need to use it? If so, why?
The Cloud SQL proxy allows you to authenticate a connection to connect to your database using IAM permissions of a service account.
Since Cloud SQL is a cloud database, it requires (by default) some form of authentication to help it remain secure. The proxy is a more secure method of connecting compared to a self-managed SSL Certificate or a whitelisted IP address.
Would I need to do any extra work to connect to my Cloud SQL instance from Cloud Run?
Cloud Run takes care of running the proxy for you, but you need to do the following:
Enable the Cloud SQL Admin API
Add the Cloud SQL instance to your Run deployment(follow these steps).
Ensure that the service account running your code has the Cloud SQL Client IAM permissions (this is done for the default service account by step 2)
Configure your application to connect with /cloudsql/INSTANCE_CONNECTION_NAME
I have created a cluster in MongoDB Atlas and can't seem to be able to write data to it.
const uri = "mongodb+srv://myUser:myPass#myDB-4myav.mongodb.net/portfolio";
I have this as my uri to connect to, and every time I try to write to the database, I get this error:
{ MongoError: not authorized on admin to execute command { insert:
"users", documents: [[{name Daniel} {_id
ObjectIdHex("5aa6d7d6396deb25844ccb52")} {__v 0}]], ordered: false }
I have read that I need to create an admin user with the role of "root" but when I connect to my database using the mongo shell and try creating it, I get this:
Error: couldn't add user: not authorized on admin to execute command
So basically I don't have a user that can write to my database.
I've also tried making a user with every role possible on the MongoDB Atlas website (for my cluster of course) and then connecting through the mongo shell with it, but that failed as well.
To summarize: I've made a new cluster on MongoDB Atlas. How do I write data to it?
Thanks in advance, feel free to point out if I'm missing something simple and stupid.
{ MongoError: not authorized on admin to execute command { insert: "users", documents: [[{name Daniel} {_id ObjectIdHex("5aa6d7d6396deb25844ccb52")} {__v 0}]], ordered: false }
This error indicates your code is trying to insert documents into the admin database, which is a system database reserved for user and role information.
In your connection string you intended to use the portfolio database. However, the srv connection string format does not support specifying a database so your option was ignored and the default database of admin was used. You need to select the portfolio database after connecting.
This was also reported in the Mongoose issue tracker (GitHub issue #6106) where a user posted a workaround you could adapt:
mongoose.connection.on('connected', function() {
// Hack the database back to the right one, because when using mongodb+srv as protocol.
if (mongoose.connection.client.s.url.startsWith('mongodb+srv')) {
mongoose.connection.db = mongoose.connection.client.db('portfolio');
}
console.log('Connection to MongoDB established.')
});
mongoose.connect(...);
I have read that I need to create an admin user with the role of "root" but when I connect to my database using the mongo shell and try creating it, I get this:
You need to manage users via MongoDB Atlas rather than the mongo shell. You can create a user account with the "Read and write to any database" role (or more refined privileges using Advanced Options).
You need to create a user with root role from Mongo Shell,
use admin
db.createUser(
{
user: "admin",
pwd: "password",
roles: [ { role: "root", db: "admin" } ]
}
);
exit;
or you need to add new user through atlas(refer below snapshot)
Goto Clusters
click Security
Click ADD NEW USER
type username
create Password based on SCRAM-SHA1 method
choose ROLE Atlas admin
click Add User
After created user, goto Clusters-->OverView --> CONNECT
Verify the IP Whitelist
Choose a connection method:
Connect with Mongo Shell (I am choosing)
Connect your Application
Connect with MongoDB Compass
$ mongo "mongodb+srv://project-u-s3cgp.azure.mongodb.net/test" --username dbAdmin
MongoDB shell version v3.6.1
Enter password:
connecting to: mongodb+srv://project-u-s3cgp.azure.mongodb.net/test
.........
.........
MongoDB Enterprise project-U-shard-0:PRIMARY>db.user.insert({name:"Daniel"})
WriteResult({ "nInserted" : 1 })
MongoDB Enterprise O2Ci-U-shard-0:PRIMARY>
I hope, you can able to insert the record after creation of root user.
Thanks,
Karthick
TL;DR: I can get my parse dashboard talking to my locally-hosted Parse server and mongo db instance but cannot get the parse server to talk to the mLab-hosted database.
I am going through the Parse migration guide and have got mongo DB, parse-server-example and parse-dashboard running locally. When I use the following details in the parse index.js file I can successfully connect the dashboard and see the test items in the database:
databaseURI: 'mongodb://localhost:27017/dev',
cloud: __dirname + '/cloud/main.js',
appId: '1',
masterKey: '1',
serverURL: 'http://localhost:1337/parse'
I have installed mongo db locally and when connecting to my mLab instance with the shell I can see the database content. When I use that same mLab connection string in the databaseURI parameter within index.js the dashboard can no longer see the database content and the /test page on the locally-hosted parse server.
The Parse Migration Guide states...
Go to the Security & Keys section of App Settings in your Dashboard
and take note of the File Key and Master Key values. Pass that into
the ParseServer constructor in index.js. You no longer need to use a
client key with Parse Server.
I can find those keys but I cannot see where to put the File Key into the index.js.
I also do not understand why those keys are required if the locally-hosted Parse server and mLab database know nothing about them.
steps :
create your user/pwd in the mLab/mongo instance
get the db URL from mLab dashboard
connect using a command lib client to verify what parse-server will
use. this verifies the user/pwd you will use below...
go back to 'parse-server.js' to config it for mongo/remote
var databaseUri = $what-was-on-cli-client-above
var api = new ParseServer({
databaseURI: databaseUri || 'mongodb://<db.....
....