I'm trying to make rds with postgres work with lambda, but no luck so far. I've read all other threads about it here, double-checked my Lambda VPC + Subnet config, it's the same as the RDS one, but still no luck connecting, what am I missing here?
Some screenshots to clarify:
Before, I enabled the Public access and I could connect through serverless offline.
Thanks!
EDIT ----
Have you verified your security group for your RDS service? It needs to allow access from the security groups given to your Lambda function. It is not enough that they are in the same VPC/subnets. The security group still needs to allow traffic on the ports for postgres (5432).
Note that for security groups you don't have to select an origin IP (which can be tricky for Lambda). But i notice you are giving your Lambda function the group sg-29aac25d. You can use that ID to give access to the RDS.
IAM policies should be irrelevant as you are authenticating against postgres. Unless your IAM doesn't allow your Lambda to execute, the problem is not IAM.
Related
This is driving me crazy, been trying to get this to work for 3 days now: I'm trying to connect a kubernetes deployment to my Cloud SQL database in GCP.
Here's what I've done so far:
Set up the cloud SQL proxy to work as a sidecar in my deployment
Created a GKE service account and attached it to my deployment
Bound the GKE service account to my GCP service account
Edited to the service account (to what I can tell) is owner permission
Yet what I run the deployment in GKE I still get:
the default Compute Engine service account is not configured with sufficient permissions to access the Cloud SQL API from this VM. Please create a new VM with Cloud SQL access (scope) enabled under "Identity and API access". Alternatively, create a new "service account key" and specify it using the -credential_file parameter
How can I fix this? I can't find any documentation on how to set up the service account to have the correct permissions with Cloud SQL or how to debug this issue. Every single tutorial I can find ends with "bind your service account" and then stops. Nothing that describes what permissions are needed, and nothing about how to actually connect to the DB from my code (how would my code talk to the proxy?).
Please help
FINALLY got it to work!
Two major pieces that the main article on this (cloud.google.com/sql/docs/mysql/connect-kubernetes-engine) glosses over:
Properly setting up workload identity, for which I found these links to be very helpful:
a) https://cloud.google.com/kubernetes-engine/docs/how-to/workload-identity
b) https://www.youtube.com/watch?v=l-nws1e4B8M
To connect to the DB you have to have your code use the DB host 127.0.0.1
I'd like to limit the privileges afforded to any given user that I create via the Google Terraform provider. By default, any user created is placed in the cloudsqlsuperuser group, and any new database created has that role/group as owner. This gives any user created via the GCP console or google_sql_user Terraform resource total control over any database that is (or was) created in a similar fashion.
So far, the best we've been able to come up with is creating and altering a user via a single-run k8s job. This seems circuitous, at best, especially given that that resource must then be manually imported later if we want to manage it via Terraform.
Is there a better way to create a user that has privileges limited to a single, application-specific database?
I was puzzled by this behaviour too. Its probably not the answer you want but if you can use GCP IAM accounts the user gets created in the PostgreSQL instance with NO roles.
There are 3 types of account you can create from "gcloud sql users create" or terraform module "google_sql_user"
"CLOUD_IAM_USER", "CLOUD_IAM_SERVICE_ACCOUNT" or "BUILT_IN"
The default is the built_in type if not specified.
CLOUD_IAM_USER and CLOUD_IAM_SERVICE_ACCOUNTS get created with NO roles.
We are using these as integration with IAM is useful in lots of ways (no managing passwords at database level is a major plus esp. when used in conjunction with SQL Auth Proxy).
BUILT_IN accounts (ie old school need a postgres username and password) for some reason are granted the "cloudsqlsuperuser" role.
In the absence of being allowed the superuser role on GCP this is about as privileged as you can get so to me (and you) seems a bizarre default.
I'm following the instructions on how to connect from GKE to Cloud SQL: https://cloud.google.com/sql/docs/postgres/connect-kubernetes-engine
It talks about YOUR-GSA-NAME. Google cloud creates "Compute Engine default service account" by default. Should I pick this one or create another service account for GKE only? What is the recommended way?
The Compute Engine default service account won't be able to connect to Cloud SQL out of the box, you'll have to add permissions to it (Cloud SQL Client role) for it to be able to connect.
I would create a new one however, as you likely don't want all GCE instances to be able to connect to Cloud SQL, and for permissions, best practice is to limit access. So just create a new SA (service account) with the Cloud SQL Client role (and any other permissions you might need GKE to access) and use that one.
This is all found in IAM -> Service Accounts in the console.
I want to give access to my developer to my MongoDB which is hosted by an EC2 Instance on AWS.
He should be able to make mongodump, upload the new backend and do some changes on our control Panel.
I created an IAM User with EC2FullAccess Permissions - I have seen that he was able to add his own IP to the Security Group so he could connect.
I don't feel so comfortable with that - what should I do, to secure myself that he has just enough access to do the necessary work:
Upload new code to server
Do MongoDB dump
I don't want him to be able to switch off/delete my instance or be able to delete my database at all.
Looking at your use case, you do not need to give any EC2 permissions, your developer does not even need IAM user, he can simply have the IP of the instance and the login credentials to the EC2 Instance, that should be suffice to log in to the instance and make the required changes. No need for an IAM user or AWS Console access.
IAM roles are for the purpose of accessing a service on behalf of another. Say, you want to access AWS DynamoDB or S3 from EC2 instance. In this case, an IAM role with required permissions attached to EC2 will server the purpose.
IAM User is for users who need access to AWS services either through Console or through API (programmatic). AWS credentials are required to access the service.
In your case, MongoDB is installed on EC2 and your developer needs access to "the server on which MongoDB is installed" and is not required any access of "AWS EC2 Service".
As correctly pointed out in answer by #X-Men, IAM role or IAM user is not at all required. What required is, your developer to have the IP of server and credentials to login to that server. Username-password or username-key.
Restriction which you need on developer related to MongoDB are to be configured on MongoDB itself and not on EC2 level.
Is it possible to use AWS KMS and a tool like credstash without the use of EC2 or equivalent or does it rely solely on IAM roles?
I've got a server elsewhere where I am testing some things out and ultimately I will be looking at migrating an app to EC2 etc. to make use of scaling. But for now whilst I'm setting up my deployment pipeline etc. I wondered if it was still possible to make use of KMS on my non-aws provisioned server?
The only possible way I can think of is by installing the AWS CLI tools on the server in question. Does this sounds like the right approach?
What #Viccari said is correct (in the comments). In terms of what you want to do (store passwords), the AWS Parameter Store would be a good fit for you. See https://docs.aws.amazon.com/systems-manager/latest/userguide/systems-manager-paramstore.html for more information. The guide explicitly calls out your use-case:
Parameter Store offers the following benefits and features.
Use a secure, scalable, hosted secrets management service (No servers to manage).
In the end, if you end up using Parameter Store or KMS, you will need some sort of credentials stored somewhere to grab an AWS STS token to use to call the underlying AWS services. If working outside of AWS EC2, you will need the AWS Access Key and AWS Secret Key from an IAM user. If you are in EC2, the IAM instance role will magically provide you the credentials and use that role to call those AWS services. The AWS SDK does this for you behind the scenes.
But, as you state, you don't want to run this in EC2 (to save money, or other reasons). The quickest way to store these credentials is to have them in a un-tracked file (added to your .gitignore) you can source from as environment variables, which your program will then read. This allows you to do local testing, and easily run it in EC2
with zero code changes. See https://docs.aws.amazon.com/cli/latest/userguide/cli-environment.html for what variables to set. Note that this doc talks about the CLI; the SDK's follow the same behavior.