I am writing a Python app to run as lambda function and want to connect to an RDS DB instance without making it publicly accessible.
The RDS DB instance was already created under the default VPC with security group "sg-abcd".
So I have:
created a lambda function under the same default VPC
created a role with the following permission AWSLambdaVPCAccessExecutionRole and assigned it to the lambda function as in https://docs.aws.amazon.com/lambda/latest/dg/services-rds-tutorial.html
set sg-abcd as the lambda function's security group
added sg-abcd as source in the security group's inbound rules
added the CIDR range of the lambda function's subnet as source in the security group's inbound rules
However, when I invoke the lambda function it times out.
I can connect to the RDS DB from my laptop (after setting my IP as source in the sg's inbound rules), so I now that it is not an authentication problem. Also, for the RDS DB "Publicly Accessible" is set to "Yes".
Here's part of the app's code (where I try to connect):
rds_host = "xxx.rds.amazonaws.com"
port = xxxx
name = rds_config.db_username
password = rds_config.db_password
db_name = rds_config.db_name
logger = logging.getLogger()
logger.setLevel(logging.INFO)
try:
conn = psycopg2.connect(host=rds_host, database=db_name, user=name, password=password, connect_timeout=20)
except psycopg2.Error as e:
logger.error("ERROR: Unexpected error: Could not connect to PostgreSQL instance.")
logger.error(e)
sys.exit()
I really can't understand what I am missing. Any suggestion is welcomed, please help me figure it out!
Edit: the inbound rules that I have set look like this:
Security group rule ID: sgr-123456789
Type Info: PostgreSQL
Protocol Info: TPC
Port range Info: 5432
Source: sg-abcd OR IP or CIDR range
This document should help you out. Just make sure to get the suggestions for your specific scenario, whether the lambda function and the RDS instance are in the same VPC or not.
In my case I have the lambda function and the RDS instance in the same VPC and also both have the same subnets and SGs. But just make sure to follow the instructions in that document for the configurations needed for each scenario.
Related
Hi all,
I am trying to execute queries on a postgresql database I created in AWS.
I added a data source to Athena, I created the data source for postgresql and I created the lambda function.
In Lambda function I set:
default connection string
spill_bucket and spill prefix (I set the same for both: 'athena-spill'. In the S3 page I cannot see any athena-spill bucket)
the security group --> I set the security group I created to access the db
the subnet --> I set one of the database subnet
I deployed the lambda function but I received an error and I had to add a new environment variable created with the connection string but named as 'dbname_connection_string'.
After adding this new env variable I am able to see the database in Athena but when I try to execute any query on this database as:
select * from tests_summary limit 10;
I receive this error after running query:
GENERIC_USER_ERROR: Encountered an exception[com.amazonaws.SdkClientException] from your LambdaFunction[arn:aws:lambda:eu-central-1:449809321626:function:data-production-athena-connector-nina-lambda] executed in context[retrieving meta-data] with message[Unable to execute HTTP request: Connect to s3.eu-central-1.amazonaws.com:443 [s3.eu-central-1.amazonaws.com/52.219.170.25] failed: connect timed out]
This query ran against the "public" database, unless qualified by the query. Please post the error message on our forum or contact customer support with Query Id: 3366bd80-143e-459c-a4da-5350b5ab4a77
What could be causing the problem?
Thanks a lot!
Root Cause:
VPC have no internet connection issue, causing Lambda can't access S3.
Solution:
Add VPC Gateway Endpoint (Select com.amazonaws.eu-central-1.s3) in Lambda associated VPC.
I have a query I'd like to run regularly in Redshift. I've set up an AWS Data Pipeline for it.
My problem is that I cannot figure out how to access Redshift. I keep getting "Unable to establish connection" errors. I have an Ec2Resource and I've tried including a subnet from our cluster's VPC and using the Security Group Id that Redshift uses, while also adding that sg-id to the inbound part of the rules. No luck.
Does anyone have a from-scratch way to set up a data pipeline to run against Redshift?
How I currently have my pipeline set up
RedshiftDatabase
Connection String: jdbc:redshift://[host]:[port]/[database]
Username, Password
Ec2Resource
Resource Role: DataPipelineDefaultResourceRole
Role: DataPipelineDefaultRole
Terminate after: 20 minutes
SqlActivity
Database: [database] (from Connection String)
Runs on: Ec2Resource
Script: SQL query
Error message
Unable to establish connection to jdbc:postgresql://[host]:[port]/[database] Connection refused. Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connections.
Ok, so the answer lies in Security Groups. I had to find the Security Group my Redshift cluster is in, and then add that as a value to "Security Group" parameter on the Ec2Resource in the DataPipeline.
Ec2Resource
Resource Role: DataPipelineDefaultResourceRole
Role: DataPipelineDefaultRole
Terminate after: 20 minutes
Security Group: sg-XXXXX [pull from Redshift]
Try opening inbound rules to all sources, just to narrow down possible causes. You've probably done this, but make sure you've set up your jdbc driver and configurations according to this.
(I'm afraid I'm probably about to reveal myself as completely unfit for the task at hand!)
I'm trying to setup a Redshift cluster and database to help manage data for a class/group project.
I have a dc2.large cluster running with either default options, or what looked like the most generic in the couple of place I was forced to make entries.
I have downloaded Aginity (Win64) as it is described as being specialized for Redshift. That said, I can't find any instructions for connecting using it. The connection dialog requests the follwoing:
Server: using the endpoint for my cluster (less :57xx at the end).
UserID: the Master username for the database defined for the cluster.
Password: to match the UserID
SSL Mode (Disable, Allow, Prefer, Require): trying various options
Database: as named in cluster setup
Port: as defined in cluster setup
I can't get it to connect ("failed to establish connection") and don't know if I'm entering something wrong in Aginity or if I haven't set up my cluster properly.
Message: Failed to establish a connection to 'abc1234-smtm.crone7m2jcwv.us-east-1.redshift.amazonaws.com'.
Type : Npgsql.NpgsqlException
Source : Npgsql
Trace : at Npgsql.NpgsqlClosedState.Open(NpgsqlConnector context, Int32 timeout)
at Npgsql.NpgsqlConnector.Open()
at Npgsql.NpgsqlConnection.Open()
at Aginity.MPP.Common.BaseDataProvider.get_Connection()
at Aginity.MPP.Common.BaseDataProvider.CreateCommand(String commandText, CommandType commandType, IDataParameter[] commandParams)
at Aginity.MPP.Common.BaseDataProvider.ExecuteReader(String commandText, CommandType commandType, IDataParameter[] commandParams)
--- Inner Exception: ---
......
It seems there is not enough information going into Aginity to authorize connection to my cluster - no account credential are supplied. For UserID, am I meant to enter the ID of a valid user? Can I use the root account? What would the ID look like? I have setup a User with FullAccess to S3 and Redshift, then entered the UserID in this format
arn:aws:iam::600123456789:user/john
along with the matching password, but that hasn't worked either.
The only training/tutorial I have been able to find/do on this is the Intro AWS direct you to, at https://qwiklabs.com/focuses/2366, which uses a web-based client that I can't find outside of the tutorial (pgweb).
Any advice what I am doing wrong, and how to do it right?
Well, I think I got it working - I haven't had a chance to see if I can actually create table yet, but it seems to be connected. I had to allow inbound traffic from outside the VPC, as per the above snapshot.
I'm guessing there's a better way than opening it up to all IP addresses, but I don't know the users' (fellow team members) IPs, and aren't they all subject to change depending on the device they're using to connect?
How does one go about getting inside the VPC to connect that way, presumably more securely?
I want to delete an AWS VPC which I don't know how it came into existence. When I try to delete it in AWS Console, it says:
We could not delete the following VPC (vpc-0a72ac71) Network interface
'eni-ce2a0d10' is currently in use. (Service: AmazonEC2; Status Code:
400; Error Code: InvalidParameterValue; Request ID:
821d8a6d-3d9b-4c24-b372-314ea9b18b23)
As it mentions "AmazonEC2" in the error message, I suspected there might be some EC2 instances residing in this VPC. So I went into EC2 dashboard but found no EC2 exist there. However, I found there are two security groups associated with this vpc. So I decided to delete them hoping that's the cause of the error. But when I tried to do so, I got this message:
As the message says, these security groups are associated with some network interfaces. Therefore, I decided to 'Detach' those but I got this error message:
Error deleting network interfaces eni-ce2a0d10: You do not have
permission to access the specified resource. eni-0b7ff712: You do not
have permission to access the specified resource.
But I'm the root user so I assume I should be able to do whatever I want to do except if the resource is made by aws itself or another root account.
I know somewhere this network interface is being used but it will be very time-consuming to go through each aws service and check that.
I've already checked AWS RDS service and no instance or rds subnet is made.
I've already checked this question and this with no luck.
I found the root cause of this issue.
Short Answer:
That VPC was created solely for the WorkDocs service instance. So AWS was preventing me to delete its VPC and any of its dependent services and pieces.
How I figured it out:
First, I noticed something interesting has been written in the 'Description' column of the 'undeletable' Network Interfaces (you can see them in the last OP's figure):
"AWS created network interface for directory d-90672d6b72."
From "directory", I suspected that this might have something do to with AWS Directory Service. So I went to this service and noticed there is a directory associated with the VPC:
So I tried to remove this directory but I got this error message:
Error - Directory cannot be deleted This directory still has
authorized applications, and cannot be deleted. To delete this
directory, complete all of the following steps: • Delete the WorkDocs
site attached to this directory.
Therefore, I went to AWS WorkDocs Service and found it and deleted it:
So now the directory is also deleted (circled in red), I went back to delete those network interfaces. However I realized that they are vanished! (I guess Amazon removed them on its own). I went to VPC service to see whether I'm now able to delete the VPC. Guess what? That VPC was vanished too!
Now I understand what was happening. That VPC was created solely for the WorkDocs service instance. I wish Amazon was more transparent about it.
As a more generic answer to the "Error deleting network interface" issue, it happens when a network interface was created automatically for a higher-level AWS resource.
The Generic solution is to manage the network interface in the higher level resource directly such as WorkDocs or EFS.
In my case it happened when I wanted to delete a security group assigned to network interfaces created by an EFS volume.
So I went in the EFS console and removed the security group from the EFS.
I have a Python Lambda function that's not able to connect to MongoDB (private IP) through a VPC.
I've ported out the part that's failing in the snippet below.
import pymongo
def handler(event, context):
try:
client = pymongo.MongoClient(host=['10.155.152.15:27017'], document_class=dict, tz_aware=False, connect=True, serverselectiontimeoutms=5)
except pymongo.errors.ServerSelectionTimeoutError as err:
print err
Result:
No servers found yet
For the user role, it has this Policy attached: (AWSLambdaVPCAccessExecutionRole) and (AWSLambdaBasicExecutionRole)
. I've even allowed (AmazonVPCFullAccess) just to see, but it didn't help.
I've selected the same VPC as what MongoDB server using, along with all of the subnets.
Security Group is BasicLambda which allows all traffic.
This has worked for me yesterday but somehow it's not working anymore. It seems to be some kind of connection permission issues.