Connectivity issue with AWS DMS with Postgresql on RDS - postgresql

I have 2 Aws RDS instances,(Run on Postgresql). Both are on Different accounts and different regions. I want to set up data replication between them, using AWS DMS.
I tried doing VPC peering.
I saw the following video to enable VPC peering :-
https://www.youtube.com/watch?v=KmCEFGDTb8U
The Problem:-
When I try creating the AWS DMS service, I add the Hostname, Username and Password, etc for the source(Which exists on the other account), and when I hit Test Connection, I get the following error.
Test Endpoint failed: Application-Status: 1020912, Application-Message: Failed to connect Network error has occurred, Application-Detailed-Message: RetCode: SQL_ERROR SqlState: 08001 NativeError: 101 Message: [unixODBC]timeout expired ODBC general error.
To my surprise, I get a similar error when I hit the Test Connection for the Target RDS instance, which is in the same account. i.e.:-
Test Endpoint failed: Application-Status: 1020912, Application-Message: Cannot connect to ODBC provider Network error has occurred, Application-Detailed-Message: RetCode: SQL_ERROR SqlState: 08001 NativeError: 101 Message: [unixODBC]timeout expired ODBC general error.
Google suggests that we are having some sort of Firewall, but looking at the NACLs I can see we allow 0.0.0.0/0 for both the VPC's.

If you're attempting to access the private IP ranges in one IP from another IP, in addition to creating the VPC Peering connections, you'll have to:
create route table entries in both VPCs to route traffic to the remote VPC's IP range(s) through the Peering Connection,
allow connections within the security groups, both from the source CIDR range in the destination security group, and, if you're filtering outgoing connections from the source, also in it's outbound rules. Note that you can't use Security Group Id to allow this traffic because it doesn't apply to cross region peering;
allow the connection in the underyling software ( probably allowed by default ),
allow the network ACL to pass the traffic ( you've verified that's also allowed by default)
Since you're seeing timeouts, I'd suspect the security group rules. But, it could also be a bad route.

As suggested here https://aws.amazon.com/premiumsupport/knowledge-center/dms-endpoint-connectivity-failures/
When modifying the Replication Instance used to test connection to the Endpoint, take note of:
Private IP Address
VPC Security Group
Either change the Security Group to a suitable one or edit the Security Group being used adding an Inbound Rule to allow PostgreSQL traffic Type from the Private IP Address of the Replication Instance.

The below solution worked for me.
Create replication instance, then endpoints.
If the test endpoints fails - then ensure to pick up the private IP from the instance(if DMS replication instance and the DB are located within the same VPC) and add it to the inbound rules of the corresponding security ID.
If the VPC's are in different region, you might need VPC Peering to get this sorted.
Since I had both running in the same VPC, adding the private IP to inbound rules worked fine and the connection is successful.

Related

Unable to connect to CloudSQL instance that is part of a VPC network

I have created a CloudSQL instance which was part of a VPC I have created.
I'm able to connect to this CloudSQL using CloudSQL Proxy service. But I'm unable to connect to this instance using public IP of the instance though I added the firewall rule to this VPC.
The error I'm getting:
Unable to connect to host <public-ip-of-cloudsql>, or the request timed out.
Be sure that the address is correct and that you have the necessary privileges, or try increasing the connection timeout (currently 10 seconds).
MySQL said:
Can't connect to MySQL server on '<public-ip-of-cloudsql>' (4)
Following is the firewall rule I added and provided my home IP address in the blocked out area.
Please let me know if I'm missing something. I can provide more details if needed.
These are the steps you should follow in order to connect to Cloud SQL using the public IP:
Created a Cloud SQL instance, including configuring the default user.
Assuming you use a local client:
2.Install the client.
3.Configure access to your Cloud SQL instance.
4.Connect to your Cloud SQL instance.
You can find a detailed explanation here: Connecting MySQL client using public IP
If you are using the Cloud SQL proxy to connect via public ip, it requires port 3307 to be open to the address.
If you aren't using the Cloud SQL proxy to connect via public ip, you need to authorize your external IP.
I was able to connect CloudSQL which is part of a VPC by just adding the client IP address as Authorized networks.
It's weird, I tried many times before but couldn't succeed. It is working now.
Thanks, guys for answers.

connect to amazon RDS on scaleway server

I wanna try amazon RDS for my project, so I created micro instance in amazon AWS. I am new to AWS, I added my server IP to security group and VPC. I tried to connect from server:
psql -h digrin.asdada.us-west-2.rds.amazonaws.com -p 5432 -d digrin -U my_username -W
psql: could not connect to server: Connection timed out
Is the server running on host "digrin.asdada.us-west-2.rds.amazonaws.com" (54.71.212.12) and accepting
TCP/IP connections on port 5432?
I can login from localhost, so I was going through amazon AWS settings multiple times and added IP of server everywhere I thought might be causing problems. No luck.
Then I tried to login from another server, which IP I did not add anywhere to AWS and I did connect successfully - so my database is pulicly available.
So I thought my server provider is blocking 5432 port or something such. I created ticket with them (Scaleway) and this is what I got:
Hello,
Please understand that we do not provide support regarding software management / Firewall configuration.
On our end, nothing is locked network-wise on any port unless you modify your security group's configuration.
I don't understand where the issue could be. Any ideas?
Network access to your RDS instances is controlled by a security group which acts somewhat like a firewall. Rules can be crafted in security groups to allow other members of that security group to connect to each other, which is the default policy in the default security group. Thus because your RDS instance and your EC2 instance are members of the same security group they can communicate freely but nothing else outside of that group can communicate with either host.
An instance can be a member of more than one security group at a time and if there are rules allowing the traffic in any of the security groups for which the instance is a member then the traffic will be allowed. In the spirit of security I would suggest not opening more ports than necessary and would not recommend opening ports within the default security group as that would effectively make those ports accessible on all instances in that security group.
So, to allow access to your Scaleway host:
Create a new security group from the VPC console, add a rule to this group that allows the MySQL port to be accessed from the IP address of your Scaleway host (or from anywhere, depending on your level of security concerns)
Attach that security group to your database in addition to the default security group through the RDS console

External Redshift connection only works upon full access instead of ip restricted

I want to connect externally to my Redshift cluster (VPC, NOT classic) using aginity workbench. Hence I added my public ip address to the ec2 inbound rules, but I get a connection timeout.
When I allow all traffic to the inbound rules (0.0.0.0/0) it is possible to connect.Off course this is not a preferred solution because of security.
Does anybody have an idea why/where it is failing with my public ip (grabbed from whatismyipaddress.com)?

AWS RDS Postgresql Pgadmin - Server doesn't listen

I followed the aws tutorial found here.
Everything went smoothly up until connecting to the postgresql instance via pgadmin.
I entered the appropriate user/pw info and copy/pasted the address of the db appropriately.
The port is indeed 5432 on my aws dashboard.
I am receiving the following error message:
Server doesn't listen
The server doesn't accept connections: the connection library reports
could not connect to server: Operation timed out Is the server running on host "my_database_name.some_stuff.us-west-2.rds.amazonaws.com" (52.10.228.18) and accepting TCP/IP connections on port 5432?
If you encounter this message, please check if the server you're trying to contact is actually running PostgreSQL on the given port. Test if you have network connectivity from your client to the server host using ping or equivalent tools. Is your network / VPN / SSH tunnel / firewall configured correctly?
For security reasons, PostgreSQL does not listen on all available IP addresses on the server machine initially. In order to access the server over the network, you need to enable listening on the address first.
For PostgreSQL servers starting with version 8.0, this is controlled using the "listen_addresses" parameter in the postgresql.conf file. Here, you can enter a list of IP addresses the server should listen on, or simply use '*' to listen on all available IP addresses. For earlier servers (Version 7.3 or 7.4), you'll need to set the "tcpip_socket" parameter to 'true'.
You can use the postgresql.conf editor that is built into pgAdmin III to edit the postgresql.conf configuration file. After changing this file, you need to restart the server process to make the setting effective.
If you double-checked your configuration but still get this error message, it's still unlikely that you encounter a fatal PostgreSQL misbehaviour. You probably have some low level network connectivity problems (e.g. firewall configuration). Please check this thoroughly before reporting a bug to the PostgreSQL community.
Step 1
You are getting the same dialog I was seeing above. Crap!
Step 2
Go to your RDS instances
Step 3
Go to your security groups
Step 4
If your account was like mine you see this text:
Your account does not support the EC2-Classic Platform in this region.
DB Security Groups are only needed when the EC2-Classic Platform is supported.
Instead, use VPC Security Groups to control access to your DB Instances.
Go to the EC2 Console to view and manage your VPC Security Groups.
For more information, see AWS Documentation on Supported Platforms and Using RDS in VPC.
Step 5 Go back and check your RDS security group name (RDS->instances right click your instance). You will see something like Security GroupsList of VPC Security Groups associated with this DB Instance.
You will see something like:
default (sg-********) ( active )
Step 6 In your VPC security groups find your sg-******** that matches your database. Right click that. Edit inbound/outbound rules to add postgresql.
Try to connect again.
This solved my problem.
If this does not solve your problem I am very sorry, but I hope this documentation brings me some debugging karma.
go to AWS services in security group click on the security group id . from the "actions" button click on "edit inbound roles" and then change the "source" to "my ip"

Cannot access google cloud SQL from google container engine

I'm still having problems accessing the cloud SQL instance from a GCE container. When I try to open up mysql, I get the following error:
ERROR 2013 (HY000): Lost connection to MySQL server at 'reading initial
communication packet', system error: 0
The connection works fine from my local machine, though (The instance has a public IP and I have added my office's IP to the 'allowed Networks'). So, the instance is accessible through the internet just fine.
I guess the db's access control is blocking my access from the gce network, but I'm unable to figure out how to configure this.
I added my project to "Authorized App Engine Applications" in the Cloud SQL control panel, but that doesn't seem to help.
EDIT:
If I add "0.0.0.0/0" to Allowed Networks, all works well. This is obviously not what I want, so what do I need to enter instead?
EDIT2: I could also add all public IPs from my kubernetes cluster (obtained through gcloud compute instances list) and add them to the cloud sql access list manually. But, this doesn't seem to be right, does it?
The recommended solution is to use SSL connection with that 0.0.0.0/0 CIDR. This is to limit the connection to the correct key. I also read that they won't promise you a specific IP range so the CIDR /14 might not work some times. I had to do the SSL connection with my Cloud SQL for the same reasons.
You should use the public IP addresses of the GCE instances to correctly allow traffic to your Cloud SQL instance (as you mentioned in EDIT2).
You can find more information in Cloud SQL documentation: https://cloud.google.com/sql/docs/gce-access
If you add the /14 CIDR block for your Container Engine cluster as the source address range does that work?
To find the CIDR block for your cluster, click on the cluster name in the Google Cloud Console and find the row labeled "Container address range".