Synapse Notebook throws timeout error while connecting to AWS RDS SQL Server - pyspark

I am working in the Synapse Workspace and trying to connect to AWS RDS from the Synapse Notebook.
Whenever I try to connect, it throws the below timeout error -
The TCP/IP connection to the host my-host, port 1433 has failed.
Error: "connect timed out.
Verify the connection properties. Make sure that an instance of SQL Server is running on the host and accepting TCP/IP connections at the port.
Make sure that TCP connections to the port are not blocked by a firewall.
To check whether I can ping the host from the Synapse Notebook - I tried the below code -
import subprocess
temp = subprocess.Popen(
['ping', '-c 1', 'my-host'], stdout = subprocess.PIPE)
output = str(temp.communicate())
print(output)
and this throws
ping statistics ---\n1 packets transmitted, 0 received, 100% packet loss
I get that this is the timeout error and the notebook cannot reach the server.
What is surprising is, if I try to connect to the same AWS RDS Server by creating a linked service from the Synapse pipeline, it connects successfully.
On my source AWS RDS, do I need to open the firewall for Synapse notebooks specifically? Is there any endpoint that I should mention in my notebook?
Also, Isn't it handled at the resource group level?
Any help is appreciated.
Thank you,
Sanket Kelkar

If you have already configured your database to listen on TCP/IP traffic on port 1433 then it could be any of following three reasons:
JDBC connection string might be incorrect.
Firewall is blocking the incoming connection. Make sure that it is publicly accessible. You can check this when you check the availability.
AWS RDS SQL database is not running. Ensure that "available" is shown as the status.
make sure you specify the port 1433 while creating SQL server.
Check to see if your DB instance can be accessed by the inbound rules of your VPC security group. For more information, see Can't connect to Amazon RDS DB instance.

Related

psql with source IP anchoring

I am using PostgreSQL flexible server by MSAzure.
The server is not setting any connection restriction and allows connections from all IPs.
I can access to the server with psql command normally.
psql "host=.postgres.database.azure.com port=5432 dbname=postgres user=myadmin password= sslmode=require"
However, with the Source IP anchoring function by Zscaler Internet Access, it fails.
psql: server closed the connection unexpectedly This probably means the server terminated abnormally before or while processing the request.
I could not find the error logs from server side.
Also I checked the packet and it seems the server sends RST packet as soon as it received connection request message from client.
enter image description here
Source IP Anchoring(SIPA) is a function which fixes source IP by transferring the packet to the proxy cloud, and from proxy cloud to private VM. The VM takes over the connection to the destination server.
Client <-> Proxy Cloud <-> VM <-> sql server
So from client side, it looks like the client is connecting to the proxy cloud.
And from sql server, it seems the server is connecting to VM.
The VM only performs outbound connection and never allows inbound connection.
This means connection pushed from server to client is not supported (like P2P, Application lelvel gateway, SIP, RTSP, Bittorent).
Does Postgresql connection requires inbound connection from server to client?
I tried disabling SSL connection, but the result was same.
The client version and server version are both same.
Also I tried connecting from pgAdmin 4, but failed again.
I'll appreciate any clue on this.

Connecting to GCP Cloud SQL using cloud sql proxy

I am following this guide to connect to my GCP Cloud Postgresql - I have a private IP only. When I run
./cloud_sql_proxy -instances=<Instance connection name>=tcp:5432
I get the following (actual references replaced by <Instance connection name>)
2019/04/01 11:46:45 failed to setup file descriptor limits: failed to set rlimit {&{8500 4096}} for max file descriptors: invalid argument
2019/04/01 11:46:45 Listening on 127.0.0.1:5432 for <Instance connection name>
2019/04/01 11:46:45 Ready for new connections
2019/04/01 11:47:28 New connection for <Instance connection name>
2019/04/01 11:49:38 couldn't connect to <Instance connection name>: dial tcp 172.27.160.3:3307: connect: connection timed out
If I run
psql "host=127.0.0.1 sslmode=disable dbname=<db> user=<user> password=<password>"
then I get the following
psql: FATAL: password authentication failed for user "prodigy"
If run the same psql command from a VM instance on the same network as my cloud SQL then connection is just fine.
Your Cloud SQL instance needs to be in the same client's network in order to connect using its private IP.
It will not be possible to connect Cloud SQL (as is) through its private IP outside of the Google Cloud Platform network. In such a case, you could try Cloud Interconnect [2] in order to extend your on-premise network to Google's network so that you can try your connections and see if that works for you.
On the other hand, when using the Cloud SQL Proxy, you need to state being using private IP [3]. If not specified, the proxy will look for the public IP by default.
Let me know how it goes.
[1] https://cloud.google.com/sql/docs/mysql/private-ip
[2] https://cloud.google.com/interconnect/docs/
[3] https://cloud.google.com/sql/docs/postgres/sql-proxy#private-ip
The connection to a Cloud SQL instance using cloud_proxy can happen only if Cloud SQL and cloud_proxy belong the same VPC network [1].
Otherwise cloud_proxy cannot reach Cloud SQL
[1] https://github.com/GoogleCloudPlatform/cloudsql-proxy/issues/164

aws datapipeline - psycopg2.OperationalError: could not connect to server: Connection timed out

I am trying to load data from redshift to dynamodb via Data Pipeline shellcommand activity using python script.
The hostname and port details are hardcoded in the script.
Issue arises when I try to run the Datapipeline job. I receive the below error:
psycopg2.OperationalError: could not connect to server: Connection timed out
Is the server running on host "1*.***.**.*" and accepting
TCP/IP connections on port 5***?
I understand the redshift is not accessible to ec2 instance and hence this issue. So I passed the subnet Id and security groups created for redshift cluster as parameters to EC2 instance to be created. But now I see my shellcommand activity "waiting_for_runner" state,despite the ec2 instance is active and running.
Anyone has faced similar issues and resolved ?
Any suggestions would be appreciated !

PGAdmin III cannot connect AWS RDS

I am trying to connect AWS RDS PostgreSql from PgAdmin 3. I followed the below link
http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_ConnectToPostgreSQLInstance.html
In Security Group, I also added PostgreSQL and All traffic as below
The "publicly accessible" flag was enabled (updated after Mark B's comment)
I got the error from PGAdmin3
Very appreciate for any suggestion
******UPDATE*******
I can connect pgAdminIII to AWS RDS successfully using home wifi, but cannot connect using office wifi.
My concern is:
Was the port 5432 blocked by office wifi?
How can I configure/update the port without impacting to current API?
Note: My current API is working well (CRUD)
Can you can test your connection to a DB instance using common Linux or Windows tools first?
From a Linux or Unix terminal, you can test the connection by typing the following (replace with the endpoint and with the port of your DB instance):
$nc -zv DB-instance-endpoint port
For example, the following shows a sample command and the return value:
$nc -zv postgresql1.c6c8mn7tsdgv0.us-west-2.rds.amazonaws.com 8299
Connection to postgresql1.c6c8mn7tsdgv0.us-west-2.rds.amazonaws.com
8299 port [tcp/vvr-data] succeeded!
Windows users can use Telnet to test the connection to a DB instance. Note that Telnet actions are not supported other than for testing the connection. If a connection is successful, the action returns no message. If a connection is not successful, you receive an error message such as the following:
C:>telnet sg-postgresql1.c6c8mntzhgv0.us-west-2.rds.amazonaws.com
8299
Connecting To sg-postgresql1.c6c8mntzhgv0.us-west-2.rds.amazonaws.com...Could not
open connection to the host, on port 819: Connect failed
If Telnet actions return success, then you are good to go.
If you are trying to access it from a network which is not listed for that port. you need to add inbound rules for those network IPs from AMAZON RDS system
You will also need to set Public accessibility true under Connect & security tab in RDS console.
Read this post.In your security group go to unbound rules and add my ip.
and make sure your database is public.
https://serverfault.com/questions/656079/unable-to-connect-to-public-postgresql-rds-instance

RDS Postgresql DB connection timeout from Heroku

I'm unable to connect to a new PostgreSQL in AWS RDS.
I have a Heroku app and I would like to use Amazon RDS for my database instead of Heroku. For that I've been following this guide: https://www.reinteractive.net/posts/128-heroku-app-backed-by-an-aws-rds-postgres-database
I've made a backup from my current Heroku DB and want to load it on the new database.
My security group for the database allows all inbound connections for port 5432 (0.0.0.0/0) and I've made a new VPC to have my DB set as Publicly Accesible (DNS hostnames and DNS resolution enabled). I created the database on postgres version 9.4.9.
However when I do:
-f latest.sql --host=xxx.xxx.us-west-2.rds.amazonaws.com --port=5432 --username=awsuser --password --dbname=mydatabase
from my computer, I only get a connection time out error:
psql: could not connect to server: Connection timed out
Is the server running on host xxx.xxx.us-west-2.rds.amazonaws.com" (1.2.3.4) and accepting
TCP/IP connections on port 5432?
The server is indeed running. In this case latest.sql is the backup I did. After this I edited the Database security groups to accept all connections (0.0.0.0/0) too.
Database Rules
(from what I've read this should not be necessary because I already have the VPC Security Group), but the result is the same.
Is there any way to trace what's going on / why is my connection getting blocked?