Connecting to GCP Cloud SQL using cloud sql proxy - postgresql

I am following this guide to connect to my GCP Cloud Postgresql - I have a private IP only. When I run
./cloud_sql_proxy -instances=<Instance connection name>=tcp:5432
I get the following (actual references replaced by <Instance connection name>)
2019/04/01 11:46:45 failed to setup file descriptor limits: failed to set rlimit {&{8500 4096}} for max file descriptors: invalid argument
2019/04/01 11:46:45 Listening on 127.0.0.1:5432 for <Instance connection name>
2019/04/01 11:46:45 Ready for new connections
2019/04/01 11:47:28 New connection for <Instance connection name>
2019/04/01 11:49:38 couldn't connect to <Instance connection name>: dial tcp 172.27.160.3:3307: connect: connection timed out
If I run
psql "host=127.0.0.1 sslmode=disable dbname=<db> user=<user> password=<password>"
then I get the following
psql: FATAL: password authentication failed for user "prodigy"
If run the same psql command from a VM instance on the same network as my cloud SQL then connection is just fine.

Your Cloud SQL instance needs to be in the same client's network in order to connect using its private IP.
It will not be possible to connect Cloud SQL (as is) through its private IP outside of the Google Cloud Platform network. In such a case, you could try Cloud Interconnect [2] in order to extend your on-premise network to Google's network so that you can try your connections and see if that works for you.
On the other hand, when using the Cloud SQL Proxy, you need to state being using private IP [3]. If not specified, the proxy will look for the public IP by default.
Let me know how it goes.
[1] https://cloud.google.com/sql/docs/mysql/private-ip
[2] https://cloud.google.com/interconnect/docs/
[3] https://cloud.google.com/sql/docs/postgres/sql-proxy#private-ip

The connection to a Cloud SQL instance using cloud_proxy can happen only if Cloud SQL and cloud_proxy belong the same VPC network [1].
Otherwise cloud_proxy cannot reach Cloud SQL
[1] https://github.com/GoogleCloudPlatform/cloudsql-proxy/issues/164

Related

Synapse Notebook throws timeout error while connecting to AWS RDS SQL Server

I am working in the Synapse Workspace and trying to connect to AWS RDS from the Synapse Notebook.
Whenever I try to connect, it throws the below timeout error -
The TCP/IP connection to the host my-host, port 1433 has failed.
Error: "connect timed out.
Verify the connection properties. Make sure that an instance of SQL Server is running on the host and accepting TCP/IP connections at the port.
Make sure that TCP connections to the port are not blocked by a firewall.
To check whether I can ping the host from the Synapse Notebook - I tried the below code -
import subprocess
temp = subprocess.Popen(
['ping', '-c 1', 'my-host'], stdout = subprocess.PIPE)
output = str(temp.communicate())
print(output)
and this throws
ping statistics ---\n1 packets transmitted, 0 received, 100% packet loss
I get that this is the timeout error and the notebook cannot reach the server.
What is surprising is, if I try to connect to the same AWS RDS Server by creating a linked service from the Synapse pipeline, it connects successfully.
On my source AWS RDS, do I need to open the firewall for Synapse notebooks specifically? Is there any endpoint that I should mention in my notebook?
Also, Isn't it handled at the resource group level?
Any help is appreciated.
Thank you,
Sanket Kelkar
If you have already configured your database to listen on TCP/IP traffic on port 1433 then it could be any of following three reasons:
JDBC connection string might be incorrect.
Firewall is blocking the incoming connection. Make sure that it is publicly accessible. You can check this when you check the availability.
AWS RDS SQL database is not running. Ensure that "available" is shown as the status.
make sure you specify the port 1433 while creating SQL server.
Check to see if your DB instance can be accessed by the inbound rules of your VPC security group. For more information, see Can't connect to Amazon RDS DB instance.

Mongo DB archive unable to connect from ec2 with vpc Peering

We have configured mongo atlas archiving and trying to connect with aws ec2 with the help of VPC peering. We have already established vpc peering to connect Primary database and it is working fine. But in case of archive db connection we are having connection issue, if we try with ip whitelist then it works fine.
Can any one help us to connect mongo archive db with vpc peering.
Error while connecting with mongo shell
2020-12-12T05:41:15.283+0000 W NETWORK [thread1] Failed to connect to xx.xxx.xx.xx:port after 5000ms milliseconds, giving up.
2020-12-12T05:41:15.283+0000 E QUERY [thread1] Error: couldn't connect to server atlas-online-archive-5da8542bcf09a2545f54fa24-jsraa.a.query.mongodb.net:27017, connection attempt failed :
connect#src/mongo/shell/mongo.js:275:13
#(connect):1:6
exception: connect failed
Atlas provides a unified endpoint through which you can query both your live cluster and archived data using the same database and collection name you use in your Atlas cluster. You can't use the unified endpoint over a private connection such as Peering or AWS PrivateLink. You must use a standard internet connection over TLS.
https://docs.atlas.mongodb.com/online-archive/manage-online-archive/#how-service-archives-data
Unfortunately VPC peering is not supported when connecting to the Online Archive

Connect postgres cloud sql through cloud sql proxy

I created a Single Zone postgres db instance on Cloud Sql, and I am trying to connect by cloud sql proxy.
/cloud_sql_proxy -instances=<PROJECT_ID>:us-central1:staging=tcp:5432 -credential_file=./<SERVICE_ACCOUNT_KEY_FILE>
This is running well. But when i run below command,
psql "host=127.0.0.1 sslmode=disable dbname=postgres user=postgres"
the proxy shows this error:
2019/11/14 15:20:10 using credential file for authentication; email=<SERVICE_ACCOUNT_EMAIL>
2019/11/14 15:20:13 Listening on 127.0.0.1:5432 for <PROJECT_ID>:us-central1:staging
2019/11/14 15:20:13 Ready for new connections
2019/11/14 15:20:34 New connection for "<PROJECT_ID>:us-central1:staging"
2019/11/14 15:22:45 couldn't connect to "<PROJECT_ID>:us-central1:staging": dial tcp 34.70.245.249:3307: connect: connection timed out
Why is this happening?
I am doing this from my local.
I've just followed this tutorial step by step and it worked perfectly for me.
I did not have to do any extra steps(whitelisting ip, opening port etc...) and this was done in a clean project.
Are you trying to do this from local with the SDK or from Cloud Shell? Do you have any firewall restrictions in place?
Any further information about specific setup from your side that might affect will surely help.
Let us know.
EDIT:
Make sure your port 3307 is not blocked by anything.
Have a look at this official documentation specifying that.
Make sure you have all the required IAM roles attached to the service account before you connect to it:
For instance, the list of roles for cloudsql can be retrieved from gcloud with:
$ gcloud iam roles list --filter 'name~"roles/cloudsql"' --format 'table(name, description)'
NAME DESCRIPTION
roles/cloudsql.admin Full control of Cloud SQL resources.
roles/cloudsql.client Connectivity access to Cloud SQL instances.
roles/cloudsql.editor Full control of existing Cloud SQL instances excluding modifying users, SSL certificates or deleting resources.
roles/cloudsql.instanceUser Role allowing access to a Cloud SQL instance
roles/cloudsql.serviceAgent Grants Cloud SQL access to services and APIs in the user project
roles/cloudsql.viewer Read-only access to Cloud SQL resources.
If your service account is lacking the appropriate roles, it won't be able to connect to the instance for IAM authentication to work.
The issue is probably that you are not in the VPC network, like when you connect from localhost, so what happens is the cloud proxy showing it cannot connect to the remote IP.
Read this carefully if you use a private IP :
https://cloud.google.com/sql/docs/postgres/private-ip
Note that the Cloud SQL instance is in a Google managed network and the proxy is meant to be used to simplify connections to the DB within the VPC network.
In short: running cloud-sql-proxy from a local machine will not work, because it's not in the VPC network. It should work from a Compute Engine VM that is connected to the same VPC as the DB.
What I usually do as a workaround is use gcloud ssh from a local machine and port forward over a small VM in compute engine, like:
gcloud beta compute ssh --zone "europe-north1-b" "instance-1" --project "my-project" -- -L 5432:cloud_sql_server_ip:5432
Then you can connect to localhost:5432 (make sure nothing else is running or change first port number to one that is free locally)
What should also work is to setup a VPN connection to the VPC network and then run the cloud proxy in node in that network.
I have to say I found this really confusing because it gives the impression the proxy does similar magic like gloud does. It's beyond me why some Google engineers have not wired that together yet, can't be too hard.
I had this issue previously when I didn't specify the port argument to psql for some reason, try this:
psql "host=127.0.0.1 port=5432 sslmode=disable user=postgres"
Don't specify the db, and see if that lets you get to the prompt.

PGAdmin III cannot connect AWS RDS

I am trying to connect AWS RDS PostgreSql from PgAdmin 3. I followed the below link
http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_ConnectToPostgreSQLInstance.html
In Security Group, I also added PostgreSQL and All traffic as below
The "publicly accessible" flag was enabled (updated after Mark B's comment)
I got the error from PGAdmin3
Very appreciate for any suggestion
******UPDATE*******
I can connect pgAdminIII to AWS RDS successfully using home wifi, but cannot connect using office wifi.
My concern is:
Was the port 5432 blocked by office wifi?
How can I configure/update the port without impacting to current API?
Note: My current API is working well (CRUD)
Can you can test your connection to a DB instance using common Linux or Windows tools first?
From a Linux or Unix terminal, you can test the connection by typing the following (replace with the endpoint and with the port of your DB instance):
$nc -zv DB-instance-endpoint port
For example, the following shows a sample command and the return value:
$nc -zv postgresql1.c6c8mn7tsdgv0.us-west-2.rds.amazonaws.com 8299
Connection to postgresql1.c6c8mn7tsdgv0.us-west-2.rds.amazonaws.com
8299 port [tcp/vvr-data] succeeded!
Windows users can use Telnet to test the connection to a DB instance. Note that Telnet actions are not supported other than for testing the connection. If a connection is successful, the action returns no message. If a connection is not successful, you receive an error message such as the following:
C:>telnet sg-postgresql1.c6c8mntzhgv0.us-west-2.rds.amazonaws.com
8299
Connecting To sg-postgresql1.c6c8mntzhgv0.us-west-2.rds.amazonaws.com...Could not
open connection to the host, on port 819: Connect failed
If Telnet actions return success, then you are good to go.
If you are trying to access it from a network which is not listed for that port. you need to add inbound rules for those network IPs from AMAZON RDS system
You will also need to set Public accessibility true under Connect & security tab in RDS console.
Read this post.In your security group go to unbound rules and add my ip.
and make sure your database is public.
https://serverfault.com/questions/656079/unable-to-connect-to-public-postgresql-rds-instance

Connecting to Mongo DB on GCE instance

I have a Google Compute Engine instance which is running an active MongoDB server. I can connect to and interact with the server on the local machine via SSH.
However, when I attempt to connect to the database instance from an external source, like my home computer, I'm getting a connection refused. I have setup a firewall rule on the default network that looks like this:
Source Ranges:
0.0.0.0/0
Allowed Protocols or Ports:
tcp:27017
udp:27017
However, even after adding this rule to the default network I am still getting a connection refused when running:
$ mongo [ipaddress]
I'm not sure why the instance is refusing the incoming connection.