Trying to connect from a datalab notebook with PostgreSQL database hosted on Google Cloud SQL. Try both direct IP and instance connection ways but both give us an exception.
direct connection URI:
"{engine}://{user}:{password}#{host}:{port}/{database}"
using gcloud sql connect
"{engine}://{user}:{password}#/{database}?host=/cloudsql/{instance_connection_name}"
both give us this exception:
OperationalError: (psycopg2.OperationalError) could not connect to
server: Connection timed out
Is the server running on host "***.***.***.***" and accepting
TCP/IP connections on port ****?
Any idea if it need a cloud sql proxy as in Collab proxy connection? And if it is needed how to do it with datalab libraries?
I finally got it.
Assuming that datalab VM is already authenticated on Gcloud i try to use cloud_sql_proxy to connect without auth python commands that appear on Collab proxy connection and fix the error that still appears by crating missing directory. Si i got this:
!wget https://dl.google.com/cloudsql/cloud_sql_proxy.linux.amd64 -O cloud_sql_proxy
!mkdir -p /cloudsql
!chmod +x cloud_sql_proxy
.
!./cloud_sql_proxy --instances=project-id:europe-west1:posty --dir /cloudsql
as with Collab solution we need to let the notebook running in alternate window to keep proxy. With other notebooks on the same machine we finally obtain access to the database.
Note: Probably a better solution could be to edit the docker image of datalab machines to include this behaviour as noted here.
IT may be that your VM machine ip isn't whitelisted on the database.
You can access the list and add new ips on the google cloud console SQL > yor_database > authorization.
Check this link for details https://cloud.google.com/sql/docs/mysql/connect-external-app?hl=en_US&_ga=2.178999533.-851571953.1521816449#appaccessIP
Related
Hello all
I have a dabase seted up in RDS. Right now there is a ec-instance which is my test enviroment which connects to that RDS and working on a particular DB.
Now something with a migration might have messed up the data in that database.
What I am trying to do is connect to that database using pgAdmin 3 to check what's going on.
What I did is used the connect to a new server of pgAdmin
and used the SSH Tunnel tab.
In the properties I used the credentials of the RDS and in the SSH Tunnel(tab)
I used the ssh files I am using to connect to the testing instance I already know has access to the RDS.
Problem is I am getting
Error: SSH error: Error when starting up SSH session with error code
-8 [Unable to exchange encryption keys]
Does my logic make sense ?
Meaning to use the SSH tunnel to configure a connection to an ec2 instance and assume that throught this tunnel the pgAdmin is going to be able to connect to the actuall RDS hosted database ?
Any direction on how to proceed further are greatly appriciated !
I created a Single Zone postgres db instance on Cloud Sql, and I am trying to connect by cloud sql proxy.
/cloud_sql_proxy -instances=<PROJECT_ID>:us-central1:staging=tcp:5432 -credential_file=./<SERVICE_ACCOUNT_KEY_FILE>
This is running well. But when i run below command,
psql "host=127.0.0.1 sslmode=disable dbname=postgres user=postgres"
the proxy shows this error:
2019/11/14 15:20:10 using credential file for authentication; email=<SERVICE_ACCOUNT_EMAIL>
2019/11/14 15:20:13 Listening on 127.0.0.1:5432 for <PROJECT_ID>:us-central1:staging
2019/11/14 15:20:13 Ready for new connections
2019/11/14 15:20:34 New connection for "<PROJECT_ID>:us-central1:staging"
2019/11/14 15:22:45 couldn't connect to "<PROJECT_ID>:us-central1:staging": dial tcp 34.70.245.249:3307: connect: connection timed out
Why is this happening?
I am doing this from my local.
I've just followed this tutorial step by step and it worked perfectly for me.
I did not have to do any extra steps(whitelisting ip, opening port etc...) and this was done in a clean project.
Are you trying to do this from local with the SDK or from Cloud Shell? Do you have any firewall restrictions in place?
Any further information about specific setup from your side that might affect will surely help.
Let us know.
EDIT:
Make sure your port 3307 is not blocked by anything.
Have a look at this official documentation specifying that.
Make sure you have all the required IAM roles attached to the service account before you connect to it:
For instance, the list of roles for cloudsql can be retrieved from gcloud with:
$ gcloud iam roles list --filter 'name~"roles/cloudsql"' --format 'table(name, description)'
NAME DESCRIPTION
roles/cloudsql.admin Full control of Cloud SQL resources.
roles/cloudsql.client Connectivity access to Cloud SQL instances.
roles/cloudsql.editor Full control of existing Cloud SQL instances excluding modifying users, SSL certificates or deleting resources.
roles/cloudsql.instanceUser Role allowing access to a Cloud SQL instance
roles/cloudsql.serviceAgent Grants Cloud SQL access to services and APIs in the user project
roles/cloudsql.viewer Read-only access to Cloud SQL resources.
If your service account is lacking the appropriate roles, it won't be able to connect to the instance for IAM authentication to work.
The issue is probably that you are not in the VPC network, like when you connect from localhost, so what happens is the cloud proxy showing it cannot connect to the remote IP.
Read this carefully if you use a private IP :
https://cloud.google.com/sql/docs/postgres/private-ip
Note that the Cloud SQL instance is in a Google managed network and the proxy is meant to be used to simplify connections to the DB within the VPC network.
In short: running cloud-sql-proxy from a local machine will not work, because it's not in the VPC network. It should work from a Compute Engine VM that is connected to the same VPC as the DB.
What I usually do as a workaround is use gcloud ssh from a local machine and port forward over a small VM in compute engine, like:
gcloud beta compute ssh --zone "europe-north1-b" "instance-1" --project "my-project" -- -L 5432:cloud_sql_server_ip:5432
Then you can connect to localhost:5432 (make sure nothing else is running or change first port number to one that is free locally)
What should also work is to setup a VPN connection to the VPC network and then run the cloud proxy in node in that network.
I have to say I found this really confusing because it gives the impression the proxy does similar magic like gloud does. It's beyond me why some Google engineers have not wired that together yet, can't be too hard.
I had this issue previously when I didn't specify the port argument to psql for some reason, try this:
psql "host=127.0.0.1 port=5432 sslmode=disable user=postgres"
Don't specify the db, and see if that lets you get to the prompt.
Ubuntu 16.04 LTS
I have followed the guides which all say the same thing; to enable remote connection to a postgres server, update the postgresql.conf file, update the pg_hba.conf file and make sure the port (5432) is open and firewall is not blocking.
When I attempt to connect to my server from the remote machine using the following command, I receive no response (for example, 'Connection refused...'). It hangs as if the firewall has DROP policy, but I checked and the host's firewall is ACCEPT all. Here is the command:
psql -h 45.67.82.123 -U postgres -p 5432 -d mydatabase
I have googled extensively and can't find anyone else who's psql request sits with no response from the host server.
Edit: I should mention I have been connecting locally on the host machine. I should also mention that the data directory on the host machine is in a non-default location. I have my cluster on a mounted drive, in case this could affect the remote connection.
Solution:
It is my first AWS instance and I didn't know they have their own firewall rules on the platform. So I was highly confused by the fact all my policies were ACCEPT on my server. Turns out you are behind AWS firewall and you have to go onto the platform to add/change security groups etc. In the past when I've used Digital Ocean droplets or Linodes, the firewall policy on the vps is all I need to change. AWS threw me another curveball there.
I am trying to connect AWS RDS PostgreSql from PgAdmin 3. I followed the below link
http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_ConnectToPostgreSQLInstance.html
In Security Group, I also added PostgreSQL and All traffic as below
The "publicly accessible" flag was enabled (updated after Mark B's comment)
I got the error from PGAdmin3
Very appreciate for any suggestion
******UPDATE*******
I can connect pgAdminIII to AWS RDS successfully using home wifi, but cannot connect using office wifi.
My concern is:
Was the port 5432 blocked by office wifi?
How can I configure/update the port without impacting to current API?
Note: My current API is working well (CRUD)
Can you can test your connection to a DB instance using common Linux or Windows tools first?
From a Linux or Unix terminal, you can test the connection by typing the following (replace with the endpoint and with the port of your DB instance):
$nc -zv DB-instance-endpoint port
For example, the following shows a sample command and the return value:
$nc -zv postgresql1.c6c8mn7tsdgv0.us-west-2.rds.amazonaws.com 8299
Connection to postgresql1.c6c8mn7tsdgv0.us-west-2.rds.amazonaws.com
8299 port [tcp/vvr-data] succeeded!
Windows users can use Telnet to test the connection to a DB instance. Note that Telnet actions are not supported other than for testing the connection. If a connection is successful, the action returns no message. If a connection is not successful, you receive an error message such as the following:
C:>telnet sg-postgresql1.c6c8mntzhgv0.us-west-2.rds.amazonaws.com
8299
Connecting To sg-postgresql1.c6c8mntzhgv0.us-west-2.rds.amazonaws.com...Could not
open connection to the host, on port 819: Connect failed
If Telnet actions return success, then you are good to go.
If you are trying to access it from a network which is not listed for that port. you need to add inbound rules for those network IPs from AMAZON RDS system
You will also need to set Public accessibility true under Connect & security tab in RDS console.
Read this post.In your security group go to unbound rules and add my ip.
and make sure your database is public.
https://serverfault.com/questions/656079/unable-to-connect-to-public-postgresql-rds-instance
I'm using Google Cloud SQL 2nd generation and installed cloud-sql-proxy on my local machine.
On my local machine I'd simply connect to 127.0.0.1:3306 and this has been working fine, in NodeJs, Php and using the mysql command line client.
On Google App Engine Managed VM (flexible environment) i'm using unix_socket or socketPath '/cloudsql/MY_PROJECT_ID:us-central1:SQL_INSTANCE' this has been working fine too, in both, Php and NodeJs.
What doesn't work is MySQL Workbench, I can't figure out how to get it to connect. Does it use another protocol or is cloud-sql-proxy for command line only ?
Here is how I start cloud-sql-proxy
this works:
./cloud_sql_proxy \
-instances=MY_PROJECT:us-central1:MY_SQL_INSTANCE=tcp:3306 \
-credential_file='/Users/ME/SomeFolder/MY_SERVICE_ACC_KEY.json'
after that I'd use MySQL Workbench to try to connect to 127.0.0.1:3306, but I always get an error :
SSL connection error: socket layer receive error
local PHP, NodeJS and mysql client work though.
Any help would be appreciated
Ok, I got it working and believe it might be useful for others too:
I couldn't get it working over the tcp connection, but I figured out how to use the socket method running without fuse:
sudo ./cloud_sql_proxy \
-dir=/cloudsql \
-instances=MY_PROJECT:MY_SQL_REGION:MY_SQL_INSTANCE \
-credential_file='/Users/ME/some_folder/MY_SERVICE_ACC_KEY.json'
couple of things here:
The folder /cloudsql has to already exists e.g. sudo mkdir /cloudsql
Don't mistype that folder name, really don't
Don't specify the tcp port after instances or it will use a tcp connection instead.
sudo is necessary
In MySQL Workbench:
Select Database > Manage Connections...
Under Connection > Connection Method choose Standard (TCP/IP)
Under Parameters set Host: localhost Port:3306 (although I don't think it matters)
Goto Advanced > Others: enter socket=/cloudsql/MY_PROJECT:MY_SQL_REGION:MY_SQL_INSTANCE
Click Test Connection it should as for your username and password, success.
This error indicates that MySQL Workbench is requesting an SSL connection, which is not supported via the proxy. The proxy always uses SSL between the local machine and the instance so there's no need to enable SSL at the MySQL protocol level.
Can you try turning off SSL in MySQL Workbench connection settings?