I'm using pgAdmin to connect to my RDS instance, which has an ssh tunnel through a vpc. That element is mostly fine; I can connect, and query the tables, and create tables. The issue is the I can not import CSVs into the newly created tables; the import wizard is rejected with 'exit code 2' (connection rejected).
I've tried opening a new port specifically to Postgres traffic, but no dice. this seems like it should be fairly basic; I can do it in my RDS instance that doesn't have SSH no problem (succeeded with the same file, same terminal, etc). It really seems like a basic item but I can't seem to even find a guide.
EDIT/Addition: I've made some progress. if I run a command like so:
ssh -N -L 1234:<rds-endpoint>:5432 ec2-user#<ec2-public-ip-addres> -i /Users/user/.ssh/permissionsfile.pem
and then in pgadmin run as a normal connection to:
host: localhost
port: 1234
username: rds-username
maintenancedatabasae: rds-database
password: rds-usernames-password
this works. I'm mostly just baffled that I can't figure out how to put that same ssh info into pgadmin. typically it's a super simple copy and paste operation.
First, if RDS is located in private subnet, pgAdmin cannot connect to RDS even if the private subnet is connected to NAT Gatway which is in public subnet. So to make the private subnet public, set Internet Gateway to the private subnet instead of NAT Gatway.
Second, check the security group's inbound rule of 5432(RDS postgresql port) is open to your computer which has pgAdmin.
Third, check Public access of RDS setting is Yes showing below. If it's No, tools outside your VPC cannot connect to RDS.
Tell me you could successfully connect to RDS with pgAdmin or not with 3 solutions above.
Related
We are looking to connect dbt to Postgres using SSH bastion.
I followed the comments left under this issue, but I get a timeout error.
A few questions:
How should the profiles.yml be configure to connect via SSH? I added ssh-host but that did not get it working.
Is there any other configurations that I'd need to set up?
I just hacked my way through figuring this out and the steps listed in the above comment were very helpful for someone with zero experience in this realm who still needs to use dbt with a bastion host. Here is specifically how I did this and some helpful resources I came across. Hopefully others will find these examples helpful.
You register a public SSH key with the remote location, tied to a
private key that lives on your machine
Github has a helpful guide for how to do this: https://docs.github.com/en/authentication/connecting-to-github-with-ssh/generating-a-new-ssh-key-and-adding-it-to-the-ssh-agent
Add keys to ~/.ssh/config: Adding an RSA key without overwriting.
I also had to add IgnoreUnknown UseKeychain to ~/.ssh/config
You use a CLI tool (e.g. ssh, autossh) to "forward" a local port to
the remote location (bastion host)
To forward the local port to the bastion host, save your user/bastion host/db host into environment variables. I used Postgres so it looked like this.
ssh -l $BASTION_USER $BASTION_HOST -p 22 -N -C -L "5432:${POSTGRES_HOST}:5432";
In profiles.yml, instead of putting the host/port of a remote
database, you put localhost and the number of the "forwarding" port
Then in my ~/.dbt/profiles.yml looks includes this:
dev:
type: postgres
threads: 1
host: localhost
port: 5432
user: POSTGRES_USER
pass: POSTGRES_PWD
dbname: POSTGRES_DB_NAME
schema: dbt_tmp
Voila! Your connection is forwarded to the bastion host, authenticated
via SSH, and passed along to the database
At that point I ran dbt debug against my target and it connected with all checks passed.
I think you need to follow Jeremy's instructions from this comment:
The basic idea, as I remember it:
You register a public SSH key with the remote location, tied to a
private key that lives on your machine
You use a CLI tool (e.g. ssh,
autossh) to "forward" a local port to the remote location (bastion
host)
In profiles.yml, instead of putting the host/port of a remote
database, you put localhost and the number of the "forwarding" port
Voila! Your connection is forwarded to the bastion host, authenticated
via SSH, and passed along to the database
To be fair, he was also asking for definitive walkthroughs and included the caveat that this has had varying levels of success based on the particulars of the client, host, environment etc.
I created a Single Zone postgres db instance on Cloud Sql, and I am trying to connect by cloud sql proxy.
/cloud_sql_proxy -instances=<PROJECT_ID>:us-central1:staging=tcp:5432 -credential_file=./<SERVICE_ACCOUNT_KEY_FILE>
This is running well. But when i run below command,
psql "host=127.0.0.1 sslmode=disable dbname=postgres user=postgres"
the proxy shows this error:
2019/11/14 15:20:10 using credential file for authentication; email=<SERVICE_ACCOUNT_EMAIL>
2019/11/14 15:20:13 Listening on 127.0.0.1:5432 for <PROJECT_ID>:us-central1:staging
2019/11/14 15:20:13 Ready for new connections
2019/11/14 15:20:34 New connection for "<PROJECT_ID>:us-central1:staging"
2019/11/14 15:22:45 couldn't connect to "<PROJECT_ID>:us-central1:staging": dial tcp 34.70.245.249:3307: connect: connection timed out
Why is this happening?
I am doing this from my local.
I've just followed this tutorial step by step and it worked perfectly for me.
I did not have to do any extra steps(whitelisting ip, opening port etc...) and this was done in a clean project.
Are you trying to do this from local with the SDK or from Cloud Shell? Do you have any firewall restrictions in place?
Any further information about specific setup from your side that might affect will surely help.
Let us know.
EDIT:
Make sure your port 3307 is not blocked by anything.
Have a look at this official documentation specifying that.
Make sure you have all the required IAM roles attached to the service account before you connect to it:
For instance, the list of roles for cloudsql can be retrieved from gcloud with:
$ gcloud iam roles list --filter 'name~"roles/cloudsql"' --format 'table(name, description)'
NAME DESCRIPTION
roles/cloudsql.admin Full control of Cloud SQL resources.
roles/cloudsql.client Connectivity access to Cloud SQL instances.
roles/cloudsql.editor Full control of existing Cloud SQL instances excluding modifying users, SSL certificates or deleting resources.
roles/cloudsql.instanceUser Role allowing access to a Cloud SQL instance
roles/cloudsql.serviceAgent Grants Cloud SQL access to services and APIs in the user project
roles/cloudsql.viewer Read-only access to Cloud SQL resources.
If your service account is lacking the appropriate roles, it won't be able to connect to the instance for IAM authentication to work.
The issue is probably that you are not in the VPC network, like when you connect from localhost, so what happens is the cloud proxy showing it cannot connect to the remote IP.
Read this carefully if you use a private IP :
https://cloud.google.com/sql/docs/postgres/private-ip
Note that the Cloud SQL instance is in a Google managed network and the proxy is meant to be used to simplify connections to the DB within the VPC network.
In short: running cloud-sql-proxy from a local machine will not work, because it's not in the VPC network. It should work from a Compute Engine VM that is connected to the same VPC as the DB.
What I usually do as a workaround is use gcloud ssh from a local machine and port forward over a small VM in compute engine, like:
gcloud beta compute ssh --zone "europe-north1-b" "instance-1" --project "my-project" -- -L 5432:cloud_sql_server_ip:5432
Then you can connect to localhost:5432 (make sure nothing else is running or change first port number to one that is free locally)
What should also work is to setup a VPN connection to the VPC network and then run the cloud proxy in node in that network.
I have to say I found this really confusing because it gives the impression the proxy does similar magic like gloud does. It's beyond me why some Google engineers have not wired that together yet, can't be too hard.
I had this issue previously when I didn't specify the port argument to psql for some reason, try this:
psql "host=127.0.0.1 port=5432 sslmode=disable user=postgres"
Don't specify the db, and see if that lets you get to the prompt.
Ubuntu 16.04 LTS
I have followed the guides which all say the same thing; to enable remote connection to a postgres server, update the postgresql.conf file, update the pg_hba.conf file and make sure the port (5432) is open and firewall is not blocking.
When I attempt to connect to my server from the remote machine using the following command, I receive no response (for example, 'Connection refused...'). It hangs as if the firewall has DROP policy, but I checked and the host's firewall is ACCEPT all. Here is the command:
psql -h 45.67.82.123 -U postgres -p 5432 -d mydatabase
I have googled extensively and can't find anyone else who's psql request sits with no response from the host server.
Edit: I should mention I have been connecting locally on the host machine. I should also mention that the data directory on the host machine is in a non-default location. I have my cluster on a mounted drive, in case this could affect the remote connection.
Solution:
It is my first AWS instance and I didn't know they have their own firewall rules on the platform. So I was highly confused by the fact all my policies were ACCEPT on my server. Turns out you are behind AWS firewall and you have to go onto the platform to add/change security groups etc. In the past when I've used Digital Ocean droplets or Linodes, the firewall policy on the vps is all I need to change. AWS threw me another curveball there.
I am trying to connect AWS RDS PostgreSql from PgAdmin 3. I followed the below link
http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_ConnectToPostgreSQLInstance.html
In Security Group, I also added PostgreSQL and All traffic as below
The "publicly accessible" flag was enabled (updated after Mark B's comment)
I got the error from PGAdmin3
Very appreciate for any suggestion
******UPDATE*******
I can connect pgAdminIII to AWS RDS successfully using home wifi, but cannot connect using office wifi.
My concern is:
Was the port 5432 blocked by office wifi?
How can I configure/update the port without impacting to current API?
Note: My current API is working well (CRUD)
Can you can test your connection to a DB instance using common Linux or Windows tools first?
From a Linux or Unix terminal, you can test the connection by typing the following (replace with the endpoint and with the port of your DB instance):
$nc -zv DB-instance-endpoint port
For example, the following shows a sample command and the return value:
$nc -zv postgresql1.c6c8mn7tsdgv0.us-west-2.rds.amazonaws.com 8299
Connection to postgresql1.c6c8mn7tsdgv0.us-west-2.rds.amazonaws.com
8299 port [tcp/vvr-data] succeeded!
Windows users can use Telnet to test the connection to a DB instance. Note that Telnet actions are not supported other than for testing the connection. If a connection is successful, the action returns no message. If a connection is not successful, you receive an error message such as the following:
C:>telnet sg-postgresql1.c6c8mntzhgv0.us-west-2.rds.amazonaws.com
8299
Connecting To sg-postgresql1.c6c8mntzhgv0.us-west-2.rds.amazonaws.com...Could not
open connection to the host, on port 819: Connect failed
If Telnet actions return success, then you are good to go.
If you are trying to access it from a network which is not listed for that port. you need to add inbound rules for those network IPs from AMAZON RDS system
You will also need to set Public accessibility true under Connect & security tab in RDS console.
Read this post.In your security group go to unbound rules and add my ip.
and make sure your database is public.
https://serverfault.com/questions/656079/unable-to-connect-to-public-postgresql-rds-instance
I have configured an RDS Postgres micro instance for testing but am having problem connecting. I have created a security group and allow all hosts (0.0.0.0/0) by default to port 5432 but for some reason I am not able to connect. I tried connecting to the port 5432 on the machine but it appears to be closed. Any ideas? I have verified that the security group is using the correct security group.
This is my configuration
TCP
Port (Service) Source Action
5432 0.0.0.0/0 Delete
Here is how I am trying to connect to the rds service
psql -p 5432 -h example.cs945smhrv09.us-west-2.rds.amazonaws.com -U example example
UPDATE
OK feel a little silly after figuring out that rds starts up on a private 172 subnet. I was able to connect from my ec2 instance.
Question now is there a way to put that on a public address that I can connect to? I would like to configure it with a security group that limits the source ip but then be able to connect to it with psql from my desktop.
Update 2
I don't remember all the details when I originally posted this question but it is no longer an issue. We now have a management vpc that we connect to using a vpn server. The management vpc network is trusted by the prod and devel security groups for the rds instances. We have no problem communicating with them without adding any public ips.
Step 4 of the RDS setup wizard ("Additional Config") lets you select whether the DB is Publicly Accessible or not (if it's not, access is limited to within your VPC, regardless of your security group settings). I'd imagine that determines whether the hostname resolves to 54.* vs 172.* .
Your first instance may have been set to private - as far as I can tell this can't be changed after initialization. I had a similar issue and creating a new instance set to be publicly accessible solved it for me.
My company's firewall had blocked outgoing traffic to the default postgres port, 5432. I didn't realise this until I tested the port using netcat in a terminal window, as follows:
nc -zv portquiz.net 5432
Hopefully this saves someone half a day of messing around with AWS security groups.
Absolutely there is. I have pgadmin loaded on my local machine connected to my RDS instance. |
I just confirmed that I can connect using the same syntax you used for psql, although the -U was not necessary for me (I still included both "example" parameters though)
I did edit the "default" security group and added my IP (Ip of the machine I'm connecting from)/32 to be allowed, although I'm sure you could get less restrictive than that, but I'd start there and see if it helps.
This is likely caused by your outbound firewall rules. If you're behind a corporate firewall, they will often block outbound ports above a certain number. You need to setup an SSH tunnel of some kind to avoid this.
http://conoroneill.net/accessing-amazon-rds-instances-from-your-desktop/
I had this problem. This isn't a direct solution but might work for some. I set the Postgres security group to accept connections coming from the server that will be using it (same VPC). I then set up my Postgres client to use ssh tunneling. If I need to use the command line, I simply SSH into that server and then psql from there.
You can simply use workbench recommended by aws but make sure your all postgres availibily is set to everywhere..while adding PostgrSQL 5432
https://aws.amazon.com/getting-started/tutorials/create-connect-postgresql-db/