I'm running a Superset instance via Docker on a MacBook Air (2019, v11.5.2, Intel i5). I'm trying to set up a remote connection to a PostgreSQL database via an AWS endpoint. I entered the credentials via the dynamic form, as such:
HOST: {dbalias}.{xyz}.us-east-1.rds.amazonaws.com
PORT: 5432
DATABASE: {dbname}
USERNAME: {username}
PASSWORD: {password}
I'm sure that my credentials are valid because I used them to connect from both Databox and DBeaver. But when I try to connect here, Superset tells me that port 5432 is closed. A little digging tells me that this is likely a firewall issue.
I know that our database is set up to only allow connections from allowed IPs, and of course my machine's IP is whitelisted, so I assume I need to whitelist the IP that is sending the connection request (i.e., Superset). However, I cannot seem to find that information. Indeed, even Superset's PostgreSQL connection instructions seem to be incomplete vis-a-vis connecting to an AWS endpoint.
Assuming I've diagnosed the problem correctly (which is by no means a guarantee), the key question is: Where can I find my Superset instance's IP to add to my PostgreSQL IP whitelist? Relatedly, would this IP change next time I launch Superset from Docker, or will it persist?
Many thanks for any consideration.
Related
Is it possible possible to connect to Fly.io Postgres from a NEXT.js API route served from a Vercel severless function, and if so, how?
I've read and reread the Fly.io Postgres docs, focusing especially on Connecting from outside Fly.io and (since my API routes use Prisma to connect to Postgres) Connecting with Prisma, but it seems I would need to either install flyctl or a set up a WireGuard Tunnel in Vercel, neither of which I could figure out how to do from web searches.
I found essentially the same question on the Fly.io community forum, but unfortunately, no one has answered it after 26 days (as of this writing).
To add a bit more detail, I'm familiar with Heroku Postgres, where a connection string is essentially all that's needed to connect, but it seems connecting to Fly.io Postgres requires a connection string plus a proxy created with flyctl. This was easy to get working on my local machine, but it's not clear how to do this on a remote host like Vercel.
One of the solutions is to allocate an ip address to your app with link here
fly ips allocate-v4 --app <pg-app-name>
Then add these service configurations to the fly.toml file
[[services]]
internal_port = 5432 # Postgres instance
protocol = "tcp"
[[services.ports]]
handlers = ["pg_tls"]
port = 5432
Redeploy your app with fly deploy
Go to your fly dashboard you will be able to see Ip Addresses that you can use as host to connect to the database
Problem
In general, there have been a ton of issues in connecting a remote service to a PostgreSQL database. The documentation for most services doesn't really have documentation for this since the task of connecting to a PostgreSQL database requires the Admin to modify both the postgresql.conf and pg_hba.conf files.
The current scenario is giving AppSmith remote access to the given server.
PostgreSQL needs to allow the following IP addresses at the pg_hba.conf: 18.223.74.85 and 3.131.104.27
Research SSL Connections and if one would be required in this case
Success Criteria:
Appsmith is able to connect to given database
Appsmith is able to read/write data to given database
Resolution Research
Allowing the above IP addresses to connect to the PostgreSQL database in the pg_hba.conf file and changing the postgresql.conf file to allow remote connections using listening_addresses = '*' did not resolve the issue. Appsmith is unable to connect to the database.
SSL Connection: SSL stands for Secure Sockets Layer and, in short, it's the standard technology for keeping an internet connection secure and safeguarding any sensitive data that is being sent between two systems, preventing criminals from reading and modifying any information transferred, including potential personal details.
SSL proved to be a dead-end for this issue with no resolution.
In order to create an SSL connection, the following steps must be followed (PostgreSQL SSL documentation: https://www.postgresql.org/docs/9.1/ssl-tcp.html):
OpenSSL needs to be installed on the host server (https://fedingo.com/how-to-install-openssl-in-ubuntu/). Determined that OpenSSL is already installed on the host server using openssl version -a.
Following the above steps from the postgresql documentation does not produce the desired result.
Following steps from https://www.cyberciti.biz/faq/postgresql-remote-access-or-connection/ does not work either.
Error in all cases on Appsmith: Failed to initialize pool: The connection attempt failed
Next-steps
Consult with the Stackoverflow community to see if anyone else is having a similar issue.
Similar issues have been found, but Appsmith documentation and the Stackoverflow community did not have the information needed to fulfill this issue. I will post the full case to the community.
I resolved this problem using ngrok to make a tcp tunnel on your localhost.
Donwload and conifg ngrok and create a new tunnel on terminal, just run ngrok tcp 5432 at terminal
After this, get the host/port and insert at Appsmith PostgreSQL Connection.
make sure your username/password is correct.
Image from host/port create on ngrok
Image to how config this host/port on Appsmith
I cannot connect to a restored rds instance.
I create a new rds instance by restoring_from_snapshot.
It copies all settings except the security group so I modify it after boot.
Now I cannot get psql to connect to 5432 on the endpoint.
Despite identical settings
old working instance (can be access is in live usage)
new test instance based on snapshot
I have the exact same settings on both instances. The new instance starts with the default vpc sec group but I modified it to have the same.
The Instance also had many hours trying to propagate dns info. I have been trying the whole day to fix this.
What could I have possibly missed here.
The difference between the old(live) and new(test) is that the latter has no Multi AZ, no Backups, no Minor Version Upgrade. Ports are all default postgres 5432
psql response is that it asks for password with the new it says
psql: could not connect to server: Connection refused
Is the server running on host "**********.eu-central-1.rds.amazonaws.com" (**.**.**.**) and accepting
TCP/IP connections on port 5432?
It seems our company firewall won't let me out on port 5432.
I am trying to connect AWS RDS PostgreSql from PgAdmin 3. I followed the below link
http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_ConnectToPostgreSQLInstance.html
In Security Group, I also added PostgreSQL and All traffic as below
The "publicly accessible" flag was enabled (updated after Mark B's comment)
I got the error from PGAdmin3
Very appreciate for any suggestion
******UPDATE*******
I can connect pgAdminIII to AWS RDS successfully using home wifi, but cannot connect using office wifi.
My concern is:
Was the port 5432 blocked by office wifi?
How can I configure/update the port without impacting to current API?
Note: My current API is working well (CRUD)
Can you can test your connection to a DB instance using common Linux or Windows tools first?
From a Linux or Unix terminal, you can test the connection by typing the following (replace with the endpoint and with the port of your DB instance):
$nc -zv DB-instance-endpoint port
For example, the following shows a sample command and the return value:
$nc -zv postgresql1.c6c8mn7tsdgv0.us-west-2.rds.amazonaws.com 8299
Connection to postgresql1.c6c8mn7tsdgv0.us-west-2.rds.amazonaws.com
8299 port [tcp/vvr-data] succeeded!
Windows users can use Telnet to test the connection to a DB instance. Note that Telnet actions are not supported other than for testing the connection. If a connection is successful, the action returns no message. If a connection is not successful, you receive an error message such as the following:
C:>telnet sg-postgresql1.c6c8mntzhgv0.us-west-2.rds.amazonaws.com
8299
Connecting To sg-postgresql1.c6c8mntzhgv0.us-west-2.rds.amazonaws.com...Could not
open connection to the host, on port 819: Connect failed
If Telnet actions return success, then you are good to go.
If you are trying to access it from a network which is not listed for that port. you need to add inbound rules for those network IPs from AMAZON RDS system
You will also need to set Public accessibility true under Connect & security tab in RDS console.
Read this post.In your security group go to unbound rules and add my ip.
and make sure your database is public.
https://serverfault.com/questions/656079/unable-to-connect-to-public-postgresql-rds-instance
I have configured an RDS Postgres micro instance for testing but am having problem connecting. I have created a security group and allow all hosts (0.0.0.0/0) by default to port 5432 but for some reason I am not able to connect. I tried connecting to the port 5432 on the machine but it appears to be closed. Any ideas? I have verified that the security group is using the correct security group.
This is my configuration
TCP
Port (Service) Source Action
5432 0.0.0.0/0 Delete
Here is how I am trying to connect to the rds service
psql -p 5432 -h example.cs945smhrv09.us-west-2.rds.amazonaws.com -U example example
UPDATE
OK feel a little silly after figuring out that rds starts up on a private 172 subnet. I was able to connect from my ec2 instance.
Question now is there a way to put that on a public address that I can connect to? I would like to configure it with a security group that limits the source ip but then be able to connect to it with psql from my desktop.
Update 2
I don't remember all the details when I originally posted this question but it is no longer an issue. We now have a management vpc that we connect to using a vpn server. The management vpc network is trusted by the prod and devel security groups for the rds instances. We have no problem communicating with them without adding any public ips.
Step 4 of the RDS setup wizard ("Additional Config") lets you select whether the DB is Publicly Accessible or not (if it's not, access is limited to within your VPC, regardless of your security group settings). I'd imagine that determines whether the hostname resolves to 54.* vs 172.* .
Your first instance may have been set to private - as far as I can tell this can't be changed after initialization. I had a similar issue and creating a new instance set to be publicly accessible solved it for me.
My company's firewall had blocked outgoing traffic to the default postgres port, 5432. I didn't realise this until I tested the port using netcat in a terminal window, as follows:
nc -zv portquiz.net 5432
Hopefully this saves someone half a day of messing around with AWS security groups.
Absolutely there is. I have pgadmin loaded on my local machine connected to my RDS instance. |
I just confirmed that I can connect using the same syntax you used for psql, although the -U was not necessary for me (I still included both "example" parameters though)
I did edit the "default" security group and added my IP (Ip of the machine I'm connecting from)/32 to be allowed, although I'm sure you could get less restrictive than that, but I'd start there and see if it helps.
This is likely caused by your outbound firewall rules. If you're behind a corporate firewall, they will often block outbound ports above a certain number. You need to setup an SSH tunnel of some kind to avoid this.
http://conoroneill.net/accessing-amazon-rds-instances-from-your-desktop/
I had this problem. This isn't a direct solution but might work for some. I set the Postgres security group to accept connections coming from the server that will be using it (same VPC). I then set up my Postgres client to use ssh tunneling. If I need to use the command line, I simply SSH into that server and then psql from there.
You can simply use workbench recommended by aws but make sure your all postgres availibily is set to everywhere..while adding PostgrSQL 5432
https://aws.amazon.com/getting-started/tutorials/create-connect-postgresql-db/