I am trying to use the PostgreSQL foreign data wrapper (postgres_fdw) extension from an Azure DB for PostgreSQL instance. The foreign data wrapper should be supported as mentioned in Azure DB for PostgreSQL: https://learn.microsoft.com/en-us/azure/postgresql/concepts-extensions
I manage to create a server and a mapping for the user, but when I try to import a table or a schema I have the following error:
ERROR: could not connect to server "<server with public hostname on AWS>"
DETAIL: could not translate host name "<server with public hostname on AWS>" to address: Unknown host
When replacing the hostname with a public IP I get the following:
ERROR: could not connect to server "<SERVER NAME>"
DETAIL: could not connect to server: Network is down (0x00002742/10050)
Is the server running on host "<public IP of the server" and accepting
TCP/IP connections on port 5432?
In the firewall options of the Azure DB for PostgreSQL instance, I already have the AllowAllIps rule starting 0.0.0.0 and ending 255.255.255.255. is there any thing else to configure to get access to servers outside of Azure, DNS or firewall configuration?
Not entirely sure about this, but we were trying to do something like this the other day, and stumbled upon this (somewhat disheartening) quote:
"Currently, outbound connections from Azure Database for PostgreSQL are
not supported, except for connections to other Azure Database for
PostgreSQL servers."
Source: https://learn.microsoft.com/en-us/azure/postgresql/concepts-extensions#dblink-and-postgres_fdw
Related
I'm trying to create a subscription.
It works well when I create it on a PostgreSQL instance or cluster (to PostgreSQL).
But when I try to do logical replication from PostgreSQL to AWS Aurora PostgreSQL I see the following error:
ERROR: could not connect to the publisher: could not connect to server: No route to host
Is the server running on host "my-db.dfsfdsfsdfd.us-east-1.rds.amazonaws.com" (10.2.5.7) and accepting
TCP/IP connections on port 5432?
Some notes:
I updated the parameter group to enable replication: rds.logical_replication (from 0 to 1)
The RDS policy has enough permissions.
Both DBs are in the same VPC, same subnets and share the security group
I can connect to both DBs, and as I said, subscription works if instead of using Aurora I use PostgreSQL
Any idea why this could be happening?
Thanks!
I am trying to pull data from Postgresql on Azure using Azure Data Factory but have no luck. Both ADF and VM are in the same Azure Vnet.
In my VM networking settings I allowed port 5432 inbound access from ADF to VirtualNetwork. Similar outbound vnet to vnet.
I also installed ODBC driver on VM and added it to firewall.
I installed also the selfhosted integration runtime on VM which connects to ADF.
When creating a new Linked Service on ADF I chose PostgreSQL as source.
Not able to establish connection though.
Tried different settings:
-IP and DNS as servername
-self hosted and auto resolve integration runtime
-public and private IP
-no encryption and encryption
-temporarily turned off firewall on vm to test
still no success.
What am I missing?
password and database names are correct.
The error message I am getting when testing connection:
Error code
9603
Details
ERROR [08001] [Microsoft][ODBC PostgreSQL Wire Protocol driver]Connection refused. Verify Host Name and Port Number.
ERROR [HY000] [Microsoft][ODBC PostgreSQL Wire Protocol driver]Can't connect to server on 'xx.xxx.xxx.xx'
I would appreciate it if anyone could help.
Thank you
I found the solution. I need to use the ODBC connector as a linked service on ADF, not the PostgreSQL connector. Bingo!
I am trying to connect to my RDS database from my computer with a python script using psycopg2.
python code:
import psycopg2
from db_credentials import *
import logging
def get_psql_conn():
conn = psycopg2.connect(dbname=DB_NAME, user=DB_USER, password=DB_PASS, host=DB_HOST)
logging.info("connected to DB!")
return conn
I get the following error:
psycopg2.OperationalError: could not connect to server: Operation timed out
Is the server running on host ********* and accepting
TCP/IP connections on port 5432?
My security groups assigned to the RDS database:
SG 1:
SG 2:
Now i tried to make a security group which allows my computer IP to access the DB.
SG 3:
I can connect to the DB from my ec2 instances, running the same python script as above. This seemingly has to do with the 2nd security group, as when i remove it, i can no longer connect from my ec2 instances either. It then throws the same error i get when trying to connect from my computer.
I have little understanding of RDS or security groups, i just followed internet tutorials, but seemingly couldnt make much sense out of it.
Any help is greatly appreciated! Thanks
When accessing an Amazon RDS database from the Internet, the database needs to be configured for Publicly Accessible = Yes.
This will assign a Public IP address to the database instance. The DNS Name of the instance will also resolve to the public IP address.
For good security on publicly-accessible databases, ensure that the Security Group only permits access from your personal IP address.
When I select "sslmode=verify-full" when connecting to my postgres server in Azure, the connection fails with the following error:
server certificate for "<region>.control.database.windows.net" (and 7 other names) does not match host name "<servername>.postgres.database.azure.com".
Is "sslmode=verify-full" not allowed for Azure Database for PostgreSQL service? How do I connect to my postgres server in Azure if I have a requirement to use "sslmode=verify-full"?
"sslmode=verify-full" is supported in Azure Database for PostgreSQL service, however there is a limitation in the current implementation of the service.
If you require using "sslmode=verify-full", please use the server naming convention <servername>.database.windows.net as your connection string host name instead of the preferred host naming convention <servername>.postgres.database.azure.com.
This limitation is expected to be removed in the future. Connections using other SSL modes should continue to use the preferred host naming convention <servername>.postgres.database.azure.com.
Please refer to How to Configure SSL for Postgres in Azure for more details.
I have two Postgres 9.3.5 instances in RDS, both in one security group that allows all inbound traffic from within the security group and all outbound traffic. I'm trying to set up one database to be able to select from a few tables from the other via postgres_fdw.
I've created the server -
create server master
foreign data wrapper postgres_fdw
OPTIONS (dbname 'main',
host 'myinstance.xxxxx.amazonaws.com');
as well as the requisite user mapping and foreign table -
create foreign table condition_fdw (
cond_id integer,
cond_name text
) server master options(table_name 'condition', schema_name 'data');
However, a simple select count(*) from condition_fdw gives me
ERROR: could not connect to server "master"
DETAIL: could not connect to server: Connection timed out
Is the server running on host "myinstance.xxxxxx.amazonaws.com" (xx.xx.xx.xx) and accepting
TCP/IP connections on port 5432?
I can connect to both databases via psql from an EC2 instance. I know until recently RDS didn't support postgres_fdw, but I'm running the newer versions that do.
In the create server statement, I have tried replacing "myinstance.xxxxxx.amazonaws.com" with the IP address it resolves to, no luck.
Any ideas?
Further Testing
I installed postgres on an ec2 instance with the same security group, foreign tables to the master server behave as expected.
postgres_fdw between databases on the same RDS instance works.
This all leads me to think it must some issue with outgoing connections from postgres_fdw on my Postgres RDS instance.
To get postgres_fdw to work between two instances in AWS RDS (with VPC) I had to:
Enable custom_dns_resolution https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Appendix.PostgreSQL.CommonDBATasks.html#Appendix.PostgreSQL.CommonDBATasks.CustomDNS
Add the public IP of the querying database in the inbound rules of the Security group OR add the internal IP of the RDS instance as server host. https://forums.aws.amazon.com/thread.jspa?threadID=235154
The first one is documented pretty well but I could not get it to work until stumbling over the second one.
It would appear that Amazon does not allow outgoing connections from RDS instances, so until that changes using postgres_fdw across RDS instances is not possible. I'll have to run an ec2 instance as my postgres server in order to use a foreign table to a database on another server.
http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Overview.RDSSecurityGroups.html
I experienced a similar problem and found that postgres_fdw would only work in the EU-WEST-1 RDS region when the master and remote instances were in the same availability zone and on 9.3.5, if you crossed AZs it wouldn't connect. My security group was 5432 TCP inbound only and all outbound.
If you are in the same availability zone, there is no reason why it wouldn't work.