I am trying to set up firehose to send data from a kinesis stream to a redshift cluster. Firehose successfully inserts the data to my s3 bucket, but I am receiving the following error when firehose attempts to execute the s3->Redshift copy command:
The connection to the specified Amazon Redshift cluster failed. Ensure that security settings allow Firehose connections, that the cluster or database specified in the Amazon Redshift destination configuration JDBC URL is correct, and that the cluster is available.
I have performed every setup step according to this except for one: I did not make my Redshift cluster publicly accessible. I am unable to do this bc the cluster is in a private VPC that does not have an internet gateway attached.
After researching the issue, I found this article which provides insight for how to set up an AWS PrivateLink with firehose. However, I have heard that some AWS services support PrivateLink and others do not. Would PrivateLink work for this case?
I am also concerned with how this would affect the security of my VPC. Could anyone provide insight to possible risks to using a PrivateLink?
I was able to solve this issue. Add an Internet gateway to your VPC route table.
Go to Redshift VPC.
On the Routes tab (you must have 3 private routes), choose Edit, Add another route, and add the following routes as necessary. Choose Save when you're done.
For IPv4 traffic, specify 0.0.0.0/0 in the Destination box, and select the internet gateway ID in the Target list.
If you add internet gateway ID to all 3 private routes, you might see Failure in other applications that are using the same route/VPC. To fix that, update only 1 route with internet gateway ID and the other two will have nat as destination for (0.0.0.0/0).
Related
I am trying to enable ZoneRedundant High Availability on our Azure PostgreSQL Flexible server.
The Azure documentation mentions the following important step:
High availability Features of Azure Database for PostgreSQL - Flexible Server require ability to send\receive traffic to destination ports 5432, 6432 within Azure virtual network subnet where Azure Database for PostgreSQL - Flexible Server is deployed , as well as to Azure storage for log archival. If you create Network Security Groups (NSG) to deny traffic flow to or from your Azure Database for PostgreSQL - Flexible Server within the subnet where its deployed, please make sure to allow traffic to destination ports 5432 and 6432 within the subnet, and also to Azure storage by using service tag Azure Storage as a destination.
I'm finding it hard to get my head around this from the way it is written and can't find many details about it elsewhere.
From what I understand, the first requirement is to add an inbound NSG rule as follows:
Source IP: [CIDR of the database subnet]
Source Port Range: *
Destination IP addresses: [CIDR of the database subnet]
Destination Port Ranges: 5432,6432
Action: Allow
Priority: [Any number before the default inbound DENY rules]
Have I read this correctly? We are effectively allowing traffic between the instance and the replication instance within the subnet, so the source IP and the destination IP should both be the database subnet CIDR.
I know that NSGs are stateful. So am I right in saying I don't need any Outbound rule for this case?
The second requirement to allow traffic 'to Azure storage by using service tag Azure storage as a destination' is confusing me. From the way this is written I don't know if it should be an inbound or an outbound rule. My first guess was outbound as logs would be moving FROM the database to Azure storage. Is that correct?
• As per the statement from the Microsoft documentation, it is imminent that all the resources dealing with or related to the Azure PostgreSQL Database – Flexible Server should be deployed within the same virtual network. The statement accordingly states that for Azure PostgreSQL Database – Flexible server, the incoming as well as outgoing traffic over the ports 5432 and 6432 should be allowed within the same subnet.
It is because the PostgreSQL Database Server listens to the localhost IP, i.e., ‘127.0.0.1’ through these ports only, i.e., 5432 and 6432. Also, since these ports are by default not open and thus, are secure in these scenarios, the related Microsoft documentation text states accordingly.
Thus, based on that, the NSG rule that you have created is clearly sufficient and correct to allow the traffic on these ports for the PostgreSQL database server. And yes, the traffic rule needs to be created to the Azure storage from the Azure PostgreSQL database subnet and from the trusted network ranges/subnets to the Azure storage with the destination as Azure Storage as a service tag if only logs are to be transferred to that storage account. If it is provisioned for some other purpose other than this, then you will have to create rules accordingly.
Ensure to create an outbound NSG rule for the Azure storage account which will suffice the requirement from the Azure PostgreSQL DB Server subnet.
I read the VPC flow logs blog post here. It excludes traffic to Amazon DNS servers.
Also DHCP traffic is excluded and requests for instance metadata.
Just curious to know why that is.
If you reach aws about it and they will just refer it as system limitation at this point and will be added as a road map
So, I created a postgreSQL instance in Google Cloud, and I have a Kubernetes Cluster with containers that I would like to connect to it. I know that the cloud sql proxy sidecar is one method, but the documentation says that I should be able to connect to the private IP as well.
I notice that a VPC peering connection was automatically created for me. It's set for a destination network of 10.108.224.0/24, which is where the instance is, with a "Next hop region" of us-central1, where my K8s cluster is.
And yet when I try the private IP via TCP on port 5432, I time out. I see nothing in the documentation about have to modify firewall rules to make this work, but I tried that anyway, finding the firewall interface in GCP rather clumsy and confusing compared with writing my own rules using iptables, but my attempts failed.
Beyond going to the cloud sql sidecar, does anyone have an idea why this would not work?
Thanks.
Does your GKE cluster meet the environment requirements for private IP? It needs to be a VPC enabled cluster on the same VPC and region as your Cloud SQL instance.
In the end, the simplest thing to do was to just use the google cloud sql proxy. As opposed to a sidecar, I have multiple containers needing db access so I put the proxy into my cluster as its own container with a service, and it seems to just work.
If your instance of cloud SQL or compute both in the same VPC then only you can create a VPC peering over private IP.
From cloud SQL compute VM you can choose the VPC and subnet and also setup same for the GKE and you can make the connection from pod to cloud sql.
I am fairly new to AWS, so I am sure that I am just missing something, but here is my problem:
I have created a VPC with 3 subnets and one security group linked to all of them. The security group accepts inbound from my machine. Next I have created two RDS instances (both PostgreSQL), put them into that VPC and linked them to the VPC security group. Weirdly, I can only connect to one of them, for the other one I get a generic time-out error.
Any idea on what I am missing? I can share any more details if needed.
EDIT: Both RDS instances are deployed on the same subnets and I am trying to connect from my machine on the internet.
Please verify that to fix your issue:
Both RDS instance have been deployed into the same Subnet.
If not check that both subnets are public subnets and have a route to your internet gateway
If one RDS (the not working one) is in a private subnets you should consider using a bastion to access it because by default you should not have a route to your Internet Gateway.
But still you will find a bellow a simple subnet design if you want to build something secure:
Create 2 public subnets if you want to deploy something directly accessible through internet (one good practice is to deploy only managed instance there (like load balancer)
Create 2 private subnets with NATGateway and correct route configuration to it
Create a bastion in you public subnets to be able to access your instance in private
Deploy your RDS into Private subnets and create one security group for each (or one for both if they are really linked)
You will find an AWS QuickStart which deploy all network stack for you on VPC Architecture - AWS Quick Start.
I am following the link below to create a Cloud SQL Proxy.
https://cloud.google.com/sql/docs/mysql/connect-container-engine
When I get to the step to create the service account, I am unable to see any Cloud SQL roles, even though I have a MySQL instance associated with the project and I have enabled Cloud SQL administration API as described in the previous step.
Also, the whole process seems to be quite long-winded. Is there a way to connect directly from the container cluster to Cloud SQL without using the proxy? If yes, how do I find the IP address of the Cloud SQL instance? Also, how do I get the container cluster IP to white-list?
Many thanks