I have an ECS Fargate container app that serves the API request over the public internet.
My understanding is that this API service container can be deployed on the public subnet and that is configured with ALB DNS and target group. As we can see target group redirects the traffic to private IP of the ECS task, I guess we don't need public IP to be enabled when launching the task. However when I attempt this on ECS task launch getting an error "Resourceinitializationerror: unable to pull secrets or registry auth: execution resource retrieval failed: unable to retrieve ecr registry auth: service call has been retried 3 time(s): RequestError: send request failed caused by: Post "https://api.ecr.eu-west-2.amazonaws.com/": dial tcp 52.94.53.88:443: i/o timeout"
If this is not workable and we need to enable public ip on the task launch, I'd prefer to restrict the public IP port access only to web service ALB for best security practice. Could someone suggest me a workable approach on this use-case pls? Thanks.
"I'd prefer to restrict the public IP port access only to web service ALB for best security practice."
Have you tried doing that? It should work fine. Since security groups are stateful, as long as the outbound rules are open, you should be able to lock down the inbound rules on the security group.
If you want remove the public IP completely, then you will need to either deploy Fargate task to a private subnet, with a route to a NAT Gateway, or add VPC endpoints to your VPC for the AWS services that the task needs to access, like ECR.
Related
I deployed a containerized app to my IBM Cloud Kubernetes service in a VPC. The app uses App ID for authentication. The deployment pipeline ran successfully. The app seems ready, but when accessing its URL it gives an internal server error (500 status code).
From the Kubernetes dashboard I found that the ALB Oauth Proxy add-on is failing. It is deployed, but does not start.
The deployment seems to fail in the health checks (ping not successful). From the POD logs I found the following as last (and only) entry:
[provider.go:55] Performing OIDC Discovery...
Else, there is not much. Any advise?
Guessing from the missing logs and the failing pings, it seemed related to some network setup. Checking the VPC itself, I found that there was no Public Gateway attached to the subnet. Enabling it allowed outbound traffic. The oauth proxy could contact the App ID instance. The app is working as expected now.
Make sure that the VPC subnets allow outbound traffic and have a Public Gateway enabled.
I'm trying to setup Identity Aware Proxy for my backend services parts of which resides in GCP and other on on-prem,according to the instruction given in the following link
Enabling IAP for on-premises apps and
Overview of IAP for on-premises apps
After, following the guide I ended up in a partial state where services running on GCP serving at https endpoint is perfectly accessible via IAP. However, the app which is running on on-prem is not reachable through pods* and external loadbalancer*.
Current Architecture followed:
Steps Followed
On GCP project
Created a VPC network in any region with one subnet in my case (asia-southeast1)
Used IAP connector https://github.com/GoogleCloudPlatform/iap-connector
Configured the mapping for 2 domains.
For app in GCP
source: gcp.domain.com
destination: app1.domain.com (serving at https endpoint)
For app in on-prem(Another GCP project)
source: onprem.domain.com
destination: app2.domain.com (serving at https endpoint but not exposed to internet)
Configured VPN Tunnel between both the project so the network gets peered
Enabled IAP for the loadbalancer which is created by the deployment.
Added corresponding accounts to allow access to the services with IAP web-user role.
On-prem
Created VPC network in a region with one subnet (asia-southeast1)
Created VM on VPC in that region
Assigned that VM to an instance group
Created Internal Https loadbalancer and chose instance group as backend
Secured load balancer http with ssl
Setup VPN tunnel to the first project
What I have tried?
logged in to pods and pinged different pods. All pods were reachable.
logged in to nodes and pinged the remote VM on port 80 and 443 both are reachable.
pinged remote VM from inside the pods. Not reachable.
Expected Behaviour:
User requests to loadbalancer on the app1.domain.com which IAP authenticates and authorizes user with OAuth and grant access to the webapp.
User requests to loadbalancer on the app2.domain.com which IAP authenticates and authorizes user with OAuth and grant access to the webapp running on on-prem.
Actual Behaviour
Request to the app1.domain.com prompts OAuth screen after authenticating the website is returned to the user.
Request to the app2.domain.com prompts OAuth screen after authenticating the browser returns 503 - "No healthy upstream"
Note:
I am using a separate GCP project to simulate on-premise.
Both projects are peered via VPN tunnel.
Both peering projects have subnets in the same region.
I have used internal https loadbalancer in my on-prem project to make my VM visible in my host project so that the external loadbalancer can route request to the VM's https endpoint.
** I'm suspecting that if pod could able to reach the remote VM the problem might as well be resolved. It's just a wild guess.
Thank you so much guys. I'm looking forward for your responses.
I have 2 GKE cluster both private and public and using cloudproxy as sidecar container for gke app to access cloudsql instance.
public cluster setup for development/testing
Cloud SQL is enabled with both private and public IP.
GKE app is using cloudproxy with default option of ip types (public,private) as below
Cloud SQL doesn't have any authorized network.
In this case, my app is able to connect CloudSQL and works smoothly. As far as I understand, here connection to cloudsql should be happening with private becuase there is no authorised network configured.
private cluster setup for production
Cloud SQL is enabled with both private and public IP.
GKE app is using cloudproxy with default option of ip types (public,private)
cloudsql-proxy setting in deployment file
- name: cloudsql-proxy
image: gcr.io/cloudsql-docker/gce-proxy:1.11
command: ["/cloud_sql_proxy"]
args: ["-instances=$(REAL_DB_HOST)=tcp:$(REAL_DB_PORT)","-credential_file=/secrets/cloudsql/credentials.json"]
case 1
Cloud SQL doesn't have any authorized network.
Result: Application is not able to connect with Cloud SQL
case 2
Cloud SQL have private GKE NAT gateway as authorized network
Result: Application is not able to connect with Cloud SQL
May be removing cloudproxy from application will work (I am yet to test) but it discourages the usage of proxy during dev env as it will need changes in deployment file during production deployment.
I am not able to understand what is causing the connection failure with cloudproxy in gke private cluster. Should we not use cloudproxy in private cluster?
Update
The reason due to which cloud proxy not able to connect cloud sql was disabled Cloud SQL Admin API. I have updated my answer in answer section.
It looks like the question here is "Should we use the Cloud SQL proxy in a private cluster?" and that answer is "it depends". It's not required to connect, but it allows for more security because you can restrict unnecessary access to your Cloud SQL server.
The Cloud SQL proxy doesn't provide connectivity for you application - it only provides authentication. It has to be able to connect via the existing path, but then uses the Service Account's IAM roles to authenticate the connection. This also means that it doesn't have to come from a whitelisted network because it's been authenticated by a different means.
If you want to use the proxy to connect via Private IP (instead of defaulting to public), use the -ip_address_types=PRIVATE - this will tell the proxy to connect with the instance's Private IP instead. (Please note that if the proxy lacks a network path (eg, isn't on the VPC) that proxy will still be unable to connect.)
#kurtisvg has provided an informative answer to it.
However the real issue was SQL Admin API and enabling it fixed the issue. After looking into the logs I found below entry.
Error 403: Access Not Configured. Cloud SQL Admin API has not been used in project XXXXXX before or it is disabled. Enable it by visiting https://console.developers.google.com/apis/api/sqladmin.googleapis.com/overview?
The issue for me was enabling Private cluster in GKE cluster :(
Because of private GKE cluster it wasn't having access to external IP addresses and fix was to create a NAT gateway with cloud router as per https://cloud.google.com/nat/docs/gke-example.
Hint if it's the issue is you won't be able to ping to google.com etc from the container after logging into it.
I am new to Networking and have some questions regarding some of the service tags in Azure NSG.
If you see below, Azure has multiple options for service tags while defining inbound NSG rules. But I failed to understand the scenarios for AzureKeyVault, Storge, Cosmos DB etc. in which scenarios these services initiate the request? Why do we need these service tags in the inbound NSG.
But I failed to understand the scenarios for AzureKeyVault, Storage,
Cosmos DB etc. in which scenarios these services initiate the request?
Why do we need these service tags in the inbound NSG.
It's not so good understanding for service tags in the inbound NSG as outbound NSG. For example, If you want to deny all outbound internet traffic and allow only traffic to specific Azure services such as AzurekeyVault or AzureCosmosDB. You can do so using service tags as the destination in your NSG outbound rules.
Similarly, If you want to allow or deny traffic from Azure service in a virtual network, Ip address or Application security group. You can do so using service tags as the source in your NSG inbound rules. For example, you can set the service tag AppService and specific IP addresses(some specific VM IP address) as the destination, then you could restrict the AppService to access the resources in your VM like API or database.
For more details, you can view scenarios for securing your Azure service.
I can't connect from azure resource (aks node) to Azure postgres using pgcli. I also tried directly from node and got the same error message:
FATAL: Client from Azure Virtual Networks is not allowed to access the server. Please make sure your Virtual Network is correctly configured.
Firewall rules in the resource are on:
Allow access to Azure services: ON
Running the same pgcli login command on my computer and on another azure resource seems to work fine.
Adding Firewall rules to all IPs return the same error.
Curl from the problematic server (host:5432) returns a reply, so it's not an outbound issue.
What does the error mean?
A VM where the connection originates from is deployed to a virtual network subnet where Microsoft.Sql service endpoint is turned on. Per documentation:
If Microsoft.Sql is enabled in a subnet, it indicates that you only want to use VNet rules to connect. Non-VNet firewall rules of resources in that subnet will not work.
For connection to succeed there must be a VNet rule added on PostgreSQL side. At the time the question was asked VNet Service Endpoints for Azure Database for PostgreSQL just got to public preview so I assume it might not have been available for the OP.
Solution
As of November 2020, Service Endpoints for Postgres is GA and instead of disabling the service endpoint one can add a missing VNet rule to the PostgreSQL server instance and reference the service endpoint-enabled subnet. It can be done via Portal or Azure CLI
Apparently, the vm is part of a vnet that a service endpoint tag Microsoft.sql was enabled.
I found this answer. To solve the problem I disabled the service endpoint and added the public IP to the Connection Security section.
I encountered the same problem.
All I did was to switch Allow access to Azure services to ON .