IAP connector not routing request to on-prem. "No healthy upstream" - kubernetes

I'm trying to setup Identity Aware Proxy for my backend services parts of which resides in GCP and other on on-prem,according to the instruction given in the following link
Enabling IAP for on-premises apps and
Overview of IAP for on-premises apps
After, following the guide I ended up in a partial state where services running on GCP serving at https endpoint is perfectly accessible via IAP. However, the app which is running on on-prem is not reachable through pods* and external loadbalancer*.
Current Architecture followed:
Steps Followed
On GCP project
Created a VPC network in any region with one subnet in my case (asia-southeast1)
Used IAP connector https://github.com/GoogleCloudPlatform/iap-connector
Configured the mapping for 2 domains.
For app in GCP
source: gcp.domain.com
destination: app1.domain.com (serving at https endpoint)
For app in on-prem(Another GCP project)
source: onprem.domain.com
destination: app2.domain.com (serving at https endpoint but not exposed to internet)
Configured VPN Tunnel between both the project so the network gets peered
Enabled IAP for the loadbalancer which is created by the deployment.
Added corresponding accounts to allow access to the services with IAP web-user role.
On-prem
Created VPC network in a region with one subnet (asia-southeast1)
Created VM on VPC in that region
Assigned that VM to an instance group
Created Internal Https loadbalancer and chose instance group as backend
Secured load balancer http with ssl
Setup VPN tunnel to the first project
What I have tried?
logged in to pods and pinged different pods. All pods were reachable.
logged in to nodes and pinged the remote VM on port 80 and 443 both are reachable.
pinged remote VM from inside the pods. Not reachable.
Expected Behaviour:
User requests to loadbalancer on the app1.domain.com which IAP authenticates and authorizes user with OAuth and grant access to the webapp.
User requests to loadbalancer on the app2.domain.com which IAP authenticates and authorizes user with OAuth and grant access to the webapp running on on-prem.
Actual Behaviour
Request to the app1.domain.com prompts OAuth screen after authenticating the website is returned to the user.
Request to the app2.domain.com prompts OAuth screen after authenticating the browser returns 503 - "No healthy upstream"
Note:
I am using a separate GCP project to simulate on-premise.
Both projects are peered via VPN tunnel.
Both peering projects have subnets in the same region.
I have used internal https loadbalancer in my on-prem project to make my VM visible in my host project so that the external loadbalancer can route request to the VM's https endpoint.
** I'm suspecting that if pod could able to reach the remote VM the problem might as well be resolved. It's just a wild guess.
Thank you so much guys. I'm looking forward for your responses.

Related

IBM Cloud: Kubernetes add-on ALB Oauth2 Proxy for App ID integration fails to start

I deployed a containerized app to my IBM Cloud Kubernetes service in a VPC. The app uses App ID for authentication. The deployment pipeline ran successfully. The app seems ready, but when accessing its URL it gives an internal server error (500 status code).
From the Kubernetes dashboard I found that the ALB Oauth Proxy add-on is failing. It is deployed, but does not start.
The deployment seems to fail in the health checks (ping not successful). From the POD logs I found the following as last (and only) entry:
[provider.go:55] Performing OIDC Discovery...
Else, there is not much. Any advise?
Guessing from the missing logs and the failing pings, it seemed related to some network setup. Checking the VPC itself, I found that there was no Public Gateway attached to the subnet. Enabling it allowed outbound traffic. The oauth proxy could contact the App ID instance. The app is working as expected now.
Make sure that the VPC subnets allow outbound traffic and have a Public Gateway enabled.

How to setup k8s istio aspnet identity server for internal requests

We have a cluster that runs a number of dotnet apps, one of which runs the identity server. All the other apps need to authenticate with the identity server. If the identity server was external this wouldn't be an issue as it would have an HTTPS endpoint, but internally they are all running HTTP.
With istio adding MTLS security to all the comms, do all the apps just get set with RequireHttpsMetadata = false?
Is this the correct way to setup the network with internal requests being sent as http://auth-server.default.svc.cluster.local/...?
or should they be sent as https://auth-server.default.svc.cluster.local/..., if so how?

Forwarding All Traffic from Global External IP to a Domain on GCP

I have an Autopilot GKE cluster set up. There is an Ingress which is an entry point to the app deployed in the cluster. I managed to configure SSL and HTTP -> HTTPS redirection with ease.
I also configured Cloud DNS that resolves my domain name to the cluster's IP (global static IP, let's name it global-front-app-ip).
This works without any problems. I'm able to access the app with the domain I own. My setup is very simillar to the one described in this article.
What I'm trying to achieve now is to redirect all the clients that try to access the app with LB IP global-front-app-ip to the domain name (http://global-front-app-ip -> http://my-domain.com).
I played with LB forwarding rules and Cloud Armor but I haven't found a working solution.

How do you deploy Identity Server on Kubernetes?

I want to deploy Identity Server 4 on Kubernetes 1.8, and use this as a Federation Gateway between my web application and Azure Active Directory (to begin with).
If I call Identity Server from my web application using the local k8s service name, my users are redirected to the wrong Identity Server URL (containing the local k8s service name) during Sign in which clearly won't work. We are using an implicit flow.
I therefore setup a Azure Load balancer with dns name and configured Identity Server to be externally accessible with the domain name as the PublicOrigin URL.
However, my web application which runs in the same cluster cannot access Identity Server using the external URL of the Identity Server (discovery fails).
If I run Identity Server on another Kubernetes cluster then everything works fine.
My question is:
How do you properly deploy Identity Server in Kubernetes? Do I really need another Kubernetes cluster?
Note: I am using Kubernetes on Azure created with ACS engine (because we have mixed windows and linux containers).
I'm using AKS (Azure managed kubernetes) and have a single client asp.net core 2 web app in the same cluster as my IS4 service with no issues. Both webapps are fronted by Nginx with kube-lego for LetsEncrpyt TLS support, and DNS is provided by Azure DNS.
I'm not using the PublicOrigin but instead the client app's Authority (in the openidconnect setup) uses the full (external Azure) DNS name of the IS4 service. You can use PublicOrigin if you want to use the cluster service naming from your clients

For the Bluemix Secure Gateway service, how does the data center's network need to be configured?

I am going to use Secure Gateway service in Bluemix and I have some questions about how I should make it work.
Systems in my data center's intranet access the Internet through a proxy (with no authentication). Can Secure Gateway connect to Bluemix via a proxy?
Does it connect to Bluemix via HTTPS protocol?
The network admins asked me: What are the IPs (or the IP range) of Bluemix, any idea?
Thank you very much.
A Secure Gateway instance runs in two parts, as shown in "Reaching enterprise backend with Bluemix Secure Gateway via console": the gateway and the gateway client. The gateway runs in Bluemix, the gateway client runs in the data center containing one or more systems of record to connect to. The gateway client needs network access to the Bluemix data center (typically via the Internet) and to the systems of record (via the data center's internal network). The gateway client initiates the connection, so it needs to know Bluemix's address, but Bluemix doesn't need to know the gateway client's address.
To answer your questions specifically:
A proxy isn't supported. The gateway and its client need direct access to each other.
The connection uses HTTPS for SSL encryption. The transport level security (TLS) options can be used to add authentication.
Bluemix's IP addresses aren't published.
For point 3:
The client connects outbound to the cloud services. Once the SecGW is connected, all additional Destination connects flow through that connection, no additional firewall or iptables rules are needed. If they have a rule in-place so that the on-premises machine where the SecureGateway client is installed can use the outbound port 443 (HTTPS) to make connections, that is all they need.