I am running a Kubernetes Cluster with an Nginx-ingress fronting couple of web apps. Because Nginx doesn't support SSO/OIDC by default, I use an oauth_proxy for authentication.
Everything is working, only verified users are able to access the web pages.
Is it possible to pass or request information from the Identity Provider to the client?
Edit
I already use oauth2_proxy (https://github.com/pusher/oauth2_proxy) with Azure AD. The issue is that I need all user details from the IP.
Logs from my oauth2_proxy:
$ kubectl logs oauth2-proxy-7ddc97f9d5-ckm29
[oauthproxy.go:846] Error loading cookied session: Cookie "_oauth2_proxy" not present
...
[requests.go:25] 200 GET https://graph.windows.net/me?api-version=1.6 {
"odata.metadata":"https://graph.windows.net/myorganization/$metadata#directoryObjects/#Element",
"odata.type":"Microsoft.DirectoryServices.User",
"objectType":"User",
... ,
"sipProxyAddress":"nico.schuck#example.com",
"streetAddress":"my stree",
"surname":"Schuck",
"telephoneNumber":55512345,
"usageLocation":"DE",
"userType":"Member"
}
165.xxx.xxx.214 - nico.schuck#example.com [2020/01/17 11:22:02] [AuthSuccess] Authenticated via OAuth2: Session{email:nico.schuck#example.com user: token:true id_token:true created:2020-01-17 11:22:02.28839851 +0000 UTC m=+181.592452463 expires:2020-01-17 12:22:02 +0000 UTC refresh_token:true}
Consider oauth2_proxy which works well with nginx ingress for SSO. Follow the below link
https://github.com/bitly/oauth2_proxy
You should be using below configuration in your Ingress Rule
metadata:
name: application
annotations:
nginx.ingress.kubernetes.io/auth-url: "https://$host/oauth2/auth"
nginx.ingress.kubernetes.io/auth-signin: "https://$host/oauth2/start?rd=$escaped_request_uri"
Related
We have a dozen of services exposed using a ingress-nginx controller in GKE.
In order to route the traffic correctly on the same domain name, we need to use a rewrite-target rule.
The services worked well without any maintenance since their launch in 2019, that is until recently; when cert-manager suddenly stopped renewing the Let's Encrypt certificates, we "resolved" this by temporarily removing the "tls" section from the ingress definition, forcing our clients to use the http version.
After that we removed all traces of cert-manager attempting to set it up from scratch.
Now, the cert-manager is creating the certificate signing request, spawns an acme http solver pod and adds it to the ingress, however upon accessing its url I can see that it returns an empty response, and not the expected token.
This has to do with the rewrite-target annotation that messes up the routing of the acme challenge.
What puzzles me the most, is that this used to work before. (It was set up by a former employee)
Disabling rewrite-target is unfortunately not an option, because it will stop the routing from working correctly.
Using dns01 won't work because our ISP does not support programmatic changes of the DNS records.
Is there a way to make this work without disabling rewrite-target?
P.S.
Here's a number of similar cases reported on Github:
https://github.com/cert-manager/cert-manager/issues/2826
https://github.com/cert-manager/cert-manager/issues/286
https://github.com/cert-manager/cert-manager/issues/487
None of them help.
Here's the definition of my ClusterIssuer
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
name: letsencrypt-prod
spec:
acme:
# The ACME server URL
server: https://acme-v02.api.letsencrypt.org/directory
# Email address used for ACME registration
email: mail#domain.com
# Name of a secret used to store the ACME account private key
privateKeySecretRef:
name: letsencrypt-prod
# Enable the HTTP-01 challenge provider
solvers:
- http01:
ingress:
class: nginx
Please share the cluster issuer or issue you are using.
ingressClass
If the ingressClass field is specified, cert-manager will create
new Ingress resources in order to route traffic to the
acmesolver pods, which are responsible for responding to ACME
challenge validation requests.
Ref : https://cert-manager.io/v0.12-docs/configuration/acme/http01/#ingressclass
Mostly we don't see the HTTP solver challenge it comes and get removed if DNS or HTTP working fine.
Also, make sure your ingress doesn't have SSL-redirect annotation that could be also once reason behind certs not getting generated.
Did you try checking the other object of cert-manager like order and certificate status request ? kubectl describe challenge are you getting 404 there ?
If you are trying continuously there could be chance you hit rate limit of let's encrypt to request generating certificates.
Troubleshooting : https://cert-manager.io/docs/faq/troubleshooting/#troubleshooting-a-failed-certificate-request
When you configure an Issuer with http01, the default serviceType is NodePort. This means, it won't even go through the ingress controller. From the docs:
By default, type NodePort will be used when you don't set HTTP01 or when you set serviceType to an empty string. Normally there's no need to change this.
I'm not sure how the rest of your setup looks like, but http01 cause the acme server to make HTTP requests (not https). You need to make sure your nginx has listener for http (80). It does follow redirects, so you can listen on http and redirect all traffic to https, this is legit and working.
The cert-manager creates an ingress resource for validation. It directs traffic to the temporary pod. This ingress has it's own set of rules, and you can control it using this setting. You can try and disable or modify the rewrite-targets on this resource.
Another thing I would try is to access this URL from inside the cluster (bypassing the ingress nginx). If it works directly, then it's an ingress / networking problem, otherwise it's something else.
Please share the relevant nginx and cert-manager logs, it might be useful for debugging or understanding where your problem exist.
I am new at kubernetes and I am trying to setup a clsuter on a remote server.
For this I use microk8s and a server from hetzner-cloud (https://www.hetzner.com/de/cloud).
I entered the server with ssh and followd the microk8s installation instructions for linux(https://microk8s.io/). That all seemed to work fine. My problem is now that I have not found a way to access the kubernetes-dashboard.
I've tryed the workaround with NodePort and microk8s kubectl proxy --disable-filter = true but it does not work and is not recommended for security reasons. With the disable Filter Method it is possible to access the login page but it does not respond.
I've also tryed to access the dashbourd from outside using a ssh tunnel and this tutorial: How can I remotely access kubernetes dashboard with token
The tunnel seems to works fine, but I still cannot access the port.
Now I have two main questions:
1: How do you usualy use kubernetes, if kubernetes does not wan you to access the dashboard from outside. Because don't you run your services usualy on a rented server that is not in you living room? What's the point I simply do not get?
2: How can I access the Dashborad?
I would be really happy, if anybody could help me. I am already struggling with this problem since a month now. :)
best greetings,
mamo
In order to access the K8s services from outside using HTTP, you should configure and use ingress controller.
after ingress is running, you will be able to specify a "path" or route and a port and name that points to your service.
Once this is done, you should be able to access the dashboard
Sample configuration (reference)
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: dashboard-google
namespace: kube-system
annotations:
nginx.ingress.kubernetes.io/secure-backends: "true"
nginx.ingress.kubernetes.io/ssl-passthrough: "true"
spec:
tls:
- hosts:
- kube.mydomain.com
secretName: tls-secret
rules:
- host: kube.mydomain.com
http:
paths:
- path: /
backend:
serviceName: kubernetes-dashboard
servicePort: 443
Is there a way to force an SSL upgrade for incoming connections on the ingress load-balancer? Or if that is not possible with, can I disable port :80? I haven't found a good documentation pages that outlines such an option in the YAML file. Thanks a lot in advance!
https://github.com/kubernetes/ingress-gce#frontend-https
You can block HTTP through the annotation kubernetes.io/ingress.allow-http: "false" or redirect HTTP to HTTPS by specifying a custom backend. Unfortunately GCE doesn't handle redirection or rewriting at the L7 layer directly for you, yet. (see https://github.com/kubernetes/ingress-gce#ingress-cannot-redirect-http-to-https)
Update: GCP now handles redirection rules for load balancers, including HTTP to HTTPS. There doesn't appear to be a method to create these through Kubernetes YAML yet.
This was already correctly answered by a comment on the accepted answer. But since the comment is buried I missed it several times.
As of GKE version 1.18.10-gke.600 you can add a k8s frontend config to redirect from http to https.
https://cloud.google.com/kubernetes-engine/docs/how-to/ingress-features#https_redirect
apiVersion: networking.gke.io/v1beta1
kind: FrontendConfig
metadata:
name: ssl-redirect
spec:
redirectToHttps:
enabled: true
# add below to ingress
# metadata:
# annotations:
# networking.gke.io/v1beta1.FrontendConfig: ssl-redirect
The annotation has changed:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: test
annotations:
kubernetes.io/ingress.allow-http: "false"
spec:
...
Here is the annotation change PR:
https://github.com/kubernetes/contrib/pull/1462/files
If you are not bound to the GCLB Ingress Controller you could have a look at the Nginx Ingress Controller. This controller is different to the builtin one in multiple ways. First and foremost you need to deploy and manage one by yourself. But if you are willing to do so, you get the benefit of not depending on the GCE LB (20$/month) and getting support for IPv6/websockets.
The documentation states:
By default the controller redirects (301) to HTTPS if TLS is enabled for that ingress . If you want to disable that behaviour globally, you
can use ssl-redirect: "false" in the NGINX config map.
The recently released 0.9.0-beta.3 comes with an additional annotation for explicitly enforcing this redirect:
Force redirect to SSL using the annotation ingress.kubernetes.io/force-ssl-redirect
Google has responded to our requests and is testing HTTP->HTTPS SSL redirection on their load balancers. Their latest answer said it should be in Alpha sometime before the end of January 2020.
Their comment:
Thank you for your patience on this issue. The feature is currently in testing and we expect to enter Alpha phase before the end of January. Our PM team will have an announcement with more details as we get closer to the Alpha launch.
My fingers are crossed that we'll have a straightforward solution to this very common feature in the near future.
UPDATE (April 2020):
HTTP(S) rewrites is now a Generally Available feature. It's still a bit rough around the edges and does not work out-of-the-box with the GCE Ingress Controller unfortunately. But time will tell and hopefully a native solution will appear.
A quick update. Here
Now a FrontEndConfig can be make to configure the ingress. Hopes it helps.
Example:
apiVersion: networking.gke.io/v1beta1
kind: FrontendConfig
metadata:
name: my-frontend-config
spec:
redirectToHttps:
enabled: true
responseCodeName: 301
You'll need to make sure that your load balancer supports HTTP and HTTPS
Worked on this for a long time. In case anyone isn't clear on the post above. You would rebuild your ingress with annotation -- kubernetes.io/ingress.allow-http: "false” --
Then delete your ingress and redeploy. The annotation will have the ingress only create a LB for 443, instead of both 443 and 80.
Then you do a compute HTTP LB, not one for GKE.
Gui directions:
Create a load balancer and choose HTTP(S) Load Balancing -- Start configuration.
choose - From Internet to my VMs and continue
Choose a name for the LB
leave the backend configuration blank.
Under Host and path rules, select Advanced host and path rules with the action set to
Redirect the client to different host/path.
Leave the Host redirect field blank.
Select Prefix Redirect and leave the Path value blank.
Chose the redirect response code as 308.
Tick the Enable box for HTTPS redirect.
For the Frontend configuration, leave http and port 80, for ip address select the static
IP address being used for your GKE ingress.
Create this LB.
You will now have all http traffic go to this and 308 redirect to your https ingress for GKE. Super simple config setup and works well.
Note: If you just try to delete the port 80 LB that GKE makes (not doing the annotation change and rebuilding the ingress) and then adding the new redirect compute LB it does work, but you will start to see error messages on your Ingress saying error 400 invalid value for field 'resource.ipAddress " " is in use and would result in a conflict, invalid. It is trying to spin up the port 80 LB and can't because you already have an LB on port 80 using the same IP. It does work but the error is annoying and GKE keeps trying to build it (I think).
Thanks to the comment of #Andrej Palicka and according to the page he provided: https://cloud.google.com/kubernetes-engine/docs/how-to/ingress-features#https_redirect now I have an updated and working solution.
First we need to define a FrontendConfig resource and then we need to tell the Ingress resource to use this FrontendConfig.
Example:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: myapp-app-ingress
annotations:
kubernetes.io/ingress.global-static-ip-name: myapp-prd
networking.gke.io/managed-certificates: managed-cert
kubernetes.io/ingress.class: "gce"
networking.gke.io/v1beta1.FrontendConfig: myapp-frontend-config
spec:
defaultBackend:
service:
name: myapp-app-service
port:
number: 80
---
apiVersion: networking.gke.io/v1beta1
kind: FrontendConfig
metadata:
name: myapp-frontend-config
spec:
redirectToHttps:
enabled: true
responseCodeName: MOVED_PERMANENTLY_DEFAULT
You can disable HTTP on your cluster (note that you'll need to recreate your cluster for this change to be applied on the load balancer) and then set HTTP-to-HTTPS redirect by creating an additional load balancer on the same IP address.
I spend couple of hours on the same question, and ended up doing what I've just described. It works perfectly.
Redirecting to HTTPS in Kubernetes is somewhat complicated. In my experience, you'll probably want to use an ingress controller such as Ambassador or ingress-nginx to control routing to your services, as opposed to having your load balancer route directly to your services.
Assuming you're using an ingress controller, then:
If you're terminating TLS at the external load balancer and the LB is running in L7 mode (i.e., HTTP/HTTPS), then your ingress controller needs to use X-Forwarded-Proto, and issue a redirect accordingly.
If you're terminating TLS at the external load balancer and the LB is running in TCP/L4 mode, then your ingress controller needs to use the PROXY protocol to do the redirect.
You can also terminate TLS directly in your ingress controller, in which case it has all the necessary information to do the redirect.
Here's a tutorial on how to do this in Ambassador.
I have seen example for EnvoyFilter in ISTIO where grpc_service is supported as filterconfig for external service call out.
kind: EnvoyFilter
metadata:
name: ext-authz
namespace: istio-system
spec:
filters:
- insertPosition:
index: FIRST
listenerMatch:
listenerType: SIDECAR_INBOUND
listenerProtocol: HTTP
filterType: HTTP
filterName: "envoy.ext_authz"
filterConfig:
grpc_service:
google_grpc:
target_uri: 127.0.0.1:9191
stat_prefix: "ext_authz"
But I want to use my external service as filterconfig using http_service instead of grpc_service but everytime I get 404 not found error.
Is http_service supported as filterConfig in Istio's envoyFilter?
version info : GKE is 14 and istio is 1.1.17
Update: modified entire answer.
After further verification it appears that Istio had http_service authorization service in the past it was not fully functional.
There were attempts to implement external HTTP service authorization for older versions of Istio, however it did work and the only workaround solutions were to use http lua filter or Nginx-Ingress Controller as Ingress Gateway that delegates the authentication part.
All of above cases can be found in this github issue. The HTTP call was successful but the headers were not being passed.
Here is another attempt in running http_service as authorization service.
As You noticed the Envoy documentation for Envoy 1.11.0 http_service filterConfig has different syntax. Therefore I suggest trying the configuration for filter from the github issue. And if It doesnt't work, try the http lua filter as a workaround.
The HTTP service as an external authorization service its not mentioned in Istio documentation so, I think its safe to say its not fully supported.
I have a sample application (web-app, backend-1, backend-2) deployed on minikube all under a JWT policy, and they all have proper destination rules, Istio sidecar and MTLS enabled in order to secure the east-west traffic.
apiVersion: authentication.istio.io/v1alpha1
kind: Policy
metadata:
name: oidc
spec:
targets:
- name: web-app
- name: backend-1
- name: backend-2
peers:
- mtls: {}
origins:
- jwt:
issuer: "http://myurl/auth/realms/test"
jwksUri: "http://myurl/auth/realms/test/protocol/openid-connect/certs"
principalBinding: USE_ORIGIN
When I run the following command I receive a 401 unauthorized response when requesting the data from the backend, which is due to $TOKEN not being forwarded to backend-1 and backend-2 headers during the http request.
$> curl http://minikubeip/api "Authorization: Bearer $TOKEN"
Is there a way to forward http headers to backend-1 and backend-2 using native kubernetes/istio? Am I forced to make application code changes to accomplish this?
Edit:
This is the error I get after applying my oidc policy. When I curl web-app with the auth token I get
{"errors":[{"code":"APP_ERROR_CODE","message":"401 Unauthorized"}
Note that when I curl backend-1 or backend-2 with the same auth-token I get the appropriate data. Also, there is no other destination rule/policy applied to these services currently, policy enforcement is on, and my istio version is 1.1.15.
This is the policy I am applying:
apiVersion: authentication.istio.io/v1alpha1
kind: Policy
metadata:
name: default
namespace: default
spec:
# peers:
# - mtls: {}
origins:
- jwt:
issuer: "http://10.148.199.140:8080/auth/realms/test"
jwksUri: "http://10.148.199.140:8080/auth/realms/test/protocol/openid-connect/certs"
principalBinding: USE_ORIGIN
should the token be propagated to backend-1 and backend-2 without any other changes?
Yes, policy should transfer token to both backend-1 and backend-2
There is a github issue , where users had same issue like You
A few informations from there:
The JWT is verified by an Envoy filter, so you'll have to check the Envoy logs. For the code, see https://github.com/istio/proxy/tree/master/src/envoy/http/jwt_auth
Pilot retrieves the JWKS to be used by the filter (it is inlined into the Envoy config), you can find the code for that in pilot/pkg/security
And another problem with that in stackoverflow
where accepted answer is:
The problem was resolved with two options: 1. Replace Service Name and port by external server ip and external port (for issuer and jwksUri) 2. Disable the usage of mTLS and its policy (Known issue: https://github.com/istio/istio/issues/10062).
From istio documentation
For each service, Istio applies the narrowest matching policy. The order is: service-specific > namespace-wide > mesh-wide. If more than one service-specific policy matches a service, Istio selects one of them at random. Operators must avoid such conflicts when configuring their policies.
To enforce uniqueness for mesh-wide and namespace-wide policies, Istio accepts only one authentication policy per mesh and one authentication policy per namespace. Istio also requires mesh-wide and namespace-wide policies to have the specific name default.
If a service has no matching policies, both transport authentication and origin authentication are disabled.
Istio supports header propagation. Probably didn't support when this thread was created.
You can allow the original header to be forwarded by using forwardOriginalToken: true in JWTRules or forward a valid JWT payload using outputPayloadToHeader in JWTRules.
Reference: ISTIO JWTRule documentation