I'm using Keycloak (latest) for Auth 2.0, to validate authentication, provide a token (JWT) and with the token provided, allows the access to the application URLs, based in the permissions.
Keycloak is currently running in Kubernates, with Istio as Gateway. For Keycloak, this is the policy being used:
apiVersion: authentication.istio.io/v1alpha1
kind: Policy
metadata:
name: application-auth-policy
spec:
targets:
- name: notification
origins:
- jwt:
issuer: http://<service_name>http.<namespace>.svc.cluster.local:8080/auth/realms/istio
jwksUri: http://<service_name>http.<namespace>.svc.cluster.local:8080/auth/realms/istio/protocol/openid-connect/certs
principalBinding: USE_ORIGIN
An client was registered in this Keycloak and a RSA created for it.
The issuer can generates a token normally and the policy was applied successfully.
Problem:
Even with everything set, the token provided by Keycloak has the signature invalid according to JWT Validator.
This token doesn't allow any access for the URLs, as it should be, with 401 code.
Anyone else had a similar issue?
The problem was resolved with two options:
1. Replace Service Name and port by external server ip and external port (for issuer and jwksUri)
2. Disable the usage of mTLS and its policy (Known issue: https://github.com/istio/istio/issues/10062).
Related
We have a dozen of services exposed using a ingress-nginx controller in GKE.
In order to route the traffic correctly on the same domain name, we need to use a rewrite-target rule.
The services worked well without any maintenance since their launch in 2019, that is until recently; when cert-manager suddenly stopped renewing the Let's Encrypt certificates, we "resolved" this by temporarily removing the "tls" section from the ingress definition, forcing our clients to use the http version.
After that we removed all traces of cert-manager attempting to set it up from scratch.
Now, the cert-manager is creating the certificate signing request, spawns an acme http solver pod and adds it to the ingress, however upon accessing its url I can see that it returns an empty response, and not the expected token.
This has to do with the rewrite-target annotation that messes up the routing of the acme challenge.
What puzzles me the most, is that this used to work before. (It was set up by a former employee)
Disabling rewrite-target is unfortunately not an option, because it will stop the routing from working correctly.
Using dns01 won't work because our ISP does not support programmatic changes of the DNS records.
Is there a way to make this work without disabling rewrite-target?
P.S.
Here's a number of similar cases reported on Github:
https://github.com/cert-manager/cert-manager/issues/2826
https://github.com/cert-manager/cert-manager/issues/286
https://github.com/cert-manager/cert-manager/issues/487
None of them help.
Here's the definition of my ClusterIssuer
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
name: letsencrypt-prod
spec:
acme:
# The ACME server URL
server: https://acme-v02.api.letsencrypt.org/directory
# Email address used for ACME registration
email: mail#domain.com
# Name of a secret used to store the ACME account private key
privateKeySecretRef:
name: letsencrypt-prod
# Enable the HTTP-01 challenge provider
solvers:
- http01:
ingress:
class: nginx
Please share the cluster issuer or issue you are using.
ingressClass
If the ingressClass field is specified, cert-manager will create
new Ingress resources in order to route traffic to the
acmesolver pods, which are responsible for responding to ACME
challenge validation requests.
Ref : https://cert-manager.io/v0.12-docs/configuration/acme/http01/#ingressclass
Mostly we don't see the HTTP solver challenge it comes and get removed if DNS or HTTP working fine.
Also, make sure your ingress doesn't have SSL-redirect annotation that could be also once reason behind certs not getting generated.
Did you try checking the other object of cert-manager like order and certificate status request ? kubectl describe challenge are you getting 404 there ?
If you are trying continuously there could be chance you hit rate limit of let's encrypt to request generating certificates.
Troubleshooting : https://cert-manager.io/docs/faq/troubleshooting/#troubleshooting-a-failed-certificate-request
When you configure an Issuer with http01, the default serviceType is NodePort. This means, it won't even go through the ingress controller. From the docs:
By default, type NodePort will be used when you don't set HTTP01 or when you set serviceType to an empty string. Normally there's no need to change this.
I'm not sure how the rest of your setup looks like, but http01 cause the acme server to make HTTP requests (not https). You need to make sure your nginx has listener for http (80). It does follow redirects, so you can listen on http and redirect all traffic to https, this is legit and working.
The cert-manager creates an ingress resource for validation. It directs traffic to the temporary pod. This ingress has it's own set of rules, and you can control it using this setting. You can try and disable or modify the rewrite-targets on this resource.
Another thing I would try is to access this URL from inside the cluster (bypassing the ingress nginx). If it works directly, then it's an ingress / networking problem, otherwise it's something else.
Please share the relevant nginx and cert-manager logs, it might be useful for debugging or understanding where your problem exist.
I run a kubernetes cluster with cert-manager installed for managing ACME (Let's Encrypt) certificates. I'm using DNS domain validation with Route 53 and it works all fine.
The problem comes when I try to issue a certificate for a cluster internal domain. In this case domain validation does not pass since the validation challenge is presented on external Route53 zone only, while cert-manager is trying to look for domain name via cluster internal DNS.
Any hints on how this can be solved are welcome.
Assuming that you don't control public DNS for your cluster internal domain, you will not be able to receive LetsEncrypt certificates for it.
You may however set up another issuer that will grant you certificates for this domain, e.g. the SelfSigned issuer: https://cert-manager.io/docs/configuration/selfsigned/
Then set the issuerRef of your certificate object to point to your SelfSigned issuer:
(...)
issuerRef:
name: selfsigned-issuer
kind: ClusterIssuer
group: cert-manager.io
I am implementing a microservices architecture, I have an oauth2 service that does the authentication and generates a token for me.
I am using Treafik as an API-Gateway
How do I validate that a token has expired and that the token signature is valid?
I am testing with those tags in my docker-compose but it seems to ignore it
labels:
- "traefik.http.middlewares.test-jwtAuth.plugin.jwtAuth.source=jwtSource"
- "traefik.http.middlewares.test-jwtAuth.plugin.jwtAuth.forwardHeaders.Group=grp"
- "traefik.http.middlewares.test-jwtAuth.plugin.jwtAuth.forwardHeaders.Expires-At=exp"
- "traefik.http.middlewares.test-jwtAuth.plugin.jwtAuth.claims: Equals(`grp`, `admin`)"
I am running a Kubernetes Cluster with an Nginx-ingress fronting couple of web apps. Because Nginx doesn't support SSO/OIDC by default, I use an oauth_proxy for authentication.
Everything is working, only verified users are able to access the web pages.
Is it possible to pass or request information from the Identity Provider to the client?
Edit
I already use oauth2_proxy (https://github.com/pusher/oauth2_proxy) with Azure AD. The issue is that I need all user details from the IP.
Logs from my oauth2_proxy:
$ kubectl logs oauth2-proxy-7ddc97f9d5-ckm29
[oauthproxy.go:846] Error loading cookied session: Cookie "_oauth2_proxy" not present
...
[requests.go:25] 200 GET https://graph.windows.net/me?api-version=1.6 {
"odata.metadata":"https://graph.windows.net/myorganization/$metadata#directoryObjects/#Element",
"odata.type":"Microsoft.DirectoryServices.User",
"objectType":"User",
... ,
"sipProxyAddress":"nico.schuck#example.com",
"streetAddress":"my stree",
"surname":"Schuck",
"telephoneNumber":55512345,
"usageLocation":"DE",
"userType":"Member"
}
165.xxx.xxx.214 - nico.schuck#example.com [2020/01/17 11:22:02] [AuthSuccess] Authenticated via OAuth2: Session{email:nico.schuck#example.com user: token:true id_token:true created:2020-01-17 11:22:02.28839851 +0000 UTC m=+181.592452463 expires:2020-01-17 12:22:02 +0000 UTC refresh_token:true}
Consider oauth2_proxy which works well with nginx ingress for SSO. Follow the below link
https://github.com/bitly/oauth2_proxy
You should be using below configuration in your Ingress Rule
metadata:
name: application
annotations:
nginx.ingress.kubernetes.io/auth-url: "https://$host/oauth2/auth"
nginx.ingress.kubernetes.io/auth-signin: "https://$host/oauth2/start?rd=$escaped_request_uri"
I have a sample application (web-app, backend-1, backend-2) deployed on minikube all under a JWT policy, and they all have proper destination rules, Istio sidecar and MTLS enabled in order to secure the east-west traffic.
apiVersion: authentication.istio.io/v1alpha1
kind: Policy
metadata:
name: oidc
spec:
targets:
- name: web-app
- name: backend-1
- name: backend-2
peers:
- mtls: {}
origins:
- jwt:
issuer: "http://myurl/auth/realms/test"
jwksUri: "http://myurl/auth/realms/test/protocol/openid-connect/certs"
principalBinding: USE_ORIGIN
When I run the following command I receive a 401 unauthorized response when requesting the data from the backend, which is due to $TOKEN not being forwarded to backend-1 and backend-2 headers during the http request.
$> curl http://minikubeip/api "Authorization: Bearer $TOKEN"
Is there a way to forward http headers to backend-1 and backend-2 using native kubernetes/istio? Am I forced to make application code changes to accomplish this?
Edit:
This is the error I get after applying my oidc policy. When I curl web-app with the auth token I get
{"errors":[{"code":"APP_ERROR_CODE","message":"401 Unauthorized"}
Note that when I curl backend-1 or backend-2 with the same auth-token I get the appropriate data. Also, there is no other destination rule/policy applied to these services currently, policy enforcement is on, and my istio version is 1.1.15.
This is the policy I am applying:
apiVersion: authentication.istio.io/v1alpha1
kind: Policy
metadata:
name: default
namespace: default
spec:
# peers:
# - mtls: {}
origins:
- jwt:
issuer: "http://10.148.199.140:8080/auth/realms/test"
jwksUri: "http://10.148.199.140:8080/auth/realms/test/protocol/openid-connect/certs"
principalBinding: USE_ORIGIN
should the token be propagated to backend-1 and backend-2 without any other changes?
Yes, policy should transfer token to both backend-1 and backend-2
There is a github issue , where users had same issue like You
A few informations from there:
The JWT is verified by an Envoy filter, so you'll have to check the Envoy logs. For the code, see https://github.com/istio/proxy/tree/master/src/envoy/http/jwt_auth
Pilot retrieves the JWKS to be used by the filter (it is inlined into the Envoy config), you can find the code for that in pilot/pkg/security
And another problem with that in stackoverflow
where accepted answer is:
The problem was resolved with two options: 1. Replace Service Name and port by external server ip and external port (for issuer and jwksUri) 2. Disable the usage of mTLS and its policy (Known issue: https://github.com/istio/istio/issues/10062).
From istio documentation
For each service, Istio applies the narrowest matching policy. The order is: service-specific > namespace-wide > mesh-wide. If more than one service-specific policy matches a service, Istio selects one of them at random. Operators must avoid such conflicts when configuring their policies.
To enforce uniqueness for mesh-wide and namespace-wide policies, Istio accepts only one authentication policy per mesh and one authentication policy per namespace. Istio also requires mesh-wide and namespace-wide policies to have the specific name default.
If a service has no matching policies, both transport authentication and origin authentication are disabled.
Istio supports header propagation. Probably didn't support when this thread was created.
You can allow the original header to be forwarded by using forwardOriginalToken: true in JWTRules or forward a valid JWT payload using outputPayloadToHeader in JWTRules.
Reference: ISTIO JWTRule documentation