I would like to add ssl support to my web app (wordpress) deployed on kubernetes. for that i deployed cert-manager using helm like following:
helm upgrade \
cert-manager \
--namespace cert-manager \
--version v1.9.1 \
--set installCRDs=true \
--set ingressShim.defaultIssuerName=letsencrypt-prod \
--set ingressShim.defaultIssuerKind=ClusterIssuer \
--set ingressShim.defaultIssuerGroup=cert-manager.io \
--create-namespace \
jetstack/cert-manager --install
Then i deployed wordpress using helm as well, while values.yml look like :
#Change default svc type
service:
type: ClusterIP
#ingress resource
ingress:
enabled: true
hostname: app.benighil-mohamed.com
path: /
annotations:
#kubernetes.io/ingress.class: azure/application-gateway
kubernetes.io/ingress.class: "nginx"
cert-manager.io/cluster-issuer: letsencrypt-prod
extraTls:
- hosts:
- "{{ .Values.ingress.hostname }}" # ie: app.benighil-mohamed.com
secretName: "{{ .Release.Name }}-{{ .Values.ingress.hostname }}" #ie: wp-app.benighil-mohamed.com
However, when i check certifiactes and certificaterequests i got the following:
vscode ➜ /workspaces/flux/ingress $ kubectl get certificate -n app -owide
NAME READY SECRET ISSUER STATUS AGE
wp-benighil.benighil-mohamed.com False wp-benighil.benighil-mohamed.com letsencrypt-prod Issuing certificate as Secret does not exist 25m
vscode ➜ /workspaces/flux/ingress
vscode ➜ /workspaces/flux/ingress $ kubectl get certificaterequests -n app -owide
NAME APPROVED DENIED READY ISSUER REQUESTOR STATUS AGE
wp-benighil.benighil-mohamed.com-45d6s True False letsencrypt-prod system:serviceaccount:cert-manager:cert-manager Referenced "ClusterIssuer" not found: clusterissuer.cert-manager.io "letsencrypt-prod" not found 27m
vscode ➜ /workspaces/flux/ingress
Any idea please ?
Related
I'm trying to set up a K3s cluster. When I had a single master and agent setup cert-manager had no issues. Now I'm trying a 2 master setup with embedded etcd. I opened TCP ports 6443 and 2379-2380 for both VMs and did the following:
VM1: curl -sfL https://get.k3s.io | sh -s server --token TOKEN --cluster-init
VM2: curl -sfL https://get.k3s.io | sh -s server --token TOKEN --server https://MASTER_IP:6443
# k3s kubectl get nodes
NAME STATUS ROLES AGE VERSION
VM1 Ready control-plane,etcd,master 130m v1.22.7+k3s1
VM2 Ready control-plane,etcd,master 128m v1.22.7+k3s1
Installing cert-manager works fine:
# k3s kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.8.0/cert-manager.yaml
# k3s kubectl get pods --namespace cert-manager
NAME READY STATUS
cert-manager-b4d6fd99b-c6fpc 1/1 Running
cert-manager-cainjector-74bfccdfdf-gtmrd 1/1 Running
cert-manager-webhook-65b766b5f8-brb76 1/1 Running
My manifest has the following definition:
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
name: letsencrypt
spec:
acme:
server: https://acme-v02.api.letsencrypt.org/directory
email: info#example.org
privateKeySecretRef:
name: letsencrypt-account-key
solvers:
- selector: {}
http01:
ingress: {}
Which results in the following error:
# k3s kubectl apply -f manifest.yaml
Error from server (InternalError): error when creating "manifest.yaml": Internal error occurred: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/mutate?timeout=10s": context deadline exceeded
I tried disabling both firewalls, waiting a day, reset and re-setup, but the error persists. Google hasn't been much help either. The little info I can find goes over my head for the most part and no tutorial seems to do any extra steps.
Try to specify the proper ingress class name in your Cluster Issuer, like this:
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
name: letsencrypt
spec:
acme:
server: https://acme-v02.api.letsencrypt.org/directory
email: info#example.org
privateKeySecretRef:
name: letsencrypt-account-key
solvers:
- http01:
ingress:
class: nginx
Also, make sure that you have the cert manager annotation and the tls secret name specified in your Ingress like this:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
cert-manager.io/cluster-issuer: letsencrypt
...
spec:
tls:
- hosts:
- domain.com
secretName: letsencrypt-account-key
A good starting point for troubleshooting issues with the webhook can be found int the docs, e.g. there is a section for problems on GKE private clusters.
In my case, however, this didn't really solve the problem. For me the issue was that when I played around with cert-manager I happen to install and uninstall it multiple times. It turned out that just removing the namespace, e.g. kubectl delete namespace cert-manager didn't remove the webhooks and other non-obvious resources.
Following the official guide for uninstalling cert-manager and applying the manifests again solved the issue.
I do this it, and work for me.
helm install
cert-manager jetstack/cert-manager
--namespace cert-manager
--create-namespace
--version v1.8.0
--set webhook.securePort=10260
source: https://hackmd.io/#maelvls/debug-cert-manager-webhook
I'm getting this issue below. Anyone has an idea what could be wrong?
user#master-1:~$ kubectl logs -n velero velero-77b544f457-dw4hf
# REMOVED
An error occurred: some backup storage locations are invalid: backup store for location "aws" is invalid: rpc error: code = Unknown desc = AuthorizationHeaderMalformed: The authorization header is malformed; the region 'us-east-1' is wrong; expecting 'us-west-2'
status code: 400, request id: A3Q97JKM6GQRNABA, host id: b6g0on189w6hYgCrId/Xr0BP44pXjZPy2SqK2t7bn/+Ggq9FUY2N3KQHYRcMEuCCHY2L2vfsYEo=; backup store for location "velero" is invalid: rpc error: code = Unknown desc = AuthorizationHeaderMalformed: The authorization header is malformed; the region 'us-east-1' is wrong; expecting 'us-west-2'
status code: 400, request id: YF6DRKN7MYSXVBV4, host id: Y8/Gufd7R0BlZCIZqbJPfdAjVqK8+WLfWoANBDnipDkH421/vGt0Ne2E/yZw2bYf7rfms+rGxsg=
user#master-1:~$
I have installed Velero 1.4.2 with Helm chart:
user#master-1:~$ helm search repo velero --versions | grep -e 2.12.17 -e NAME
NAME CHART VERSION APP VERSION DESCRIPTION
vmware-tanzu/velero 2.12.17 1.4.2 A Helm chart for velero
user#master-1:~$
I used this command to install:
helm install velero vmware-tanzu/velero --namespace velero --version 2.12.17 -f velero-values.yaml \
--set-file credentials.secretContents.cloud=/home/era/creds-root.txt \
--set configuration.provider=aws \
--set configuration.backupStorageLocation.name=velero \
--set configuration.backupStorageLocation.bucket="velero" \
--set configuration.backupStorageLocation.prefix="" \
--set configuration.backupStorageLocation.config.region="us-east-1" \
--set image.repository=velero/velero \
--set image.tag=v1.4.2 \
--set image.pullPolicy=IfNotPresent \
--set initContainers[0].name=velero-plugin-for-aws \
--set initContainers[0].image=velero/velero-plugin-for-aws:v1.1.0 \
--set initContainers[0].volumeMounts[0].mountPath=/target \
--set initContainers[0].volumeMounts[0].name=plugins \
--replace
My credential files passed:
$ cat creds-root.txt
[default]
aws_access_key_id=12345678
aws_secret_access_key=12345678
Helm values file:
user#master-1:~$ cat velero-values.yaml
configuration:
provider: aws
backupStorageLocation:
name: minio
provider: aws
# caCert: null
bucket: velero
config:
region: us-east-1
credentials:
useSecret: true
existingSecret: cloud-credentials
secretContents: {}
extraEnvVars: {}
backupsEnabled: true
snapshotsEnabled: true
deployRestic: true
MinIO snapshot resource (MinIO is working at 192.168.2.239:9000):
# For MinIO
---
apiVersion: velero.io/v1
kind: VolumeSnapshotLocation
metadata:
name: minio
namespace: velero
spec:
provider: openebs.io/cstor-blockstore
config:
bucket: velero
prefix: cstor
provider: aws
# The region where the server is located.
region: us-east-1
# profile for credential, if not mentioned then plugin will use profile=default
profile: user1
# Whether to use path-style addressing instead of virtual hosted bucket addressing.
# Set to "true"
s3ForcePathStyle: "true"
# S3 URL, By default it will be generated from "region" and "bucket"
s3Url: http://192.168.2.239:9000
# You can specify the multipart_chunksize here for explicitness.
# multiPartChunkSize can be from 5Mi(5*1024*1024 Bytes) to 5Gi
# For more information: https://docs.min.io/docs/minio-server-limits-per-tenant.html
# If not set then it will be calculated from the file size
multiPartChunkSize: 64Mi
# If MinIO is configured with custom certificate then certificate can be passed to plugin through caCert
# Value of caCert must be base64 encoded
# To encode, execute command: cat ca.crt |base64 -w 0
# caCert: LS0tLS1CRU...tRU5EIENFUlRJRklDQVRFLS0tLS0K
# If you want to disable certificate verification then set insecureSkipTLSVerify to "true"
# By default insecureSkipTLSVerify is set to "false"
insecureSkipTLSVerify: "true"
aws resource which seems failing:
$ k get backupstoragelocation -n velero aws -o yaml
apiVersion: velero.io/v1
kind: BackupStorageLocation
metadata:
annotations:
helm.sh/hook: post-install,post-upgrade
helm.sh/hook-delete-policy: before-hook-creation
creationTimestamp: "2021-04-15T08:23:38Z"
generation: 3
labels:
app.kubernetes.io/instance: velero
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/name: velero
helm.sh/chart: velero-2.12.17
managedFields:
- apiVersion: velero.io/v1
fieldsType: FieldsV1
fieldsV1:
f:metadata:
f:annotations:
.: {}
f:helm.sh/hook: {}
f:helm.sh/hook-delete-policy: {}
f:labels:
.: {}
f:app.kubernetes.io/instance: {}
f:app.kubernetes.io/managed-by: {}
f:app.kubernetes.io/name: {}
f:helm.sh/chart: {}
f:spec:
.: {}
f:config:
.: {}
f:region: {}
f:objectStorage:
.: {}
f:prefix: {}
f:provider: {}
manager: Go-http-client
operation: Update
time: "2021-04-15T08:23:38Z"
- apiVersion: velero.io/v1
fieldsType: FieldsV1
fieldsV1:
f:spec:
f:objectStorage:
f:bucket: {}
manager: kubectl-edit
operation: Update
time: "2021-04-15T17:52:46Z"
name: aws
namespace: velero
resourceVersion: "1333724"
selfLink: /apis/velero.io/v1/namespaces/velero/backupstoragelocations/aws
uid: a51033b2-e53d-4751-9110-c9649de6aa67
spec:
config:
region: us-east-1
objectStorage:
bucket: velero
prefix: backup
provider: aws
user#master-1:~$
For some reason no plugins are listed:
user#master-1:~$ velero plugin get
user#master-1:~$
Velero is obviously crashing because of original issue:
user#master-1:~$ kubectl get pods -n velero
NAME READY STATUS RESTARTS AGE
restic-nqpsl 1/1 Running 0 7m52s
restic-pw897 1/1 Running 0 7m52s
restic-rtwzd 1/1 Running 0 7m52s
velero-77b544f457-dw4hf 0/1 CrashLoopBackOff 5 5m59s
user#master-1:~$
More resources:
user#master-1:~$ k get BackupStorageLocation -n velero
NAME PHASE LAST VALIDATED AGE
aws 10h
velero 11m
user#master-1:~$ k get volumesnapshotlocation -n velero
NAME AGE
default 11m
minio 39h
velero-snapshot 9h
user#master-1:~$
My MinIO service is started using Docker Compose and working fine:
version: '3.8'
services:
minio:
container_name: minio
hostname: minio
build:
context: .
dockerfile: Dockerfile
restart: always
ports:
- "0.0.0.0:9000:9000"
environment:
# ROOT
MINIO_ACCESS_KEY: 12345678
MINIO_SECRET_KEY: 12345678
MINIO_REGION: us-east-1
command: server --address :9000 /data
volumes:
- ./data:/data
Unknown PHASE for backup locations:
user#master-1:~$ velero get backup-locations
NAME PROVIDER BUCKET/PREFIX PHASE LAST VALIDATED ACCESS MODE
aws aws velero/backup Unknown Unknown ReadWrite
velero aws velero Unknown Unknown ReadWrite
user#master-1:~$
Test MinIO access separately:
bash-4.3# AWS_ACCESS_KEY_ID=12345678 AWS_SECRET_ACCESS_KEY=12345678 aws s3api get-bucket-location --endpoint-url http://192.168.2.239:9000 --bucket velero
{
"LocationConstraint": "us-east-1"
}
bash-4.3#
Secrets are correct:
user#master-1:~$ k get secret -n velero cloud-credentials -o yaml | head -n 4
apiVersion: v1
data:
cloud: W2RlZmF-REMOVED
kind: Secret
user#master-1:~$
user#master-1:~$ k get secret -n velero
NAME TYPE DATA AGE
cloud-credentials Opaque 1 91m
default-token-8rwhg kubernetes.io/service-account-token 3 2d20h
sh.helm.release.v1.velero.v1 helm.sh/release.v1 1 45m
velero Opaque 0 2d19h
velero-restic-credentials Opaque 1 40h
velero-server-token-8zm9k kubernetes.io/service-account-token 3 45m
user#master-1:~$
The problem was missing configuration:
--set configuration.backupStorageLocation.config.s3Url="http://192.168.2.239:9000" \
--set configuration.backupStorageLocation.config.s3ForcePathStyle=true \
There were a namespace "sandbox" on the node which was deleted, but there is still a challenge for a certificate "echo-tls".
But i can not access anymore sandbox namespace to delete this cert.
Could anyone help me deleting this resource ?
Here are the logs of the cert-manager :
Found status change for Certificate "echo-tls" condition "Ready": "True" -> "False"; setting lastTransitionTime to...
cert-manager/controller/CertificateReadiness "msg"="re-queuing item due to error processing" "error"="Operation cannot be fulfilled on certificates.cert-manager.io \"echo-tls\": StorageError: invalid object, Code: 4, Key: /cert-manager.io/certificates/sandbox/echo-tls, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: ..., UID in object meta: " "key"="sandbox/echo-tls"
After restarting the pod cert-manager here are the logs :
cert-manager/controller/certificaterequests/handleOwnedResource "msg"="error getting referenced owning resource" "error"="certificaterequest.cert-manager.io \"echo-tls-bkmm8\" not found" "related_resource_kind"="CertificateRequest" "related_resource_name"="echo-tls-bkmm8" "related_resource_namespace"="sandbox" "resource_kind"="Order" "resource_name"="echo-tls-bkmm8-1177139468" "resource_namespace"="sandbox" "resource_version"="v1"
cert-manager/controller/orders "msg"="re-queuing item due to error processing" "error"="ACME client for issuer not initialised/available" "key"="sandbox/echo-tls-dwpt4-1177139468"
And then the same logs as before
The issuer :
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
name: letsencrypt-prod
spec:
acme:
email: ***
server: https://acme-v02.api.letsencrypt.org/directory
privateKeySecretRef:
name: letsencrypt-prod
solvers:
- http01:
ingress: {}
The configs for deployment :
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: <APP_NAME>
annotations:
kubernetes.io/tls-acme: "true"
kubernetes.io/ingress.class: nginx-<ENV>
acme.cert-manager.io/http01-ingress-class: nginx-<ENV>
cert-manager.io/cluster-issuer: "letsencrypt-prod"
spec:
tls:
- hosts:
- ***.fr
secretName: <APP_NAME>-tls
rules:
- host: ***.fr
http:
paths:
- backend:
serviceName: <APP_NAME>
servicePort: 80
.k8s_config: &k8s_config
before_script:
- export HOME=/tmp
- export K8S_NAMESPACE="${APP_NAME}"
- kubectl config set-cluster k8s --server="${K8S_SERVER}"
- kubectl config set clusters.k8s.certificate-authority-data ${K8S_CA_DATA}
- kubectl config set-credentials default --token="${K8S_USER_TOKEN}"
- kubectl config set-context default --cluster=k8s --user=default --namespace=default
- kubectl config set-context ${K8S_NAMESPACE} --cluster=k8s --user=default --namespace=${K8S_NAMESPACE}
- kubectl config use-context default
- if [ -z `kubectl get namespace ${K8S_NAMESPACE} --no-headers --output=go-template={{.metadata.name}} 2>/dev/null` ]; then kubectl create namespace ${K8S_NAMESPACE}; fi
- if [ -z `kubectl --namespace=${K8S_NAMESPACE} get secret *** --no-headers --output=go-template={{.metadata.name}} 2>/dev/null` ]; then kubectl get secret *** --output yaml | sed "s/namespace:\ default/namespace:\ ${K8S_NAMESPACE}/" | kubectl create -f - ; fi
- kubectl config use-context ${K8S_NAMESPACE}
Usually certificates are stored inside Kubernete secrets: https://kubernetes.io/docs/concepts/configuration/secret/#tls-secrets. You can retrieve secrets using kubectl get secrets --all-namespaces. You can also check which secrets are used by a given pod by checking its yaml description: kubectl get pods -n <pod-namespace> -o yaml (additional informations: https://kubernetes.io/docs/concepts/configuration/secret/#using-secrets-as-files-from-a-pod)
A namespace is cluster-wide, it is not located on any node. So deleting a node does not delete any namespace.
If above tracks does not fit your need, could you please provide some yaml files and some command-line instructions which would allow reproducing the problem?
Finally this sunday the cert-manger has stop challenges on the old tls without any other action.
I've been trying for the last 3 days to setup cert-manager on a K8S cluster (v1.19.8) in an OpenStack environment with 1 master and 2 nodes.
It worked before (like 1 month ago), but since I re-created the cluster, pod ACME challenges cannot be created due to this error:
Status:
Presented: false
Processing: true
Reason: pods "cm-acme-http-solver-" is forbidden: PodSecurityPolicy: unable to admit pod: []
State: pending
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Started 8m25s cert-manager Challenge scheduled for processing
Warning PresentError 3m18s (x7 over 8m23s) cert-manager Error presenting challenge: pods "cm-acme-http-solver-" is forbidden: PodSecurityPolicy: unable to admit pod: []
I've tried different versions of the ingress-nginx, different versions of cert-manager, different versions of k8s, but to no avail. I'm getting crazy..., please help. Many thanks :)
Cluster setup
kubectl create namespace ingress-nginx && \
helm install ingress-nginx ingress-nginx/ingress-nginx -n ingress-nginx && \
kubectl create namespace cert-manager && \
helm install cert-manager jetstack/cert-manager \
--namespace cert-manager \
--version v1.1.0 \
--set installCRDs=true
Issuer
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
name: letsencrypt-prod
spec:
acme:
server: https://acme-v02.api.letsencrypt.org/directory
email: email#example.com
preferredChain: "ISRG Root X1"
privateKeySecretRef:
name: letsencrypt-prod
solvers:
- http01:
ingress:
class: nginx
Ingress
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: main-ingress
annotations:
kubernetes.io/ingress.class: "nginx"
nginx.ingress.kubernetes.io/from-to-www-redirect: "true"
cert-manager.io/issuer: "letsencrypt-prod"
spec:
tls:
- hosts:
- host.com
secretName: the-secret-name
rules:
- host: host.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: api-nginx
port:
number: 80
After some debugging and much help from the hosting provider, we found the problem and the solution.
We were using the latest (from master) version of Magnum/OpenStack, which got an update that installed by default a PodSecurityPolicy controller. That prevented ACME pods to be created by cert-manager.
Recreating the cluster without a policy controller solved the issue:
openstack coe cluster create \
--cluster-template v1.kube1.20.4 \
--labels \
admission_control_list="NodeRestriction,NamespaceLifecycle,LimitRanger,ServiceAccount,ResourceQuota,TaintNodesByCondition,Priority,DefaultTolerationSeconds,DefaultStorageClass,StorageObjectInUseProtection,PersistentVolumeClaimResize,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,RuntimeClass" \
--merge-labels
...
A year late, but adding another solution in case it helps others finding this. I had the same issue of the challenge pod being blocked by PSP, but really didn't want to have to recreate/reconfigure my cluster, so I eventually solved the issue by adding this to the helm chart values.yaml:
https://github.com/cert-manager/cert-manager/blob/master/deploy/charts/cert-manager/values.yaml
global:
podSecurityPolicy:
enabled: true
useAppArmor: false
In my case, this is part of a Gitlab deployment so I added it under the certmanager key, as follows:
certmanager:
install: true
global:
podSecurityPolicy:
enabled: true
useAppArmor: false
(tags for search: gitlab helm chart certmanager PodSecurityPolicy "unable to admit pod" blocked)
I'm following the official Helm documentation for "Deploy Tiller in a namespace, restricted to deploying resources only in that namespace". Here is my bash script:
Namespace="$1"
kubectl create namespace $Namespace
kubectl create serviceaccount "tiller-$Namespace" --namespace $Namespace
kubectl create role "tiller-role-$Namespace" /
--namespace $Namespace /
--verb=* /
--resource=*.,*.apps,*.batch,*.extensions
kubectl create rolebinding "tiller-rolebinding-$Namespace" /
--namespace $Namespace /
--role="tiller-role-$Namespace" /
--serviceaccount="$Namespace:tiller-$Namespace"
helm init /
--service-account "tiller-$Namespace" /
--tiller-namespace $Namespace
--override "spec.template.spec.containers[0].command'='{/tiller,--storage=secret}"
--upgrade
--wait
Running helm upgrade gives me the following error:
Error: UPGRADE FAILED: configmaps is forbidden: User "system:serviceaccount:kube-system:default" cannot list configmaps in the namespace "kube-system"
Is there a bug in the official documentation? Have I read it wrong?
I'm not sure about --resource flag correct syntax in your script, whether asterisk symbols "*" are allowed here, look at this issue reported on GitHub.
$ kubectl create role "tiller-role-$Namespace" \
--namespace $Namespace \
--verb=* \
--resource=*.,*.apps,*.batch,*.extensions
the server doesn't have a resource type "*"
But you can check this role object in your cluster:
kubectl get role tiller-role-$Namespace -n $Namespace -o yaml
Otherwise, try to create the role for tiller within yaml file as guided in the documentation:
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: tiller-manager
namespace: tiller-world
rules:
- apiGroups: ["", "batch", "extensions", "apps"]
resources: ["*"]
verbs: ["*"]
Moreover, keep in mind that if you have installed tiller in the non-default namespace (default), it is necessary to specify namespace where tiller resides on, when you invoke Helm command:
$ helm --tiller-namespace $Namespace version
Client: &version.Version{SemVer:"v2.11.0", GitCommit:"2e55dbe1fdb5fdb96b75ff144a339489417b146b", GitTreeState:"clean"}
Server: &version.Version{SemVer:"v2.11.0", GitCommit:"2e55dbe1fdb5fdb96b75ff144a339489417b146b", GitTreeState:"clean"}