I'm trying to build an admission controller to enforce pod annotations on our cluster.
I was able to build a webhook service and deploy it. For testing purposes, I'm changing the server code to respond with Allowed: false as a default response to any request but it doesn't stop pods from getting created.
In the logs, I'm seeing the request hit the server but it seems as though the kubeapi-server is not receiving or not adhering to the response.
2021/01/25 02:29:15 &AdmissionResponse{UID:,Allowed:false,Result:&v1.Status{ListMeta:ListMeta{SelfLink:,ResourceVersion:,Continue:,RemainingItemCount:nil,},Status:,Message:,Reason:,Details:nil,Code:0,},Patch:nil,PatchType:nil,AuditAnnotations:map[string]string{},Warnings:[],}
Below are the service deployment file and the validating webhook configuration.
Appreciate any ideas/recommendations!
apiVersion: apps/v1
kind: Deployment
metadata:
name: webhook-server
namespace: webhook-demo
labels:
app: webhook-server
spec:
replicas: 1
selector:
matchLabels:
app: webhook-server
template:
metadata:
labels:
app: webhook-server
spec:
securityContext:
runAsNonRoot: true
runAsUser: 1234
containers:
- name: webhook-server
image: demo/admission-controller-webhook-demo:latest
imagePullPolicy: Never
ports:
- containerPort: 8443
name: webhook-api
resources:
requests:
cpu: "100m"
memory: "128M"
limits:
cpu: "250m"
memory: "256M"
volumeMounts:
- name: webhook-tls-certs
mountPath: /run/secrets/tls
readOnly: true
volumes:
- name: webhook-tls-certs
secret:
secretName: webhook-server-tls
---
apiVersion: v1
kind: Service
metadata:
name: webhook-server
namespace: webhook-demo
spec:
selector:
app: webhook-server
ports:
- port: 443
targetPort: webhook-api
---
apiVersion: admissionregistration.k8s.io/v1beta1
kind: ValidatingWebhookConfiguration
metadata:
name: webhook-server
webhooks:
- name: webhook-server.webhook-demo.svc
rules:
- apiGroups: ["*"]
apiVersions: ["*"]
operations: ["CREATE","UPDATE"]
resources: ["pods","deployments", "replicasets"]
timeoutSeconds: 5
clientConfig:
service:
name: webhook-server
namespace: webhook-demo
path: "/validate"
caBundle: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0t<truncated>
sideEffects: None
admissionReviewVersions: ["v1beta1"]
I was sending the Admission Response object in the response but the API server actually needs the Admission Review object which encapsulates the admission response and request objects. :facepalm
Related
Steps i have done :
I have two namespaces one with istio injected and another not
Now deploy simple nginx server using this yaml in both namespace
apiVersion: v1
kind: Service
metadata:
name: software-upgrader
labels:
app: software-upgrader
service: software-upgrader
spec:
ports:
- name: http
port: 25301
selector:
app: software-upgrader
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: software-upgrader
spec:
selector:
matchLabels:
app: software-upgrader
version: v1
template:
metadata:
labels:
app: software-upgrader
version: v1
spec:
containers:
- image: gcr.io/mesh7-public-images/scalability/nginx
imagePullPolicy: IfNotPresent
name: software-upgrader
resources:
limits:
cpu: 20m
memory: 32Mi
requests:
cpu: 20m
memory: 32Mi
now deploy HTTPS servers in both namespaces by this steps Steps to deploy HTTPS server
now curl it from another pod in both namespace
The Pod with istio not injected would get 200 OK , while istio-injected pod would get
curl: (56) OpenSSL SSL_read: error:1409445C:SSL routines:ssl3_read_bytes:tlsv13 alert certificate required, errno 0
command terminated with exit code 56
Pardon me of my ignorance do i have to create some Service-entry or Virtual Service for HTTPS to happen between Pods in same namespace to happen if istio is injected?
You have to add Protocol to Service port Definition
apiVersion: v1
kind: Service
metadata:
name: test-https-server
labels:
app: test-https-server
service: test-https-server
spec:
ports:
- name: test-https
port: 25302
appProtocol: https
selector:
app: test-https-server
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: test-https-server
spec:
selector:
matchLabels:
app: test-https-server
template:
metadata:
labels:
app: test-https-server
spec:
containers:
- image: gcr.io/mesh7-public-images/scalability/nginx
command: ["bash", "-c", "python3 ThreadedHTTPSServer.py 25302"]
imagePullPolicy: Always
name: test-https-server
resources:
limits:
cpu: 20m
memory: 32Mi
requests:
cpu: 20m
memory: 32Mi
This has a example of working example
ports:
- name: http
port: 25302
appProtocol: https # Should Specify Protocol
Istio appProtocol configuration doc
I have installed on K3S raspberry pi cluster nexus with the following setups for kubernetes learning purposes. First I created a StatefulSet:
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: nexus
namespace: dev-ops
spec:
serviceName: "nexus"
replicas: 1
selector:
matchLabels:
app: nexus-server
template:
metadata:
labels:
app: nexus-server
spec:
containers:
- name: nexus
image: klo2k/nexus3:latest
env:
- name: MAX_HEAP
value: "800m"
- name: MIN_HEAP
value: "300m"
resources:
limits:
memory: "4Gi"
cpu: "1000m"
requests:
memory: "2Gi"
cpu: "500m"
ports:
- containerPort: 8081
volumeMounts:
- name: nexusstorage
mountPath: /sonatype-work
volumes:
- name: nexusstorage
persistentVolumeClaim:
claimName: nexusstorage
Storage class
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: nexusstorage
provisioner: driver.longhorn.io
allowVolumeExpansion: true
reclaimPolicy: Delete
volumeBindingMode: Immediate
parameters:
numberOfReplicas: "3"
staleReplicaTimeout: "30"
fsType: "ext4"
diskSelector: "ssd"
nodeSelector: "ssd"
pvc
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: nexusstorage
namespace: dev-ops
spec:
accessModes:
- ReadWriteOnce
storageClassName: nexusstorage
resources:
requests:
storage: 50Gi
Service
apiVersion: v1
kind: Service
metadata:
name: nexus-server
namespace: dev-ops
annotations:
prometheus.io/scrape: 'true'
prometheus.io/path: /
prometheus.io/port: '8081'
spec:
selector:
app: nexus-server
type: LoadBalancer
ports:
- port: 8081
targetPort: 8081
nodePort: 32000
this setup will spin up nexus, but if I restart the pod the data will not persist and I have to create all the setups and users from scratch.
What I'm missing in this case?
UPDATE
I got it working, nexus needs on mount permissions on directory. The working StatefulSet looks as it follow
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: nexus
namespace: dev-ops
spec:
serviceName: "nexus"
replicas: 1
selector:
matchLabels:
app: nexus-server
template:
metadata:
labels:
app: nexus-server
spec:
securityContext:
runAsUser: 200
runAsGroup: 200
fsGroup: 200
containers:
- name: nexus
image: klo2k/nexus3:latest
env:
- name: MAX_HEAP
value: "800m"
- name: MIN_HEAP
value: "300m"
resources:
limits:
memory: "4Gi"
cpu: "1000m"
requests:
memory: "2Gi"
cpu: "500m"
ports:
- containerPort: 8081
volumeMounts:
- name: nexus-storage
mountPath: /nexus-data
volumes:
- name: nexus-storage
persistentVolumeClaim:
claimName: nexus-storage
important snippet to get it working
securityContext:
runAsUser: 200
runAsGroup: 200
fsGroup: 200
I'm not familiar with that image, although checking dockerhub, they mention using a Dockerfile similar to that of Sonatype. Then, I would change the mountpoint for your volume, to /nexus-data
This is the default path storing data (they set this env var, then declare a VOLUME). Which we can confirm, looking at the repository that most likely produced your arm-capable image
And following up on your last comment, let's try to also mount it in /opt/sonatype/sonatype-work/nexus3...
In your statefulset, change volumeMounts, to this:
volumeMounts:
- name: nexusstorage
mountPath: /nexus-data
- name: nexusstorage
mountPath: /opt/sonatype/sonatype-work/nexus3
volumes:
- name: nexusstorage
persistentVolumeClaim:
claimName: nexusstorage
Although the second volumeMount entry should not be necessary, as far as I understand. Maybe something's wrong with your storage provider?
Are you sure your PVC is write-able? Reverting back to your initial configuration, enter your pod (kubectl exec -it) and try to write a file at the root of your PVC.
I have a kubernetes deployment file user.yaml -
apiVersion: apps/v1
kind: Deployment
metadata:
name: user-deployment
namespace: stage
spec:
replicas: 1
selector:
matchLabels:
app: user
template:
metadata:
labels:
app: user
annotations:
prometheus.io/scrape: 'true'
prometheus.io/port: '9022'
spec:
nodeSelector:
env: stage
containers:
- name: user
image: <docker image path>
imagePullPolicy: Always
resources:
limits:
memory: "512Mi"
cpu: "250m"
requests:
memory: "256Mi"
cpu: "200m"
ports:
- containerPort: 8080
env:
- name: MODE
value: "local"
- name: PORT
value: ":8080"
- name: REDIS_HOST
value: "xxx"
- name: KAFKA_ENABLED
value: "true"
- name: BROKERS
value: "xxx"
imagePullSecrets:
- name: regcred
---
apiVersion: v1
kind: Service
metadata:
namespace: stage
name: user
spec:
selector:
app: user
ports:
- protocol: TCP
port: 8080
targetPort: 8080
---
apiVersion: autoscaling/v2beta2
kind: HorizontalPodAutoscaler
metadata:
name: user
namespace: stage
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: user-deployment
minReplicas: 1
maxReplicas: 10
metrics:
- type: Resource
resource:
name: cpu
target:
type: Utilization
averageUtilization: 50
- type: Resource
resource:
name: memory
target:
type: AverageValue
averageValue: 300Mi
This deployment is already running with linkerd injected with command cat user.yaml | linkerd inject - | kubectl apply -f -
Now I wanted to add linkerd inject annotation (as mentioned here) and use command kubectl apply -f user.yaml just like I use for a deployment without linkerd injected.
However, with modified user.yaml (after adding linkerd.io/inject annotation in deployment) -
apiVersion: apps/v1
kind: Deployment
metadata:
name: user-deployment
namespace: stage
spec:
replicas: 1
selector:
matchLabels:
app: user
template:
metadata:
labels:
app: user
annotations:
prometheus.io/scrape: 'true'
prometheus.io/port: '9022'
linkerd.io/inject: enabled
spec:
nodeSelector:
env: stage
containers:
- name: user
image: <docker image path>
imagePullPolicy: Always
resources:
limits:
memory: "512Mi"
cpu: "250m"
requests:
memory: "256Mi"
cpu: "200m"
ports:
- containerPort: 8080
env:
- name: MODE
value: "local"
- name: PORT
value: ":8080"
- name: REDIS_HOST
value: "xxx"
- name: KAFKA_ENABLED
value: "true"
- name: BROKERS
value: "xxx"
imagePullSecrets:
- name: regcred
---
apiVersion: v1
kind: Service
metadata:
namespace: stage
name: user
spec:
selector:
app: user
ports:
- protocol: TCP
port: 8080
targetPort: 8080
---
apiVersion: autoscaling/v2beta2
kind: HorizontalPodAutoscaler
metadata:
name: user
namespace: stage
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: user-deployment
minReplicas: 1
maxReplicas: 10
metrics:
- type: Resource
resource:
name: cpu
target:
type: Utilization
averageUtilization: 50
- type: Resource
resource:
name: memory
target:
type: AverageValue
averageValue: 300Mi
When I run kubectl apply -f user.yaml, it throws error -
service/user unchanged
horizontalpodautoscaler.autoscaling/user configured
Error from server (BadRequest): error when creating "user.yaml": Deployment in version "v1" cannot be handled as a Deployment: v1.Deployment.Spec: v1.DeploymentSpec.Template: v1.PodTemplateSpec.Spec: v1.PodSpec.Containers: []v1.Container: v1.Container.Env: []v1.EnvVar: v1.EnvVar.v1.EnvVar.Value: ReadString: expects " or n, but found 1, error found in #10 byte of ...|,"value":1},{"name":|..., bigger context ...|ue":":8080"}
Can anyone please point out where I have gone wrong in adding annotation?
Thanks
Try with double quotes like below
annotations:
prometheus.io/scrape: "true"
prometheus.io/port: "9022"
linkerd.io/inject: enabled
I have deployed JFrog Container Registry to my Kubernetes cluster, which all comes up fine but when I try to access it via browser, it redirects to /ui which returns a 404 but nothing seems to show in the logs.
I have not used the Helm chart as I do not need the nginx or Postgres etc just to try it out.
My deployment is this:
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: jcr
namespace: <REDACTED>
spec:
replicas: 1
template:
metadata:
labels:
app: jcr
spec:
containers:
- name: jcr
image: docker.bintray.io/jfrog/artifactory-jcr:latest
ports:
- containerPort: 8081
volumeMounts:
- name: jcr-data
mountPath: /jcr-data
volumes:
- name: jcr-data
persistentVolumeClaim:
claimName: jcr-data
securityContext:
fsGroup: 2000
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: jcr-data
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 10Gi
---
apiVersion: v1
kind: Service
metadata:
name: jcr
namespace: <REDACTED>
annotations:
prometheus.io/scrape: 'true'
prometheus.io/path: /
prometheus.io/port: '8081'
spec:
selector:
app: jcr
ports:
- port: 80
targetPort: 8081
sessionAffinity: None
type: ClusterIP
---
apiVersion: contour.heptio.com/v1beta1
kind: IngressRoute
metadata:
labels:
app: jcr
name: jcr
namespace: <REDACTED>
spec:
virtualhost:
fqdn: <REDACTED>
tls:
secretName: jcr-live
routes:
- match: /
services:
- name: jcr
port: 80
Looks like your port configuration is missing some changes.
You need to expose port 8082 in the jcr container, which is now the main UI port
Once port is exposed, you should add this port to your service.
So your revised yaml should look something like (Deployment and Service):
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: jcr
namespace: <REDACTED>
spec:
replicas: 1
template:
metadata:
labels:
app: jcr
spec:
containers:
- name: jcr
image: docker.bintray.io/jfrog/artifactory-jcr:latest
ports:
- containerPort: 8081
- containerPort: 8082
volumeMounts:
- name: jcr-data
mountPath: /jcr-data
volumes:
- name: jcr-data
persistentVolumeClaim:
claimName: jcr-data
securityContext:
fsGroup: 2000
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: jcr-data
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 10Gi
---
apiVersion: v1
kind: Service
metadata:
name: jcr
namespace: <REDACTED>
annotations:
prometheus.io/scrape: 'true'
prometheus.io/path: /
prometheus.io/port: '8081'
spec:
selector:
app: jcr
ports:
- port: 80
targetPort: 8082
- port: 8081
targetPort: 8081
sessionAffinity: None
type: ClusterIP
Notice I left 8081 open, which allows for direct access to Artifactory if needed for better performance (Artifactory is now running behind a router service).
NOTE - I recommend using the official JFrog Container Registry Helm chart, which greatly simplifies the process of configuring and managing your JCR deployment lifecycle.
in my env file for my angular frontend I have the API endpoint set as localhost:8000 because my API listens on that port, but it is in a separate pod is this correct or am I meant to use the name I gave to the backend service in the deployment file. Second, how do I connect the backend service is how I have it done in the deployment file below correct?
apiVersion: apps/v1
kind: Deployment
metadata:
name: ai-api
spec:
replicas: 1
selector:
matchLabels:
app: ai-api
template:
metadata:
labels:
app: ai-api
spec:
nodeSelector:
"beta.kubernetes.io/os": linux
containers:
- name: ai-api
image: test.azurecr.io/api:v5
resources:
requests:
cpu: 100m
memory: 128Mi
limits:
cpu: 250m
memory: 256Mi
ports:
- containerPort: 8000
name: ai-api
---
apiVersion: v1
kind: Service
metadata:
name: ai-api
spec:
ports:
- port: 8000
selector:
app: ai-api
---
# Frontend
apiVersion: apps/v1
kind: Deployment
metadata:
name: ai-front
spec:
replicas: 1
selector:
matchLabels:
app: ai-front
template:
metadata:
labels:
app: ai-front
spec:
nodeSelector:
"beta.kubernetes.io/os": linux
containers:
- name: ai-front
image: test.azurecr.io/front-end:v5.1
resources:
requests:
cpu: 100m
memory: 128Mi
limits:
cpu: 250m
memory: 256Mi
ports:
- containerPort: 80
env:
- name: api
value: "ai-api"
---
apiVersion: v1
kind: Service
metadata:
name: ai-front
spec:
type: LoadBalancer
ports:
- port: 80
#Tells loadbalancer which deployment to use
selector:
app: ai-front
You mentioned that you have API endpoint set as localhost:8000 for your frontend which is not correct as localhost is referring to the same pod from which the request is send from (so it means "connect to myself"). Change it to ai-api:8000. And also make sure that your api server is listening on 0.0.0.0:8000 and not on localhost:8000.
I also see that you are passing the name of your backend service to the frontend pod:
env:
- name: api
value: "ai-api"
and if you are using this env to connect to your backend app it would stand in contradiction with your earlier statement that you are connecting to localhost:8000.