I'm trying out KServe. I've followed installation instructions as per the official docs. I'm trying to create a sample HTTP InferenceService exactly same as this. by doing kubectl apply -f tensorflow.yaml
apiVersion: "serving.kserve.io/v1beta1"
kind: "InferenceService"
metadata:
name: "flower-sample"
spec:
predictor:
tensorflow:
storageUri: "gs://kfserving-samples/models/tensorflow/flowers"
This yaml creates an InferenceService and assosicated deployments, replicaset, and a pod. However, the InferenceService remains in Unknown state, upon investiagating, I found that 1 container (queue-proxy container) out of 2 wasn't able to do Readiness probe, Readiness probe failed: HTTP probe failed with statuscode: 503 .
Upon further investigation, I saw the following logs from kserve-container container
2022-04-07 17:14:46.482205: E tensorflow_serving/sources/storage_path/file_system_storage_path_source.cc:365] FileSystemStoragePathSource encountered a filesystem access error: Could not find base path /mnt/models for servable flower-sample with error Not found: /mnt/models not found
So I understood models not present. I manually downloaded the same gs://kfserving-samples/models/tensorflow/flowers and copied in the correct path. Finally InferenceService started working.
But I want to avoid doing this manual copying of models into pod's container.
It should ideally be done by init-container - storage-initializer which is missing in my case.
I did kubectl describe flower-sample-predictor-default-00001-deployment-5db9d7d9fgqfn4 and I don't see any initContainers section.
All these workloads are running in kserve namespace. I've a configmap inferenceservice-config which has the following (default) storageInitializer
{
"image" : "kserve/storage-initializer:v0.8.0",
"memoryRequest": "100Mi",
"memoryLimit": "1Gi",
"cpuRequest": "100m",
"cpuLimit": "1"
}
But still when I do kubectl apply -f tensorflow.yaml I'm facing the same error. Could anyone help me on how to fix this ?
Related
I did the following to deploy a helm chart (you can copy-and-paste my sequence of commands to reproduce this error).
$ flux --version
flux version 0.16.1
$ kubectl create ns traefik
$ flux create source helm traefik --url https://helm.traefik.io/traefik --namespace traefik
$ cat values-6666.yaml
ports:
traefik:
healthchecksPort: 6666 # !!! Deliberately wrong port number!!!
$ flux create helmrelease my-traefik --chart traefik --source HelmRepository/traefik --chart-version 9.18.2 --namespace traefik --values=./values-6666.yaml
✚ generating HelmRelease
► applying HelmRelease
✔ HelmRelease created
◎ waiting for HelmRelease reconciliation
✔ HelmRelease my-traefik is ready
✔ applied revision 9.18.2
So Flux reports it as a success, and can be confirmed like this:
$ flux get helmrelease --namespace traefik
NAME READY MESSAGE REVISION SUSPENDED
my-traefik True Release reconciliation succeeded 9.18.2 False
But in fact, as shown above, values-6666.yaml contains a deliberately wrong port number 6666 for pod's readiness probe (as well as liveness probe):
$ kubectl -n traefik describe pod my-traefik-8488cc49b8-qf5zz
...
Type Reason ... From Message
---- ------ ... ---- -------
Warning Unhealthy ... kubelet Liveness probe failed: Get "http://172.31.61.133:6666/ping": dial tcp 172.31.61.133:6666: connect: connection refused
Warning Unhealthy ... kubelet Readiness probe failed: Get "http://172.31.61.133:6666/ping": dial tcp 172.31.61.133:6666: connect: connection refused
Warning BackOff ... kubelet Back-off restarting failed container
My goal is to have FluxCD automatically detect the above error. But, as shown above, FluxCD deems it a success.
Either of the following deployment methods would have detected that failure:
$ helm upgrade --wait ...
or
$ argocd app sync ... && argocd app wait ...
So, is there something similar in FluxCD to achieve the same effect?
====================================================================
P.S. Flux docs here seems to suggest that the equivalent to helm --wait is already the default behaviour in FluxCD. My test above shows that it isn't. Furthermore, in the following example, I explicitly set it to disableWait: false but the result is the same.
$ cat helmrelease.yaml
---
apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
name: my-traefik
namespace: traefik
spec:
chart:
spec:
chart: traefik
sourceRef:
kind: HelmRepository
name: traefik
version: 9.18.2
install:
disableWait: false # !!! Explicitly set this flag !!!
interval: 1m0s
values:
ports:
traefik:
healthchecksPort: 6666
$ kubectl -n traefik create -f helmrelease.yaml
helmrelease.helm.toolkit.fluxcd.io/my-traefik created
## Again, Flux deems it a success:
$ flux get hr -n traefik
NAME READY MESSAGE REVISION SUSPENDED
my-traefik True Release reconciliation succeeded 9.18.2 False
## Again, the pod actually failed:
$ kubectl -n traefik describe pod my-traefik-8488cc49b8-bmxnv
... // Same error as earlier
Helm considers a deployment with one replica and strategy rollingUpdate with maxUnavailable of 1 to be ready when it has been deployed and there is 1 unavailable pod. If you test Helm itself, I believe you will find the same behavior exists in the Helm CLI / Helm SDK package upstream.
(Even if the deployment's one and only pod has entered CrashLoopBackOff and readiness and liveness checks have all failed... with maxUnavailable of 1 and replicas of 1, the deployment technically has no more than the allowed number of unavailable pods, so it is considered ready.)
This question was re-raised recently at: https://github.com/fluxcd/helm-controller/issues/355 and I provided more in-depth feedback there.
Anyway, as for the source of this behavior which is seemingly/clearly not what the user wanted (even if it appears to be specifically what the user has asked for, which is perhaps debatable):
As for Helm, this appears to be the same issue reported at GitHub here:
helm install --wait does not wait for deployment pod readiness properly - (helm/helm#3173)
and resurrected here:
helm upgrade --wait does not wait on newer versions - (helm/helm#10061)
FluxCD v2 is by default using the --wait option of Helm.
In general, you can use any Helm parameter of the CLI in the HelmRelease object:
https://fluxcd.io/docs/components/helm/helmreleases/
I recommend to invest in proper readiness probes of your pods. Helm/FluxCDv2 will wait for all pods to become ready. Liveness probes have a different purpose. Kubelet uses liveness probes to know when to restart a container. Usually they are not of interest for Helm/Flux.
If you have a complex application lifecycle, then take a look into Kubernetes Operators - (C) Jason Dobies, Joshua Wood. With the help of kstatus and kustomize you can let flux wait for your custom ressource to become ready.
Problem summary:
I am following the Kubernetes guide to set up a sample Cassandra cluster. The cluster is up and running, and I would like to add a second volume to each node in order to try enable backups for Cassandra that would be stored on a separate volume.
My attempt to a solution:
I tried editing my cassandra-statefulset.yaml file by adding a new volumeMounts and volumeClaimTemplates entry, and reapplying it, but got the following error message:
$ kubectl apply -f cassandra-statefulset.yaml
storageclass.storage.k8s.io/fast unchanged
The StatefulSet "cassandra" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'template', and 'updateStrategy' are forbidden
I then tried to enable rolling updates and patch my configuration following the documentation here:
https://kubernetes.io/docs/tasks/run-application/update-api-object-kubectl-patch/
$ kubectl patch statefulset cassandra -p '{"spec":{"updateStrategy":{"type":"RollingUpdate"}}}'
statefulset.apps/cassandra patched (no change)
My cassandra-backup-patch.yaml:
spec:
template:
spec:
containers:
volumeMounts:
- name: cassandra-backup
mountPath: /cassandra_backup
volumeClaimTemplates:
- metadata:
name: cassandra-backup
spec:
accessModes: [ "ReadWriteOnce" ]
storageClassName: fast
resources:
requests:
storage: 1Gi
However this resulted in the following error:
$ kubectl patch statefulset cassandra --patch "$(cat cassandra-backup-patch.yaml)"
The request is invalid: patch: Invalid value: "map[spec:map[template:map[spec:map[containers:map[volumeMounts:[map[mountPath:/cassandra_backup name:cassandra-backup]]]]] volumeClaimTemplates:[map[metadata:map[name:cassandra-backup] spec:map[accessModes:[ReadWriteOnce] resources:map[requests:map[storage:1Gi]] storageClassName:fast]]]]]": cannot restore slice from map
Could anyone please point me to the correct way of adding an additional volume for each node or explain why the patch does not work? This is my first time using Kubernetes so my approach may be completely wrong. Any comment or help is very welcome, thanks in advance.
The answer is in your first log:
The StatefulSet "cassandra" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'template', and 'updateStrategy'
You can't change some fields in a statefulset after creation. You will likely need to delete and recreate the statefulset to add a new volumeClaimTemplate.
edit:
It can many times be useful to leave your pods running even when you delete the statefulset. To accomplish this use the --cascade=false flag on the delete operation.
kubectl delete statefulset <name> --cascade=false
Then your workload will stay running while you recreate your statefulset with the updated VPC.
As mentioned by switchboard.op, deleting is the answer.
but
Watch out for deleting these objects:
PersistentVolumeClaim (kubectl get pvc)
PersistentVolume (kubectl get pv)
which for example in case you'd want to do just helm uninstall instead of kubectl delete statefulset/<item> will be deleted thus unless there's any other reference for the volumes and in case you don't have backups of the previous YAMLs that contain the IDs (i.e. not just generated from Helm templates, but from the orchestrator) you might have a painful day ahead of you.
PVCs and PVs hold IDs and other reference properties for the underlying (probably/mostly?) vendor specific volume referencing by e.g. S3 or other object or file storage implementation used in the background as a volume in a Pod or other resources.
Deleting or otherwise modifying a StatefulSet if you preserve the PVC name within the spec doesn't affect mounting of the correct resource.
If in doubt, always just copy locally the whole volume prior to doing destructive action to PVCs and PVs if you need them in the future or running commands without knowing the underlying source code e.g. by:
kubectl cp <some-namespace>/<some-pod>:/var/lib/something /tmp/backup-something
and then just load it back by reversing the arguments.
Also for Helm usage, delete the StatefulSet, then issue helm upgrade command and it'll fix the missing StatefulSet without touching PVCs and PVs.
I have a kubernetes cluster set up and I would like to use local images. I have configured .yaml file so that it contains (in containers -> image -section) "imagePullPolicy: Never" like this:
spec:
containers:
- image: <name>:<version>
name: <name>
imagePullPolicy: Never
resources: {}
restartPolicy: Always
I have deployed this service to kubernetes but image cannot be pulled (getting ImagePullBackOff -error when viewing pods with kubectl get pod) since image cannot be found from internet/registry and for some unknown reason imagePullPolicy is in Always-value. This can be seen e.g. from /var/log/messages from text:
"spec":{"containers":[{"image":"<name>","imagePullPolicy":"Always","name":"<name>","
So my question is: Why is this imagePullPolicy in Always-value though I have set imagePullPolicy to Never in my .yaml file (which has of course been taken into use)? Is there some default value for imagePullPolicy that runs over the value described in .yaml file?
My environment is Centos7 and I'm using Kontena Pharos 2.2.0 (uses e.g. docker 1.13.1 (Apache License 2.0) and kubernetes 1.13.2 (Apache License 2.0)).
I expected that when I set "imagePullPolicy: Never" in .yaml file the value should now be Never (and not Always).
Thank you so much for helping!
welcome on StackOverflow.
It happens so, because your Kubernetes cluster has presumably enabled admission control plugin in api server, called 'AlwaysPullImages', which role is to overwrite (mutate) objects prior storing them in Kubernetes data store - etcd.
This is a default behavior of clusters bootstrapped with Kontena Pharos since version v2.0.0-alpha.2.
You can disable this admission plugin in your main cluster.yml config file:
...
addons:
ingress-nginx:
enabled: true
admission_plugins:
- name: AlwaysPullImages
enabled: false
...
You should expect then to see PODs failing with different status reason, if image is not found on local registry:
client-deployment-99699599d-lfqmr 0/1 ErrImageNeverPull 0 42s
Please read more on using of Admission Controllers here
I am new to kubernetes and docker. I am trying to chain 2 containers in a pod such that the second container should not be up until the first one is running. I searched and got a solution here. It says to add "depends" field in YAML file for the container which is dependent on another container. Following is a sample of my YAML file:
apiVersion: v1beta4
kind: Pod
metadata:
name: test
labels:
apps: test
spec:
containers:
- name: container1
image: <image-name>
ports:
- containerPort: 8080
hostPort: 8080
- name: container2
image: <image-name>
depends: ["container1"]
Kubernetes gives me following error after running the above yaml file:
Error from server (BadRequest): error when creating "new.yaml": Pod in version "v1beta4" cannot be handled as a Pod: no kind "Pod" is registered for version "v1beta4"
Is the apiVersion problem here? I even tried v1, apps/v1, extensions/v1 but got following errors (respectively):
error: error validating "new.yaml": error validating data: ValidationError(Pod.spec.containers[1]): unknown field "depends" in io.k8s.api.core.v1.Container; if you choose to ignore these errors, turn validation off with --validate=false
error: unable to recognize "new.yaml": no matches for apps/, Kind=Pod
error: unable to recognize "new.yaml": no matches for extensions/, Kind=Pod
What am I doing wrong here?
As I understand there is no field called depends in the Pod Specification.
You can verify and validate by following command:
kubectl explain pod.spec --recursive
I have attached a link to understand the structure of the k8s resources.
kubectl-explain
There is no property "depends" in the Container API object.
You split your containers in two different pods and let the kubernetes cli wait for the first container to become available:
kubectl create -f container1.yaml --wait # run command until the pod is available.
kubectl create -f container2.yaml --wait
I have just installed a basic kubernetes cluster the manual way, to better understand the components, and to later automate this installation. I followed this guide: https://kubernetes.io/docs/getting-started-guides/centos/centos_manual_config/
The cluster is completely empty without addons after this. I've already deployed kubernetes-dashboard succesfully, however, when trying to deploy kube-dns, it fails with the log:
2017-01-11T15:09:35.982973000Z F0111 15:09:35.978104 1 server.go:55]
Failed to create a kubernetes client:
invalid configuration: no configuration has been provided
I used the following yaml template for kube-dns without modification, only filling in the cluster IP:
https://coreos.com/kubernetes/docs/latest/deploy-addons.html
What did I do wrong?
Experimenting with kubedns arguments, I added --kube-master-url=http://mykubemaster.mydomain:8080 to the yaml file, and suddenly it reported in green.
How did this solve it? Was the container not aware of the master for some reason?
In my case, I had to put numeric IP on "--kube-master-url=http://X.X.X.X:8080". It's on yaml file of RC (ReplicationController), just like:
...
spec:
containers:
- name: kubedns
...
args:
# command = "/kube-dns"
- --domain=cluster.local
- --dns-port=10053
- --kube-master-url=http://192.168.99.100:8080