I have a pod running in openshift 3.11, I wish to increase the pod memory limit from 2GB to 4GB. How to do it via Openshift Web Console or via OC command line?
When I try to edit the yaml file in Openshift Web Console I got the following exception
Reason: Pod "hjaha" is invalid: spec: Forbidden: pod updates may not
change fields other than spec.containers[*].image,
spec.initContainers[*].image, spec.activeDeadlineSeconds or
spec.tolerations (only additions to existing tolerations)...
Basically Pods are deployed using containers template of the controllers of the Pods, such as DeploymentConfig, Deployment, DaemonSet, StatefulSet and so on. First of all, you should verify what controller is used for your Pod deployment and modify the resources section on the controller yaml, not running Pod yaml. Look at the following example, if you modify the memory limit on the deployment controller yaml using oc CLI or web console, then it will deploy new pod with new configuration after that.
// List some deployment controller resources as follows.
// Then you can see one which is similar name with running pod name.
$ oc get deploymentconfig,deployment,statefulset,daemonset -n <your project name>
$ oc edit <deployment controller type> <the resource name>
:
kind: <deployment controller type>
metadata:
name: <the resource name>
spec:
:
template:
:
spec:
containers:
- name: <the container name>
resources:
limits:
// modify the memory size from 2Gi to 4Gi.
memory: 4Gi
You have to edit the yaml file and add this resources section under your containers part
containers:
- image: nginx
imagePullPolicy: Always
name: default-mem-demo-ctr
resources:
limits:
memory: 4Gi #<--------------This is limit
requests:
memory: 2Gi #<--------------Your applictaion will use memory in between 2Gb to upto 4GB
Related
I want deploy a mongodb chart using helm on my local dev environment.
I found all the possibile values on bitnami, but is overwhelming!
How can i configure something like that:
template:
metadata:
labels:
app: mongodb
spec:
containers:
- name: mongodb
image: mongo
ports:
- containerPort: 27017
volumeMounts:
- name: mongo-data
mountPath: /data/db/
volumes:
- name: mongo-data
hostPath:
path: /app/db
Using value.yml configuration file?
The best approach here is to deploy something like the Bitnami MongoDB chart that you reference in the question with its default options
helm install mongodb bitnami/mongodb
The chart will create a PersistentVolumeClaim for you, and a standard piece of Kubernetes called the persistent volume provisioner will create the corresponding PersistentVolume. The actual storage will be "somewhere inside Kubernetes", but for database storage there's little you can do with the actual files directly, so this isn't usually a practical problem.
If you can't use this approach, then you need to manually create the storage and then tell the chart to use it. You need to create a pair of a PersistentVolumeClaim and a PersistentVolume, for example as shown in the start of Kubernetes Persistent Volume and hostpath, and manually submit these using kubectl apply -f pv-pvc.yaml. You then need to tell the Bitnami chart about that PersistentVolume:
helm install mongodb bitnami/mongodb \
--set persistence.existingClaim=your-pvc-name
I'd avoid this sequence in a non-development environment. The cluster should normally have a persistent volume provisioner set up and so you shouldn't need to manually create PersistentVolumes, and host-path volumes are unreliable in multi-node environments (they refer to a fixed path on whichever node the pod happens to be running on, so data can get misplaced if a pod is rescheduled on a different node).
You need first to create a persistent volume claim, where it will create a persistent volume only if needed by a specific deployement( here you mondb helm chart):
kubectl -n $NAMESPACE apply -f persistent-volume-claim.yaml
For example: (or check https://kubernetes.io/docs/concepts/storage/persistent-volumes/)
#persistent-volume-claim.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: mongo-data
spec:
accessModes:
- ReadWriteMany
storageClassName: default
resources:
requests:
storage: 10Gi
Check your volume is well created
kubectl -n $NAMESPACE get pv
Now, Even if you delete your mongodb, your volume will persist and can be accessed by any other deployment
I want that every time I create a new image with the tag latest Kubernetes automatically pull the new image. I added imagePullPolicy: Always in pod spec but it doesn't update the old image with new image.
apiVersion: apps/v1
kind: Deployment
metadata:
name: node
namespace: dev
labels:
app: my-node-app
spec:
replicas: 2
selector:
matchLabels:
app: my-node-app
template:
metadata:
labels:
app: my-node-app
spec:
hostNetwork: true
securityContext:
fsGroup: 1000
containers:
- name: node
imagePullPolicy: Always
image: gcr.io/my-repo/my-node-app:latest
ports:
- containerPort: 3000
envFrom:
- configMapRef:
name: my-configmap
resources:
requests:
cpu: 100m
memory: 128Mi
limits:
cpu: 2
memory: 8Gi
restartPolicy: Always
imagePullPolicy is only taken into account by Kubernetes when a POD is created or re-started. It is NOT taken into account while a POD is running, which means it does NOT check for image updates at any time while a POD is running.
Even if another POD with the same image would be scheduled onto the same Kubernetes node, the already running POD is not affected, even though Kubernetes does a pull and then uses the new image for the new POD.
If you want the desired functionality, you will have to implement your own solution. You could do this by implementing a sidecar that regularly checks the Docker Repository for changes to the given tag. When it detects such a change, it could trigger a restart of the POD, which would then force the image to be re-pulled.
A restart of the POD can either be triggered by simply exiting the sidecar or by utilizing the Kubernetes API inside the sidecar. The latter solution however gets more complicated as you will also need service accounts and RBAC rules to get proper permissions inside the sidecar container. It also has security implications you'd have to give the whole POD escalated permissions.
Setting imagePullPolicy: Always does not mean an image will be pulled automatically without any trigger.
I would recommend to use tagged image with semvar. Since you are using deployment you can perform rolling update of the pod which will pull new image and rollout the change across all the replica pod one by one in a graceful way without causing any downtime.
Let's say initially the image is gcr.io/my-repo/my-node-app:v1 and you want to update it to v2
kubectl set image deployment/node nginx=gcr.io/my-repo/my-node-app:v2 --record
Check the rollout history
kubectl rollout history deployment.v1.apps/node
In case of any issue rollback to previous version
kubectl rollout undo deployment.v1.apps/node
Also if you want to be more advanced you could do GitOps using FluxCD which supports triggering a rollout automatically whenever a new version of an image is pushed to a container registry.
Kubernetes will pull image only upon Pod creation which means it does not check for image updates while a POD is in running state.
I would recommend to use Semantic Versioning for the image tag and use a CI/CD pipeline which build, tag, and push to your registry. Then use a CD tool such as keel to re-create your pods in the last step of the pipeline.
I'm very new to Kubernetes, and trying to get node-red running on a small cluster of raspberry pi's
I happily managed that, but noticed that once the cluster is powered down, next time I bring it up, the flows in node-red have vanished.
So, I've create a NFS share on a freenas box on my local network and can mount it from another RPI, so I know the permissions work.
However I cannot get my mount to work in a kubernetes deployment.
Any help as to where I have gone wrong please?
apiVersion: apps/v1
kind: Deployment
metadata:
name: node-red
labels:
app: node-red
spec:
replicas: 1
selector:
matchLabels:
app: node-red
template:
metadata:
labels:
app: node-red
spec:
containers:
- name: node-red
image: nodered/node-red:latest
ports:
- containerPort: 1880
name: node-red-ui
securityContext:
privileged: true
volumeMounts:
- name: node-red-data
mountPath: /data
env:
- name: NODE_NAME
valueFrom:
fieldRef:
fieldPath: spec.nodeName
- name: TZ
value: Europe/London
volumes:
- name: node-red-data
nfs:
server: 192.168.1.96
path: /mnt/Pool1/ClusterStore/nodered
The error I am getting is
error: error validating "node-red-deploy.yml": error validating data:
ValidationError(Deployment.spec.template.spec): unknown field "nfs" in io.k8s.api.core.v1.PodSpec; if
you choose to ignore these errors, turn validation off with --validate=false
New Information
I now have the following
apiVersion: v1
kind: PersistentVolume
metadata:
name: clusterstore-nodered
labels:
type: nfs
spec:
capacity:
storage: 1Gi
accessModes:
- ReadWriteMany
nfs:
path: /mnt/Pool1/ClusterStore/nodered
server: 192.168.1.96
persistentVolumeReclaimPolicy: Recycle
claim.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: clusterstore-nodered-claim
spec:
accessModes:
- ReadWriteMany
resources:
requests:
storage: 1Gi
Now when I start the deployment it waits at pending forever and I see the following the the events for the PVC
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal WaitForFirstConsumer 5m47s (x7 over 7m3s) persistentvolume-controller waiting for first consumer to be created before binding
Normal Provisioning 119s (x5 over 5m44s) rancher.io/local-path_local-path-provisioner-58fb86bdfd-rtcls_506528ac-afd0-11ea-930d-52d0b85bb2c2 External provisioner is provisioning volume for claim "default/clusterstore-nodered-claim"
Warning ProvisioningFailed 119s (x5 over 5m44s) rancher.io/local-path_local-path-provisioner-58fb86bdfd-rtcls_506528ac-afd0-11ea-930d-52d0b85bb2c2 failed to provision volume with StorageClass "local-path": Only support ReadWriteOnce access mode
Normal ExternalProvisioning 92s (x19 over 5m44s) persistentvolume-controller
waiting for a volume to be created, either by external provisioner "rancher.io/local-path" or manually created by system administrator
I assume that this is becuase I don't have a nfs provider, in fact if I do kubectl get storageclass I only see local-path
New question, how do I a add a storageclass for NFS? A little googleing around has left me without a clue.
Ok, solved the issue. Kubernetes tutorials are really esoteric and missing lots of assumed steps.
My problem was down to k3s on the pi only shipping with a local-path storage provider.
I finally found a tutorial that installed an nfs client storage provider, and now my cluster works!
This was the tutorial I found the information in.
In the stated Tutorial there are basically these steps to fulfill:
1.
showmount -e 192.168.1.XY
to check if the share is reachable from outside the NAS
2.
helm install nfs-provisioner stable/nfs-client-provisioner --set nfs.server=192.168.1.**XY** --set nfs.path=/samplevolume/k3s --set image.repository=quay.io/external_storage/nfs-client-provisioner-arm
Whereas you replace the IP with your NFS Server and the NFS path with your specific Path on your synology (both should be visible from your showmount -e IP command
Update 23.02.2021
It seems that you have to use another Chart and Image too:
helm install nfs-subdir-external-provisioner nfs-subdir-external-provisioner/nfs-subdir-external-provisioner --set nfs.server=192.168.1.**XY** --set nfs.path=/samplevolume/k3s --set image.repository=gcr.io/k8s-staging-sig-storage/nfs-subdir-external-provisioner
kubectl get storageclass
To check if the storageclass now exists
4.
kubectl patch storageclass nfs-client -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}' && kubectl patch storageclass local-path -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"false"}}}'
To configure the new Storage class as "default". Replace nfs-client and local-path with what kubectl get storageclass tells
5.
kubectl get storageclass
Final check, if it's marked as "default"
This is a validation error pointing at the very last part of your Deployment yaml, therefore making it an invalid object. It looks like you've made a mistake with indentations. It should look more like this:
volumes:
- name: node-red-data
nfs:
server: 192.168.1.96
path: /mnt/Pool1/ClusterStore/nodered
Also, as you are new to Kubernetes, I strongly recommend getting familiar with the concepts of PersistentVolumes and its claims. PVs are volume plugins like Volumes, but have a lifecycle independent of any individual Pod that uses the PV.
Please let me know if that helped.
I have 1 node K8 cluster on digitalocean with 1cpu/2gbRAM
and 3 node cluster on google cloud with 1cpu/2gbRAM
I ran two jobs separatley on each cloud platform with auto-scaling enabled.
First job had memory request of 200Mi
apiVersion: batch/v1
kind: Job
metadata:
name: scaling-test
spec:
parallelism: 16
template:
metadata:
name: scaling-test
spec:
containers:
- name: debian
image: debian
command: ["/bin/sh","-c"]
args: ["sleep 300"]
resources:
requests:
cpu: "100m"
memory: "200Mi"
restartPolicy: Never
More nodes of (1cpu/2gbRAM) were added to cluster automatically and after job completion extra node were deleted automatically.
After that, i ran second job with memory request 4500Mi
apiVersion: batch/v1
kind: Job
metadata:
name: scaling-test2
spec:
parallelism: 3
template:
metadata:
name: scaling-test2
spec:
containers:
- name: debian
image: debian
command: ["/bin/sh","-c"]
args: ["sleep 5"]
resources:
requests:
cpu: "100m"
memory: "4500Mi"
restartPolicy: Never
After checking later job remained in pending state . I checked pods Events log and i'm seeing following error.
0/5 nodes are available: 5 Insufficient memory **source: default-scheduler**
pod didn't trigger scale-up (it wouldn't fit if a new node is added): 1 Insufficient memory **source:cluster-autoscaler**
cluster did not auto-scaled for required requested resource for job. Is this possible using kubernetes?
CA doesn't add nodes to the cluster if it wouldn't make a pod schedulable. It will only consider adding nodes to node groups for which it was configured. So one of the reasons it doesn't scale up the cluster may be that the pod has too large (e.g. 4500Mi memory). Another possible reason is that all suitable node groups are already at their maximum size.
I have a pretty simple Kubernetes pod. I want a stateful set and want the following process:
I want to have an initcontainer download and uncompress a tarball from s3 into a volume mounted to the initcontainer
I want to mount that volume to my main container to be used
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: app
namespace: test
labels:
name: app
spec:
serviceName: app
replicas: 1
selector:
matchLabels:
app: app
template:
metadata:
labels:
app: app
spec:
initContainers:
- name: preparing
image: alpine:3.8
imagePullPolicy: IfNotPresent
command:
- "sh"
- "-c"
- |
echo "Downloading data"
wget https://s3.amazonaws.com/.........
tar -xvzf xxxx-........ -C /root/
volumeMounts:
- name: node-volume
mountPath: /root/data/
containers:
- name: main-container
image: ecr.us-west-2.amazonaws.com/image/:latest
imagePullPolicy: Always
volumeMounts:
- name: node-volume
mountPath: /root/data/
volumeClaimTemplates:
- metadata:
name: node-volume
spec:
accessModes: [ "ReadWriteOnce" ]
storageClassName: gp2-b
resources:
requests:
storage: 80Gi
I continue to get the following error:
At first I run this and I can see the logs flowing of my tarball being downloaded by the initcontainer. About half way done it terminates and gives me the following error:
Multi-Attach error for volume "pvc-faedc8" Volume is
already exclusively attached to one node and can't be
attached to another
Looks like you have a dangling PVC and/or PV that is attached to one of your nodes. You can ssh into the node and run a df or mount to check.
If you look at this the PVCs in a StatefulSet are always mapped to their pod names, so it may be possible that you still have a dangling pod(?)
If you have a dangling pod:
$ kubectl -n test delete pod <pod-name>
You may have to force it:
$ kubectl -n test delete pod <pod-name> --grace-period=0 --force
Then, you can try deleting the PVC and it's corresponding PV:
$ kubectl delete pvc pvc-faedc8
$ kubectl delete pv <pv-name>
I had the same issue right now and the problem was, that the node on which the pod is usually running on was down and another one took over (which didn't work as expected for whatever reason). Had the "node down" scenario a few times before already and it never caused any issues. Couldn't get the StatefulSet and Deployment back up and running without booting the node that was down. But as soon as the node was up and running again the StatefulSet and Deployment immediately came back to life as well.
I had a similar error:
The volume pvc-2885ea01-f4fb-11eb-9528-00505698bd8b
cannot be attached to the node node1 since it is already attached to the node node2*
I use longhorn as a storage provisioner and manager. So I just detached this pv in the error and restarted the stateful set. It automatically was able to attach to the pv correctly this time.
I'll add an answer that will prevent this from happening again.
Short answer
Access modes: Switch from ReadWriteOnce to ReadWriteMany.
In a bit more details
You're usng a StatefulSet where each replica has its own state, with a unique persistent volume claim (PVC) created for each pod.
Each PVC is referring to a Persistent Volume where you decided that the access mode is ReadWriteOnce.
Which as you can see from here:
ReadWriteOnce
the volume can be mounted as read-write by a single
node. ReadWriteOnce access mode still can allow multiple pods to
access the volume when the pods are running on the same node.
So in case K8S Scheduler (due to priorities or resource calculations or due to a Cluster autoscaler which decided to shift the pod to a different node) - you will receive an error that the volume is already exclusively attached to one node and can't be
attached to another node.
Please consider using ReadWriteMany where the volume can be mounted as read-write by many nodes.