I am looking for a way to install an Antivirus in Google Kubernetes Engine(GKE). However, I could only find this Kubernetes ClamAV tutorial.
Here is the link of Installing antivirus and file integrity monitoring on Container-Optimized OS.
There are two issues I am facing with this ClamAV on GKE:
Ensure scan-required paths within other pods are mounted as named volumes so they will be included in the scan of the node. This requirement in the overview suggests that only named volumes can be scanned and we want to scan almost every file inside the container.
Trying the steps mentioned in this tutorial I got stuck on the step which says get all pods and exec into one to initiate a full scan.
After creating the resources using kubectl apply -f clamav-daemonset.yaml
I don't see any pods being created.
How do I solve the above-mentioned issues?
Also, is there any other alternative for Antivirus (and file integrity) on GKE apart from ClamAV?
Related
I have installed Kubeflow using manifest. After installing ml-pipeline, the pod is in "CrashLoopBackOff" state. I changed the destinationrule for ml-pipeline, ml-pipeline-ui and ml-pipeline-msql to DISABLE but no luck. Can anyone help with this?
Thanks in advance.
There are a bunch of possible root causes for this POD’s status, but I am going to try to focus on the most common ones. To choose the correct one for your accurate situation, you are going to need to take a look into the “describe” and the log from the POD with "CrashLoopBackOff" state.
Verify if the “describe” says something like “Back-off restarting failed container” and the log says something like “a container name must be specified for …”, “F ml_metadata/metadata_store/metadata_store_server_main.cc:219] Non-OK-status …”.
If yes, the problem is the dynamic volume provisioning regularly, maybe because no volume provisioner is installed.
On the other hand, you can verify your cluster’s size, because anything less than 8 CPUs is going to run only if you reduce each service’s requested cpu in the manifest files.
You do not give details on the affected POD yet; but another option is to try to install Katib only (without Kubeflow or other resources) on your K8s cluster to verify other Kubernetes resources do not affect this connection. You can use the following URL’s information for more empirical cases’ troubleshooting and solutions: Multiple Pods stuck in CrashLoopBackOff, katib-mysql , ml-pipeline-persistenceagent pod keeps crashing.
Finally just confirm that you followed the correct instructions, based on the Distribution you used to deploy Kubeflow, you can visit the following URL: Kubeflow Distributions
Environment: external NFS share for persistent storage, accessible to all, R/W, Centos7 VMs (NFS share and K8s cluster), NFS utils installed on all workers.
Mount on a VM, e.g. a K8s worker node, works correctly, the share is R/W
Deployed in the K8s cluster: PV, PVC, Deployment (Volumes - referenced to PVC, VolumeMount)
The structure of the YAML files corresponds to the various instructions and postings, including the postings here on the site.
The pod starts, the share is mounted. Unfortunately, it is read-only. All the suggestions from the postings I have found about this did not work so far.
Any idea what else I could look out for, what else I could try?
Thanks. Thomas
After digging deep, I found the cause of the problem. Apparently, the syntax for the NFS export is very sensitive. One more space can be problematic.
On the NFS server, two export entries were stored in the kernel tables. The first R/O and the second R/W. I don't know whether this is a Centos bug because of the syntax in /etc/exports.
On another Centos machine I was able to mount the share without any problems (r/w). In the container (Debian-based image), however, not (only r/o). I have not investigated whether this is due to Kubernetes or Debian behaves differently.
After correcting the /etc/exports file and restarting the NFS server, there was only one correct entry in the kernel table. After that, mounting R/W worked on a Centos machine as well as in the Debian-based container inside K8s.
Here are the files / table:
privious /etc/exports:
/nfsshare 172.16.6.* (rw,sync,no_root_squash,no_all_squash,no_acl)
==> kernel:
/nfsshare 172.16.6.*(ro, ...
/nfsshare *(rw, ...
corrected /etc/exports (w/o the blank):
/nfsshare *(rw,sync,no_root_squash,no_all_squash,no_acl)
In principle, the idea of using an init container is a good one. Thank you for reminding me of this.
I have tried it.
Unfortunately, it doesn't change the basic problem. The file system is mounted "read-only" by Kubernetes. The init container returns the following error message (from the log):
chmod: /var/opt/dilax/enumeris: Read-only file system
What can be done to backup kubernetes PVC regularly for GCP and AWS?
GCP has VolumeSnapshot but I'm not sure how to schedule it, like every hour or every day.
I also tried Gemini/fairwinds but I get the following error when for GCP. I installed the charts as mentioned in README.MD and I can't find anyone else encountering the same error.
error: unable to recognize "backup-test.yml": no matches for kind "SnapshotGroup" in version "gemini.fairwinds.com/v1beta1"
You can implement Velero, which gives you tools to back up and restore your Kubernetes cluster resources and persistent volumes.
Unfortunately, Velero only allows you to backup & restore PV, not PVCs.
Velero’s restic integration backs up data from volumes by accessing the node’s filesystem, on which the pod is running. For this reason, restic integration can only backup volumes that are mounted by a pod and not directly from the PVC.
Might wanna look into stash.run
Agree with #hdhruna - Velero is really the most popular tool for doing that task.
However, you can also try miracle2k/k8s-snapshots
Automatic Volume Snapshots on Kubernetes
How is it useful? Simply add
an annotation to your PersistentVolume or PersistentVolumeClaim
resources, and let this tool create and expire snapshots according to
your specifications.
Supported Environments:
Google Compute Engine disks,
AWS EBS disks.
I evaluated multiple solutions including k8s CSI VolumeSnapshots, https://stash.run/, https://github.com/miracle2k/k8s-snapshots and CGP disks snapshots.
The best one in my opinion, is using k8s native implementation of snapshots via CSI driver, that is if you have a cluster version > = 1.17. This allows snapshoting volumes while in use, doesn't require having a read many or write many volume like stash.
I chose gemini by fairwinds also to automate backup creation and deletion and restoration and it works like a charm.
I believe your problem is caused by that missing CRD from gemini in your cluster. Verify that the CRD is installed correctly and also that the version installed is indeed the version you are trying to use.
My installation went flawlessly using their install guide with Helm.
I launched a MongoDB replica set on Kubernetes (GKE as well as kubeadm). I faced no problems with the pods accessing the storage.
However, when I used Helm to deploy the same, I face this problem.
When I run this command-
(
kubectl describe po mongodb-shard1-0 --namespace=kube-system
)
(Here mongodb-shard1-0 is the first and only pod (of the desired three) which was created)
I get the error-
Events
Error: failed to start container "mongodb-shard1-container": Error
response from daemon: error while creating mount source path
'/mongo/data': mkdir /mongo: read-only file system
I noticed one major difference between the two ways of creating MongoDB cluster (without Helm, and with Helm)- when using Helm, I had to create a service account and install the Helm chart using that service account. Without Helm, I did not need that.
I used different mongo docker images, I faced the same error every time.
Can anybody help why I am facing this issue?
Docker exports volumes from filesystem using -v command line option. i.e. -v /var/tmp:/tmp
Can you check if the containers/pods are writing to shared volumes, not to the root filesystem?
I have a gcloud Kubernetes cluster running and a Google bucket that holds some data I want to run on the cluster.
In order to use the data in the bucket, I need gcsfs installed on the nodes. How do I install packages like this on the cluster using gcloud, kubectl, etc.?
Check if a recipe like "Launch development cluster on Google Cloud Platform with Kubernetes and Helm" could help.
Using Helm, you can define workers with additional pip packages:
worker:
replicas: 8
limits:
cpu: 2
memory: 7500000000
pipPackages: >-
git+https://github.com/gcsfs/gcsfs.git
git+https://github.com/xarray/xarray.git
condaPackages: >-
-c conda-forge
zarr
blosc
I don't know if the suggestion given by VonC will actually work, but what I do know is that you're not really supposed to install stuff onto a Kubernetes Engine's worker node. This is evident by the fact that neither does it have a package manager nor does it allow to update individual programs separately.
Container-Optimized OS does not support traditional package managers (...) This design prevents updating individual software packages in the root filesystem independent of other packages.
Having said that, you can customize the worker nodes of a node pool if the number of nodes is static for that node pool via startup scripts. These still work as intended, but since you can't edit the instance template being used by the node pool, you'll have to edit those into the instances manually. So again, this clearly is not a very good way of doing things.
Finally, worker nodes have something called a "Toolbox", which is basically a special container you can run to get access to debugging tools. This container is ran directly on Docker, it's not scheduled by Kubernetes. You can customize this container image, so you can add some extra tools into it.