We are Containerizing dotnet application on GKE cluster(Windows node-pool). We have a requirement, where multiple pods can access same shared space(persistent volume). Also it should support "ReadWriteMany" AccessMode. We have explored below option:
GCE Persistent disk accessed by Persistent volume.(It doesn't support ReadWriteMany. Only one pod can access the disk).
Network File Share(NFS). Currently not supported for windows node pools.
Filestore fits the solutions but expensive and managed by google.
We are looking other options to fit our requirement. Please help.
You are right by saying that NFS isn't yet supported on Windows, at least, not for the built-in client v4. So as long as there is no support for NFS v4, Kubernetes team could not start up this work in k8s. source
With this constraint, the only solution I can see remains the Filestore.
I've been trying to solve the same problem - accessing shared filesystem from 2 Windows pods (ASP.NET application on IIS + console application). I wasn't able to use the Filestore because it requires an NFSClient (Install-WindowsFeature NFS-Client) and I couldn't install it into the containers (during container build or runtime) since it requires a computer restart - maybe i'm missing sth here.
The options I've found:
If you need to create a simple temporary demo application that can run on single VM you can run both pods on a single instance, create a Persistent Disk, attach it to the instance with gcloud compute instances attach-disk, RDP into the instance, mount the disk and provide the disk to the pods as a hostPath.
That's the solution I'm using now.
Create an SMB share (on a separate VM or using a Docker container https://hub.docker.com/r/dperson/samba/ and access it from the pods using New-SmbMapping -LocalPath $shareletter -RemotePath $dhcpshare -Username $shareuser -Password $sharepasswd -Persistent $true. This solution worked for my console application but the web application couldn't access the files (even though I've set the application pool on IIS to run as Local System). The SMB could also be mounted from the instance using the New-SmbGlobalMapping - the flexvolume does that https://github.com/microsoft/K8s-Storage-Plugins/tree/master/flexvolume/windows. I haven't explored that option and I think it would have the same problem (IIS not seeing the files).
I think the best (most secure and reliable) solution would be to setup an Active Directory Domain Controller and SMB share on separate VM and provide access to it to the containers using gMSA: https://learn.microsoft.com/en-us/virtualization/windowscontainers/manage-containers/manage-serviceaccounts
https://kubernetes.io/docs/tasks/configure-pod-container/configure-gmsa/
That doesn't seem easy though.
Related
I'm using GKE to run K8 workloads and want to add TPU support. From GCP docs, I "need" to attach a GCS bucket so the Job can read models and store logs. However, we already create shared NSF mounts for our k8 clusters. How hard of a requirement is it to "need" GCS to use TPUs? Can shared Filestore NFS mounts work just fine? What about using GCS Fuse?
I'm trying to avoid having the cluster user know about the back end file system (NFS vs GCS), and just know that that the files they provide will be available at "/home/job". Since the linked docs show passing a gs://mybucket/some/path value as needed for file system parameters, I'm not sure if a /home/job value will still work. Does the TPU access the filesystem directly and is only compatible with GCS? Or do the nodes access the filesystem (preferring GCS) and then share the data (in memory) with the TPUs?
I'll try it out to learn the hard way (and report back), but curious if others have experience with this already.
I am using Docker desktop to run a application in kubernetes platform where i need location to store files how can i use my local directory(c:\app-data) to be pointed to application running in kubernetes.
I had a similar problem. Docker contains are usually meant to be throwaway/gateway containers normally, so people don't usually use them for storing files.
That being said, you have two options:
Add path and files to docker container, which will cause your docker container to be massive in size (NOT RECOMMENDED). Docker build will require substantial time and memory, as all the files will be copied. Here's an example of creating a local ubuntu container with docker. https://thenewstack.io/docker-basics-how-to-share-data-between-a-docker-container-and-host/
Host your files through another server/api, and fetch those files using simple requests in your app. I used this solution. The only caveat is you need
to be able to host your files somehow. This is easy enough, but may require extra payment. https://www.techradar.com/best/file-hosting-and-sharing-services
You can't really do this. The right approach depends on what the data you're trying to store is.
If you're just trying to store data somewhere – perhaps it's the backing data for a MySQL StatefulSet – you can create a PersistentVolumeClaim like normal. Minikube includes a minimal volume provisioner so you should automatically get a PersistentVolume created; you don't need to do any special setup for this. But, the PersistentVolume will live within the minikube container/VM; if you completely delete the minikube setup, it could delete that data, and you won't be able to directly access the data from the host.
If you have a data set on the host that your container needs to access, there are a couple of ways to do it. Keep in mind that, in a "real" Kubernetes cluster, you won't be able to access your local filesystem at all. Creating a PersistentVolume as above and then running a pod to copy the data into it could be one approach; as #ParmandeepChaddha suggests in their answer, baking the data into the image is another reasonable approach (this can be very reasonable if the data is only a couple of megabytes).
If the data is the input or output data to your process, you can also consider restructuring your application so that it transfers that data over a protocol like HTTP. Set up a NodePort Service in front of your application, and use a tool like curl to HTTP POST the data into the service.
Finally, you could be considering a setup where all of the important data is local: you have some batch files on the local system, the job's purpose is to convert some local files to other local files, and it's just that the program is in minikube. (Or, similarly, you're trying to develop your application and the source files are on your local system.) In this case Kubernetes, as a distributed, clustered container system, isn't the right tool. Running the application directly on your system is the best approach; you can simulate this with a docker run -v bind mount, but this is inconvenient and can lead to permission and environment problems.
(In theory you can use a hostPath volume too, and minikube has some support to mount a host directory into the VM. In practice, the setup required to do this is as complex as the rest of your Kubernetes setup combined, and it won't be portable to any other Kubernetes installation. I wouldn't attempt this.)
You can mount your local directory to your kubernetes Pod using hostPath. Your path c:\app-data on your Windows host should be represented as either /C/app-data or /host_mnt/c/app-data, depending on your Docker Desktop version as suggested in this comment.
You may also want to take a look at this answer.
I'm working on a project where we're planning to migrate existing Dot net applications to GKE (Windows nodes). The problem is, in the existing state, different applications connect a windows shared drive(external) and perform simultaneous read/write operations on files. Now we want to achieve the same thing on GKE/GCP where multiple pods interact with a file system in readWrite mode in parallel.
We've explored the below options:
GCE PD (ReadWriteMany is not supported) https://kubernetes.io/docs/concepts/storage/persistent-volumes/#access-modes
NFS File Storage (Windows containers not supported) https://github.com/kubernetes/enhancements/blob/master/keps/sig-windows/20190103-windows-node-support.md
Google Filestore (Internally uses NFS & may not be supported by Windows)
I've verified multiple sources but unable to figure what solution fits in our requirement. Request for any help for the above mentioned scenario.
Our company use Kubernetes in all our environments. as well as on our local Macbook using minikube.
We have many microservices and most of them are running JVM which require a large amount of memory. We started to face an issue that we cannot run our stack on minikube due to out of memory of the local machine.
We thought about multiple solutions:
the first was to create a k8s cloud development environment and when a developer is working on a single microservice on his local macbook he will redirect the outbound traffic into the cloud instead of the local minikube. but this solution will create new problems:
how a pod inside the cloud dev env will send data to the local developer machine? its not just a single request/response scenario
We have many developers, they can overlap each other with different versions of each service they need to be deploy on the cloud. (We can set each developer separate namespace but we will need a huge cluster to support it)
The second solution was maybe we should use a tools like skaffold or draft to deploy our current code into the cloud development environment. that will solve issue #1 but again we see problems:
Slow development cycle - building a java image and push to remote cloud and wait for init will take too much time for developer to work.
And we still facing issue #2
Antoher though was, kubernetes support multiple nodes, why won't we just add another node, a remote node that sit on the cloud, to our local minikube? The main issue is that minikube is a single node solution. Also, we didn't find any resources for it on the web.
Last thought was to connect minikube docker daemon to a remote machine. so we will use minikube on the local machine but the docker will run the containers on a remote cloud server. But no luck so far, minikube crush when we do this manipulate. and we didn't find any resources for it on the web as well.
Any thought how to solve our issue? Thank you!
Do you know if it is possible to mount a local folder to a Kubernetes running container.
Like docker run -it -v .:/dev some-image bash I am doing this on my local machine and then remote debug into the container from VS Code.
Update: This might be a solution: telepresence
Link: https://kubernetes.io/docs/tasks/debug-application-cluster/local-debugging/
Do you know it it is possible to mount a local computer to Kubernetes. This container should have access to a Cassandra IP address.
Do you know if it is possible?
Kubernetes Volume
Using hostPath would be a solution: https://kubernetes.io/docs/concepts/storage/volumes/#hostpath
However, it will only work if your cluster runs on the same machine as your mounted folder.
Another but probably slightly over-powered method would be to use a distributed or parallel filesystem and mount it into your container as well as to mount it on your local host machine. An example would be CephFS which allows multi-read-write mounts. You could start a ceph cluster with rook: https://github.com/rook/rook
Kubernetes Native Dev Tools with File Sync Functionality
A solution would be to use a dev tool that allows you to sync the contents of the local folder to the folder inside a kubernetes pod. There, for example, is ksync: https://github.com/vapor-ware/ksync
I have tested ksync and many kubernetes native dev tools (e.g. telepresence, skaffold, draft) but I found them very hard to configure and time-consuming to use. That's why I created an open source project called DevSpace together with a colleague: https://github.com/loft-sh/devspace
It allows you to configure a real-time two-way sync between local folders and folders within containers running inside k8s pods. It is the only tool that is able to let you use hot reloading tools such as nodemon for nodejs. It works with volumes as well as with ephemeral / non-persistent folders and lets you directly enter the containers similar to kubectl exec and much more. It works with minikube and any other self-hosted or cloud-based kubernetes clusters.
Let me know if that helps you and feel free to open an issue if you are missing something you need for your optimal dev workflow with Kubernetes. We will be happy to work on it.
As long as we talk about doing stuff like docker -v a hostPath volume type should do the trick. But that means that you need to have the content you want to use stored on the Node that the Pod will run upon. Meaning that in case of GKE it would mean the code needs to exist on google compute node, not on your workstation. If you have local k8s cluster provisioned (minikube, kubeadm...) for local dev, that could be set to work as well.