Since volume service is not yet available on Swisscom is it possible to do something like this with docker on Swisscom cloud foundry:
docker run -d --volume /mydata --name elastic-shared alpine echo My Data Container
docker run -d --volumes-from elastic-shared --name myelastic elasticsearch:latest
One container will be used as a volume for the other container running elasticseach as persistent data storage.
Alright, I see what you're trying to do but there's sone major issue with it: All application containers MUST be stateless. Just saying "I'll never restart the app that serves as a persistent volume" isn't going to make it persistent. Your app can (and will!) be rescheduled or restarted by CF at any time for various reasons.
That's also one of the major reasons why there is no volume sharing between applications, meaning that no, there's no CF equivalent to docker's --volumes-from.
Obviously, this also means using logstash in a way where it stores its state in the local filesystem is a bad idea.
Since there's no NFS volumes available on the Swisscom Application Cloud, your best bet might be to somehow store the state in S3, but I'm not sure that can be achieved (other than storing snapshots).
Related
I believe this requirement seems pretty straight forward for anyone trying to host their Tier3 i.e. database in a container.
I have MVP 3x Tier MERN app using -
1x Container instance
3x ECS services (Frontend, Backend and Database)
3x Tasks (1x running task per service)
The Database Task (mongodb) has its task definition updated to use EFS and have tested stopping the task and re-starting a new one for data persistence.
Question - How to ensure auto mount of EFS volume on the ECS container host (SPOT instance). If ECS leverages cloud formation template under the covers, do I need to update or modify this template to gain this persistent efs volume auto mounted on all container ec2 instances? I have come across various articles talking about a script in the ec2 launch config but I don't see any launch config created by ECS / cloud formation.
What is the easiest and simplest way to achieve something as trivial as persistent efs volume across my container host instances. Am guessing task definition alone doesn't solve this problem?
Thanks
Actually, I think below steps achieved persistence for the db task using efs -
Updated task definition for the database container to use EFS.
Mounted the EFS vol on container instance
sudo mount -t efs -o tls fs-:/ /database/data
The above mount command did not add any entries within the /etc/fstab but still seems to be persistent on the new ECS SPOT instance.
I am using Docker desktop to run a application in kubernetes platform where i need location to store files how can i use my local directory(c:\app-data) to be pointed to application running in kubernetes.
I had a similar problem. Docker contains are usually meant to be throwaway/gateway containers normally, so people don't usually use them for storing files.
That being said, you have two options:
Add path and files to docker container, which will cause your docker container to be massive in size (NOT RECOMMENDED). Docker build will require substantial time and memory, as all the files will be copied. Here's an example of creating a local ubuntu container with docker. https://thenewstack.io/docker-basics-how-to-share-data-between-a-docker-container-and-host/
Host your files through another server/api, and fetch those files using simple requests in your app. I used this solution. The only caveat is you need
to be able to host your files somehow. This is easy enough, but may require extra payment. https://www.techradar.com/best/file-hosting-and-sharing-services
You can't really do this. The right approach depends on what the data you're trying to store is.
If you're just trying to store data somewhere – perhaps it's the backing data for a MySQL StatefulSet – you can create a PersistentVolumeClaim like normal. Minikube includes a minimal volume provisioner so you should automatically get a PersistentVolume created; you don't need to do any special setup for this. But, the PersistentVolume will live within the minikube container/VM; if you completely delete the minikube setup, it could delete that data, and you won't be able to directly access the data from the host.
If you have a data set on the host that your container needs to access, there are a couple of ways to do it. Keep in mind that, in a "real" Kubernetes cluster, you won't be able to access your local filesystem at all. Creating a PersistentVolume as above and then running a pod to copy the data into it could be one approach; as #ParmandeepChaddha suggests in their answer, baking the data into the image is another reasonable approach (this can be very reasonable if the data is only a couple of megabytes).
If the data is the input or output data to your process, you can also consider restructuring your application so that it transfers that data over a protocol like HTTP. Set up a NodePort Service in front of your application, and use a tool like curl to HTTP POST the data into the service.
Finally, you could be considering a setup where all of the important data is local: you have some batch files on the local system, the job's purpose is to convert some local files to other local files, and it's just that the program is in minikube. (Or, similarly, you're trying to develop your application and the source files are on your local system.) In this case Kubernetes, as a distributed, clustered container system, isn't the right tool. Running the application directly on your system is the best approach; you can simulate this with a docker run -v bind mount, but this is inconvenient and can lead to permission and environment problems.
(In theory you can use a hostPath volume too, and minikube has some support to mount a host directory into the VM. In practice, the setup required to do this is as complex as the rest of your Kubernetes setup combined, and it won't be portable to any other Kubernetes installation. I wouldn't attempt this.)
You can mount your local directory to your kubernetes Pod using hostPath. Your path c:\app-data on your Windows host should be represented as either /C/app-data or /host_mnt/c/app-data, depending on your Docker Desktop version as suggested in this comment.
You may also want to take a look at this answer.
Do you know if it is possible to mount a local folder to a Kubernetes running container.
Like docker run -it -v .:/dev some-image bash I am doing this on my local machine and then remote debug into the container from VS Code.
Update: This might be a solution: telepresence
Link: https://kubernetes.io/docs/tasks/debug-application-cluster/local-debugging/
Do you know it it is possible to mount a local computer to Kubernetes. This container should have access to a Cassandra IP address.
Do you know if it is possible?
Kubernetes Volume
Using hostPath would be a solution: https://kubernetes.io/docs/concepts/storage/volumes/#hostpath
However, it will only work if your cluster runs on the same machine as your mounted folder.
Another but probably slightly over-powered method would be to use a distributed or parallel filesystem and mount it into your container as well as to mount it on your local host machine. An example would be CephFS which allows multi-read-write mounts. You could start a ceph cluster with rook: https://github.com/rook/rook
Kubernetes Native Dev Tools with File Sync Functionality
A solution would be to use a dev tool that allows you to sync the contents of the local folder to the folder inside a kubernetes pod. There, for example, is ksync: https://github.com/vapor-ware/ksync
I have tested ksync and many kubernetes native dev tools (e.g. telepresence, skaffold, draft) but I found them very hard to configure and time-consuming to use. That's why I created an open source project called DevSpace together with a colleague: https://github.com/loft-sh/devspace
It allows you to configure a real-time two-way sync between local folders and folders within containers running inside k8s pods. It is the only tool that is able to let you use hot reloading tools such as nodemon for nodejs. It works with volumes as well as with ephemeral / non-persistent folders and lets you directly enter the containers similar to kubectl exec and much more. It works with minikube and any other self-hosted or cloud-based kubernetes clusters.
Let me know if that helps you and feel free to open an issue if you are missing something you need for your optimal dev workflow with Kubernetes. We will be happy to work on it.
As long as we talk about doing stuff like docker -v a hostPath volume type should do the trick. But that means that you need to have the content you want to use stored on the Node that the Pod will run upon. Meaning that in case of GKE it would mean the code needs to exist on google compute node, not on your workstation. If you have local k8s cluster provisioned (minikube, kubeadm...) for local dev, that could be set to work as well.
I have deployed a 4 node Couchbase cluster using Docker images on the Google Container Engine with Kubernetes. I was able to access the Couchbase Console, look at the buckets, query etc. Now, after a couple of days, I go the Console URL and the Couchbase initial setup screen comes up! As though this is a fresh install. I can see that the nodes and pods are all still up and running.
Had a similar problem on my Windows box with Docker cluster (No Kubernetes). I redeployed that again.
Anyone else experienced this?
When you destroy and recreate container instances all the underlying state is lost.
If you want to preserve the state of your couchbase installation you'll need to use a docker data volume. Just create one and mount your couchbase data file directory.
On gcp, you'll additionally want to map a directory on the data volume to a persistent disk.
Background
I have a Java-Servlet application that runs in tomcat, which runs in a docker container, which runs on the Google Container Engine. It is no big deal to extend the docker image so that it also fetches and refreshes the certificates (there is only a single pod per domain, so no inter-pod-communication is required). However certbot needs to save it's credentials and certificates somewhere and the pod's filesystem seems like a bad idea because it is ephemeral and won't survive a pod restart. According to the table with storage options. Google Cloud storage seems like a good idea, because it is very cheap, the volume is auto sized and I can also access it from multiple locations (I don't need to create one disk for each individual pod which will be pretty much empty) including the web-UI (the later may be useful for debugging) and throuput and latency are really no issue for this usecase.
Question
I created a bucket and now I want to access that bucket from a container. Google describes here and yet again here that I can mount the buckets using FUSE. What they don't mention is that you need to make the container privileged to use FUSE which does not feel quite right for me. Additionally I need to install the whole google cloud SDK and set up authentication (which I am going to store... where?). But actually I don't really need fuse access. Just downloading the config on startup and uploading the config after each refresh would be enough. So something that works similar to SCP would do...
There is gcloud which can access files from command line without the need for FUSE, but it still needs to be initialized somehow with credentials.
Here user326502 mentions
It won't work with zero configuration if the App Engine SDK is installed [..] As long as the container lives on a Google Compute Engine instance you can access any bucket in the same project.
He explains further that I magically don't need any credentials when I just use the library. I guess I could write my own copy application with those libraries, but it feels like the fact that I did not find something like this from anyone on the net makes me feel that I am completely on the wrong track.
So how would one actually access a google cloud storage bucket from within a container (as simple as possible)?
You can use gsutil to copy from the bucket to the local disk when the container starts up.
If you are running in Google Container Engine, gsutil will use the service account of the cluster's nodes (to do this, you'll need to specify the storage-ro scope when you create your cluster).
Alternatively, you can create a new service account, generating a JSON key. In Container Engine, you can store that key as a Kubernetes secret, and then mount the secret in the pod that needs to use it. From that pod, you'd configure gsutil to use the service account by calling gcloud auth activate-service-account--key-file /path/to/my/mounted/secret-key.json