I have a container type that I will need to intermittently add and delete from my cluster, but each different container instance will need a unique configuration in the form of environment variables.
What is the best way to structure this with Kubernetes? Should I have a separate workload for each container? Should I have one common workload and update the pod with new containers as needed?
The containers are isolated applications that don't have anything to do with their siblings.
If you want to do this you can use the jobs in kubernetes, if there is no use case of PVC or PV
For each different type create a new job with the different type of environment variables.
I have a container type that I will need to intermittently add and
delete from my cluster, but each different container instance will
need a unique configuration in the form of environment variables.
configure multiple jobs or deployment in which you can give different options of environment variables.
I do it by
Having a common deployment yaml. You should mount configmaps as environment variables
Packaging with Helm
Deploying with different values.yaml for each instance.
As Harsh stated, you can also use Jobs pattern but I prefer having single file & many values files instead of different jobs definitions.. Because if a key changes, you will need to update all job definitions..
Related
What is the best approach to passing multiple configuration files into a POD?
Assume that we have a legacy application that we have to dockerize and run in a Kubernetes environment. This application requires more than 100 configuration files to be passed. What is the best solution to do that? Create hostPath volume and mount it to some directory containing config files on the host machine? Or maybe config maps allow passing everything as a single compressed file, and then extracting it in the pod volume?
Maybe helm allows somehow to iterate over some directory, and create automatically one big configMap that will act as a directory?
Any suggestions are welcomed
Create hostPath volume and mount it to some directory containing config files on the host machine
This should be avoided.
Accessing hostPaths may not always be allowed. Kubernetes may use PodSecurityPolicies (soon to be replaced by OPA/Gatekeeper/whatever admission controller you want ...), OpenShift has a similar SecurityContextConstraint objects, allowing to define policies for which user can do what. As a general rule: accessing hostPaths would be forbidden.
Besides, hostPaths devices are local to one of your node. You won't be able to schedule your Pod some place else, if there's any outage. Either you've set a nodeSelector restricting its deployment to a single node, and your application would be done as long as your node is. Or there's no placement rule, and your application may restart without its configuration.
Now you could say: "if I mount my volume from an NFS share of some sort, ...". Which is true. But then, you would probably be better using a PersistentVolumeClaim.
Create automatically one big configMap that will act as a directory
This could be an option. Although as noted by #larsks in comments to your post: beware that ConfigMaps are limited in terms of size. While manipulating large objects (frequent edit/updates) could grow your etcd database size.
If you really have ~100 files, ConfigMaps may not be the best choice here.
What next?
There's no one good answer, not knowing exactly what we're talking about.
If you want to allow editing those configurations without restarting containers, it would make sense to use some PersistentVolumeClaim.
If that's not needed, ConfigMaps could be helpful, if you can somewhat limit their volume, and stick with non-critical data. While Secrets could be used storing passwords or any sensitive configuration snippet.
Some emptyDir could also be used, assuming you can figure out a way to automate provisioning of those configurations during container startup (eg: git clone in some initContainer, and/or some shell script contextualizing your configuration based on some environment variables)
If there are files that are not expected to change over time, or whose lifecycle is closely related to that of the application version shipping in your container image: I would consider adding them to my Dockerfile. Maybe even add some startup script -- something you could easily call from an initContainer, generating whichever configuration you couldn't ship in the image.
Depending on what you're dealing with, you could combine PVC, emptyDirs, ConfigMaps, Secrets, git stored configurations, scripts, ...
Problem:
I want every pod created in my cluster to hold\point the same data
e.g. let's say I want all of them to have an env vars like "OWNER=MYNAME".
there are multiple users in my cluster and I don't want them to start changing their YAMLs and manually assign OWNER:MYNAME to env.
Is there a way to have all current/future pods to be assigned automatically with a predefined value or mount a configmap so that the same information will be available in every single pod?
can this be done on the cluster level? namespace level?
I want it to be transparent to the user, meaning a user would apply whatever pod to the cluster, and the info could be available to him without even asking.
Thanks, everyone!
Pod Preset might help you here to partially achieve what you need. Pod Preset resource allows injecting additional runtime requirements into a Pod at creation time. You use label selectors to specify the Pods to which a given PodPreset applies.
Check this to know how pod preset works.
First you need to enable pod preset in your cluster.
You can use Pod Preset to inject env variables or volumes in your pod.
You can also inject configmap in your pod.
Make use of some common label for all the pods which you want to have common config, use this common label in your pod preset resource.
Unfortunately there are plans to remove pod presets altogether in coming releases, but I guess you can still use it with current releases. Although there are other implementations similar to pod presets, which you can try.
I found specifying like kubectl --context dev --namespace default {other commands} before kubectl client in many examples. Can I get a clear difference between them in a k8's environment?
The kubernetes concept (and term) context only applies in the kubernetes client-side, i.e. the place where you run the kubectl command, e.g. your command prompt.
The kubernetes server-side doesn't recognise this term 'context'.
As an example, in the command prompt, i.e. as the client:
when calling the kubectl get pods -n dev, you're
retrieving the list of the pods located under the namespace 'dev'.
when calling the kubectl get deployments -n dev, you're
retrieving the list of the deployments located under the namespace 'dev'.
If you know that you're targetting basically only the 'dev' namespace at the moment, then instead of adding "-n dev" all the time in each of your kubectl commands, you can just:
Create a context named 'context-dev'.
Specify the namespace='dev' for this context.
Set the current-context='context-dev'.
This way, your commands above will be simplified to as followings:
kubectl get pods
kubectl get deployments
You can set different contexts, such as 'context-dev', 'context-staging', etc., whereby each of them is targeting different namespace. BTW it's not obligatory to prefix the name with 'context'. You can just set the name with 'dev', 'staging', etc.
Just as an analogy where a group of people are talking about films. So somewhere within the conversation the word 'Rocky' was used. Since they're talking about films, it's clear and there's no ambiguity that 'Rocky' here refers to the boxing film 'Rocky' and not about the "bumpy, stony" terrains. It's redundant and unnecessary to mention 'the movie Rocky' each time. Just one word, 'Rocky', is enough. The context is obviously about film.
The same thing with Kubernetes and with the example above. If the context is already set to a certain cluster and namespace, it's redundant and unnecessary to set and / or mention these parameters in each of your commands.
My explanation here is just revolving around namespace, but this is just an example. Other than specifying the namespace, within the context you will actually also specify which cluster you're targeting and the user info used to access the cluster. You can have a look inside the ~/.kube/config file to see what information other than the namespace is associated to each context.
In the sample command in your question above, both the namespace and the context are specified. In this case, kubectl will use whatever configuration values set within the 'dev' context, but the namespace value specified within this context (if it exists) will be ignored as it will be overriden by the value explicitly set in the command, i.e. 'default'.
Meanwhile, the namespace concept is used in both sides: server and client sides. It's a logical grouping of Kubernetes objects. Just like how we group files inside different folders in Operating Systems.
You use multiple contexts to target multiple different Kubernetes clusters.You can quickly switch between clusters by using the kubectl config use-context command.
Namespaces are a way to divide cluster resources between multiple users (via resource quota).Namespaces are intended for use in environments with many users spread across multiple teams, or projects.
A context in Kubernetes is a group of access parameters. Each context contains a Kubernetes cluster, a user, and a namespace. The current context is the cluster that is currently the default for kubectl: all kubectl commands run against that cluster. Each of the context that have been used will be available on your .kubeconfig.
Meanwhile a namespace is a way to support multiple virtual cluster within the same physical cluster. This usually will be related to resource quota as well as RBAC management.
A context is the connection to a specific cluster (username/apiserver host) used by kubectl. You can manage multiple clusters that way.
Namespace is a logical partition inside a specific cluster to manage resources and constraints.
I have a containerized daemon that I need to run one instance of for every thing. Each thing has a unique set of configs associated with it, but the container image is the same. The configs can be set simply as environment variables. I have a list of the configs, and I need to define the desired state as having exactly 1 pod running for each thing. What is the appropriate way to construct this in Kubernetes with or without Helm?
My understanding is that ReplicaSets and Deployments work on identical containers, in other words they would all be spun up with the same environment variables? I understand that StatefulSet may be able to represent this, but the deamons do not need to hold state really, they do not need persistent storage, they can be killed at will, so long as another with the same configs comes up soon afterwards.
One clue I was given by somebody was to use Helmfile or Helm partials. That is the extent of what they told me. I have not yet investigated whether those are appropriate or not.
You are correct saying that Deployment and ReplicaSets are running on identical containers, so the way I see it you have 2 options:
Deploy multiple deployments with different configs defined in the values file:
You can see an example here, where multiple configs are set in the values file and using {{ range }} to iterate and create multiple deployments
Iterate over you configurations names/files using scripting language of your choice and create separate release for each of your configuration via the command line for example: --set configName=
Personally, I would go with the 2nd option since multiple helm releases can harness the helm cli to better understand what is running and it's state. also, any CRUD action you would like to do would be less dangerous since the deployments are decoupled
I have a requirement where my client applications are having almost same properties and even the URL is same, as they are running behind a load balancer, the only change they have is a particular set of environment properties that differ.
Is it possible to register them uniquely based on that property.
I would say there are a few approaches.
One would be loading Environment Variables from a Kubernetes Secret.
Second using helm(https://helm.sh/)
Helm helps you manage Kubernetes applications — Helm Charts help you define, install, and upgrade even the most complex Kubernetes application.
Charts are easy to create, version, share, and publish — so start using Helm and stop the copy-and-paste.
Explanation:
If you would use a secret option, you would probably create two separate secrets with env variables that you need and load those based on the app name, or if you have them setup in different namespaces then copy the secret over to each as those resources will not work between different namespaces.
If you would use helm, you will have to write your chart and put the env variables into values.yaml or mix it together and load secret from inside Kubernetes.
This will work on Kubernetes, I do not know (based on your tags) if it's the same on OpenShift.
Please provide some samples of what you have already done and I'll provide more details.