Is there an easy way to share a set of environment variables (coming in from various config maps and secrets) between the same container in a deployment and a cron job?
I'm using Kustomize, but I can't figure out how to approach since with a patch the patch itself would be a bit different depending on if patching a deployment or a cron job.
For example, in my deployment I have something like this:
spec:
containers:
- name: my_app
env:
name: SOME_SECRET
valueFrom:
secretKeyRef:
name: my_secret
key: my_key
envFrom:
- configmapRef:
name: some_config_map
I'd like to also have this applied to a cron job, but since the YAML schema is different between a deploymnet and a cron job I'm not sure if this is something that is possible/supported.
Thanks!
Don't know if it can be done with Kustomize or not...
But have you considered using helm instead? You can have dedicated templates for deployment and cron job and use the same value for both of their container env field...
Related
We want to use the official Tekton buildpacks task from Tekton Hub to run our builds using Cloud Native Buildpacks. The buildpacks documentation for Tekton tells us to install the buildpacks & git-clone Task from Tekton Hub, create Secret, ServiceAccount, PersistentVolumeClaim and a Tekton Pipeline.
As the configuration is parameterized, we don't want to start our Tekton pipelines using a huge kubectl command but instead configure the PipelineRun using a separate pipeline-run.yml YAML file (as also stated in the docs) containing the references to the ServiceAccount, workspaces, image name and so on:
apiVersion: tekton.dev/v1beta1
kind: PipelineRun
metadata:
name: buildpacks-test-pipeline-run
spec:
serviceAccountName: buildpacks-service-account # Only needed if you set up authorization
pipelineRef:
name: buildpacks-test-pipeline
workspaces:
- name: source-workspace
subPath: source
persistentVolumeClaim:
claimName: buildpacks-source-pvc
- name: cache-workspace
subPath: cache
persistentVolumeClaim:
claimName: buildpacks-source-pvc
params:
- name: image
value: <REGISTRY/IMAGE NAME, eg gcr.io/test/image > # This defines the name of output image
Now running the Tekton pipeline once is no problem using kubectl apply -f pipeline-run.yml. But how can we restart or reuse this YAML-based configuration for all the other pipelines runs?
There are some discussions about that topic in the Tekton GitHub project - see tektoncd/pipeline/issues/664 and tektoncd/pipeline/issues/685. Since Tekton is heavily based on Kubernetes, all Tekton objects are Kubernetes CRDs - which are in fact immutable. So it is intended to not be able to re-run an already run PipelineRun.
But as also discussed in tektoncd/pipeline/issues/685 we can simply use the generateName variable of the metadata field like this:
apiVersion: tekton.dev/v1beta1
kind: PipelineRun
metadata:
generateName: buildpacks-test-pipeline-run-
spec:
serviceAccountName: buildpacks-service-account # Only needed if you set up authorization
pipelineRef:
name: buildpacks-test-pipeline
workspaces:
- name: source-workspace
subPath: source
persistentVolumeClaim:
claimName: buildpacks-source-pvc
- name: cache-workspace
subPath: cache
persistentVolumeClaim:
claimName: buildpacks-source-pvc
params:
- name: image
value: <REGISTRY/IMAGE NAME, eg gcr.io/test/image > # This defines the name of output image
Running kubectl create -f pipeline-run.yml will now work multiple times and kind of "restart" our Pipeline, while creating a new PipelineRun object like buildpacks-test-pipeline-run-dxcq6 everytime the command is issued.
Keep in mind to delete old PipelineRun objects once in a while though.
tkn cli has the switch --use-pipelinerun to the command tkn pipeline start, what this command does is to reuse the params/workspaces from that pipelinerun and create a new one, so effectively "restarting" it.
so to 'restart' the pipelinerun pr1 which belong to the pipeline p1 you would do:
tkn pipeline start p1 --use-pipelinerun pr1
maybe we should have a easier named command, I kicked the discussion sometime ago feel free to contribute a feedback :
https://github.com/tektoncd/cli/issues/1091
You cannot restart a pipelinerun.
Since in tekton, a pipelinerun is one time execution for a pipeline(treat as template), so it should not able to be restart, another kubectl apply for pipelinerun is another execution...
I am currently creating pods on AKS from a net core project. The problem is that I have a secret generated from appsettings.json that I created previously in the pipeline. During the deployment phase I load this secret inside a volume of the pod itself. What I want to achieve is to read the values from the Kubernetes secret and load them as env variables inside the helm chart. Any help is appreciated Thanks :)
Please see how you can use secret as environmental variable
As a single variable
containers:
- name: mycontainer
image: redis
env:
- name: SECRET_USERNAME
valueFrom:
secretKeyRef:
name: mysecret
key: username
Or the whole secret
containers:
- name: test-container
image: k8s.gcr.io/busybox
command: [ "/bin/sh", "-c", "env" ]
envFrom:
- secretRef:
name: mysecret
Your secrets should not be in your appsettings.json because they will end up in your source control repository.
Reading secrets from k8s into helm chart is something you should never attempt to do.
Ideally your secrets sit in a secure secret store (a vault) that either has an API that your k8s hosted app(s) can call into
Or (the vault) has an integration with k8s which mounts your secrets as a volume in your pods (the volume is an in-memory read-only storage).
This way your secrets are only kept in the vault which ensures the secrets are encrypted while at rest and in transit.
I am creating a Helm chart that depends on several Helm charts that are not maintained by me, and I would like to make some configurations to these subcharts. The configurations are not too complex, I just want to add a several environmental variables to each of the containers. However, the env fields of the containers are not already templated out in the Helm charts. I would like to avoid forking these charts and maintaining them myself since this is such a trivial change.
Is there an easy way to provide environmental variables to several containers in Kubernetes in a flexible way, either through Helm or another tool?
I am currently looking into using Kustomize to do the last mile changes after Helm fills out the templates, but I am getting hung up on setting up Kustomize patches. In my scenario, I have the environmental variables being filled out by Helm in a ConfigMap. I would like to add an envFrom field to read the ConfigMap and add the given environment variables to the containers. I want to add the envFrom to the resource YAML files through Kustomize. The snag I am hitting is that Kustomize patch.yaml files are resource specific. Below is an example of my patch.yaml and my kustomization.yaml respectively.
patch.yaml:
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: does-not-matter
spec:
template:
spec:
containers:
- name: server
envFrom:
- configMapRef:
name: my-env
kustomization.yaml:
resources:
- all.yaml
patches:
- path: patch.yaml
target:
kind: "StatefulSet"
name: "*"
To perform the Kustomization, I run:
helm install perceptor ../ --post-renderer ./kustomize
Which basically just fills out the Helm templates and passes them to Kustomize to do the last-mile patches.
In the patch, I have to specify the name of the container ("server") to properly inject my configMap. What I would really like to do is be able to provide those environment variables to all containers in a given deployment (as defined by the target constraints in kustomization.yaml), regardless of their name. From what I have seen, it almost looks like I will have to write a separate patch for each container, which is suboptimal. I just start working with Kubernetes, so it is possible that I am missing something that would easily solve this problem.
I understand, that you don't want to break the open/closed principle of subchart your umbrella chart depends on by forking it, but still you have a right to propose a changes to it by making it more extension-able and flexible. Yes, I would suggest you to submit a Pull Request/request a new Feature to the helm chart project in context.
The following code snippet won't break current functionality, and give users a chance to introduce custom environment variables based on existing ConfigMap(s) in desired resource's Spec.
helm_template.yaml
#helm template
...
env:
- name: POD_NAME
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.name
- name: POD_NAMESPACE
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.namespace
{{- if .Values.envConfigs }}
{{- range $key, $config := $.Values.envConfigs }}
- name: {{ $key }}
valueFrom:
configMapKeyRef:
name: {{ $config }}
key: {{ $key | quote }}
{{- end }}
{{- end }}
values.yaml
#
# values.yaml
#
envConfigs:
Q3_CFG_MAP: Q3DM17
Q3_CFG_TIMEOUT: 30
# if empty use:
# envConfigs: {}
I have some CATALINA_OPTS properties (regarding database port, user and so on) set up in ConfigMap file. Then, this file is added to the docker image via Pod environment variable.
One of the CATALINA_OPTS properties is database password, and it is required to move this from ConfigMap to the Secrets file.
I can expose key from Secrets file through environment variable:
apiVersion: v1
kind: Pod
...
containers:
- name: myContainer
image: myImage
env:
- name: CATALINA_OPTS
valueFrom:
configMapKeyRef:
name: catalina_opts
key: CATALINA_OPTS
- name: MY_ENV_PASSWORD
valueFrom:
secretKeyRef:
name: db-pass
key: my-pass
Thing is, i need to append this password to the CATALINA_OPTS. I tried to do it in Dockerfile:
RUN export CATALINA_OPTS="$CATALINA_OPTS -Dmy.password=$MY_ENV_PASSWORD"
However, MY_ENV_PASSWORD is not appending to the existing CATALINA_OPTS. When I list my environment variables (i'm checking the log in Jenkins) i cannot see the password.
Am I doing something wrong here? Is there any 'regular' way to do this?
Dockerfile RUN steps are run as part of your image build step and NOT during your image execution. Hence, you cannot rely on RUN export (build step) to set K8S environment variables for your container (run step).
Remove the RUN export from your Dockerfile and Ensure you are setting CATALINA_OPTS in your catalina_opts ConfigMap like this:
apiVersion: v1
kind: ConfigMap
metadata:
name: catalina_opts
data:
SOME_ENV_VAR: INFO
CATALINA_OPTS: opts... -Dmy.password=$MY_ENV_PASSWORD
Use case:
I want to be able to re-run a job from where the first job left off. I am using Helm to deploy into Kubernetes.
I have the idea of saving the state of the first job in a ConfigMap. The ConfigMap yaml defining the ConfigMap is packaged up with the job and both are deployed at the same time with Helm.
apiVersion: v1
kind: ConfigMap
metadata:
name: NameOfMyConfigMap
data:
someKey: someValue
MY_STATE: state <---- See below as to whether this should be included or not
The job is run with an ENV variable set from the ConfigMap:
env:
- name: MY_STATE
valueFrom:
configMapKeyRef:
name: NameOfMyConfigMap
key: MY_STATE
The job runs a script that looks to see if $MY_STATE is set and if it is not set then the job is being run for the first time, otherwise the job closes down the already running first job, saves the first job's state into the MY_STATE ConfigMap variable and launches the job again using the saved state.
If I don't declare the MY_STATE key in the initial ConfigMap definition then the first run of the job will fail, as the ENV definition above cannot find the ConfigMap variable.
If I do declare the value (MY_STATE: "") in the ConfigMap definition, then the first deployment will work. However, if I re-deploy the job with helm upgrade then does the value I enter in the definition not overwrite an existing value in the existing ConfigMap?
What is the best method of storing state in between runs of the same job?
Have you tried using volumes? In this case it should not be overwritten when using helm upgrade.
Could an example like this work? (From
https://groups.google.com/forum/#!msg/kubernetes-users/v2806ezEdPk/1geJCO8-AQAJ)
apiVersion: batch/v1
kind: Job
metadata:
name: keystore-configmap-job
spec:
template:
metadata:
name: keystore-configmap
spec:
containers:
- name: keystore
image: ubuntu
volumeMounts:
- name: keystore-configmap-volume
mountPath: /config-base64
command: [ "sh", "-c", "cat /config-base64/keystore.jks | base64 --decode | sha256sum" ]
restartPolicy: Never
volumes:
- name: keystore-configmap-volume
configMap:
name: keystore-configmap