I'm trying to set multiline values (contents of a ca certificate file) to kustomize environment file for a particular key as displayed in the code below.
Is there a way to achieve this?
Note: Adding quotes to the value in some_params.env isn't working.
kustomize.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
commonLabels:
role: authentication
bases:
- ./somebase
configMapGenerator:
- name: some-parameters
env: some_params.env
vars:
- name: ca_contents
objref:
kind: ConfigMap
name: some-parameters
apiVersion: v1
fieldref:
fieldpath: data.ca_contents
configurations:
- some_params.yaml
some_params.yaml
varReference:
- path: data/ca.pem
kind: ConfigMap
some_params.env
ca_contents= |
-----BEGIN CERTIFICATE-----
YOUR CA CERTIFICATE CONTENTS
-----END CERTIFICATE-----
Running the following command:
kustomize build base
Returns:
Error: NewResMapFromConfigMapArgs: NewResMapFromConfigMapArgs: env source file: some_params.env: "-----BEGIN CERTIFICATE-----" is not a valid key name: a valid environment variable name must consist of alphabetic characters, digits, '_', '-', or '.', and must not start with a digit (e.g. 'my.env-name', or 'MY_ENV.NAME', or 'MyEnvName1', regex used for validation is '[-._a-zA-Z][-._a-zA-Z0-9]*')
So it looks like you're creating ConfigMaps using Generators in Kustomize. And in doing so you're attempting to pass the contents of the desired ConfigMap to it directly in your kustomization.yml file.
Better practice would be to instead save the contents of these configuration files to disk and maintain them in the same location as your manifests. Then import them from file into your generators in Kustomize. For example:
configMapGenerator:
- name: ca_contents
files:
- ca.crt
This will produce a ConfigMap with the name ca.crt with the contents of the file.
However, if you wanted, you can do the same thing with the literals operator and use multiline configuration directly in your kustomization.yml. For example:
configMapGenerator:
- name: ca_contents
literals:
- ca.crt=|
CERT_CONTENTS
EACH_NEWLINE_IS_INDENTED_ANOTHER_TIME
You_Can_Add_Additional_literals
- ca.key=>
AND_EVEN
CONCATENATE_THEM
_ONTO_ONE_LINE_WITH_OTHER_OPERATORS_LIKE_>
- cert.crt="You can even mix literals with files in generators."
files:
- cert.key
Let me know if this makes sense. I advise you keep your configuration in files and import them into generators but either solution should work. Feel free to reach out if you need any other help. Kustomize is a really cool project!
Related
A common requirement when deploying Kubernetes manifests to a cluster is to prefix the container names with a trusted registry prefix that mirrors the allowed images. Usually used along with an admission controller.
Is there a sensible way to do this using Kustomize without having to list every single image by name in the kustomization.yaml images: transformer stanza?
Given this kustomization.yaml:
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- "https://github.com/prometheus-operator/kube-prometheus"
if I want to prefix all the images it references with mytrusted.registry/ I need to append this to my kustomization.yaml:
images:
- name: grafana/grafana
newName: mytrusted.registry/grafana/grafana
- name: jimmidyson/configmap-reload
newName: mytrusted.registry/jimmidyson/configmap-reload
- name: k8s.gcr.io/kube-state-metrics/kube-state-metrics
newName: mytrusted.registry/k8s.gcr.io/kube-state-metrics/kube-state-metrics
- name: k8s.gcr.io/prometheus-adapter/prometheus-adapter
newName: mytrusted.registry/k8s.gcr.io/prometheus-adapter/prometheus-adapter
- name: quay.io/brancz/kube-rbac-proxy
newName: mytrusted.registry/quay.io/brancz/kube-rbac-proxy
- name: quay.io/prometheus/alertmanager
newName: mytrusted.registry/quay.io/prometheus/alertmanager
- name: quay.io/prometheus/blackbox-exporter
newName: mytrusted.registry/quay.io/prometheus/blackbox-exporter
- name: quay.io/prometheus/node-exporter
newName: mytrusted.registry/quay.io/prometheus/node-exporter
- name: quay.io/prometheus-operator/prometheus-operator
newName: mytrusted.registry/quay.io/prometheus-operator/prometheus-operator
- name: quay.io/prometheus/prometheus
newName: mytrusted.registry/quay.io/prometheus/prometheus
which I generated with this putrid, fragile monstrosity (which WILL break if your containers are specified by hash, or you have a port in your registry prefix):
kustomize build | \
grep 'image:' | \
awk '$2 != "" { print $2}' | \
sort -u | \
cut -d : -f 1 | \
jq --raw-input '{ name: ., newName: ("mytrusted.registry/" + .) }' | \
faq -s -fjson -oyaml '{ images: .}'
(Note that the above will also NOT WORK completely, because Kustomize doesn't recognise images outside PodTemplates, such as those in the kind: Alertmanager spec.image or the kind: Prometheus spec.image; it'd still be better than the current situation).
What I want instead is to able to express this in the image transformer without generating and maintaining lists of images, with something like the imaginary, does not work example:
images:
- name: "(*)"
newName: "mytrusted.registry/$1"
i.e. use a capture group. Or something functionally similar, like an image transformer "prependName" option or similar.
This must be such a common problem to have, but I can't for the life of me find a well established way this is done by convention in the k8s world. Just lots of DIY fragile hacks.
This answer is probably too late to help the original asker, but maybe it will help others who stumble upon this question through Google, like I did.
Kustomize has a built-in PrefixTransformer that can add a prefix to all your images, or indeed to any arbitrary field in your specs.
Create a file named image-prefix.yaml with the following contents:
apiVersion: builtin
kind: PrefixTransformer
metadata:
name: image-prefix
prefix: mytrusted.registry/
fieldSpecs:
- path: spec/template/spec/containers/image
- path: spec/template/spec/initContainers/image
- path: spec/image # for kind Prometheus and Alertmanager
Then add this transformer to your kustomization.yaml as follows:
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- "https://github.com/prometheus-operator/kube-prometheus"
transformers:
- image-prefix.yaml
That should do it.
When you build this, you should see your prefix automatically added to all the images:
$ kubectl kustomize | grep image:
...
image: mytrusted.registry/quay.io/prometheus/blackbox-exporter:v0.22.0
image: mytrusted.registry/jimmidyson/configmap-reload:v0.5.0
image: mytrusted.registry/quay.io/brancz/kube-rbac-proxy:v0.13.0
image: mytrusted.registry/grafana/grafana:my-tag
image: mytrusted.registry/k8s.gcr.io/kube-state-metrics/kube-state-metrics:v2.6.0
...
I tested this with kubectl 1.25 and the version of Kustomize that comes bundled with it:
$ kubectl version --short --client
...
Client Version: v1.25.0
Kustomize Version: v4.5.7
You can further restrict the PrefixTransformer by using GVK (group/version/kind) triplets. For example, if for some reason you wanted to apply your image prefix only to Deployments, but not to DaemonSets, StatefulSets, or others, you would put something like this in your image-prefix.yaml file:
fieldSpecs:
- kind: Deployment
path: spec/template/spec/containers/image
- kind: Deployment
path: spec/template/spec/initContainers/image
Also note that the ImageTransformer runs before the PrefixTransformer, so if you wanted to override the tag of a particular image in your kustomization.yaml, you should use the original image name without the prefix:
images:
- name: grafana/grafana
newTag: my-tag
Unfortunately there is no clear documentation for PrefixTransformer that I could find, or I would have linked it here. I discovered all this by digging through Kustomize source code.
There are quite a few other built-in transformers that might be of interest, you can glean their usage by looking at the *_test.go files in each of the subfolders here:
https://github.com/kubernetes-sigs/kustomize/tree/master/plugin/builtin
As I know the way to create a configMap in Kubernetes from a file is to use:
--from-file option for kubectl
What I am looking for is a way to only load part of the yaml file into the configMap.
Example:
Let's say I have this yml file:
family:
Boys:
- name: Joe
- name: Bob
- name: dan
Girls:
- name: Alice
- name: Jane
Now I want to create a configMap called 'boys' which will include only the 'Boys' section.
Possible?
Another thing that could help if the above is not possible is when I am exporting the configMap as environment variables to a pod (using envFrom) to be able to only export part of the configMap.
Both options will work for me.
Any idea?
The ConfigMap uses a key and value for its configuration. Based on your example, you get multiple arrays of data where there are multiple values with their own keys. But you can create multiple ConfigMap from different file for these issues.
First you need to create .yaml files to create a ConfigMap guided by the documentation.
First file call Boys.yaml
# Env-files contain a list of environment variables.
# These syntax rules apply:
# Each line in an env file has to be in VAR=VAL format.
# Lines beginning with # (i.e. comments) are ignored.
# Blank lines are ignored.
# There is no special handling of quotation marks (i.e. they will be part of the ConfigMap value)).
name=Joe
name=Bob
name=Dan
Second file call Girls.yaml
name=Alice
name=Jane
Create your ConfigMap
kubectl create configmap NmaeOfYourConfigmap --from-env-file=PathToYourFile/Boys.yaml
where the output is similar to this:
apiVersion: v1
kind: ConfigMap
metadata:
creationTimestamp:
name: NmaeOfYourConfigmap
namespace: default
resourceVersion:
uid:
data:
name: Joe
name: Bob
name: Dan
Finally, you can pass these ConfigMap to pod or deployment using configMapRef entries:
envFrom:
- configMapRef:
name: NmaeOfYourConfigmap-Boys
- configMapRef:
name: NmaeOfYourConfigmap-Girls
Configmaps cannot contain rich yaml data. Only key value pairs. So if you want to have a list of things, you need to express this as a multiline string.
With that in mind you could use certain tools, such a yq to query your input file and select the part you want.
For example:
podman run -rm --interactive bluebrown/tpl '{{ .family.Boys | toYaml }}' < fam.yaml \
| kubectl create configmap boys --from-file=Boys=/dev/stdin
The result looks like this
apiVersion: v1
kind: ConfigMap
metadata:
name: boys
namespace: sandbox
data:
Boys: |+
- name: Joe
- name: Bob
- name: dan
You could also encode the file or part of the file with base64 and use that as an environment variable, since you get a single string, which is easily processable, out of it. For example:
$ podman run --rm --interactive bluebrown/tpl \
'{{ .family.Boys | toYaml | b64enc }}' < fam.yaml
# use this string as env variable and decode it in your app
LSBuYW1lOiBKb2UKLSBuYW1lOiBCb2IKLSBuYW1lOiBkYW4K
Or with set env which you could further combine with dry run if required.
podman run --rm --interactive bluebrown/tpl \
'YAML_BOYS={{ .family.Boys | toYaml | b64enc }}' < fam.yaml \
| kubectl set env -e - deploy/myapp
Another thing is, that YAML is a superset of JSON, in many cases you are able to convert YAML to JSON or at least use JSON like syntax.
This can be useful in such a scenario in order to express this as a single line string rather than having to use multiline syntax. It's less fragile.
Every YAML parser will be able to parse JSON just fine. So if you are parsing the string in your app, you won't have problems.
$ podman run --rm --interactive bluebrown/tpl '{{ .family.Boys | toJson }}' < fam.yaml
[{"name":"Joe"},{"name":"Bob"},{"name":"dan"}]
Disclaimer, I created the above used tool tpl. As mentioned, you might as well use alternative tools such as yq.
I use kustomize to generate various configMap as the following example: -
# kustomization.yaml
configMapGenerator:
- name: my-config-path # <-- My original name.
files:
- file1.txt
- file2.txt
- ...
- fileN.txt
The output is as the following example: -
# my-configmap.yaml
apiVersion: v1
data:
file1.txt: |
...
file2.txt: |
...
fileN.txt |
...
kind: ConfigMap
metadata:
name: my-config-path-mk89db6928 # <-- There is a hashed value appended at the end.
Oh the other hand, I have a third-party helm which requires to override the values.yaml as the following: -
# third-party-chart-values.yaml
customArguments:
- --config.dir=/config
...
volumes:
- mountPath: /config
name: my-config-path # <--- Here, I would like to replace with `my-config-path-mk89db6928`.
type: configMap
Do we have any way to replace the my-config-path at the third-party-chart-values.yaml with the generated value my-config-path-mk89db6928 from the file named my-configmap.yaml? (Note, if the configMap is changed the hashed, e.g. mk89db6928 also be changed, too.)
Do you want to keep the hash? Because you could prevent it with:
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
generatorOptions:
disableNameSuffixHash: true
Then you'd have a predictable name. Although that would cause a collision if you installed the same helm chart more than once in the same namespace.
I am trying to add config data as environment variables, but Kubernetes warns about invalid variable names. The configmap data contains JSON and property files.
spec:
containers:
- name: env-var-configmap
image: nginx:1.7.9
envFrom:
- configMapRef:
name: example-configmap
After deploying I do not see them added in the process environment. Instead I see a warning message like below
Config map example-configmap contains keys that are not valid environment variable names. Only config map keys with valid names will be added as environment variables.
But I see it works if I add it directly as a key-value pair
env:
# Define the environment variable
- name: SPECIAL_LEVEL_KEY
valueFrom:
configMapKeyRef:
# The ConfigMap containing the value you want to assign to SPECIAL_LEVEL_KEY
name: special-config
# Specify the key associated with the value
key: special.how
I have thousand of key values in the configmap data and I could not add them all as separate key-value pairs.
Is there any short syntax to add all values from a configmap as environment variables?
My answer, while #P-Ekambaram already helped you out, I was getting the same error message, it turned out that my issue was that I named the configMap ms-provisioning-broadsoft-adapter and was trying to use ms-provisioning-broadsoft-adapter as the key. As soon as I changed they key to ms_provisioning_broadsoft_adapter, e.g. I added the underscores instead of hyphens and it happily let me add it to an application.
Hope this might help someone else that also runs into the error invalid variable name cannot be added as environmental variable
sample reference is given below
create configmap as shown below
apiVersion: v1
kind: ConfigMap
metadata:
name: special-config
namespace: default
data:
SPECIAL_LEVEL: very
SPECIAL_TYPE: charm
load configmap data as environment variables in the pod
apiVersion: v1
kind: Pod
metadata:
name: dapi-test-pod
spec:
containers:
- name: test-container
image: k8s.gcr.io/busybox
command: [ "/bin/sh", "-c", "env" ]
envFrom:
- configMapRef:
name: special-config
restartPolicy: Never
output
master $ kubectl logs dapi-test-pod | grep SPECIAL
SPECIAL_LEVEL=very
SPECIAL_TYPE=charm
You should rename your variable.
In my case they were like this:
VV-CUSTOMER-CODE
VV-CUSTOMER-URL
I just rename to:
VV_CUSTOMER_CODE
VV_CUSTOMER_URL
Works fine. Openshift/kubernets works with underline _, but not with hyphen - .
I hope help you.
I've created a secret from a file using a command like:
kubectl create secret generic laravel-oauth \
--from-file=./.work-in-progress/oauth_private.key \
--from-file=./.work-in-progress/oauth_public.key
However it seems new lines are stripped from the files (when using the secrets as ENV variables).
There is a 'encoding' note in the docs that state:
The serialized JSON and YAML values of secret data are encoded as
base64 strings. Newlines are not valid within these strings and must
be omitted. When using the base64 utility on Darwin/macOS users should
avoid using the -b option to split long lines. Conversely Linux users
should add the option -w 0 to base64 commands or the pipeline base64 |
tr -d '\n' if -w option is not available.
However I assumed this only applies for 'manually' created secrets via YAML files.
The new lines are not stripped the files are just base64 encoded as mentioned in the other answers too. For example:
# mycert.pem
-----BEGIN CERTIFICATE-----
xxxxxx
xxxxxx
...
-----END CERTIFICATE-----
Then:
$ kubectl create secret generic mysecret --from-file=./cert.pem
Then:
$ kubectl get secret mysecret -o=yaml
apiVersion: v1
data:
cert.pem: <base64 encoded string>
kind: Secret
metadata:
creationTimestamp: 2018-11-14T18:11:46Z
name: mysecret
namespace: default
resourceVersion: "20180431"
selfLink: /api/v1/namespaces/default/secrets/mysecret
uid: xxxxxx
type: Opaque
Then if you decode it, you will get the original secret.
$ echo '<base64 encoded string>' | base64 -D
-----BEGIN CERTIFICATE-----
xxxxxx
xxxxxx
...
-----END CERTIFICATE-----
Also, this is not necessarily secure at rest. If you are looking for more security you can use something like Hashicorp Vault or as alluded by #Alex Bitnami's sealed secrets.
The note you refer to is for the base64 encoded string itself (not the content that was encoded).
Using secrets as env var will potentially expose them via the dashboard "preview eye" (if you use the Kube Dashboard), you should mount them into a directory and make the app load them from there instead; I fell for that too and was surprised I was able to view the secret.
I've not come across the stripping of new line characters, as the above command would simply do a base64 of the content (including new line chars). That said, storing the secrets b64 encoded is not exactly safe either, you should consider using sealed-secrets (bitnami) instead, it works just like normal secrets, but is actually encrypted at rest.
HTH,
Alex
It seems newlines work fine (maybe I ran into another issue earlier).
Here is a full example:
#!/usr/bin/env bash
set -euo pipefail
printf "123\n456\n789" > ./.work-in-progress/example.txt
kubectl create secret generic example-test \
--from-file=./.work-in-progress/example.txt \
--dry-run -o yaml | kubectl apply -f -
cat <<EOF | kubectl apply -f -
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: example
labels:
app: example
spec:
replicas: 1
selector:
matchLabels:
app: example
template:
metadata:
labels:
app: example
spec:
volumes:
- name: example-test-volume
secret:
secretName: example-test
containers:
- name: app
command: ["sleep", "99999999"]
image: busybox:latest
imagePullPolicy: IfNotPresent
volumeMounts:
- name: example-test-volume
mountPath: /tmp/example
env:
- name: exampleenv
valueFrom:
secretKeyRef:
name: example-test
key: example.txt
EOF