Kustomize/Kubernetes - Nested envFrom injection - kubernetes

I'm trying to add envs injection based on envFrom
A simplified structure looks something like that:
├── base
│   ├ ─ backend
│   ├── backend.properties
│   ├── app1
│   │   ├── app1_backend.properties
├ ── deployment.yaml
│   │   ├── ingress.yaml
│   │   ├── kustomization.yaml
├── common.properties
├── frontend
│   ├── app1
│ ├── app1_frontend.properties
│   │   ├── deployment.yaml
│   │   ├── ingress.yaml
│   │   ├── kustomization.yaml
│   │   └── service.yaml
│   ├── frontend.properties
│   └── kustomization.yaml
└── kustomization.yaml
I would like to generate properties on the main level(common), backend/frontend level, and particular app level.
So I was trying to add the following patch on main level and it works:
- op: add
path: /spec/template/spec/containers/0/envFrom
value:
- configMapRef:
name: common-properties
and following code to nested directories(backend/frontend/particular app)
- op: add
path: "/spec/template/spec/containers/0/envFrom/-"
value:
configMapRef:
name: backend-properties
But it doesn't work with the following error:
add operation does not apply: doc is missing path: "/spec/template/spec/containers/0/envFrom/-": missing value
I have seen some examples on GitHub where that syntax was used: https://github.com/search?l=YAML&p=1&q=%2Fspec%2Ftemplate%2Fspec%2Fcontainers%2F0%2FenvFrom%2F-&type=Code (you have to be logged in to see results) And I'm not sure this stopped work on specific Kustomize version(I'm using the newest version - 4.5.3) or it never worked
I have already written some Kustomize patches and syntax with /- to resources usually worked fine to resources that already exist on the manifest.
It's possible to inject that envFrom on different levels?

It's hard to diagnose your problem without a reproducible example, but if I start with this Deployment:
apiVersion: apps/v1
kind: Deployment
metadata:
name: example
spec:
replicas: 1
selector:
matchLabels:
app: example
template:
spec:
containers:
- name: example
image: docker.io/alpine:latest
envFrom:
- configMapRef:
name: example-config
And use this kustomization.yaml, which includes your patch without
changes:
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- deployment.yaml
patches:
- target:
kind: Deployment
name: example
patch: |-
- op: add
path: "/spec/template/spec/containers/0/envFrom/-"
value:
configMapRef:
name: backend-properties
Then everything seems to work and I get the resulting output from
kustomize build:
apiVersion: apps/v1
kind: Deployment
metadata:
name: example
spec:
replicas: 1
selector:
matchLabels:
app: example
template:
spec:
containers:
- envFrom:
- configMapRef:
name: example-config
- configMapRef:
name: backend-properties
image: docker.io/alpine:latest
name: example

Related

Handle "shared" resources with Kustomize

All namespaces in my cluster are supposed to trust the same root CA. I have a mono repo with all my Kustomize files, and I'm trying to avoid having to add the root CA certificate everywhere.
My idea was to go for something like that in my kustomization files:
# [project_name]/[namespace_name]/bases/project/kustomization.yml
configMapGenerator:
- name: trusted-root-ca
files:
- ../../../../root-ca/root-ca.pem
So at least if I want to update the root CA, I do it in one place. This results to
file sources: [../../../../root-ca/root-ca.pem]: security; file 'root-ca/root-ca.pem' is not in or below [project_name]/[namespace_name]/bases/project/
So I guess this is not the way to go. (Reading from Common config across multiple environments and applications with Kustomize I can see why it's behaving like that and disable that behavior seems to be a bad idea). I'm looking for a better way to do this.
This seems like a good place to use a component. For example, if I have my files organized like this:
.
├── components
│   └── trusted-root-ca
│   ├── kustomization.yaml
│   └── root-ca.pem
└── projects
├── kustomization.yaml
├── project1
│   ├── kustomization.yaml
│   └── namespace.yaml
└── project2
├── kustomization.yaml
└── namespace.yaml
And in components/trusted-root-ca/kustomization.yaml I have:
apiVersion: kustomize.config.k8s.io/v1alpha1
kind: Component
configMapGenerator:
- name: trusted-root-ca
files:
- root-ca.pem
generatorOptions:
disableNameSuffixHash: true
Then in projects/project1/kustomization.yaml I can write:
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: project1
components:
- ../../components/trusted-root-ca
resources:
- namespace.yaml
And similarly in projects/project2/kustomization.yaml:
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: project2
components:
- ../../components/trusted-root-ca
resources:
- namespace.yaml
And in projects/kustomization.yaml I have:
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- project1
- project2
Then if, from the top directory, I run kustomize build projects, the output will look like:
apiVersion: v1
kind: Namespace
metadata:
name: project1
spec: {}
---
apiVersion: v1
kind: Namespace
metadata:
name: project2
spec: {}
---
apiVersion: v1
data:
root-ca.pem: |
...cert data here...
kind: ConfigMap
metadata:
name: trusted-root-ca
namespace: project1
---
apiVersion: v1
data:
root-ca.pem: |
...cert data here...
kind: ConfigMap
metadata:
name: trusted-root-ca
namespace: project2

Secret hash issue when run once-off job in kustomize

When using kustomize, I am trying to use job to perform some once-off job. But somehow, the kustomize just doesn't recognise hashed secret. The below is the relevant codes.
.
├── base
│ └── postgres.yaml
├── jobs
│ ├── postgres-cli.yaml
│ └── kustomization.yaml
└── overlays
└── dev
├── kustomization.yaml
└── postgres-secrects.properties
base/postgres.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: postgres
spec:
replicas: 1
selector:
matchLabels:
name: postgres
template:
metadata:
labels:
name: postgres
spec:
containers:
- name: postgres
image: postgres:14-alpine
envFrom:
- secretRef:
name: postgres-secrets
# ...
overlays/dev/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- ../../base/postgres.yaml
secretGenerator:
- name: postgres-secrets
envs:
- postgres-secrets.properties
base/overlays/dev/postgres-secrects.properties
POSTGRES_USER=postgres
POSTGRES_PASSWORD=123
jobs/postgres-cli.yaml
apiVersion: batch/v1
kind: Job
metadata:
name: postgres-cli
spec:
template:
metadata:
name: postgres-cli
spec:
restartPolicy: Never
containers:
- image: my-own-image
name: postgres-cli
envFrom:
- secretRef:
name: postgres-secrets # errors here cannot cannot recognise
# ...
jobs/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Component
commonLabels:
environment: local
resources:
- ./postgres-cli.yaml
To start my stack, I run kubectl apply -k ./overlay/dev.
Then, when I try to run the postgres-cli, I try to run kubectl apply -k /jobs, it complains like below: secret "postgres-secrets" not found
Do we have a way to find the secret back when apply the job?
.
├── base
│ └── postgres.yaml
├── jobs
│ ├── postgres-cli.yaml
│ └── kustomization.yaml
└── overlays
└── dev
├── kustomization.yaml
└── postgres-secrects.properties
Having this structure it means that the secret is in the overlay at dev level. You try to run the job from one level below and the sercret is not there. To run this you have two ways or by kubectl apply -k ./overlay/dev. or by moving the overlay into jobs and create only one kustomization.

Kustomize: how to reference a value from a ConfigMap in another resource/overlay?

I have a couple of overlays (dev, stg, prod) pulling data from multiple bases where each base contains a single service so that each overlay can pick and choose what services it needs. I generate the manifests from the dev/stg/prod directories.
A simplified version of my Kubernetes/Kustomize directory structure looks like this:
├── base
│ ├── ServiceOne
│ │ ├── kustomization.yaml
│ │ └── service_one_config.yaml
│ ├── ServiceTwo
│ │ ├── kustomization.yaml
│ │ └── service_two_config.yaml
│ └── ConfigMap
│ ├── kustomization.yaml
│ └── config_map_constants.yaml
└── overlays
├── dev
│ ├── kustomization.yaml
│ └── dev_patch.yaml
├── stg
│ ├── kustomization.yaml
│ └── stg_patch.yaml
└── prod
├── kustomization.yaml
└── prod_patch.yaml
Under base/ConfigMap, config_map_constants.yaml file contains key/value pairs that are non-secrets:
apiVersion: v1
kind: ConfigMap
metadata:
labels:
app: myApp
name: global-config-map
namespace: myNamespace
data:
aws_region: "us-west"
env_id: "1234"
If an overlay just needs a default value, it should reference the key/value pair as is, and if it needs a custom value, I would use a patch to override the value.
kustomization.yaml from base/ConfigMap looks like this and refers to ConfigMap as a resource:
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- config_map_constants.yaml
QUESTION: how do I reference "aws_region" in my overlays' yaml files so that I can retrieve the value?
For example, I want to be able to do something like this in base/ServiceOne/service_one_config.yaml:
apiVersion: v1
kind: Service
metadata:
labels:
app: myApp
aws_region: ../ConfigMap/${aws_region} #pseudo syntax
name: service_one
spec:
env_id: ../ConfigMap/${env_id} #pseudo syntax
I am able to build the ConfigMap and append it to my services but I am struggling to find how to reference its contents within other resources.
EDIT:
Kustomize version: v4.5.2
You can try using https://kubectl.docs.kubernetes.io/references/kustomize/kustomization/replacements/
For your scenario, if you want to reference the aws-region into your Service labels. You need to create a replacement file.
replacements/region.yaml
source:
kind: ConfigMap
fieldPath: data.aws-region
targets:
- select:
kind: Service
name: service_one
fieldPaths:
- metadata.labels.aws_region
And add it to your kustomization.yaml
replacements:
- path: replacements/region.yaml
Kustomize output should be similar to this
---
apiVersion: v1
kind: Service
metadata:
labels:
app: myApp
aws_region: us-west-1
name: service_one

helm hook post-install shell script execution

I am trying to set helm "post-install" hook and seeing below error.
ERROR:
sh: script/jenkins.sh: not found
postinstall.yaml content
apiVersion: batch/v1
kind: Job
metadata:
name: "{{ .Release.Name }}"
annotations:
# This is what defines this resource as a hook. Without this line, the
# job is considered part of the release.
"helm.sh/hook": post-install
"helm.sh/hook-weight": "-5"
spec:
template:
spec:
containers:
- name: post-install-jenkins-job
image: alpine:3.3
imagePullPolicy: IfNotPresent
command: [ "/bin/sh", "-c", "scripts/jenkins.sh"]
restartPolicy: Never
terminationGracePeriodSeconds: 0
Folder structure of helm package
scripts/jenkins.h is the script that I am trying to execute with "postinstall.yaml" as post-install helm hook.
riq-agent
├── Chart.yaml
├── README.md
├── scripts
│   └── jenkins.sh
├── templates
│   ├── NOTES.txt
│   ├── Untitled-1.yml
│   ├── _helpers.tpl
│   ├── awssecret.yaml
│   ├── clusterrolebinding.yaml
│   ├── configurationFiles-configmap.yaml
│   ├── deployment.yaml
│   ├── hook-aws-ecr.yaml
│   ├── initializationFiles-configmap.yaml
│   ├── postinstall.yaml
│   ├── pvc.yaml
│   ├── secrets.yaml
│   ├── serviceaccount.yaml
│   ├── servicemonitor.yaml
│   ├── svc.yaml
│   └── tests
│   ├── test-configmap.yaml
│   └── test.yaml
└── values.yaml
Is there any mistakes in the way that I am trying to execute a shell script (stored within helm package) in helm hook?
In order to be executed, scripts/jenkins.sh should be a part of post-install-jenkins-job container, mounted as a volume. You can populate a volume with data stored in a configmap.
postinstall-configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: {{ .Release.Name }}-postinstall-configmap
data:
jenkins.sh: |-
{{ .Files.Get "scripts/jenkins.sh" | indent 4}}
postinstall.yaml
apiVersion: batch/v1
kind: Job
metadata:
name: "{{ .Release.Name }}"
annotations:
"helm.sh/hook": post-install
"helm.sh/hook-weight": "-5"
spec:
template:
spec:
containers:
- name: post-install-jenkins-job
image: alpine:3.3
imagePullPolicy: IfNotPresent
command: [ "/bin/sh", "-c", "/opt/scripts/jenkins.sh"]
volumeMounts:
- name: config-volume
mountPath: /opt/scripts
volumes:
- name: config-volume
configMap:
name: {{ .Release.Name }}-postinstall-configmap
defaultMode: 0777
restartPolicy: Never
terminationGracePeriodSeconds: 0

Access parent (chart) context from subcharts

I have a Helm Chart comprising of a few subcharts, sth like this:
├── Chart.yaml
├── README.md
├── charts
│   ├── nginx-1.0.0.tgz
│   └── redis-1.0.0.tgz
├── index.yaml
├── requirements.lock
├── requirements.yaml
├── subcharts
│   ├── nginx
│   │   ├── Chart.yaml
│   │   ├── templates
│   │   │   ├── deployment.yaml
│   │   │   ├── service.yaml
│   └── redis
│   ├── Chart.yaml
│   ├── templates
│   │   ├── deployment.yaml
│   │   └── service.yaml
│   └── values.yaml
└── values.yaml
In my root level values.yaml, I define the (per chart) values, under the corresponding yaml node, i.e. the file would look something like this:
redis:
namespace: default
replicas: 1
image: redis
tag: 5.0-alpine
port: 6379
imagePullPolicy: IfNotPresent
serviceType: ClusterIP
resources:
requests:
cpu: 200m
memory: 512Mi
limits:
cpu: 1000M
memory: 1500Mi
nginx:
namespace: default
istio:
enabled: false
replicas: 1
image: redash/nginx
tag: latest
port: 80
imagePullPolicy: IfNotPresent
serviceType: LoadBalancer
And these values (the ones in the hierarchy <subchartname>.value are accessed as follows in a subchart template:
spec:
replicas: {{ default 1 .Values.replicas }}
i.e. there is no reference to the subchart name, given that this becomes the root context for the template.
Is there a way so that my helm templates access the parent (root) context values?
I want to do this so I can share values across subcharts in a DRY mode.
i.e. more or less my question becomes:
how should I access from my subchart templates, values in the root level?
myvariable1:
value1
myvarianle2:
value2
I think you need to use global chart values which can be shared across subcharts . Link to doc https://helm.sh/docs/topics/chart_template_guide/subcharts_and_globals/#global-chart-values