apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: ns1
resources:
- r1a.yaml
- r1b.yaml
- r1c.yaml
- r1d.yaml
- r1e.yaml
- r2.yaml # needs to be placed in namespace ns2
Let's assume above situation. The problem is objects specified in r2.yaml would be place in ns1 even if ns2 is explicitely referenced in metadata.namespace.
How do I have to deal with this? Or how can I solve this (as I assume there a multiple options)?
I've looked into this and I came up with one idea.
├── base
│ ├── [nginx.yaml] Deployment nginx ns: default
| ├── [nginx2.yaml] Deployment nginx ns: default
| ├── [nginx3.yaml] Deployment nginx ns: default
| ├── [nginx4.yaml] Deployment nginx ns: default
| ├── [nginx5.yaml] Deployment nginx ns: nginx
│ └── [kustomization.yaml] Kustomization
└── prod
├── [kustomization.yaml] Kustomization
└── [patch.yaml] patching namespace
You need to have 2 directories, in this setup are: base and prod. In base directory you should use your base YAMLs and kustomization.yaml file. In my scenario I have 6 YAMLs: nginx/1/2/3/4.yaml based on Kubernetes Documentation, and nginx5.yaml which looks the same but with additional spec.namespace: nginx.
In base directory:
$ cat kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- nginx1.yaml
- nginx2.yaml
- nginx3.yaml
- nginx4.yaml
- nginx5.yaml
And 5 YAMLs with nginx.
In Prod directory:
You should have 2 files. kustomization.yaml and patch.yaml.
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: ns1
bases:
- ../base
patchesJson6902:
- target:
group: apps
version: v1
kind: Deployment
name: nginx-deployment-5
path: patch.yaml
$ cat patch.yaml
- op: replace
path: /metadata/namespace
value: nginx
When you will use kustomize build . in prod directory, all nginx-deployment/-2/-3/-4 will be in the namespace: ns1 and nginx-deployment-5 will be in namespace: nginx.
~/prod (project)$ kustomize build .
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-deployment-5
namespace: nginx
spec:
replicas: 2
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
spec:
containers:
- image: nginx:1.14.2
name: nginx
ports:
- containerPort: 80
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-deployment
namespace: ns1
spec:
replicas: 2
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
spec:
containers:
- image: nginx:1.14.2
name: nginx
ports:
- containerPort: 80
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-deployment-2
namespace: ns1
spec:
Useful links:
Kustomize Builtin Plugins
Customizong
Kustomization Patches
Related
We have a EKS cluster running with Traefik deployed in CRD style (full setup on GitHub) and wan't to deploy our app https://gitlab.com/jonashackt/microservice-api-spring-boot with the Kubernetes objects Deployment, Service and IngressRoute (see configuration repository here). The manifests look like this:
deployment.yml:
apiVersion: apps/v1
kind: Deployment
metadata:
name: microservice-api-spring-boot
spec:
replicas: 3
revisionHistoryLimit: 3
selector:
matchLabels:
app: microservice-api-spring-boot
branch: main
template:
metadata:
labels:
app: microservice-api-spring-boot
branch: main
spec:
containers:
- image: registry.gitlab.com/jonashackt/microservice-api-spring-boot:c25a74c8f919a72e3f00928917dc4ab2944ab061
name: microservice-api-spring-boot
ports:
- containerPort: 8098
imagePullSecrets:
- name: gitlab-container-registry
service.yml:
apiVersion: v1
kind: Service
metadata:
name: microservice-api-spring-boot
spec:
ports:
- port: 80
targetPort: 8098
selector:
app: microservice-api-spring-boot
branch: main
traefik-ingress-route.yml:
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
name: microservice-api-spring-boot-ingressroute
namespace: default
spec:
entryPoints:
- web
routes:
- match: Host(`microservice-api-spring-boot-BRANCHNAME.tekton-argocd.de`)
kind: Rule
services:
- name: microservice-api-spring-boot
port: 80
We already use Kustomize and especially the kustomize CLI (on a Mac or in GitHub Actions install with brew install kustomize) with the following folder structure:
├── deployment.yml
├── kustomization.yaml
├── service.yml
└── traefik-ingress-route.yml
Our kustomization.yaml looks like this:
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- deployment.yml
- service.yml
- traefik-ingress-route.yml
images:
- name: registry.gitlab.com/jonashackt/microservice-api-spring-boot
newTag: foobar
commonLabels:
branch: foobar
nameSuffix: foobar
Now changing the metadata.name dynamically to add a suffix to the Deployment's, Service's and IngressRoute's .metadata.name from within our GitHub Actions workflow is easy with kustomize CLI (because we want the suffix to use a prefixed -, we need to use the -- -barfoo syntax here):
kustomize edit set namesuffix -- -barfoo
Check the result with
kustomize build .
Also changing the .spec.selector.matchLabels.branch, .spec.template.metadata.labels.branch and .spec.selector.branch in the Deployment and Service is no problem:
kustomize edit set label branch:barfoo
Changing the .spec.template.spec.containers[0].image of our Deployment works with:
kustomize edit set image registry.gitlab.com/jonashackt/microservice-api-spring-boot:barfoo
But looking into our IngressRoute it seems that .spec.routes[0].services[0].name and .spec.routes[0].match = Host() can't be changed with Kustomize out of the box?! So how can we change both fields without the need for a replacement tooling like yq or even sed/ envsubst?
1. Change the IngressRoutes .spec.routes[0].services[0].name with Kustomize
Changing the IngressRoutes .spec.routes[0].services[0].name is possible with Kustomize using a NameReference transformer (see docs here) - luckily I found inspiration in this issue. Therefore we need to include the configurations keyword in our kustomize.yaml:
nameSuffix: foobar
configurations:
# Tie target Service metadata.name to IngressRoute's spec.routes.services.name
# Once Service name is changed, the IngressRoute referrerd service name will be changed as well.
- nameReference.yml
We also need to add file called nameReference.yml:
nameReference:
- kind: Service
fieldSpecs:
- kind: IngressRoute
path: spec/routes/services/name
As you can see we tie the Service's name to the IngressRoutes spec/routes/services/name. Now running
kustomize edit set namesuffix barfoo
will not only change the metadata.name tags of the Deployment, Service and IngressRoute - but also the .spec.routes[0].services[0].name of the IngressRoute, since it is now linked to the metadata.name of the Service. Note that this only, if both the referrer and the target's have a name tag.
2. Change a part of the IngressRoutes .spec.routes[0].match = Host()
The second part of the question ask how to change a part of the IngressRoutes .spec.routes[0].match = Host(). There's an open issue in the Kustomize GitHub project. Right now Kustomize doesn't support this use case - only writing a custom generator plugin for Kustomize. As this might not be a preferred option, there's another way inspired by this blog post. As we can create yaml files inline in our console using the syntax cat > ./myyamlfile.yml <<EOF ... EOF we could also use the inline variable substitution.
So first define the branch name as variable:
RULE_HOST_BRANCHNAME=foobar
And then use the described syntax to create a ingressroute-patch.yml file inline:
cat > ./ingressroute-patch.yml <<EOF
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
name: microservice-api-spring-boot-ingressroute
namespace: default
spec:
entryPoints:
- web
routes:
- match: Host(\`microservice-api-spring-boot-$RULE_HOST_BRANCHNAME.tekton-argocd.de\`)
kind: Rule
services:
- name: microservice-api-spring-boot
port: 80
EOF
The last step is to use the ingressroute-patch.yml file as patchesStrategicMerge inside our kustomization.yaml like this:
patchesStrategicMerge:
- ingressroute-patch.yml
Now running kustomize build . should output the correct Deployment, Service and IngressRoute for our setup:
apiVersion: v1
kind: Service
metadata:
labels:
branch: barfoo
name: microservice-api-spring-boot-barfoo
spec:
ports:
- port: 80
targetPort: 8098
selector:
app: microservice-api-spring-boot
branch: barfoo
---
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
branch: barfoo
name: microservice-api-spring-boot-barfoo
spec:
replicas: 3
revisionHistoryLimit: 3
selector:
matchLabels:
app: microservice-api-spring-boot
branch: barfoo
template:
metadata:
labels:
app: microservice-api-spring-boot
branch: barfoo
spec:
containers:
- image: registry.gitlab.com/jonashackt/microservice-api-spring-boot:barfoo
name: microservice-api-spring-boot
ports:
- containerPort: 8098
imagePullSecrets:
- name: gitlab-container-registry
---
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
labels:
branch: barfoo
name: microservice-api-spring-boot-ingressroute-barfoo
namespace: default
spec:
entryPoints:
- web
routes:
- kind: Rule
match: Host(`microservice-api-spring-boot-barfoo.tekton-argocd.de`)
services:
- name: microservice-api-spring-boot-barfoo
port: 80
I am trying to patch a cronjob, but somehow it doesn't work as I would expect. I use the same folder structure for a deployment and that works.
This is the folder structure:
.
├── base
│ ├── kustomization.yaml
│ └── war.cron.yaml
└── overlays
└── staging
├── kustomization.yaml
├── war.cron.patch.yaml
└── war.cron.staging.env
base/kustomization.yaml
---
kind: Kustomization
resources:
- war.cron.yaml
base/war.cron.yaml
---
apiVersion: batch/v1
kind: CronJob
metadata:
name: war-event-cron
spec:
schedule: "*/5 * * * *"
jobTemplate:
spec:
template:
spec:
containers:
- name: war-event-cron
image: my-registry/war-service
imagePullPolicy: IfNotPresent
command:
- python
- run.py
args:
- sync-events
envFrom:
- secretRef:
name: war-event-cron-secret
restartPolicy: OnFailure
Then I am trying to patch this in the staging overlay.
overlays/staging/kustomization.yaml
---
kind: Kustomization
namespace: staging
bases:
- "../../base"
patchesStrategicMerge:
- war.cron.patch.yaml
secretGenerator:
- name: war-event-cron-secret
behavior: create
envs:
- war.cron.staging.env
overlays/staging/war.cron.patch.yaml
---
apiVersion: batch/v1
kind: CronJob
metadata:
name: war-event-cron
spec:
jobTemplate:
spec:
template:
spec:
containers:
- name: war-event-cron
image: my-registry/war-service:nightly
args:
- sync-events
- --debug
But the result of kustomize build overlays/staging/ is not what I want. The command is gone and the secret is not referenced.
apiVersion: v1
data:
...
kind: Secret
metadata:
name: war-event-cron-secret-d8m6bh7284
namespace: staging
type: Opaque
---
apiVersion: batch/v1
kind: CronJob
metadata:
name: war-event-cron
namespace: staging
spec:
jobTemplate:
spec:
template:
spec:
containers:
- args:
- sync-events
- --debug
image: my-registry/war-service:nightly
name: war-event-cron
restartPolicy: OnFailure
schedule: '*/5 * * * *'
It's known bug in kustomize - check and follow this topic (created ~ one month ago) on GitHub for more information.
For now, fix for your issue is to use apiVersion:batch/v1beta1 instead of apiVersion: batch/v1 in base/war.cron.yaml and overlays/staging/war.cron.patch.yaml files.
I would like to set the name field in a Namespace resource and also replace the namespace field in a Deployment resource with the same value, for example my-namespace.
Here is kustomization.json:
namespace: <NAMESPACE>
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
patchesJson6902:
- patch: |-
- op: replace
path: /metadata/name
value: <NAMESPACE>
target:
kind: Namespace
name: system
version: v1
resources:
- manager.yaml
and manager.yaml:
apiVersion: v1
kind: Namespace
metadata:
labels:
control-plane: controller-manager
name: system
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: controller-manager
namespace: system
spec:
selector:
matchLabels:
control-plane: controller-manager
replicas: 1
template:
metadata:
labels:
control-plane: controller-manager
spec:
containers:
- command:
- /manager
args:
- --enable-leader-election
image: controller:latest
name: manager
I tried using kustomize edit set namespace my-namespace && kustomize build, but it only changes the namespace field in the Deployment object.
Is there a way to change both field without using sed, in 'pure' kustomize and without having to change manually value in kustomization.json?
Is there a way to change both field without using sed, in 'pure' kustomize and without having to change manually value in kustomization.json?
I managed to achieve somewhat similar with the following configuration:
kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: my-namespace
resources:
- deployment.yaml
depyloment.yaml
---
apiVersion: v1
kind: Namespace
metadata:
name: nginx
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-deployment
namespace: nginx
spec:
replicas: 2
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
spec:
containers:
- image: nginx:1.14.2
name: nginx
ports:
- containerPort: 80
And here is the output of the command that you used:
➜ kustomize kustomize edit set namespace my-namespace7 && kustomize build .
apiVersion: v1
kind: Namespace
metadata:
name: my-namespace7
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-deployment
namespace: my-namespace7
spec:
replicas: 2
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
spec:
containers:
- image: nginx:1.14.2
name: nginx
ports:
- containerPort: 80
What is happening here is that once you set the namespace globally in kustomization.yaml it will apply it to your targets which looks to me that looks an easier way to achieve what you want.
I cannot test your config without manager_patch.yaml content. If you wish to go with your way further you will have update the question with the file content.
Currently we are using ${HOME}/bin/kustomize edit set nameprefix prefix1
But it is adding nameprefix to all of our resources like deployment.yaml and service.yaml.
We want to apply nameprefix to deployment.yaml only and not apply it to service.yaml
Posting for better visibility:
If you are using:
kustomize edit set nameprefix prefix1
This command will set namePrefix inside your current kustomization.
As stated in the question - this is the way how it works, namePrefix will be used for all specified resources inside kustomization.yaml.
Please consider the following scenario using the idea of an overlay and base with kustomization.
Tested with:
kustomize/v4.0.1
Base declare resources and settings shared in common and overlay declare additional differences.
.
├── base
│ ├── [deployment.yaml] Deployment nginx
│ ├── [kustomization.yaml] Kustomization
│ └── [service.yaml] Service nginx
└── prod
├── [kustomization.yaml] Kustomization
└── kustomizeconfig
└── [deploy-prefix-transformer.yaml] PrefixSuffixTransformer customPrefixer
base: common files
#deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx
spec:
selector:
matchLabels:
run: nginx
#service.yaml
apiVersion: v1
kind: Service
metadata:
name: nginx
labels:
run: nginx
#kustomization.yaml
resources:
- deployment.yaml
- service.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
overlay/prod: kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
bases:
- ../../base
nameSuffix: -Suffix1
transformers:
- ./kustomizeconfig/deploy-prefix-transformer.yaml
overlay/prod/kustomizeconfig: deploy-prefix-transformer.yaml
apiVersion: builtin
kind: PrefixSuffixTransformer
metadata:
name: customPrefixer
prefix: "deploymentprefix-"
fieldSpecs:
- kind: Deployment
path: metadata/name
As you can see, using this structure and builtin plugin PrefixSuffixTransformer you can get the desired effect:
kustomize build overlay/prod/
apiVersion: v1
kind: Service
metadata:
labels:
run: nginx
name: nginx-Suffix1
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: deploymentprefix-nginx-Suffix1
spec:
selector:
matchLabels:
run: nginx
This configuration (overlay/prod/kustomization.yaml) will apply nameSuffix: -Suffix1 to all resources specified in base directory and using PrefixSuffixTransformer will add in this specific example prefix: "deploymentprefix-" to deployment.metadata.name
apiVersion: builtin
kind: PrefixSuffixTransformer
metadata:
name: customPrefixer
prefix: "deploymentprefix-"
fieldSpecs:
- kind: Deployment
path: metadata/name
/kustomizeconfig/deploy-prefix-transformer.yaml
There is github issue about that
is it possible to have kustomization file avoid adding prefixes to few kinds ?
And there are 2 examples provided by #jbrette with which you can achieve what you need.
no prefix to secret
canary using skip
Additionally you can take a look at these pull requests:
https://github.com/kubernetes/enhancements/pull/1232
https://github.com/kubernetes-sigs/kustomize/pull/1491
For those who stumble across this, I had an issue get it to work with ServiceAccount type.
Issue was that I needed to prevent suffix from being added. Apparently namePrefix "should" prevent that too but in actuality I had to add:
nameSuffix:
- path: metadata/name
apiVersion: v1
kind: serviceaccount
skip: true
Note kind type with lowercase. Using standard kind for ServiceAccount makes it to fail.
In my kustomization.yaml I have:
...
secretGenerator:
- name: db-env
behavior: create
envs:
- my.env
patchesStrategicMerge:
- app.yaml
And then in my app.yaml (the patch) I have:
apiVersion: apps/v1
kind: Deployment
metadata:
name: app-deployment
spec:
template:
spec:
containers:
- name: server
envFrom:
- secretRef:
name: db-env
When I try build this via kustomize build k8s/development I get back out:
apiVersion: apps/v1
kind: Deployment
...
spec:
containers:
- envFrom:
- secretRef:
name: db-env
name: server
When it should be:
- envFrom:
- secretRef:
name: db-env-4g95hhmhfc
How do I get the secretGenerator name hashing to apply to patchesStrategicMerge too?
Or alternatively, what's the proper way to inject some environment vars into a deployment for a specific overlay?
This for development.
My file structure is like:
❯ tree k8s
k8s
├── base
│ ├── app.yaml
│ └── kustomization.yaml
├── development
│ ├── app.yaml
│ ├── golinks.sql
│ ├── kustomization.yaml
│ ├── mariadb.yaml
│ ├── my.cnf
│ └── my.env
└── production
├── ingress.yaml
└── kustomization.yaml
Where base/kustomization.yaml is:
namespace: go-mpen
resources:
- app.yaml
images:
- name: server
newName: reg/proj/server
and development/kustomization.yaml is:
resources:
- ../base
- mariadb.yaml
configMapGenerator:
- name: mariadb-config
files:
- my.cnf
- name: initdb-config
files:
- golinks.sql # TODO: can we mount this w/out a config file?
secretGenerator:
- name: db-env
behavior: create
envs:
- my.env
patchesStrategicMerge:
- app.yaml
This works fine for me with kustomize v3.8.4. Can you please check your version and if disableNameSuffixHash is not perhaps set to you true.
Here are the manifests used by me to test this:
➜ app.yaml deployment.yaml kustomization.yaml my.env
app.yaml
kind: Deployment
metadata:
name: app-deployment
spec:
template:
spec:
containers:
- name: server
envFrom:
- secretRef:
name: db-env
deplyoment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: app-deployment
labels:
app: nginx
spec:
replicas: 3
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: nginx:1.14.2
ports:
- containerPort: 80
and my kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
secretGenerator:
- name: db-env
behavior: create
envs:
- my.env
patchesStrategicMerge:
- app.yaml
resources:
- deployment.yaml
And here is the result:
apiVersion: v1
data:
ASD: MTIz
kind: Secret
metadata:
name: db-env-f5tt4gtd7d
type: Opaque
---
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app: nginx
name: app-deployment
spec:
replicas: 3
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
spec:
containers:
- image: nginx:1.14.2
name: nginx
ports:
- containerPort: 80
- envFrom:
- secretRef:
name: db-env-f5tt4gtd7d
name: server