I am trying to script setup of Jenkins so that I can create and tear down Jenkins clusters programmatically with helm. I've hit an annoying snag where I cannot set a key with dots in the name. My helm values.yaml file looks like this:
---
rbac:
install: true
Master:
HostName: jenkins.mycompany.com
ServiceType: ClusterIP
ImageTag: lts
InstallPlugins:
- kubernetes
- workflow-aggregator
- workflow-job
- credentials-binding
- git
- blueocean
- github
- github-oauth
ScriptApproval:
- "method groovy.json.JsonSlurperClassic parseText java.lang.String"
- "new groovy.json.JsonSlurperClassic"
- "staticMethod org.codehaus.groovy.runtime.DefaultGroovyMethods leftShift java.util.Map java.util.Map"
- "staticMethod org.codehaus.groovy.runtime.DefaultGroovyMethods split java.lang.String"
- "method java.util.Collection toArray"
- "staticMethod org.kohsuke.groovy.sandbox.impl.Checker checkedCall java.lang.Object boolean boolean java.lang.String java.lang.Object[]"
- "staticMethod org.kohsuke.groovy.sandbox.impl.Checker checkedGetProperty java.lang.Object boolean boolean java.lang.Object"
Ingress:
Annotations:
kubernetes.io/ingress.class: nginx
kubernetes.io/tls-acme: "true"
TLS:
- secretName: jenkins-mycompany-com
hosts:
- jenkins.mycompany.com
Memory: "2Gi"
# This breaks the init container
# RunAsUser: 1000
# FSGroup: 1000
Agent:
Enabled: false
ImageTag: latest
After installing cert-manager, external-dns, nginx-ingress (for now via a bash script) I install it like so:
helm install --values helm/jenkins.yml stable/jenkins
I failed to read the letsencrypt docs at all, so throughout the course of testing I used my production quota. I want to be able to add an annotation to the Ingress: certmanager.k8s.io/cluster-issuer: letsencrypt-staging so that I can continue testing (and set this as the default in the future, overriding when I'm ready for production).
The trouble is... I can't figure out how to pass this via the --set flag, since there are periods in the key name. I've tried:
helm install --values helm/jenkins.yml stable/jenkins --set Master.Ingress.Annotations.certmanager.k8s.io/cluster-issuer=letsencrypt-staging
and
helm install --values helm/jenkins.yml stable/jenkins --set Master.Ingress.Annotations.certmanager\.k8s\.io/cluster-issuer=letsencrypt-staging
I can of course solve this by adding a value that I use as a flag, but it's less explicit. Is there any way to set it directly?
You need to enclose the key with quotations and then escape the dots
helm install --values helm/jenkins.yml stable/jenkins --set Master.Ingress.Annotations."certmanager\.k8s\.io/cluster-issuer"=letsencrypt-staging
Use \ to escape the dots in the key. Quotation marks are required to prevent shell interpreting the \ character.
helm install --values helm/jenkins.yml stable/jenkins --set 'Master.Ingress.Annotations.certmanager\.k8s\.io/cluster-issuer=letsencrypt-staging'
Helm requires these characters to be escaped: . [ , =
Source: https://paul-boone.medium.com/helm-chart-install-advanced-usage-of-the-set-argument-3e214b69c87a
If you are doing this in Jenkins within a shell script like this:
sh """
...
--set-string datasources.'datasources\\.yaml'.datasources[1].secureJsonData.token=$TOKEN \
...
"""
I'm not sure if this is the best explanation but I'll give it a shot....
Groovy interprets shell scripts with backslashes differently than a normal shell script because in Jenkins this is being written inside a shell script. The double escape is needed so Groovy ignores the first backslash and then helm ignores the second. This problem took me too long to solve so I'd figure I'd share what I found for anyone stuck in the same boat.
Related
I'm trying to add some additional annotations to the pods that are created by the istiod deployment.
I'm using the istioctl install documentation, which suggests that I can use the podAnnotations field from the istio operator documentation, but I can't see how to structure the argument correctly.
The docs say it is of type map<string, string>. How do you express that?
I've tried a few variations, e.g
./istioctl install --set profile=minimal --set components.pilot.k8s.hpaSpec.minReplicas=2 --set components.pilot.k8s.podAnnotations={"foo":"bar"} -y
I ended up solving this using an IstioOperator resource, which matches what Chris suggested in his comment.
To achieve the desired effect I used a resource which looks like this:
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
profile: minimal
components:
pilot:
k8s:
podAnnotations:
co.elastic.logs/enabled : "true"
co.elastic.logs/processors.dissect.field: "message"
co.elastic.logs/processors.dissect.tokenizer: "%{#timestamp} %{level} %{message}"
co.elastic.logs/processors.dissect.overwrite_keys: "true"
co.elastic.logs/processors.dissect.target_prefix: ""
hpaSpec:
minReplicas : 2
This file is passed to the installer on the command line:
istioctl install -f ${path.root}/k8s/istio/istioctl-config.yaml -y
This seems to be the way to provide podAnnotations. It also matches the recommendation made in the docs themselves to avoid using --set arguments on the install command in production.
I want to deploy IBM-MQ to Kubernetes (Rancher) using helmfile. I've found this link and did everything as described in the guide: https://artifacthub.io/packages/helm/ibm-charts/ibm-mqadvanced-server-dev.
But the pod is not starting with the error: "ImagePullBackOff". What could be the problem? My helmfile:
...
repositories:
- name: ibm-stable-charts
url: https://raw.githubusercontent.com/IBM/charts/master/repo/stable
releases:
- name: ibm-mq
namespace: test
createNamespace: true
chart: ibm-stable-charts/ibm-mqadvanced-server-dev
values:
- ./ibm-mq.yaml
ibm-mq.yaml:
- - -
license: accept
security:
initVolumeAsRoot: true/false // I'm not sure about this, I added it just because it wasn't working.
// Both of the options don't work too
queueManager:
name: "QM1"
dev:
secret:
adminPasswordKey: adminPassword
name: mysecret
I've created the secret and seems like it's working, so the problem is not in the secret.
The full error I'm getting:
Failed to pull image "ibmcom/mq:9.1.5.0-r1": rpc error: code = Unknown desc = Error response from daemon: manifest for ibmcom/mq:9.1.5.0-r1 not found: manifest unknown: manifest unknown
I'm using helm 3, helmfile v.0.141.0, kubectl 1.22.2
I will leave some things as an exercise to you, but here is what that tutorial says:
helm repo add ibm-stable-charts https://raw.githubusercontent.com/IBM/charts/master/repo/stable
You don't really need to do this, since you are using helmfile.
Then they say to issue:
helm install --name foo
ibm-stable-charts/ibm-mqadvanced-server-dev
--set license=accept
--set queueManager.dev.secret.name=mysecret
--set queueManager.dev.secret.adminPasswordKey=adminPassword
--tls
which is targeted towards helm2 (because of those --name and --tls), but that is irrelevant to the problem.
When I install this, I get the same issue:
Failed to pull image "ibmcom/mq:9.1.5.0-r1": rpc error: code = NotFound desc = failed to pull and unpack image "docker.io/ibmcom/mq:9.1.5.0-r1": failed to resolve reference "docker.io/ibmcom/mq:9.1.5.0-r1": docker.io/ibmcom/mq:9.1.5.0-r1: not found
I went to the docker.io page of theirs and indeed such a tag : 9.1.5.0-r1 is not present.
OK, can we update the image then?
helm show values ibm-stable-charts/ibm-mqadvanced-server-dev
reveals:
image:
# repository is the container repository to use, which must contain IBM MQ Advanced for Developers
repository: ibmcom/mq
# tag is the tag to use for the container repository
tag: 9.1.5.0-r1
good, that means we can change it via an override value:
helm install foo
ibm-stable-charts/ibm-mqadvanced-server-dev
--set license=accept
--set queueManager.dev.secret.name=mysecret
--set queueManager.dev.secret.adminPasswordKey=adminPassword
--set image.tag=latest # or any other tag
so this works.
How to set-up that tag in helmfile is left as an exercise to you, but it's pretty trivial.
I'm writing an Ansible task to deploy GitLab in my k3s environment.
According to the doc, I need to execute this to install GitLab using Helm:
$ helm install gitlab gitlab/gitlab \
--set global.hosts.domain=DOMAIN \
--set certmanager-issuer.email=me#example.com
But the community.kubernetes.helm doesn't handle --set parameters and only call helm with the --values parameter.
So my Ansible task looks like this:
- name: Deploy GitLab
community.kubernetes.helm:
update_repo_cache: yes
release_name: gitlab
chart_ref: gitlab/gitlab
release_namespace: git
release_values:
global.hosts.domain: example.com
certmanager-issuer.email: info#example.com
But the helm chart still return the error You must provide an email to associate with your TLS certificates. Please set certmanager-issuer.email.
I've tried manually in a terminal, and it seems that the GitLab helm chart requires --set parameters and fail with --values. But community.kubernetes.helm doesn't.
What can I do?
Is there a bug on GitLab helm chart side?
it seems that the GitLab helm chart requires --set parameters and fail with --values
That is an erroneous assumption; what you are running into is that --set splits on . because otherwise providing fully-formed YAML on the command line would be painful
The correct values are using sub-objects where the . occurs:
- name: Deploy GitLab
community.kubernetes.helm:
update_repo_cache: yes
release_name: gitlab
chart_ref: gitlab/gitlab
release_namespace: git
release_values:
global:
hosts:
# https://gitlab.com/gitlab-org/charts/gitlab/-/blob/v4.4.5/values.yaml#L47
domain: example.com
# https://gitlab.com/gitlab-org/charts/gitlab/-/blob/v4.4.5/values.yaml#L592-595
certmanager-issuer:
email: info#example.com
I want to pass arguments to Kubernetes deployment when I use kubectl command to apply the deployment file.
Example: In my deployment .yaml, I have arguments as below and I want to pass the argument values when I run with the kubectl apply - f <my-deployment>.yaml
So, in the below example, I want to override the args - userid and role when I run the above kubectl command.
spec:
containers:
- name: testimage
image: <my image name>:<tag>
args:
- --userid=testuser
- --role=manager
The simple answer is. You can't do that.
kubectl is not a template engine. As some people mentioned, you have options like Helm or Kustomize which can solve this. I'd encurage you to look into Helm3 since it nicely solves your problem with a command like helm upgrade --install ... --set userid=xxx --set role=yyy.
If you're stuck with kubectl only though, you might want to use it's ability to ingest yaml from stdin and pass your yaml through any type of templating first. ie. as follows :
...
args:
- --userid=$USER
- --role=$ROLE
...
cat resource.yaml | USER=testuser ROLE=manager envsubst | kubectl apply -f -
obviously any other string replacement method would do (sed, awk, etc.)
This should be added in your deployment.yml
spec:
containers:
- name: testimage
image: <my image name>:<tag>
args: ["--userid","=","testuser","--role","=","manager"]
I'm installing the prometheus-redis-exporter Helm chart. Its Deployment object has a way to inject annotations:
# deployment.yaml
...
template:
metadata:
annotations:
{{ toYaml .Values.annotations | indent 8 }}
Typically if I was providing a values file, I could just do this:
# values.yaml
annotations:
foo: bar
bash: baz
And then install the chart with:
helm install --values values.yaml
However, in some cases it is more simple for me to specify these values on the command line with --set instead, I'm just not sure how I would specify a nested set like that.
How can I set the above annotations object when installing a helm chart on the commandline:
helm install --set <what_goes_here>
The helm docu has a section The Format and Limitations of --set, which contains what you are looking for.
--set outer.inner=value results in:
outer:
inner: value
Therefore your whole helm command looks like this:
helm install --set annotations.foo=bar,annotations.bash=baz stable/prometheus-redis-exporter
Just to add, if you are looking to override a key with a "." in the key name, add a back slash ("\") before the ".".
for example, with values (taken from grafana):
grafana.ini:
server:
root_url: https://my.example.com
To edit the root_url value we would pass
--set grafana\.ini.server.root_url=https://your.example.com