I have a set of K8s YAML descriptors as part of a project and I'm using kustomization to build them. I'm also using GitOps to do pull based deployments to my K8s cluster.
I now want to add some tests for my YAML files so that if I have any errors, I want to avoid or prevent Flux from pulling my changes into the cluster. So basically I want to do some unit test like thingy for my YAML files. I came across Kubeval and this could serve my purpose well. I'm just not sure how to use it.
Anyone already tried this? I want to basically do the following:
As soon as I push some YAML files into my repo, Kubeval kicks in and validates all the YAML files in a set of folders that I specify
If all the YAML files passes lint validations, then I want to proceed to the next stage where I call kustomize to build the deployment YAML.
If the YAML files fail lint validation, then my CI fails and nothing should happen
Any ideas on how I could do this?
Since my project is hosted on GitHub, I was able to get what I want using GitHub actions and kube-tools
So basically here is what I did!
In my GitHub project, added a main.yaml under project-root/.github/workflows/main.yml
The contents of my main.yaml is:
name: ValidateKubernetesYAML
branches: [ master ] pull_request:
branches: [ master ]
runs-on: ubuntu-latest
steps:
- uses: actions/checkout#v2
- name: Kubeval
uses: stefanprodan/kube-tools#v1.2.0
with:
kubectl: 1.16.2
kustomize: 3.4.0
helm: 2.16.1
helmv3: 3.0.0
command: |
echo "Run kubeval"
kubeval -d base,dev,production --force-color --strict --ignore-missing-schemas
Now when someone issues a pull request into master, this validation kicks in and if it fails the changes does not get promoted into master branch which is what I want!
Here is the output of such a validation:
Run kubeval
WARN - Set to ignore missing schemas
PASS - base/application/plant-simulator-deployment.yaml contains a valid Deployment
PASS - base/application/plant-simulator-ingress-service.yaml contains a valid Ingress
PASS - base/application/plant-simulator-namespace.yaml contains a valid Namespace
PASS - base/application/plant-simulator-service.yaml contains a valid Service
WARN - base/kustomization.yaml containing a Kustomization was not validated against a schema
PASS - base/monitoring/grafana/grafana-deployment.yaml contains a valid Deployment
PASS - base/monitoring/grafana/grafana-service.yaml contains a valid Service
PASS - base/monitoring/plant-simulator-monitoring-namespace.yaml contains a valid Namespace
PASS - base/monitoring/prometheus/config-map.yaml contains a valid ConfigMap
PASS - base/monitoring/prometheus/prometheus-deployment.yaml contains a valid Deployment
PASS - base/monitoring/prometheus/prometheus-roles.yaml contains a valid ClusterRole
PASS - base/monitoring/prometheus/prometheus-roles.yaml contains a valid ServiceAccount
PASS - base/monitoring/prometheus/prometheus-roles.yaml contains a valid ClusterRoleBinding
PASS - base/monitoring/prometheus/prometheus-service.yaml contains a valid Service
PASS - dev/flux-patch.yaml contains a valid Deployment
WARN - dev/kustomization.yaml containing a Kustomization was not validated against a schema
PASS - production/flux-patch.yaml contains a valid Deployment
WARN - production/kustomization.yaml containing a Kustomization was not validated against a schema
Related
I've spinnaker with kubernetes manifest v2. Now, I'm creating spinnaker pipeline with manifest files in which I'd like to have namespace named as applicationName-pipelineName.
I'm looking for spinnaker environment variable for pipeline name which I can use in manifest file to would append in file.
Ex. I'm using docker registry latest tag as
imageName:${trigger['tag']}
similarly something like
apiVersion: v1
kind: Namespace
metadata:
name: appName-${pipelineName}
please advise. Thanks.
During pipeline runtime you have following parameters available:
${execution.id} - unique id of execution;
${execution.name} - name of pipeline used during creation;
${execution.pipelineConfigId} - unique id for pipeline across whole Spinnaker instance, it also could be set to something meaningful if you create pipeline using API.
Do not forget that namespace's name is very sensitive to case and non-alphanumerical symbols. Such functions like #alphanumerical(string) and ${java.lang.String.toLowerCase()} will be very helpful.
You may find this reference useful - Pipeline Expression Reference.
GoCD Version: 19.12.0
I'm trying to get environment variables defined in the Kubernetes deployment (system) in my GoCD YAML config in order to pass the GitHub authentication when pulling the resource.
I've confirmed that I'm able to call the repository using a personal access token. (via https://[TOKEN]#github.com/[COMPANY]/[REPO].git)
This, of course, also works if I do the same for the actual YAML git field.
The GoCD secrets in K8s:
apiVersion: v1
data:
GITHUB_ACCESS_KEY: base64EncodedKey
kind: Secret
type: Opaque
The GoCD deployment gets the secrets:
...
spec:
containers:
- env:
- name: GOCD_PLUGIN_INSTALL_kubernetes-elastic-agents
value: https://github.com/gocd/kubernetes-elastic-agents/releases/download/v3.4.0-196/kubernetes-elastic-agent-3.4.0-196.jar
- name: GOCD_PLUGIN_INSTALL_docker-registry-artifact-plugin
value: https://github.com/gocd/docker-registry-artifact-plugin/releases/download/v1.1.0-104/docker-registry-artifact-plugin-1.1.0-104.jar
- name: GITHUB_ACCESS_KEY
valueFrom:
secretKeyRef:
key: GITHUB_ACCESS_KEY
name: gocd-server
...
I've exec'd into the pod and echoed the variable, which returns the decoded value.
The YAML:
format_version: 9
pipelines:
db-docker-build:
group: someGroup
label_template: ${COUNT}-${git[:8]}
lock_behavior: unlockWhenFinished
display_order: 1
materials:
git:
git: 'https://$GITHUB_ACCESS_KEY#github.com/[COMPANY]/[REPO].git'
shallow_clone: true
auto_update: true
branch: master
...
I'd half expect that to work, but it doesn't, it actually just gets $GITHUB_ACCESS_KEY as the value. The jobs defined in the pipeline stages are run using an elastic agent pod which also has the required secrets defined. I've tried a few
Setting env variables -
environment_variables: GIT_KEY: ${GITHUB_ACCESS_KEY}
and then using that variable
git: 'https://$GIT_KEY#github.com/[COMPANY]/[REPO].git'
Setting env variables and no quotes -
environment_variables: GIT_KEY: ${GITHUB_ACCESS_KEY}
and then using that variable
git: https://${GIT_KEY}#github.com/[COMPANY]/[REPO].git
No quotes - git: https://$GITHUB_ACCESS_KEY#github.com/[COMPANY]/[REPO].git
No quotes with brackets - git: https://${GITHUB_ACCESS_KEY}#github.com/[COMPANY]/[REPO].git
I've seen from some YAML documentation that it is recommended to use encrypted_password for the GitHub password, but this seems unnecessary since the GUI hides the token, and that its running in Kubernetes with secrets.
The team and I researched this a little further and found a workaround. Most issues and articles explain what is written in the docs, that you really need access to /bin/bash -c in order to get the variables.
The YAML plugin creator also uses secure, encrypted variables to store sensitive data which is fine, but for our team it means that a lot of Kubernetes features are not utilised.
The workaround:
Use the GUI to create a pipeline in GoCD, enter the GitHub link, add a username and the personal access token for the user as the password, test the connection is OK. Once created, go to Admin -> Pipelines and click the Download pipeline configuration and select YAML.
The generated YAML has the token encrypted as with the GoCD servers private key.
I have a react application that is hosted in a nginx container using static files that are prepared in a build step. The problem I run in to is that the API URL is then hard coded in the js files and I get a problem when I want to deploy the application to different environments.
So basically I have put a config.js file with the localhost API URL variable in the public directory which is then loaded in the application in the section of the index.html file. This works for the local environment. The problem comes when I want to deploy it to the test or production environment.
I have found out that it is possible to use a configMap with volume mounts, but that requires me to prepare one file for each environment in advance as I understand it. I want to be able to use the variables I have set in my Azure DevOps library to populate the API URL value.
So my question is if there is a way to replace the values in the config.js file in the nginx container using Kuberentes/Helm or if I can make use of a Azure DevOps pipeline task to replace the content of a pre-prepared config.js file and mount that using Kubernetes?
Not sure if it is clear what I want to do, but hopefully you can understand it...
config.js
window.env = {
API_URL: 'http://localhost:8080'
};
index.html
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8" />
<link rel="icon" href="%PUBLIC_URL%/favicon.ico" />
<meta name="viewport" content="width=device-width, initial-scale=1" />
<title>My application</title>
<!--
config.js provides all environment specific configuration used in the client
-->
<script src="%PUBLIC_URL%/config.js"></script>
</head>
...
What I ended up doing was setting it up like this:
First I added a configmap.yaml to generate the config.js file
apiVersion: v1
kind: ConfigMap
metadata:
name: config-frontend
data:
config.js: |-
window.env = {
API_URL: "{{ .Values.service.apiUrl }}"
}
Values.service.apiUrl comes from the arguments provided in the "Package and deploy Helm charts" task --set service.apiUrl=$(backend.apiUrl)
Then I added a volume mount in the deployment.yaml to replace the config.js file in the nginx container
...
containers:
...
volumeMounts:
- name: config-frontend-volume
readOnly: true
mountPath: "/usr/share/nginx/html/config.js"
subPath: "config.js"
volumes:
- name: config-frontend-volume
configMap:
name: config-frontend
This did the trick and now I can control the variable from the Azure DevOps pipeline based on the environment I'm deploying to.
You can achieve this in several ways. Following are the few.
1.ConfigMap
Most effective and best way to achieve this, like one of the added comments. You can do something like this with a single config map.
Example ConfigMap might look something like this
apiVersion: v1
kind: ConfigMap
metadata:
name: {{ .Values.definitionName }}-{{ .Values.envName }}-configmap
namespace: {{ .Values.Namespace }}
data:
API_URL: '{{ pluck .Values.envName .Values.API_URL | first }}'
Example Values file in helm charts would look like this
API_URL:
dev: dev.mycompany.io
staging: staging.mycompany.io
test: test.mycompany.io
prod: mycompany.io
And before helm install or helm upgrade run add a step in Azure devOps to run the bash command on your CI/CD pipeline, but make sure you have yq tool installed to do the thing. Or you can use any tool to do the same.
yq w -i values.yaml envName dev
This whole process replaces your config file with API_URL to dev.mycompany.io as I gave dev in yq tool.
But if you are confused with using yq tool or something, you can have multiple values files for each environment separately and make changes to helm install step in your deployment.
helm install ./path --values ./dev-values.yaml
But your configmap should look something like this if you have multiple values files and operating which values to pick from helm install
apiVersion: v1
kind: ConfigMap
metadata:
name: {{ .Values.definitionName }}-{{ .Values.envName }}-configmap
namespace: {{ .Values.Namespace }}
data:
API_URL: '{{ .Values.API_URL }}'
Well this is one way of doing things.
2.Manipulating Dockerfile
You can also do this with dockerfile, something like this step in your dockerfile would replace the value of the file.
RUN sed -i "s/env/dev.mycompany.io/" /app/config.js
But as the url is unique to each env you can take values using ARG
ARG url
RUN sed -i "s/env/${url}" /app/config.js
And during your build pipeline you need to have a task for docker build and under that pass the value of url as an argument you can see that arguments column in your task add this --build-arg url=dev.mycompany.io
This is another way to add values to your config.js file, but it also adds four(based on four envs) docker builds. And so your agents would be busy building four different images for each git commit and queuing up others builds. If you feel that command is not working in Dockerfile add RUN cat /app/config.js in your docker file, and you can debug what's happening and check if the values are updated as you change.
Again it's debatable which is good and bad, but I personally prefer first one due to number of commits I make in an hour, but if the url changes you need not change your codebase you just need to update the docker build in your pipeline. So kinda debatable.
There are other ways to do this as well. But these two are somewhat simplest to achieve.
Hope this is helpful.
In addition to the method of #BinaryBullet provided, you can try with another way that it can make use of one Azure DevOps task to replace the content of config.js file before this .js is applied with kubernetes.
Replace Tokens
The use of this task is very simple.
Step1:
Configure yourself Token prefix:
Step2:
Then apply this Token prefix into your config.js file where you want it be replaced by various values dynamically:
Step3:
Do not forget to specify the value you want it passed to config.js into Variables tab:
Note: The variable name must same with the one you configured in config.js. During the task running, it will inject the corresponding variable value into the config.js file based on the replace format #{}# and same variable name.
For example, I use apiurl in my second screenshots, so here I add one variable apiurl and give it value which I want this value can be replaced into this config.js file at build time.
Build result:
This Replace token task do not has limitation. It can be used in various type file. See my another similar answer: #1.
Hope this is the one which can help you achieve your expectation.
In Azure pipeline I download kubernetes deployment.yml property file which contains following content.
spec:
imagePullSecrets:
- name: some-secret
containers:
- name: container-name
image: pathtoimage/data-processor:$(releaseVersion)
imagePullPolicy: Always
ports:
- containerPort: 8088
env:
My intention is to get the value from pipeline variable $(releaseVersion). But it seems like kubernetes task doesn't allow this value to be accessed from pipeline variable.
I tried using inline configuration type and it works.That means If I copy same configuration as inline content to kubernetes task configuration, it works.
Is there anyway that I can make it work for the configuration from a file?
As I understand, you may want to replace the variable of deployment.yml file content while build executed.
You can use one task which name is Replace Tokens task (Note:The token under this task name is not same with PAToken). This is the task which support replace values of files in projects with environments variables when setting up VSTS Build/Release processes.
Install Replace Tokens from marketplace first, then add Replace Tokens task into your pipeline.
Configure the .yml file path in the Root directory. For me, my target file is under the Drop folder of my local. And then, point out which file you want to operate and replace.
For more argument configured, you can check this doc which I ever refer: https://github.com/qetza/vsts-replacetokens-task#readme
Note: Please execute this task before Deploy to Kubernetes task, so that the change can be apply to the Kubernetes cluster.
Here also has another sample blog can for you refer.
You should have it as part of your pipeline, to substitute environment variables inside the deployment template
Something along the lines of:
- sed -i "s/$(releaseVersion)/${RELEASE_VERSION_IN_BUILD_RUNNER}/" deployment.yml
- kubectl apply -f deployment.yml
You can set the variables in your pipeline. https://learn.microsoft.com/en-us/azure/devops/pipelines/process/variables?view=azure-devops&tabs=yaml%2Cbatch
We have an app that runs on GKE Kubernetes and which expects an auth url (to which user will be redirected via his browser) to be passed as environment variable.
We are using different namespaces per environment
So our current pod config looks something like this:
env:
- name: ENV
valueFrom:
fieldRef:
fieldPath: metadata.namespace
- name: AUTH_URL
value: https://auth.$(ENV).example.org
And all works amazingly, we can have as many dynamic environments as we want, we just do apply -f config.yaml and it works flawlessly without changing a single config file and without any third party scripts.
Now for production we kind of want to use different domain, so the general pattern https://auth.$(ENV).example.org does not work anymore.
What options do we have?
Since configs are in git repo, create a separate branch for prod environment
Have a default ConfigMap and a specific one for prod environment, and run it via some script (if exists prod-config.yaml then use that, else use config.yaml) - but with this approach we cannot use kubectl directly anymore
Move this config to application level, and have separate config file for prod env - but this kind of goes against 12factor app?
Other...?
This seems like an ideal opportunity to use helm!
It's really easy to get started, simply install tiller into your cluster.
Helm gives you the ability to create "charts" (which are like packages) which can be installed into your cluster. You can template these really easily. As an example, you might have you config.yaml look like this:
env:
- name: AUTH_URL
value: {{ .Values.auth.url }}
Then, within the helm chart you have a values.yaml which contains defaults for the url, for example:
auth:
url: https://auth.namespace.example.org
You can use the --values option with helm to specify per environment values.yaml files, or even use the --set flag on helm to override them when using helm install.
Take a look at the documentation here for information about how values and templating works in helm. It seems perfect for your use case
jaxxstorms' answer is helpful, I just want to add what that means to the options you proposed:
Since configs are in git repo, create a separate branch for prod environment.
I would not recommend separate branches in GIT since the purpose of branches is to allow for concurrent editing of the same data, but what you have is different data (different configurations for the cluster).
Have a default ConfigMap and a specific one for prod environment, and run it via some script (if exists prod-config.yaml then use that,
else use config.yaml) - but with this approach we cannot use kubectl
directly anymore
Using Helm will solve this more elegantly. Instead of a script you use helm to generate the different files for different environments. And you can use kubectl (using the final files, which I would also check into GIT btw.).
Move this config to application level, and have separate config file for prod env - but this kind of goes against 12factor app?
This is a matter of opinion but I would recommend in general to split up the deployments by applications and technologies. For example when I deploy a cluster that runs 3 different applications A B and C and each application requires a Nginx, CockroachDB and Go app-servers then I'll have 9 configuration files, which allows me to separately deploy or update each of the technologies in the app context. This is important for allowing separate deployment actions in a CI server such as Jenkins and follows general separation of concerns.
Other...?
See jaxxstorms' answer about Helm.