I am setting up Anthos config sync for my GitHub repositories for application deployment.
I am looking for a git strategy for the following problem:
I built the docker image based on the Pull Request merge on the git repository.
The image version is to be updated in the policy folder for anthos config, where I have either helm or kustomize.
Currently, helm and kustomize are with the same source code repository. So whenever I push the code docker build happens. So, how to update the docker image version in the helm or kustomize?
I have a reservation, I don't want to use the latest tag.
So does that mean:
Do I have to separate the git repository for helm / kustomize deployment from the source code?
And I have to use Github API to commit the file in that git repository for helm / kustomize?
If I use a custom pre-commit hook then also, it does not guarantee it fires to update the git commit hash to docker image and Kubernetes manifests.
Do you have some suggestions, for the git strategy to handle the situation?
Related
My situation is as follows:
have a kubernetes cluster with a couple of nodes
have argocd installed on the cluster and working great
using gitlab for my repo and build pipelines
have another repo for storing my helm charts
have docker images being built in gitlab and pushed to my gitlab registry
have argocd able to point to my helm chart repo and sync the helm chart with my k8s cluster
have helm chart archive files pushed to my gitlab repo
While this is a decent setup, it's not ideal.
The first problem i faced with using a helm chart git repo is that I can't (or don't know) how to differentiate my staging environment with my production environment. Since I have a dev environment and prod environment in my cluster, argocd syncs both environments with the helm chart repo. I could get around this with separate charts for each environment but that isn't a valid solution.
The second problem i faced, while trying to get around the above problem, is that I can't get argocd to pull helm charts from a gitlab oci registry. I made it so that my build pipeline pushed the helm chart archive file to my gitlab container registry with the tag dev-latest or prod-latest, which is great, just what I want. The problem is that argocd, as far as I can tell, can't pull from gitlab's container registry.
How do I go about getting my pipeline automated with gitlab as my repo and build pipeline, helm for packaging my application, and argocd for syncing my helm application with my k8s cluster?
is that I can't get argocd to pull helm charts from a gitlab oci registry.
You might be interested by the latest Jul. 2021 GitLab 14.1:
Build, publish, and share Helm charts
Helm defines a chart as a Helm package that contains all of the resource definitions necessary to run an application, tool, or service inside of a Kubernetes cluster.
For organizations that create and manage their own Helm charts, it’s important to have a central repository to collect and share them.
GitLab already supports a variety of other package manager formats.
Why not also support Helm? That’s what community member and MVP from the 14.0 milestone Mathieu Parent asked several months ago before breaking ground on the new GitLab Helm chart registry. The collaboration between the community and GitLab is part of our dual flywheel strategy and one of the reasons I love working at GitLab. Chapeau Mathieu!
Now you can use your GitLab project to publish and share packaged Helm charts.
Simply add your project as a remote, authenticating with a personal access, deploy, or CI/CD job token.
Once that’s done you can use the Helm client or GitLab CI/CD to manage your Helm charts.
You can also download the charts using the API or the user interface.
What’s next? First, we’d like to present additional metadata for charts.
Then we’ll start dogfooding the feature by using it as a replacement for https://charts.gitlab.io/.
So, try out the feature and let us know how it goes by commenting in the epic GitLab-#6366.
See Documentation and issue.
I'm think I'm about to reinvent the wheel here. I have all the parts but am thinking: Somebody must have done this (properly) before me.
We have a a jenkins CI job that builds image-name:${BRANCH_NAME} and pushes it to a registry. We want to create a CD job that deploys this image-name:${BRANCH_NAME} to kubernetes cluster. And so now we run into the problem that if we call helm upgrade --install with the same image-name:${BRANCH_NAME} nothing happens, even if image-name:${BRANCH_NAME} now actually refers to a different sha256 sum. We (think we) understand this.
How is this generally solved? Are there best practices about this? I see two general approaches:
The CI job doesn't just create image-name:${BRANCH_NAME}, it also creates a unique tag, e.g. image-name:${BRANCH_NAME}-${BUILD_NUMBER}. The CD job never deploys the generic image-name:${BRANCH_NAME}, but always the unique image-name:${BRANCH_NAME}-${BUILD_NUMBER}.
After the CI job has created image-name:${BRANCH_NAME}, its SHA256 sum is retrieved somehow (e.g. with docker inspect or skopeo and helm is called with the SHA256 sum.
In both cases, we have two choices. Modify, commit and track a custom-image-tags.yaml file, or run helm with --set parameters for the image tags. If we go with option 1, we'll have to periodically remove "old tags" to save disk space.
And if we have a single CD job with a single helm chart that contains multiple images, this only gets more complicated.
Surely, there must be some opinionated tooling to do all this for us.
What are the ways to do this without re-inventing this particular wheel for the 4598734th time?
kbld gets me some of the way, but breaks helm
I've found kbld, which allows me to:
helm template my-chart --values my-vals.yml | kbld -f - | kubectl apply -f -
which basically implements 2 above, but now helm is unaware that the chart has been installed so I can't helm uninstall it. :-( I'm hoping there is some better approach...
kbld can also be used "fully" with helm...
Yes, the docs suggest:
$ helm template my-chart --values my-vals.yml | kbld -f - | kubectl apply -f -
But this also works:
$ cat kbld-stdin.sh
#!/bin/bash
kbld -f -
$ helm upgrade --install my-chart --values my-vals.yml --post-renderer ./kbld-stdin.sh
With --post-renderer, helm list, helm uninstall, etc. all still work.
One approach is that every build of Jenkins CI Job should create docker image with new semantic versioned image tag.
To generate the image tag, you need to tag every git commit with a semantic version which is an increment of the previous commit tag.
For Example :
Your first commit in a git repository master branch will be tagged as 0.0.1 and your docker image tag will be 0.0.1
Then when the CI build runs for the next git commit in master branch, that git commit in the git repository will be tagged as 0.0.2 and your docker image tag will be 0.0.2
Since you have single helm chart for multiple images, your CI Build can then download the latest version of your helm chart, change the docker image tag and upload the helm chart with same helm version.
If you create a new git release branch, then it should be tagged with 0.1.0 and docker image created for this new git release branch should be tagged as 0.1.0
You can use this tag in the Maven pom.xml for Java Applications as well.
Using the docker image tag, developers can checkout the correpsonding git tag to find what is the source code corresponding to that docker image tag. It will help them with debugging and also for providing fixes.
Please also read https://medium.com/#mccode/using-semantic-versioning-for-docker-image-tags-dfde8be06699
I have created a K8 deployment for KIE WB and KIE Server.
For KIE WB I have created a Docker Image that configures the Post Commit hook so my repos are pushed to GitHub.
All of this works great.
My question revolves around restoring a repository upon pod creation. So when my pod starts up I would like to clone my GitHub repos then they're restored without having to manually import them.
I thought I could use the GitHub API and get a tarball and drop them into the .niogit/ path but that's not working (I see the file structure in git is different, it looks like just the java source and not the required files for KIE WB recognize that it's a repo).
I know I have to be doing something that has been done but I am not finding anything to get this going. I don't want to reinvent the wheel either :)
Any ideas?
You need to use initContainers with Kubernetes git-sync https://github.com/kubernetes/git-sync/
Please see Answers to Question How to clone a private git repository into a kubernetes pod using ssh keys in secrets?
So the Kubernetes git-sync initContainer will pull a git repository into a local directory which is shared with your Drools Container via emptyDir volume mount.
For More Information on initContainers please read https://kubernetes.io/docs/concepts/workloads/pods/init-containers/
So, what I'm trying to do is use helm to install an application to my kubernetes cluster. Let's say the image tag is 1.0.0 in the chart.
Then, as part of a CI/CD build pipeline, I'd like to update the image tag using kubectl, i.e. kubectl set image deployment/myapp...
The problem is if I subsequently make any change to the helm chart (e.g. number of replicas), and I helm upgrade myapp this will revert the image tag back to 1.0.0.
I've tried passing in the --reuse-values flag to the helm upgrade command but that hasn't helped.
Anyone have any ideas? Do I need to use helm to update the image tag? I'm trying to avoid this, as the chart is not available at this stage in the pipeline.
When using CI/CD to build and deploy, you should use a single source-of-truth, that means a file versioned in e.g. Git and you do all changes in that file. So if you use Helm charts, they should be stored in e.g. Git and all changes (e.g. new image) should be done in your Git repository.
You could have a build pipeline that in the end commit the new image to a Kubernetes config repository. Then a deployment pipeline is triggered that use Helm or Kustomize to apply your changes and possibly execute tests.
I have an existing github project. I want to create/add a helm folder to the project to store the helm yaml files. I want to reference this github project/folder to act like a helm repo in my local/dev environment. I know I can add the charts to my local/default helm repo. The use case is if another developer checks out the code in github and he needs to work on the charts then he can run helm install directly from the working folder. The helm.sh website has instructions of adding a gh-pages branch but I am wondering if I can avoid it.
Can I use an existing github project and it via the helm repo add command?
Unfortunately, I wasn't able to find a way to publish helm charts via GitHub using private repositories. On a theoretical level, it might work using GitHub token and 2nd (raw URLs method), but I haven't tried it. Since you're using docker registry anyway, it might be worth trying using OCI (docker) registry to store the charts.
If that doesn't work, or you have public repos, it is possible to either use GitHub Pages, or use GitHub raw URLs. Both of the solutions require public repository.
To use GitHub pages:
Setup github pages to publish docs folder as github pages (you can use a different name, just substitue later)
Package the helm repo as .tgz (using helm package): helm package charts/mychart -d docs/. Substitute charts/mychart with a path to a chart root folder
Include an index.yaml -- an index file for the repository helm repo index ./docs --url https://<YOUR_ORG_OR_USERNAME>.github.io/<REPO_NAME>
Now you can add the repo: helm repo add <INTERNAL_NAME> https://<YOUR_ORG_OR_USERNAME>.github.io/<REPO_NAME>
To use Raw URLs:
Place index.yaml and chart TGZs into a folder called docs, just like above
Now you can add a repo: helm repo add <INTERNAL_NAME> https://raw.githubusercontent.com/<YOUR_ORG_OR_USERNAME>/<REPO_NAME>/<BRANCH_USUALLY_MASTER>/docs
Firstly make sure that you have have fully functional helm repository. The tricky part is to access it as if it was simple HTTP server hosting raw files. Fortunately Github provides such feature using raw.githubusercontent.com. In order for helm to be able to pull files from such repository you need to provide it with Github username and token (Personal Access Token):
> helm repo add - username <your_github_username> - password <your_github_token> my-github-helm-repo 'https://raw.githubusercontent.com/my_organization/my-github-helm-repo/master/'
> helm repo update
> helm repo list
NAME URL
stable https://kubernetes-charts.storage.googleapis.com
local http://127.0.0.1:8879/charts
my-github-helmrepo https://raw.githubusercontent.com/my_organization/my-github-helm-repo/master/
> helm search my-app
NAME CHART VERSION APP VERSION DESCRIPTION
my-github-helmrepo/my-app-chart 0.1.0 1.0 A Helm chart for Kubernetes
These are steps for adding new packages to existing repository
If you want to add new package to existing repository simply:
1. Place new package in your local repository root
2. Execute: helm repo index .. This will detect new file/folder and make updates.
3. Commit and push your new package
4. Finally execute command: helm repo update
Security ascpect
It is important to realize where does helm actually store your Github token. It is stored as plain text in ~/.helm/repository/repositories.yaml. In this case it will be good to generate token with as few permissions as possible.
Take a look here: hosting helm private repository.