How do I automate Kubernetes deployment YAML without relying on :latest? - kubernetes

Question

I have a repository with a Kubernetes deployment YAML. Pipelines run on each commit that builds and pushes an image into our repository, versioned with the commit (eg. my_project:bm4a83). Then I'm updating the deployment image
kubectl set image deployment/my_deployment my_deployment=my_project:bm4a83.
This works, but I also want to keep the rest of the deployment YAML specification in version control.
I thought I could just keep it in the same repository, but that means my changes that may only be infrastructure (eg, changing replicas) triggers new builds, without code changes.
What felt like it made the most sense was keeping the deployment YAML in a totally separate repository. I figured I can manage all the values from there, independently from actual code changes. The only problem with that is the image key would need to be kept up to date. The only way around that, is working with some floating :latest-type version, but I don't really think that's ideal.
What's a sensible workflow for managing this? Am I missing something entirely?

Answer 1

What's a sensible workflow for managing this? Am I missing something entirely?
Some of the answer depends on the kind of risk you're trying to drive down with any process you have in place. If it's "the cluster was wiped out by a hurricane and I need to recover my descriptors," then Heptio Ark is a good solution for that. If the risks are more "human-centric," then IMHO you will have to walk a very careful line between locking down all the things and crippling the very agile, empowering, tools that kubernetes provides to a team. A concrete example of that model running up against your model is: what happens when a developer edits a Deployment but does not (remember|know) to update the descriptor in the repo? So do you revoke the edit rights? Use some diff-esque logic to detect a changed in-cluster config?
To speak to something you said specifically: it is a highly suboptimal idea to commit a descriptor change just to resize a (Deployment|ReplicationController|StatefulSet). Separately, a well-built CI pipeline would also understand if no buildable artifact changed and bail out (either early, or not even triggering a build, if the CI tool is that smart).
Finally, if you do want to carry on with the current situation, then the best practice I can think of is textual replacement right before applying a descriptor:
$ grep "image: " the-deployment.yml
image: example.com/something:#CI_PIPELINE_IID#
$ sed -i'' -e "s/#CI_PIPELINE_IID#/${CI_PIPELINE_IID}/" the-deployment.yml
$ kubectl apply -f the-deployment.yml
so that the copy in the repo remains textually pristine, and also isn't inadvertently actually applied since it won't actually result in a runnable Deployment.

Answer 2

but I also want to keep the rest of the deployment YAML specification in version control.
Yes, you want to do that. Putting everything under version control is a good practice to achieve immutable infrastructure.
If you want the deployment to have a separate piece of metadata (for whatever auditing / change tracking reason), why can't you just leverage the Kubernetes metadata block?
metadata:
name: my_deployment
commit: bm4a83
Then you inject such information through Jinja, Ruby ERBs, Go Templates, etc.