Have successfully implemented Vault with Kubernetes and applications running in K8s are getting their environment variables from Hashicorp vault. Everything is great! But, want to take a step forward and want to restart the pod whenever a change is made to the secret in the Vault, as of now, we have to restart the pod manually to reset environment variables whenever we make changes to Vault secret. How this can be achieved? Have heard about confd but not sure how it can be implemented!
Use reloader https://github.com/stakater/Reloader. We found it quite useful in our cluster. It does a rolling update hence you can change config with zero downtime too. Also if you made some errors in configmap you can easily do a rollback.
A couple ideas, depending on how much effort you want to put into it:
Just restart the pod every so often. A hacky way to do this is with a liveness probe, like this answer. Drawback is you can't use the liveness probe as a real health check without additional scripting.
Create an operator that polls Vault for changes and instructs Kubernetes to restart the pod when a change is detected. Not sure if Vault has an events API that you could use for that.
https://www.vaultproject.io/docs/agent/template#renewals-and-updating-secrets
If a secret or token isn't renewable or leased, Vault Agent will fetch the secret every 5 minutes. This is not configurable. Non-renewable secrets include (but not limited to) KV Version 2.
Related
I deployed a k8s secret.
I deployed a pod that uses the above secret.
Now I wanted to change the secret data, so I updated my k8s secret.
And restarted my pod., so that it reads new data.
I did things in the sequence mentioned above.
But still, my service inside the pod is reading old secret data.
Now, I have two doubts:
Can someone please explain why is this happening??
Do secrets override my env variables(if same), when do they exactly
override, is it after the service comes up??
Summarize the problem:
Any way we can add an ENV to a pod or a new pod in kubernetes?
For example, I want to add HTTP_PROXY to many pods and the new pods it will generate in kubeflow 1.4. So these pods can be access to internet.
Describe what you’ve tried:
I searched and found istio maybe do that, but it's too complex for me.
The second, there are too many yamls in kubeflow, as to I cannot modify them one by one to use configmap or add ENV just in them.
So anyone has a good simle way to do this? Like doing this in kubernetes configuation.
Use "PodPreset" object to inject common environment variables and other params to all the matching pods.
Please follow below article
https://v1-19.docs.kubernetes.io/docs/tasks/inject-data-application/podpreset/
If PodPreset is indeed removed from v1.20, then you seem to need a webhook.
You will have to run an additional service in your cluster that will change the configuration of the pods.
Here is an example, on the basis of which I created my webhook, which changed the configuration of the pods in the cluster, in this example the developer used the logic adding a sidecar to the pod, but you can set your own to forward the required ENV:
https://github.com/morvencao/kube-mutating-webhook-tutorial/blob/master/medium-article.md
I'm totally new to Vault and what I want is to detect when a secret changes and execute some code in response. I've been googling for resources about how to do that but haven't found anything useful. From what I've read and learnt, I think the only way of achieving what I want is by implementing a custom secrets engine. Am I right? Do you know a better way of achieving what I want?
There is no event option in the vault as of now, so on changes, we get notified it's natively changing the Key/value pairs.
i would recommend using the polling method if you have any such scenario with the vault.
Here is one nice CRD which also does the polling option and syncs the vault secret to Kubernetes secret.
This might useful for reference : https://github.com/DaspawnW/vault-crd
There currently are no triggers that'll tell you when the secret has changed. If you're running kubernetes (sidecar) or using the Vault agent, this is minimized as the agent will auto-pull any new secrets down (configurable).
We are using secret as environment variables on pod, but every time we have updated on secrets, we are redeploying the pods to take changes effect. We are looking for a mechanism where Pods get restarted automatically whenever secrets gets updated. Any help on this?
Thanks in advance.
There are many ways to handle this.
First, use Deployment instead of "naked" Pods that are not managed. The Deployment will create new Pods for you, when the Pod template is changed.
Second, to manage Secrets may be a bit tricky. It would be great if you can use a setup where you can use Kustomize SecretGenerator - then each new Secret will get its unique name. In addition, that unique name is reflected to the Deployment automatically - and your pods will automatically be recreated when a Secret is changed - this match your origin problem. When Secret and Deployment is handled this way, you apply the changes with:
kubectl apply -k <folder>
If you mount your secrets to pod it will get updated automatically you don't have to restart your pod as mentioned here
Other approaches are staker reloader which can reload your deployments based on configs, secrets etc
There are multiple ways of doing this:
Simply restart the pod
this can be done manually, or,
you could use an operator provided by VMware carvel kapp controller (documentation), using kapp controller you can reload the secrets/ configmap without needing to restart the pods (which effectively runs helm template <package> on a periodic basis and applies the changes if it founds any differences in helm template), check out my design for reloading the log level without needing to restart the pod.
Using service bindings https://servicebinding.io/
Currenly when updating a kubernetes secrets file, in order to apply the changes, I need to run kubectl apply -f my-secrets.yaml. If there was a running container, it would still be using the old secrets. In order to apply the new secrets on the running container, I currently run the command kubectl replace -f my-pod.yaml .
I was wondering if this is the best way to update a running container secret, or am I missing something.
Thanks.
For k8s' versions >v1.15: kubectl rollout restart deployment $deploymentname: this will
restart pods incrementally without causing downtime.
The secret docs for users say this:
Mounted Secrets are updated automatically
When a secret being already consumed in a volume is updated, projected keys are eventually updated as well. The update time depends on the kubelet syncing period.
Mounted secrets are updated. The question is when. In case a the content of a secret is updated does not mean that your application automatically consumes it. It is the job of your application to watch file changes in this scenario to act accordingly. Having this in mind you currently need to do a little bit more work. One way I have in mind right now would be to run a scheduled job in Kubernetes which talks to the Kubernetes API to initiate a new rollout of your deployment. That way you could theoretically achieve what you want to renew your secrets. It is somehow not elegant, but this is the only way I have in mind at the moment. I still need to check more on the Kubernetes concepts myself. So please bear with me.
Assuming we have running pod mypod [mounted secret as mysecret in pod spec]
We can delete the existing secret
kubectl delete secret mysecret
recreate the same secret with updated file
kubectl create secret mysecret <updated file/s>
then do
kubectl apply -f ./mypod.yaml
check the secrets inside mypod, it will be updated.
In case anyone (like me) want to force rolling update pods which are using those secrets. From this issue, the trick is to update an Env variable inside the container, then k8s will automatically rolling update entire pods
kubectl patch deployment mydeployment -p '{"spec":{"template":{"spec":{"containers":[{"name":"mycontainer","env":[{"name":"RESTART_","value":"'$(date +%s)'"}]}]}}}}'
By design, Kubernetes won't push Secret updates to running Pods. If you want to update the Secret value for a Pod, you have to destroy and recreate the Pod. You can read more about it here.