We are trying to create an environment using crossplane and argocd. Once Crossplane generates the database and saves the credentials to a secret on the management cluster. After we are deploying the credentials from management cluster to our destination cluster to a secret.
Now we need to pass the credentials from secret a to secret B which the application knows about. The issue starts when argo do not use helm install but template thus lookup function don't work. We thought about using vault as a middle man but we are not sure how to load values from secret to vault.
Anyway if you encounter such an issue or have some sort of a solution we'll be very happy to hear.
Thank you
You need to commit the (encrypted) secrets somewhere for ArgoCD to pick them up. That is the whole point of GitOps.
Alternatively you can try using https://argo-cd.readthedocs.io/en/stable/user-guide/parameters/ but this is considered a temporary workaround
Related
We run a Vault cluster (Deployed by helm) and some microservices all on k8s.
Our MongoDB atlas connection string configured as ENV on microservices deployment.
We want to continue using ENV without changing the code to read the vault config file. So, we tried the examples from here:
https://www.vaultproject.io/docs/platform/k8s/injector/examples
The injection to ENV works but when the vault rotates the credentials we need to recreate the pod that it will inject again to the ENV.
I would like to know How we may use the functionality of dynamic secrets in Vault with ENV on k8s. If you have any suggestions.
Thanks
If you are using an environment variable to inject a secret, you will need to recreate the pod whenever the secret changes (as you've found), because the environment variable is only generated at startup of the pod - it is not possible to change an environment variable for a running application. If you want your application to support credentials that change while it is running, you will need to add support for that to your application I'm afraid (and change from using an env var to reading the details from the file when required).
I have come across injectors/drivers/et cetera for Kubernetes for most major secret providers, but the common theme with those solutions are that these only sync one-way, i.e., only from the vault to the cluster. I want to be able to update the secrets too, from my Kubernetes cluster.
What is the recommended pattern for doing this? (Apart from the obvious solution of writing a custom service that communicates with the vault)
I'd say that this is an anti pattern, meaning you shouldn't do that.
If you create your secret in k8s from file, that would mean you either have it in version control, something you should never do. Or you don't have it in version control or create it from literal, which is good, but than you neither have a change history/log nor a real documentation of your secret. I guess that would explain, why the major secret providers don't support that.
You should set up the secret using the key vault and apply it to your cluster using Terraform for example.
Terraform supports both azure key vault secret https://www.terraform.io/docs/providers/azurerm/r/key_vault_secret.html and Kubernetes secrets https://www.terraform.io/docs/providers/kubernetes/r/secret.html
You can simply import the key vault secret and use it in the k8s secret. Every time you update the key vault secret, you apply the changes with Terraform.
I have a couple Charts which all need access to the same Kubernetes Secret. My initial plan was to create a Chart just for those Secrets but it seems Helm doesn't like that. I am thinking this must be a common problem and am wondering what folks generally do to solve this problem?
Thanks!
Best practice is, don't save any sensitive secrets in kubernetes clusters. kubernetes secret is encode, not encrypt.
You can reference the secret via aws ssm/secrets manager, hashicorp Vault or other similars.
https://github.com/aws-samples/aws-workshop-for-kubernetes/tree/master/04-path-security-and-networking/401-configmaps-and-secrets
Most charts that follow the common chart development practices allow you to use an existing secret instead of creating one for you. This way, you can create your common secrets normally (without helm), and refer to them from the charts that need them, via a reference like existingSecret config key.
Take minio helm chart for example: it accepts an existingSecret key as an alternative to passing an accessKey and a secretKey.
As you can see in the main charts repo, this is a pretty common practice.
I would like to use a Secret inside a ConfigMap. Is this possible?
Example:
An example where this might be required is if you would like to write from Fluentd to S3. In the configuration you have to add your AWS credentials.
Alternatives:
Using environment variables on the cluster itself. I do not like this idea, because the variable would still contain the secret as plain text.
Passing the password during set-up. If you are using deployment tools it might be possible to pass the secret during the deployment of your application. This is also not a nice solution since you are still passing the secret as plain text to the deployment tool. An advantage of this approach is that you do not accidentally check-in your secret to git.
Try to avoid making use of aws credentials in kubernetes.
As you can see aws_key_id and aws_sec_key are the optional fields.
Make use of AWS IAM role and assign it to the kubernetes nodes.
And then try to run your fluentd application without aws credentials in its config.
Just give it a try.
Hope this helps.
Update:
This article explain different ways to use aws iam for kubernetes.
Kube2iam and many other tools like this, might help. Give it a try.
No, it is not possible. You should always use secret for your sensitive data.
By default, secrets are only base64 encoded content of files so you should use something like Vault to secure store you sensitive data.
I am trying to use Helm charts to install applications in Kubernetes clusters. Can someone please suggest what could be a better solution to manage secrets? Using helm secrets would be a good idea or Hashicorp Vault?
Vault is technically awesome, but it can be an administrative burden. You can get strong protection of "secrets", whatever they may be; you can avoid ever sharing magic secrets like the your central database password by generating single-use passwords; if you need something signed or encrypted, you can ask Vault to do that for you and avoid ever having to know the cryptographic secret yourself. The big downsides are that it's a separate service to manage, getting secrets out of it is not totally seamless, and you occasionally need to have an administrator party to unseal it if you need to restart the server.
Kubernetes secrets are really just ConfigMaps with a different name. With default settings it's very easy for an operator to get out the value of a Secret (kubectl get secret ... -o yaml, then base64 decode the strings), so they're not actually that secret. If you have an interesting namespace setup, you generally can't access a Secret in a different namespace, which could mean being forced to copy around Secrets a lot. Using only native tools like kubectl to manage Secrets is also a little clumsy.
Pushing credentials in via Helm is probably the most seamless path – it's very easy to convert from a Helm value to a Secret object to push into a container, and very easy to push in values from somewhere like a CI system – but also the least secure. In addition to being able to dump out the values via kubectl you can also helm get values on a Helm release to find out the values.
So it's a question of how important keeping your secrets really secret is, and how much effort you want to put in. If you want seamless integration and can limit access to your cluster to authorized operators and effectively use RBAC, a Helm value might be good enough. If you can invest in the technically best and also most complex solution and you want some of its advanced capabilities, Vault works well. Maintaining a plain Kubernetes secret is kind of a middle ground, it's a little more secure than using Helm but not nearly as manageable.