How to define Kubernetes Secret Map in a declarative way in CI without committing secrets to git? - kubernetes

Question

I want to define a Kubernetes secrets map as part of my deployment pipeline. According to the Kubernetes documentation, there are two ways to define a secret.
Declarative Using a .yml with the Secret Object
Imperative Using kubectl create secret generic
The declarative approach requires writing a YAML similar to the one below.
apiVersion: v1
kind: Secret
metadata:
name: test-secret
data:
username: bXktYXBw
password: Mzk1MjgkdmRnN0pi
I want to be able to check in all the Kubernetes YAML files to git so I can run them on a CI server. Checking in the YAML means that the secret is stored in git which is not good for security. I can put the secret into my CI systems secret's store but then how do I create a secrets YAML that references the OS environment variable at the time that kubectl is called.
Questions:
How to define a Kubernetes Secret from within a CI pipeline without having to commit the secrets into source control?
Is there a best practice approach for defining secrets as part of CI for K8s?

Answer 1

There is no really good way to managed secrets securely with a vanilla Kubernetes. If you decrypt the secret or inject an unencrypted secret in your CI/CD pipeline and create a Kubernetes Secret, you'll have a decrypted Base64 encoded string to store in your Kubernetes cluster (Etcd).
Most companies I've worked with recently deciding to either keep the secret in their Key Vault and use a Kubernetes controller to inject the secret at runtime or use a controller to be able to manage encrypted secrets like sealed-secrets or Kamus. Using encrypted secrets might be a good option if you want to keep your secrets in Git.
First-class support for Hashicorp Vault and Kubernetes : https://github.com/hashicorp/vault-k8s
Take a look at this blog post from Banzai Cloud for a more detailed explanation : Inject secrets directly into Pods from Vault revisited

Answer 2

I ended up hacking this with bash scrip to output a yaml secret secret.yaml.sh
cat <<EOF
apiVersion: v1
kind: Secret
metadata:
name: test-secret
type: Opaque
data:
username: $1
password: $2
jdbcUrl: $3
EOF
Then in my CI pipeline invoke the secret.yml.sh and pass in the base64 encoded values which are store in the CI system's credentials store then pipe to kubectl like so ./secret.yaml.sh $USERNAME $PASSWORD $URL | kubectl apply -f -
This hack makes it possible for me to run the CI pipeline and update the secrets based on what is stored in the CI systems.
As others have noted the secrets in Kubernetes etcd are not secure and it's better to use a key management system with k8s. However I don't have access to a key vault for this project.

Answer 3

You can encrypt the secret and commit the encrypted secret in git and while deployment it needs to be decrypted. For example ansible vault can be used if you are using ansible as CI tool.
If you are using Jenkins then you can use credentials or Hashicorp vault plugin for storing the secret.
If you are on public cloud then AWS KMS, Azure Vault etc are available.