When managing secrets in argocd, I encode the value in argocd-secret with base64 and set it to manifest.
In that case, although I use kusotomize, build and apply it, secret encoded by base64 will be encoded to base64 further.
$ echo -n "clientid" | base64
Y2xpZW50aWQ=
$ echo -n "clientsecret" | base64
Y2xpZW50c2VjcmV0
---
apiVersion: v1
kind: Secret
metadata:
name: argocd-secret
type: Opaque
stringData:
dex.github.clientID: Y2xpZW50aWQ=
dex.github.clientSecret: Y2xpZW50c2VjcmV0
If you use kustomize build to build and check the difference, it looks like this
$ kustomize build --load_restrictor none overlays/dev/ap-northeast-1/argocd | k diff -f -
+ dex.github.clientID: WTJ4cFpXNTBhV1E9
+ dex.github.clientSecret: WTJ4cFpXNTBjMlZqY21WMA==
I don't understand why something encoded by base64 is encoded further.
I'm going to ask for someone's help.
Thanks.
I reproduced your case and it looks like it isn't further encoded by kustomize but by kubectl (either by kubectl client itself or by kube-apiserver performing the operation requested by e.g. kubectl apply command).
And here you can find a fragment that sheds some light on why this is actually happening:
The Secret contains two maps: data and stringData. The data field is
used to store arbitrary data, encoded using base64. The stringData
field is provided for convenience, and allows you to provide secret
data as unencoded strings.
The data field serves for storing the data already encoded using base64 and value of this field isn't further encoded when we apply the Secret. The stringData however behaves quite differently. It allows you to provide secret data as unencoded strings and for that reason it is further encoded when you run kubectl apply. In your example you use stringData field and although it contains already encoded data, it is treated as any other string and in consequence is encoded again.
Solution: simply use data rather than stringData map in your Secret and your base64 encoded strings won't be encoded again during the resource creation.
It's actually secret controller who encodes the string. So, you can't pass an encoded string into a yaml. You need to leave it in plain text:
$ echo -n demo | base64
ZGVtbw==
$ echo -n ZGVtbw== | base64
WkdWdGJ3PT0=
$ kubectl create secret generic demo --from-literal=key=demo
secret/demo created
$ kubectl create secret generic demo2 --from-literal=key=ZGVtbw==
secret/demo2 created
$ kubectl get secret demo demo2 -oyaml
apiVersion: v1
items:
- apiVersion: v1
data:
key: ZGVtbw== <- word demo got encoded
kind: Secret
metadata:
creationTimestamp: "2020-06-04T13:41:27Z"
name: demo
namespace: default
resourceVersion: "10118413"
selfLink: /api/v1/namespaces/default/secrets/demo
uid: bc39444c-e4f6-43ba-b151-705e15811831
type: Opaque
- apiVersion: v1
data:
key: WkdWdGJ3PT0= <- encoded string of demo got encoded further
kind: Secret
metadata:
creationTimestamp: "2020-06-04T13:41:47Z"
name: demo2
namespace: default
resourceVersion: "10118469"
selfLink: /api/v1/namespaces/default/secrets/demo2
uid: 2fe63343-06b4-4f47-94cd-9fe8e12fb388
type: Opaque
kind: List
metadata:
resourceVersion: ""
selfLink: ""
Note: this was not always like that. And it can actually change again. Back in the days, you had to encode it, then you must not, then again...
Related
I am taking over a k3s cluster and am trying to solve a problem with this:
apiVersion: v1
kind: Secret
metadata:
name: clouddesktop-prod-secrets
namespace: clouddesktop-prod
data:
tskey: dHNrZXkta0drTWVWNkNOVFJMLVNlRkZKVFFRalM3RDgzRllvVkxCTQ==
... is used along with with this snippet in my deployment.yaml
- name: AUTHKEY
valueFrom:
secretKeyRef:
name: clouddesktop-prod-secrets
key: tskey
If I am understanding it correctly, the value under tskey will be "decrypted" and then made available as an envirnment variable called ENV_AUTHKEY.
In other words the decryption process that k3s applies will convert the encrypted value "dHNrZXkta0drTWVWNkNOVFJMLVNlRkZKVFFRalM3RDgzRllvVkxCTQ==" into a plaintext value, e.g. "tskey-abc145f" and makes it available to the runnning container as environment variable "ENV_AUTHKEY"
I have verified that indeed an environment variable called "ENV_AUTHKEY" is created, in other words that k3s appends ENV_ to the name.
But as far as I can tell the plaintext is wrong. The environment variable "ENV_AUTHKEY" is indeed created, but, it seems to not have the expected value.
Now by the documentation left for me by my predecessor, I am to create the encrypted value with this simple step:
echo -n "tskey-abc145f" | base64
So I am using base64 as the "encryption" expecting that this is what k3s expects. But the eventually decrypted value appears to me to be incorrect.
What I am trying to determine is what k3s will use to decrypt my encrypted value.
To view how kubernetes would encode the input string, you "may" run the following:
echo -n "tskey-abc145f" | base64
dHNrZXktYWJjMTQ1Zg==
In the following example, I am creating a secret from the command line and providing the key and value in plain text format.
kubectl create secret generic clouddesktop-prod-secrets -n clouddesktop-prod --from-literal=tskey=tskey-abc145f
Verify the secret and its encoded value; here, you can notice the value is encoded automatically. However, when you create a secret using a manifest file, you would have to provided base64 encoded string.
kubectl get secret clouddesktop-prod-secrets -n clouddesktop-prod
apiVersion: v1
kind: Secret
metadata:
name: clouddesktop-prod-secrets
namespace: clouddesktop-prod
data:
tskey: dHNrZXktYWJjMTQ1Zg==
Validate that the encoded value present in the secret is decoding back to the original string(tskey-abc145f):
kubectl get secrets -n clouddesktop-prod clouddesktop-prod-secrets -o go-template='{{.data.tskey|base64decode}}'
tskey-abc145f
Tested on k3s:
k3s --version
k3s version v1.23.3+k3s1 (5fb370e5)
go version go1.17.5
I am new to Kubernetes, and trying to wrap my head around how to set secrets. I will try to illustrate with an example. Let's say I have a secret resource that looks like:
apiVersion: v1
kind: Secret
metadata:
name: big-secrets
type: Opaque
data:
secret-password: "secretMeow"
secret-key: "angryWoof"
database-password: "happyWhale"
How do I set up (not sure if set up is the right word here) these secrets using kubectl, and how do I retrieve them? I have tried reading through the following documentation but I am still not sure how to set up from a yaml file as mentioned above.
Using stringData it is possible to create a secret using plain text values:
apiVersion: v1
kind: Secret
metadata:
name: big-secrets
type: Opaque
stringData:
secret-password: "secretMeow"
secret-key: "angryWoof"
database-password: "happyWhale"
However it's worth noting it's provided more for convenience and the docs say:
It is provided as a write-only convenience method. All keys and values are merged into the data field on write, overwriting any existing values. It is never output when reading from the API.
For the question of how to read the secret, if you wanted it programatically, you could use a tool like jq to parse the Kubernetes output and base64 decode:
kubectl get secret big-secrets -o json | jq -r '.data["secret-password"] | #base64d'
This would get the created secret as JSON (-o yaml is also an option), read the data field for a given secret (secret-password in this case) and then base64 decode it.
Using yaml file you cannot create secret without encoded string value. Secret data value must be in base64 encoded in yaml file. If you want to create a secret from yaml file then you need to decode data value like below.
apiVersion: v1
kind: Secret
metadata:
name: big-secrets
type: Opaque
data:
secret-password: c2VjcmV0TWVvdwo=
secret-key: YW5ncnlXb29mCg==
database-password: aGFwcHlXaGFsZQo=
Or
You can use kubectl imperative command to create a secret from literal values like below.
kubectl create secret generic big-secrets --from-literal=secret-password="secretMeow" --from-literal=secret-key="angryWoof"
I am using Kubernetes to deploy my grafana dashboard and I am trying to use Kubernetes Secrets for saving grafana admin-password .. Here is my yaml file for secret
apiVersion: v1
kind: Secret
metadata:
name: $APP_INSTANCE_NAME-grafana
labels:
app.kubernetes.io/name: $APP_INSTANCE_NAME
app.kubernetes.io/component: grafana
type: Opaque
data:
# By default, admin-user is set to `admin`
admin-user: YWRtaW4=
admin-password: "$GRAFANA_GENERATED_PASSWORD"
value for GRAFANA_GENERATED_PASSWORD is base64 encoded and exported like
export GRAFANA_GENERATED_PASSWORD="$(echo -n $PASSWORD | base64)"
where PASSWORD is a variable which i exported on my machine like
export PASSWORD=qwerty123
I am trying to pass the value of GRAFANA_GENERATED_PASSWORD to the yaml file for secret like
envsubst '$GRAFANA_GENERATED_PASSWORD' > "grafana_secret.yaml"
The yaml file after passing the base64 encoded value looks like
apiVersion: v1
kind: Secret
metadata:
name: kafka-monitor-grafana
labels:
app.kubernetes.io/name: kafka-monitor
app.kubernetes.io/component: grafana
type: Opaque
data:
# By default, admin-user is set to `admin`
admin-user: YWRtaW4=
admin-password: "cXdlcnR5MTIz"
After deploying all my objects i couldn't login to my dashboard using password qwerty123 which is encoded properly ..
But when i try to encode my password like
export GRAFANA_GENERATED_PASSWORD="$(echo -n 'qwerty123' | base64)"
It is working properly and i can login to my dashboard using the password qwerty123 ..
Looks like the problem occur when i encode my password using a variable ...
But i have encode my password using a variable
As mentioned in #Pratheesh comment, after deploy the grafana for the first time, the persistent volume was not deleted/recreated and the file grafana.db that contains the Grafana dashboard password still keeping the old password.
In order to solve, the PersistentVolume (pv) need to be deleted before apply the secret with the new password.
(using kubernetes v1.15.7 in minikube and matching client version and minikube 1.9.0)
If I kubectl apply a secret like this:
apiVersion: v1
data:
MY_KEY: dmFsdWUK
MY_SECRET: c3VwZXJzZWNyZXQK
kind: Secret
metadata:
name: my-secret
type: Opaque
then subsequently kubectl apply a secret removing the MY_SECRET field, like this:
apiVersion: v1
data:
MY_KEY: dmFsdWUK
kind: Secret
metadata:
name: my-secret
type: Opaque
The data field in the result is what I expect when I kubectl get the secret:
data:
MY_KEY: dmFsdWUK
However, if I do the same thing using stringData instead for the first kubectl apply, it does not remove the missing key on the second one:
First kubectl apply:
apiVersion: v1
stringData:
MY_KEY: value
MY_SECRET: supersecret
kind: Secret
metadata:
name: my-secret
type: Opaque
Second kubectl apply (stays the same, except replacing MY_KEY's value with b2hubyEK to show the configuration DID change)
apiVersion: v1
data:
MY_KEY: b2hubyEK
kind: Secret
metadata:
name: my-secret
type: Opaque
kubectl get result after applying the second case:
data:
MY_KEY: b2hubyEK
MY_SECRET: c3VwZXJzZWNyZXQ=
The field also does not get removed if the second case uses stringData instead. So it seems that once stringData is used once, it's impossible to remove a field without deleting the secret. Is this a bug? Or should I be doing something differently when using stringData?
kubectl apply need to merge / patch the changes here. How this works is described in How apply calculates differences and merges changes
I would recommend to use kustomize with kubectl apply -k and use the secretGenerator to create a unique secret name, for every change. Then you are practicing Immutable Infrastructure and does not get this kind of problems.
A brand new tool for config manangement is kpt, and kpt live apply may also be an interesting solution for this.
The problem is that stringData is a write only field. It doesn’t have convergent behavior so it breaks the merge patch generator system. Most high level tools fix this by converting to normal data before dealing with the patch.
I want to store files in Kubernetes Secrets but I haven't found how to do it using a yaml file.
I've been able to make it using the cli with kubectl:
kubectl create secret generic some-secret --from-file=secret1.txt=secrets/secret1.txt
But when I try something similar in a yaml:
apiVersion: v1
kind: Secret
metadata:
name: some-secret
type: Opaque
data:
secret1.txt: secrets/secret1.txt
I´ve got this error:
[pos 73]: json: error decoding base64 binary 'assets/elasticsearch.yml': illegal base64 data at input byte 20
I'm following this guide http://kubernetes.io/docs/user-guide/secrets/. It explains how to create a secret using a yaml but not how to create a secret from a file using yaml.
Is it possible? If so, how can I do it?
As answered on previous post, we need to provide the certificate/key encoded as based64 to the file.
Here is generic example for a certiticate (in this case SSL):
The secret.yml.tmpl:
apiVersion: v1
kind: Secret
metadata:
name: test-secret
namespace: default
type: Opaque
data:
server.crt: SERVER_CRT
server.key: SERVER_KEY
Pre-process the file to include the certificate/key:
sed "s/SERVER_CRT/`cat server.crt|base64 -w0`/g" secret.yml.tmpl | \
sed "s/SERVER_KEY/`cat server.key|base64 -w0`/g" | \
kubectl apply -f -
Note that the certificate/key are encoded using base64 without whitespaces (-w0).
For the TLS can be simply:
kubectl create secret tls test-secret-tls --cert=server.crt --key=server.key
You can use --dry-run flag to prepare YAML that contains data from your files.
kubectl create secret generic jwt-certificates --from-file=jwt-public.cer --from-file=jwt-private.pfx --dry-run=true --output=yaml > jwt-secrets.yaml
Edit
Thanks to #Leopd for comment about API deprecation, new kubectl uses this command:
kubectl create secret generic jwt-certificates --from-file=jwt-public.cer --from-file=jwt-private.pfx --dry-run=client --output=yaml > jwt-secrets.yaml
On my machine I still have old kubectl version
When using the CLI format basically you're using a generator of the yaml before posting it to the server-side.
Since Kubernetes is client-server app with REST API in between, and the actions need to be atomic, the posted YAML needs to contain the content of the file, and best way to do that is by embedding it as a base64 format in-line. It would be nice if the file could be otherwise embedded (indentation maybe could be used to create the boundaries of the file), but I haven't seen any example of such until now.
That being said, putting a file reference on the yaml is not possible, there is no pre-flight rendering of the yaml to include the content.
So I just learned a super useful k8s fundamental I missed, and then discovered it has a security vulnerability associated with it, and came up with a resolution.
TLDR:
You can have cleartext multiline strings/textfiles as secret.yaml's in your secret repo !!! :)
(Note I recommend storing this in Hashicorp Vault, you can store versioned config files that have secrets, and easily view/edit them through the vault webpage, and unlike a git repo, you can have fine grain access control, pipelines can use the REST API to pull updated secrets which makes password rotation mad easy too.)
cleartext-appsettings-secret.yaml
appsettings.Dummy.json is the default file name (key of the secret)
(I use the word default file name as you could override it in the yaml mount)
and the clear text json code is the file contents (value of the secret)
apiVersion: v1
kind: Secret
metadata:
name: appsettings
namespace: api
type: Opaque
stringData:
appsettings.Dummy.json: |-
{
"Dummy": {
"Placeholder": {
"Password": "blank"
}
}
}
When I
kubectl apply -f cleartext-appsettings-secret.yaml
kubectl get secret appsettings -n=api -o yaml
The secret shows up cleartext in the annotation...
apiVersion: v1
data:
appsettings.Dummy.json: ewogICJEdW1teSI6IHsKICAgICJQbGFjZWhvbGRlciI6IHsKICAgICAgIlBhc3N3b3JkIjogImJsYW5rIgogICAgfQogIH0KfQ==
kind: Secret
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"v1","kind":"Secret","metadata":{"annotations":{},"name":"appsettings","namespace":"api"},"stringData":{"appsettings.Dummy.json":"{\n \"Dummy\": {\n \"Placeholder\": {\n \"Password\": \"blank\"\n }\n }\n}"},"type":"Opaque"}
creationTimestamp: 2019-01-31T02:50:16Z
name: appsettings
namespace: api
resourceVersion: "4909"
selfLink: /api/v1/namespaces/api/secrets/appsettings
uid: f0629027-2502-11e9-9375-6eb4e0983acc
Apparently the yaml used to create the secret showing up in the annotation is expected behavior for kubectl apply -f secret.yaml since 2016/has been posted as a bug report, but issue closed without resolution/they're ignoring it vs fixing it.
If you're original secret.yaml is base64'd the annotation will at least be base64'd but in this scenario it's straight up non-base64'd human readable clear text.
Note1: it doesn't happen with imperative secret creation
kubectl create secret generic appsettings --from-file appsettings.Dummy.json --namespace=api
Note2: Another reason for favoring the declarative appsettings-secret.yaml, is that when it's time to edit kubectl apply -f will configure the secret, but if you run that create command it'll say error already exists and you'll have to delete it, before it'll let you run the create command again.
Note3: A reason for kubectl create secret generic name --from-file file --namespace / a reason against secret.yaml is that kubectl show secret won't show you the last time the secret got edited. Where as with the create command, because you have to delete it before you can recreate it, you'll know when it was last edited based on how long it's existed for, so that's good for audit trial. (But there's better ways of auditing)
kubectl apply -f cleartext-appsettings-secret.yaml
kubectl annotate secret appsettings -n=api kubectl.kubernetes.io/last-applied-configuration-
kubectl get secret appsettings -n=api -o yaml
Counteracts the leak
apiVersion: v1
data:
appsettings.Dummy.json: ewogICJEdW1teSI6IHsKICAgICJQbGFjZWhvbGRlciI6IHsKICAgICAgIlBhc3N3b3JkIjogImJsYW5rIgogICAgfQogIH0KfQ==
kind: Secret
metadata:
creationTimestamp: 2019-01-31T03:06:55Z
name: appsettings
namespace: api
resourceVersion: "6040"
selfLink: /api/v1/namespaces/api/secrets/appsettings
uid: 43f1b81c-2505-11e9-9375-6eb4e0983acc
type: Opaque
You can use secode to replace secret values with base64 encoded strings, by simply doing:
secode secrets.yaml > secrets_base64.yaml
It encodes all data fields and works with multiple secrets (kind:Secret) per yaml file, when defined in a list (kind: List).
Disclaimer: I'm the author
For the Windows users in the room, use this for each of the .cer and .key (example shows the .key being encoded for insertion in to the YAML file):
$Content = Get-Content -Raw -Path C:\ssl-cert-decrypted.key
[Convert]::ToBase64String([System.Text.Encoding]::UTF8.GetBytes($Content)) | Out-File -FilePath C:\ssl-cert-decrypted.key.b64
Open the new .b64 file and paste the (single line) output in to your YAML file - be aware that if checking in the YAML file to a source code repo with this information in it, the key would effectively be compromised since base64 isn't encryption.