kubectl apply with stringData in secret does not remove fields - kubernetes

Question

(using kubernetes v1.15.7 in minikube and matching client version and minikube 1.9.0)
If I kubectl apply a secret like this:
apiVersion: v1
data:
MY_KEY: dmFsdWUK
MY_SECRET: c3VwZXJzZWNyZXQK
kind: Secret
metadata:
name: my-secret
type: Opaque
then subsequently kubectl apply a secret removing the MY_SECRET field, like this:
apiVersion: v1
data:
MY_KEY: dmFsdWUK
kind: Secret
metadata:
name: my-secret
type: Opaque
The data field in the result is what I expect when I kubectl get the secret:
data:
MY_KEY: dmFsdWUK
However, if I do the same thing using stringData instead for the first kubectl apply, it does not remove the missing key on the second one:
First kubectl apply:
apiVersion: v1
stringData:
MY_KEY: value
MY_SECRET: supersecret
kind: Secret
metadata:
name: my-secret
type: Opaque
Second kubectl apply (stays the same, except replacing MY_KEY's value with b2hubyEK to show the configuration DID change)
apiVersion: v1
data:
MY_KEY: b2hubyEK
kind: Secret
metadata:
name: my-secret
type: Opaque
kubectl get result after applying the second case:
data:
MY_KEY: b2hubyEK
MY_SECRET: c3VwZXJzZWNyZXQ=
The field also does not get removed if the second case uses stringData instead. So it seems that once stringData is used once, it's impossible to remove a field without deleting the secret. Is this a bug? Or should I be doing something differently when using stringData?

Answer 1

kubectl apply need to merge / patch the changes here. How this works is described in How apply calculates differences and merges changes
I would recommend to use kustomize with kubectl apply -k and use the secretGenerator to create a unique secret name, for every change. Then you are practicing Immutable Infrastructure and does not get this kind of problems.
A brand new tool for config manangement is kpt, and kpt live apply may also be an interesting solution for this.

Answer 2

The problem is that stringData is a write only field. It doesn’t have convergent behavior so it breaks the merge patch generator system. Most high level tools fix this by converting to normal data before dealing with the patch.