Hi I use Gitlab I want to containerize apps that I make and automate deploy so I'm trying to play with Kubernetes on GKE.
I was following Gitlab's documentation regarding linking Cluster
I tried to create secret through K8S Dashboard (Create, Paste yaml)
apiVersion: v1
kind: Secret
metadata:
name: gitlab
annotations:
kubernetes.io/service-account.name: gitlab
type: kubernetes.io/service-account-token
press Upload and it just swallows it, no errors no new secrets.
Then I tried to add it through kubectl:
kubectl create -f /tmp/gitlab.yaml it prints secret "gitlab" created
but it didn't
What am I doing wrong?
kind: Secret
type: kubernetes.io/service-account-token
I see what's going on here: you're trying to manually create a ServiceAccount token, when those are designed to be managed by kubernetes, not you, because the token contains a correctly formatted and cryptographically signed JWT.
Independent of that, it's silly to create a service-account-token Secret that contains no secret data (s.a.t. always contain 3 bits of data: ca.crt, namespace, and token). Then, even if you did populate that Secret with an actual JWT -- which would be very weird -- you'll also want to include the annotation kubernetes.io/service-account.uid: containing the UUID of the gitlab``ServiceAccount (which you can find by kubectl get -o json sa gitlab | jq -r .metadata.uid)
Reasonable people can differ about whether this is a bug, or a crazy edge case that doesn't hurt anything. I think of this as the equivalent of INSERT INTO users; reporting OK; sure, it didn't insert anything, but the command was nonsense anyway, so it's hard to get really worked up about things.
my bad, I specified kubernetes.io/service-account.name to be gitlab even though there is no such service account, I replaced it with default and everything worked
Related
I need to fetch a credentials file from Jenkins and load it into a Kubernetes deployment.
This file needs to be loaded at the beginning, before the pods launch since it's a configuration file.
My idea was, in the Jenkins part, to do something like withCredentials([file(credentialsId: credentials, variable: 'credentials')]) { }
And use that variable to create a secret, then use it in a deployment. Right now I can only do this with stringData, something like
apiVersion: v1
kind: Secret
metadata:
name: 'metadata'
labels:
project: 'example'
type: Opaque
stringData:
TEST: '${TEST}'
I would need a way to use this, but with a file, within a Secret.
And to mount it in my deployment, I suppose with the volumeMounts option
This sounds a bit hacky in my opinion. I would rather use a configmap, which you can create with a custom (or search for one) kubernetes operator. Otherwise, a sidecar container could help as well.
https://learnk8s.io/sidecar-containers-patterns
So when creating secrets I often will use:
kubectl create secret generic super-secret --from-env-file=secrets
However, I wanted to move this to a dedicated secrets.yaml file, of kind "Secret" as per the documentation: https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands#-em-secret-generic-em-
However, to do this, I need to base64 encode all of the secrets. Wha! This is bad imho. Why can't kubectl handle this for us? (Another debate for another day).
So, I am back to using the kubectl create secret command which works really well for ease of deployment - however, I am wondering what the process is to apply said creation of secret to a specific namespace? (Again, this probably would involve something tricky to get things to work - but why it's not a standard feature yet is a little worrying?)
You can provide stringData section of a Secret instead of data section. That won't require base64 encoding. Here, is an example:
apiVersion: v1
kind: Secret
metadata:
name: secret-basic-auth
namespace: demo
type: kubernetes.io/basic-auth
stringData:
username: admin
password: t0p-Secret
Ref: https://kubernetes.io/docs/concepts/configuration/secret/#basic-authentication-secret
You can use --dry-run and -oyaml flags.
Use this command to generate your secrets.yaml file
kubectl create secret generic super-secret \
--from-env-file=secrets --namespace <your-namespace> \
--dry-run=client -oyaml > secrets.yaml
The above is pretty standard in the k8s community.
Kubectl's standard options include a --namespace or -n option*, which can achieve this with zero fuss:
kubectl create secret generic super-secret --from-env-file=secrets --namespace=whatever_namespace_you want
*This is not included or mentioned in the documentation for creating a secret, explicitly. It's assumed knowledge for that section.
I want to define a Kubernetes secrets map as part of my deployment pipeline. According to the Kubernetes documentation, there are two ways to define a secret.
Declarative Using a .yml with the Secret Object
Imperative Using kubectl create secret generic
The declarative approach requires writing a YAML similar to the one below.
apiVersion: v1
kind: Secret
metadata:
name: test-secret
data:
username: bXktYXBw
password: Mzk1MjgkdmRnN0pi
I want to be able to check in all the Kubernetes YAML files to git so I can run them on a CI server. Checking in the YAML means that the secret is stored in git which is not good for security. I can put the secret into my CI systems secret's store but then how do I create a secrets YAML that references the OS environment variable at the time that kubectl is called.
Questions:
How to define a Kubernetes Secret from within a CI pipeline without having to commit the secrets into source control?
Is there a best practice approach for defining secrets as part of CI for K8s?
There is no really good way to managed secrets securely with a vanilla Kubernetes. If you decrypt the secret or inject an unencrypted secret in your CI/CD pipeline and create a Kubernetes Secret, you'll have a decrypted Base64 encoded string to store in your Kubernetes cluster (Etcd).
Most companies I've worked with recently deciding to either keep the secret in their Key Vault and use a Kubernetes controller to inject the secret at runtime or use a controller to be able to manage encrypted secrets like sealed-secrets or Kamus. Using encrypted secrets might be a good option if you want to keep your secrets in Git.
First-class support for Hashicorp Vault and Kubernetes : https://github.com/hashicorp/vault-k8s
Take a look at this blog post from Banzai Cloud for a more detailed explanation : Inject secrets directly into Pods from Vault revisited
I ended up hacking this with bash scrip to output a yaml secret secret.yaml.sh
cat <<EOF
apiVersion: v1
kind: Secret
metadata:
name: test-secret
type: Opaque
data:
username: $1
password: $2
jdbcUrl: $3
EOF
Then in my CI pipeline invoke the secret.yml.sh and pass in the base64 encoded values which are store in the CI system's credentials store then pipe to kubectl like so ./secret.yaml.sh $USERNAME $PASSWORD $URL | kubectl apply -f -
This hack makes it possible for me to run the CI pipeline and update the secrets based on what is stored in the CI systems.
As others have noted the secrets in Kubernetes etcd are not secure and it's better to use a key management system with k8s. However I don't have access to a key vault for this project.
You can encrypt the secret and commit the encrypted secret in git and while deployment it needs to be decrypted. For example ansible vault can be used if you are using ansible as CI tool.
If you are using Jenkins then you can use credentials or Hashicorp vault plugin for storing the secret.
If you are on public cloud then AWS KMS, Azure Vault etc are available.
I'm trying to get metadata for a given kubernetes resource. Similar to a describe for a REST end-point.
Is there a kubectl to get all possible things that I could provide for any k8s resource ?
For example for the deployment resource, it could be something like this.
apiVersion: apps/v1
kind: Deployment
metadata:
name: <type:String>
<desc: name for the deployment>
namespace: <type:String>
<desc: Valid namespace>
annotations:
...
Thanks in advance !
You can use the kubectl explain CLI command:
This command describes the fields associated with each supported API
resource. Fields are identified via a simple JSONPath identifier:
<type>.<fieldName>[.<fieldName>]
Add the --recursive flag to display all of the fields at once without descriptions. Information about each field is retrieved from the server in OpenAPI format.
Example to view all Deployment associated fields:
kubectl explain deployment --recursive
You can dig into specific fields:
kubectl explain deployment.spec.template
You can also rely on Kubernetes API Reference Docs.
Are you familiar with OpenApi/Swagger? Try to open the following file in swagger-ui https://raw.githubusercontent.com/kubernetes/kubernetes/master/api/openapi-spec/swagger.json
If you have a live kubernetes api available the file should be available under /openapi/v2 like they describe here: https://kubernetes.io/docs/concepts/overview/kubernetes-api/#openapi-and-swagger-definitions
I want to store files in Kubernetes Secrets but I haven't found how to do it using a yaml file.
I've been able to make it using the cli with kubectl:
kubectl create secret generic some-secret --from-file=secret1.txt=secrets/secret1.txt
But when I try something similar in a yaml:
apiVersion: v1
kind: Secret
metadata:
name: some-secret
type: Opaque
data:
secret1.txt: secrets/secret1.txt
I´ve got this error:
[pos 73]: json: error decoding base64 binary 'assets/elasticsearch.yml': illegal base64 data at input byte 20
I'm following this guide http://kubernetes.io/docs/user-guide/secrets/. It explains how to create a secret using a yaml but not how to create a secret from a file using yaml.
Is it possible? If so, how can I do it?
As answered on previous post, we need to provide the certificate/key encoded as based64 to the file.
Here is generic example for a certiticate (in this case SSL):
The secret.yml.tmpl:
apiVersion: v1
kind: Secret
metadata:
name: test-secret
namespace: default
type: Opaque
data:
server.crt: SERVER_CRT
server.key: SERVER_KEY
Pre-process the file to include the certificate/key:
sed "s/SERVER_CRT/`cat server.crt|base64 -w0`/g" secret.yml.tmpl | \
sed "s/SERVER_KEY/`cat server.key|base64 -w0`/g" | \
kubectl apply -f -
Note that the certificate/key are encoded using base64 without whitespaces (-w0).
For the TLS can be simply:
kubectl create secret tls test-secret-tls --cert=server.crt --key=server.key
You can use --dry-run flag to prepare YAML that contains data from your files.
kubectl create secret generic jwt-certificates --from-file=jwt-public.cer --from-file=jwt-private.pfx --dry-run=true --output=yaml > jwt-secrets.yaml
Edit
Thanks to #Leopd for comment about API deprecation, new kubectl uses this command:
kubectl create secret generic jwt-certificates --from-file=jwt-public.cer --from-file=jwt-private.pfx --dry-run=client --output=yaml > jwt-secrets.yaml
On my machine I still have old kubectl version
When using the CLI format basically you're using a generator of the yaml before posting it to the server-side.
Since Kubernetes is client-server app with REST API in between, and the actions need to be atomic, the posted YAML needs to contain the content of the file, and best way to do that is by embedding it as a base64 format in-line. It would be nice if the file could be otherwise embedded (indentation maybe could be used to create the boundaries of the file), but I haven't seen any example of such until now.
That being said, putting a file reference on the yaml is not possible, there is no pre-flight rendering of the yaml to include the content.
So I just learned a super useful k8s fundamental I missed, and then discovered it has a security vulnerability associated with it, and came up with a resolution.
TLDR:
You can have cleartext multiline strings/textfiles as secret.yaml's in your secret repo !!! :)
(Note I recommend storing this in Hashicorp Vault, you can store versioned config files that have secrets, and easily view/edit them through the vault webpage, and unlike a git repo, you can have fine grain access control, pipelines can use the REST API to pull updated secrets which makes password rotation mad easy too.)
cleartext-appsettings-secret.yaml
appsettings.Dummy.json is the default file name (key of the secret)
(I use the word default file name as you could override it in the yaml mount)
and the clear text json code is the file contents (value of the secret)
apiVersion: v1
kind: Secret
metadata:
name: appsettings
namespace: api
type: Opaque
stringData:
appsettings.Dummy.json: |-
{
"Dummy": {
"Placeholder": {
"Password": "blank"
}
}
}
When I
kubectl apply -f cleartext-appsettings-secret.yaml
kubectl get secret appsettings -n=api -o yaml
The secret shows up cleartext in the annotation...
apiVersion: v1
data:
appsettings.Dummy.json: ewogICJEdW1teSI6IHsKICAgICJQbGFjZWhvbGRlciI6IHsKICAgICAgIlBhc3N3b3JkIjogImJsYW5rIgogICAgfQogIH0KfQ==
kind: Secret
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"v1","kind":"Secret","metadata":{"annotations":{},"name":"appsettings","namespace":"api"},"stringData":{"appsettings.Dummy.json":"{\n \"Dummy\": {\n \"Placeholder\": {\n \"Password\": \"blank\"\n }\n }\n}"},"type":"Opaque"}
creationTimestamp: 2019-01-31T02:50:16Z
name: appsettings
namespace: api
resourceVersion: "4909"
selfLink: /api/v1/namespaces/api/secrets/appsettings
uid: f0629027-2502-11e9-9375-6eb4e0983acc
Apparently the yaml used to create the secret showing up in the annotation is expected behavior for kubectl apply -f secret.yaml since 2016/has been posted as a bug report, but issue closed without resolution/they're ignoring it vs fixing it.
If you're original secret.yaml is base64'd the annotation will at least be base64'd but in this scenario it's straight up non-base64'd human readable clear text.
Note1: it doesn't happen with imperative secret creation
kubectl create secret generic appsettings --from-file appsettings.Dummy.json --namespace=api
Note2: Another reason for favoring the declarative appsettings-secret.yaml, is that when it's time to edit kubectl apply -f will configure the secret, but if you run that create command it'll say error already exists and you'll have to delete it, before it'll let you run the create command again.
Note3: A reason for kubectl create secret generic name --from-file file --namespace / a reason against secret.yaml is that kubectl show secret won't show you the last time the secret got edited. Where as with the create command, because you have to delete it before you can recreate it, you'll know when it was last edited based on how long it's existed for, so that's good for audit trial. (But there's better ways of auditing)
kubectl apply -f cleartext-appsettings-secret.yaml
kubectl annotate secret appsettings -n=api kubectl.kubernetes.io/last-applied-configuration-
kubectl get secret appsettings -n=api -o yaml
Counteracts the leak
apiVersion: v1
data:
appsettings.Dummy.json: ewogICJEdW1teSI6IHsKICAgICJQbGFjZWhvbGRlciI6IHsKICAgICAgIlBhc3N3b3JkIjogImJsYW5rIgogICAgfQogIH0KfQ==
kind: Secret
metadata:
creationTimestamp: 2019-01-31T03:06:55Z
name: appsettings
namespace: api
resourceVersion: "6040"
selfLink: /api/v1/namespaces/api/secrets/appsettings
uid: 43f1b81c-2505-11e9-9375-6eb4e0983acc
type: Opaque
You can use secode to replace secret values with base64 encoded strings, by simply doing:
secode secrets.yaml > secrets_base64.yaml
It encodes all data fields and works with multiple secrets (kind:Secret) per yaml file, when defined in a list (kind: List).
Disclaimer: I'm the author
For the Windows users in the room, use this for each of the .cer and .key (example shows the .key being encoded for insertion in to the YAML file):
$Content = Get-Content -Raw -Path C:\ssl-cert-decrypted.key
[Convert]::ToBase64String([System.Text.Encoding]::UTF8.GetBytes($Content)) | Out-File -FilePath C:\ssl-cert-decrypted.key.b64
Open the new .b64 file and paste the (single line) output in to your YAML file - be aware that if checking in the YAML file to a source code repo with this information in it, the key would effectively be compromised since base64 isn't encryption.