Unable to get ENV variables in GoCD Kubernetes using YAML config - kubernetes

Question

GoCD Version: 19.12.0
I'm trying to get environment variables defined in the Kubernetes deployment (system) in my GoCD YAML config in order to pass the GitHub authentication when pulling the resource.
I've confirmed that I'm able to call the repository using a personal access token. (via https://[TOKEN]#github.com/[COMPANY]/[REPO].git)
This, of course, also works if I do the same for the actual YAML git field.
The GoCD secrets in K8s:
apiVersion: v1
data:
GITHUB_ACCESS_KEY: base64EncodedKey
kind: Secret
type: Opaque
The GoCD deployment gets the secrets:
...
spec:
containers:
- env:
- name: GOCD_PLUGIN_INSTALL_kubernetes-elastic-agents
value: https://github.com/gocd/kubernetes-elastic-agents/releases/download/v3.4.0-196/kubernetes-elastic-agent-3.4.0-196.jar
- name: GOCD_PLUGIN_INSTALL_docker-registry-artifact-plugin
value: https://github.com/gocd/docker-registry-artifact-plugin/releases/download/v1.1.0-104/docker-registry-artifact-plugin-1.1.0-104.jar
- name: GITHUB_ACCESS_KEY
valueFrom:
secretKeyRef:
key: GITHUB_ACCESS_KEY
name: gocd-server
...
I've exec'd into the pod and echoed the variable, which returns the decoded value.
The YAML:
format_version: 9
pipelines:
db-docker-build:
group: someGroup
label_template: ${COUNT}-${git[:8]}
lock_behavior: unlockWhenFinished
display_order: 1
materials:
git:
git: 'https://$GITHUB_ACCESS_KEY#github.com/[COMPANY]/[REPO].git'
shallow_clone: true
auto_update: true
branch: master
...
I'd half expect that to work, but it doesn't, it actually just gets $GITHUB_ACCESS_KEY as the value. The jobs defined in the pipeline stages are run using an elastic agent pod which also has the required secrets defined. I've tried a few
Setting env variables -
environment_variables: GIT_KEY: ${GITHUB_ACCESS_KEY}
and then using that variable
git: 'https://$GIT_KEY#github.com/[COMPANY]/[REPO].git'
Setting env variables and no quotes -
environment_variables: GIT_KEY: ${GITHUB_ACCESS_KEY}
and then using that variable
git: https://${GIT_KEY}#github.com/[COMPANY]/[REPO].git
No quotes - git: https://$GITHUB_ACCESS_KEY#github.com/[COMPANY]/[REPO].git
No quotes with brackets - git: https://${GITHUB_ACCESS_KEY}#github.com/[COMPANY]/[REPO].git
I've seen from some YAML documentation that it is recommended to use encrypted_password for the GitHub password, but this seems unnecessary since the GUI hides the token, and that its running in Kubernetes with secrets.

Answer 1

The team and I researched this a little further and found a workaround. Most issues and articles explain what is written in the docs, that you really need access to /bin/bash -c in order to get the variables.
The YAML plugin creator also uses secure, encrypted variables to store sensitive data which is fine, but for our team it means that a lot of Kubernetes features are not utilised.
The workaround:
Use the GUI to create a pipeline in GoCD, enter the GitHub link, add a username and the personal access token for the user as the password, test the connection is OK. Once created, go to Admin -> Pipelines and click the Download pipeline configuration and select YAML.
The generated YAML has the token encrypted as with the GoCD servers private key.