i have the follow question. i try connect to eks cluster using a Terraform with Gitlab CI/CD , i receive the error message , but when try it in my compute , this error dont appear, someone had same error ?
$ terraform output authconfig > authconfig.yaml
$ cat authconfig.yaml
<<EOT
apiVersion: v1
kind: ConfigMap
metadata:
name: aws-auth
namespace: kube-system
data:
mapRoles: |
- rolearn: "arn:aws:iam::503655390180:role/clusters-production-workers"
username: system:node:{{EC2PrivateDNSName}}
groups:
- system:bootstrappers
- system:nodes
EOT
$ kubectl create -f authconfig.yaml -n kube-system
error: error parsing authconfig.yaml: error converting YAML to JSON: yaml: line 2: mapping values are not allowed in this context
The output is including EOT(EndOfText) marks since it is generated as a multiline string originally.
as documentation suggests (terrafom doc link)
Don't use "heredoc" strings to generate JSON or YAML. Instead, use the
jsonencode function or the yamlencode function so that Terraform can
be responsible for guaranteeing valid JSON or YAML syntax.
use json encoding or yaml encoding before building output.
If you want to continue like this with what you have now then try to give these options with output -json or -raw
terraform output -json authconfig > authconfig.yaml
or
terraform output -raw authconfig > authconfig.yaml
The error message tells you the authconfig.yaml file can not be converted from YAML to JSON, suggesting it's not a valid yaml
The cat authconfig.yaml you're showing us includes some <<EOT and EOT tags. I would suggest to remove those, before running kubectl create -f
Your comment suggests you knew this already - then why didn't you ask about terraform, rather than showing us kubectl create failing? From your post, it really sounded like you copy/pasted the output of your job, without even reading it.
So, obviously, the next step is to terraform output -raw, or -json, there are several mentions in their docs, or knowledge base, a google search would point you to:
https://discuss.hashicorp.com/t/terraform-outputs-with-heredoc-syntax-leaves-eot-in-file/18584/7
https://www.terraform.io/docs/cli/commands/output.html
Last: we could ask why? Why would you terraform output > something, when you can have terraform write a file?
While as a general rule, whenever writing terraform stdout/stderr to files, I strongly suggest going with no-color.
Related
In a shell scrip I want to assigning a variable what to use in a value in a deployment. For the life of me I can not figure out how to get it to work.
My helm deploy script file has the following in order to set the value to use my variable :
--set AuthConfValue=$AUTH_CONF_VALUE
And I have this in the deployment.yaml file in order to use the variable :
- name: KONG_SETTING
value: "{ {{ .Values.AuthConfValue }} }"
If I assign the variable in my shell script like the following :
AUTH_CONF_VALUE="ernie"
It will work and the value in the deployment will show up like so:
value: '{ ernie }'
Now if I try to assign the variable like this:
AUTH_CONF_VALUE="\\\"ernie\\\":\\\"123\\\""
I will then get the error error converting YAML to JSON: yaml: line 118: did not find expected key when the helm deploy runs.
I was hoping that this would give me the following value in the deployment :
value: "{ "ernie":"123" }"
If I hardcode the value into the deployment.yaml with this:
- name: KONG_SETTING
value: "{ \"ernie\": \"123\" }"
and then run the helm deploy it will work and populate the value in the deployment with this -
value: "{ "ernie":"123" }"
Can someone show me if/how I might be able to do this?
The Helm --set option also uses backslash escaping. So in your example, the $AUTH_CONF_VALUE variable in the host shell contains a single backslash before each quote, which is consumed by --set, so .Values.AuthConfValue contains no backslashes at all, and you get invalid YAML.
If you want to keep this as close to the existing form as you can, let's construct a string with no backslashes at all (and hopefully no commas or brackets either, since those also have special meaning to --set)
AUTH_CONF_VALUE='"ernie":"123"'
helm install --set AuthConfValue="$AUTH_CONF_VALUE" .
When Helm expands a template it doesn't know anything about the context where it might be used. In your case, you know
.Values.AuthConfValue is the body of a JSON object
If you surround it in curly braces { ... } then it should be a valid JSON object
You need to turn that into a correctly-escaped YAML string
Helm contains a lightly-documented toJson function that takes an arbitrary object and converts it to JSON; any valid JSON is also valid YAML. So the closest-to-what-you-have approach might look like
- name: KONG_SETTING
value: {{ printf "{%s}" .Values.AuthConfValue | toJson }}
If you're willing to modify your deploy process a little more, you can have less escaping and more certainty. In the sequence above, we have a string that happens to be a JSON object; what if we had an actual object? Imagine settings like
# kong-auth.yaml
authConf:
ernie: "123"
You could provide this file at install time with a helm install -f option. Since valid JSON is valid YAML, again, you could also provide a JSON file here without changing anything.
helm install -f kong-auth.yaml .
Now with this setup .Values.authConf is an object; the only escaping you need to do is standard YAML/JSON escaping (for example quoting "123" so it's a string and not a number). Now we can use toJson twice, once to get the {"ernie":"123"} JSON object string, and a second time to escape that string as a value "{\"ernie\":\"123\"}".
- name: KONG_SETTING
value: {{ .Values.authConf | toJson | toJson }}
Setting this up would require modifying your deployment script, but it would be much safer against quoting and escaping concerns.
I'm trying to put a Service Account into a secret - I did it previously a year ago and it works but now - no matter how I approach it, the application doesn't see it right and says there is Input byte array has incorrect ending byte - When creating normal secret I know you've gotta do it with a new line so
echo -n "secret" | base64
and put that value in secret and apply, but my multiline file
cat secret.json
{
"type": "service_account",
"project_id": "smth-smth",
"private_key_id": "blabla"
...
}
No matter how I approach - whether put it by hand like in the first example, or do it with
cat secret.json | base64
# or
base64 < secret.json
the secret is created but application throws
Constructor threw exception; nested exception is java.lang.IllegalArgumentException: Input byte array has incorrect ending byte at 3104
When I compare the new secret to the last one of the service account the difference is how the output looks like
The working one is smth like this - when I try to decrypt the base64
echo -n "<long string of base64 encrypred sa> | base64 -D
{ "type": "service_account", "project_id": "blabla"... }
so it's in one line, and the new SA I try to decrypt is outputed in the format as in the file - so each part of json in new line - I tried manually putting it all in one line but without success
Anyone know ? how to put a multiline file in a secret (base64) properly?
The easiest way to create a secret from a file is to use kubectl create secret generic.
Put your file secret.json in a folder config and then run:
kubectl create secret generic my-secret --from-file=config
You will get a secret my-secret with one key secret.json containing your file (which you can then mount to a pod volume).
If you cannot create files an option is to write into a variable and then load the result into a --file-literal. This may be necessary because it seems kubectl either escapes newline characters \n when inside a quoted string and ignores them if no quotes are supplied. When reading from a variable the \nare treated as expected.
EDIT: With regards to multi-line strings do take care to use correct linefeed characters, as explained here. I ran into that when trying my answer at home :)
target_string=$(echo "string1\nstring2")
kubectl create secret generic your-secret-name --from-literal=your_key=$target_string
I want to pass a certificate to the helm chart and currently I am passing using --set-file global.dbValues.dbcacertificate=./server.crt but instead i want to pass the file in values file of helm chart.
The Values.yaml file reads
global:
dbValues:
dbcacertificate: <Some Way to pass the .crt file>
According to the relevant documentation, one must pre-process a file that is external to the chart into a means that can be provided via --set or --values, since .Files.Get cannot read file paths that are external to the chart bundle.
So, given the following example template templates/secret.yaml containing:
apiVersion: v1
kind: Secret
data:
dbcacertificate: {{ .Values.dbcacertificate | b64enc }}
one can use shell interpolation as:
helm template --set dbcacertificate="$(cat ./server.crt)" .
or, if shell interpolation is not suitable for your circumstances, you can pre-process the certificate into a yaml compatible format and feed it in via --values:
$ { echo "dbcacertificate: |"; sed -e 's/^/ /' server.crt; } > ca-cert.yaml
$ helm template --values ./ca-cert.yaml .
I have developed an Openshift template which basically creates two objects (A cluster & a container operator).
I understand that templates run oc create under the hood. So, in case any of these two objects already exists then trying to create the objects through template would through an error. Is there any way to override this behaviour? I want my template to re-configure the object even if it exists.
You can use "oc process" which renders template into set of manifests:
oc process foo PARAM1=VALUE1 PARAM2=VALUE2 | oc apply -f -
or
oc process -f template.json PARAM1=VALUE1 PARAM2=VALUE2 | oc apply -f -
I'm trying to save the contents of a configmap to a file on my local hard drive. Kubectl supports selecting with JSONPath but I can't find the expression I need to select just the file contents.
The configmap was created using the command
kubectl create configmap my-configmap --from-file=my.configmap.json=my.file.json
When I run
kubectl describe configmap my-configmap
I see the following output:
Name: my-configmap
Namespace: default
Labels: <none>
Annotations: <none>
Data
====
my.file.json:
----
{
"key": "value"
}
Events: <none>
The furthest I've gotten so selecting only the file contents is this:
kubectl get configmap my-configmap -o jsonpath="{.data}"
Which outputs
map[my.file.json:{
"key": "value"
}]
The output that I want is
{
"key": "value"
}
What is the last piece of the JSONPath puzzle?
There’s an open issue at the Kubernetes GitHub repo with a list of things that needs to be fixed in regards to kubectl (and JSONpath), one of them are issue 16707 jsonpath template output should be json.
Edit:
How about this:
kubectl get cm my-configmap -o jsonpath='{.data.my\.file\.json}'
I just realized i had answered another question related (kind of) to this one. The above command should output what you had in mind!
If you have the ability to use jq, then you can use the following approach to e.g. "list" all config maps by selector, and extract the files:
readarray -d $'\0' -t a < <(kubectl get cm -l grafana=dashboards -o json | jq -cj '.items[] | . as $cm | .data | to_entries[] | [ ($cm.metadata.name + "-" + .key), .value ][]+"\u0000"') ; count=0; while [ $count -lt ${#a[#]} ]; do echo "${a[$((count + 1))]}" > ${a[$count]}; count=$(( $count + 2)); done
This uses kubectl (using -l for a label selector) to get all configmaps. Next it pipes them through jq, creating key value pairs with a null byte termination (the key also contains the name of the configmap, this way I ensured that duplicate file names are not an issue). Then it reads this into a bash array, iterating over the array in steps of 2. Creating files with the content.
This also works file config map values that contain newlines.