update nested array object if exists yq - yq

Having the following YAML how to using jq
versions:
- name: test1
version: "0.0.1"
- name: test2
id: "1"
How to update a version by its name fullfilling the following scenarios:
adding a a new object with a new name:
name: test3
version: "0.0.3"
updating an existing object with a new version:
name: test1
version: "0.1.1"
updating an existing object with no version:
name: test2
version: "0.0.2"
I have tested the following command, but i get undesired results like when replacing the version it keeps the old version and not the new one
yq -i ".versions= (.versions + {"name": "test", "version": "0.0.5"} | unique_by(.name))" test.yaml

If you care about the array's ordering (which I believe you do), the general answer would be: Find the matching index or default to a new one past the last, and add both fields to the item (or null) behind that index.
You didn't specify which implementation of yq you are using. While the solutions are similar, they are certainly not identical.
This would be a solution using kislyuk/yq:
yq='(.versions | .[(to_entries[] | select(.value.name == $name).key) // length]) += {$name, $version}'
# Adding a new object with a new name
yq -y --arg name test3 --arg version 0.0.3 "$yq"
versions:
- name: test1
version: 0.0.1
- name: test2
id: '1'
- name: test3
version: 0.0.3
# Updating an existing object with a new version:
yq -y --arg name test1 --arg version 0.1.1 "$yq"
versions:
- name: test1
version: 0.1.1
- name: test2
id: '1'
# Updating an existing object with no version:
yq -y --arg name test2 --arg version 0.0.2 "$yq"
versions:
- name: test1
version: 0.0.1
- name: test2
id: '1'
version: 0.0.2
Using mikefarah/yq, this can be translated to:
yq='(.versions | .[(to_entries | .[] | select(.value.name == env(name)).key) // length]) += {"name": env(name), "version": env(version)}'
# Adding a new object with a new name
name=test3 version=0.0.3 yq "$yq"
versions:
- name: test1
version: "0.0.1"
- name: test2
id: "1"
- name: test3
version: 0.0.3
# Updating an existing object with a new version:
name=test1 version=0.1.1 yq "$yq"
versions:
- name: test1
version: 0.1.1
- name: test2
id: "1"
# Updating an existing object with no version:
name=test2 version=0.0.2 yq "$yq"
versions:
- name: test1
version: "0.0.1"
- name: test2
id: "1"
version: 0.0.2

Related

Invoking an argo workflow container with optional flags?

Say I have a dockerized command that can take an optional flag, similar to how ls can take a flag -l.
How can I make argo invoke it properly, allowing the caller to choose whether to include the flag or not?
This does not work, but hopefully it is somewhat close and demonstrates my intent:
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
generateName: do-work-
spec:
arguments:
parameters:
- name: long-format
value: false
entrypoint: do-work
templates:
- name: do-work
container:
image: alpine:3.7
command: [ls]
args:
- "{{workflow.parameters.long-format == 'true' ? '-l' : '' }}"
- /etc
If I invoke it like argo submit do-work.yaml or argo submit do-work.yaml -p long-format=false then I want it to run ls /etc.
If I invoke it like argo submit do-work.yaml -p long-format=true then I want it to run ls -l /etc.
I figured out that something like this can work, using python to jam them all together into one string... kind of ugly.
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
generateName: do-work-
spec:
arguments:
parameters:
- name: dir
value: /etc
- name: long-format
value: false
entrypoint: main
templates:
- name: main
dag:
tasks:
- name: build-cli-args
template: build-cli-args
- name: do-work
depends: build-cli-args
template: do-work
arguments:
parameters:
- name: cli-args
value: '{{tasks.build-cli-args.outputs.result}}'
- name: build-cli-args
script:
image: python:alpine3.6
command: [python]
source: |
cli_args = [
"-l" if "{{workflow.parameters.long-format}}" == "true" else None,
"{{workflow.parameters.dir}}",
]
print(" ".join([a for a in cli_args if a is not None]))
- name: do-work
inputs:
parameters:
- name: cli-args
script:
image: alpine:3.7
command: [sh, -ex]
source: |
ls {{inputs.parameters.cli-args}}
I will leave the question open in case someone can tell me how to do it more minimally, with an expression like I was attempting.

Pass values from Helmrelease to Terraform

I have a helm release file test-helm-release.yaml as given below.
apiVersion: helm.toolkit.gitops.io/v2beta1
kind: HelmRelease
metadata:
name: "test"
namespace: "test-system"
spec:
chart:
spec:
chart: "test-environment"
version: "0.1.10"
values:
key1: "value1"
key1: "value1"
key1: "value1"
key1: "value1"
gitRepository:
url: https://github.com/test-eng/test.git
helmRepositories:
- name: testplatform
url: https://test-platform/charts
While creating the helm release I can pass the values from above helm release to the new release using the below command
chart=$(yq '.spec.chart.spec.chart' test-helm-release.yaml)
version=$(yq '.spec.chart.spec.version' test-helm-release.yaml)
yq '.spec.values' test-helm-release.yaml | helm upgrade --install --values - --version "$version" --namespace test-system --create-namespace test-platform "helm-repo/$chart"
Above code is working perfectly and I'm able to pass the values to the helm release using "yq" command. How I can do the same "yq" function with terraform "helm-release" and git repository data type given below.
data "github_repository_file" "test-platform" {
repository = "test-platform"
branch = "test"
file = "internal/default/test-helm-release.yaml"
}
resource "helm_release" "test-platform" {
name = "test-platform"
repository = "https://test-platform/charts"
chart = "test-environment"
namespace = "test-system"
create_namespace = true
timeout = 800
lifecycle {
ignore_changes = all
}
}
Note
I cannot use "set" because i want to fetch the values form test-helm-release.yaml dynamically.Any idea how I could fetch the .spec.values alone using templatefile functio or a different way?

Meltano: (tag and tagret: postgresql) temporary file size exceeds temp_file_limit

I'm getting the issue while running Meltano with tag and target "postgresql" as source database, without changing the temp_file_limit params by the tap "postgresql".
how could I fix that?
psycopg2.errors.ConfigurationLimitExceeded: temporary file size exceeds temp_file_limit (13411002kB)
Meltano meltano.yml setting file looks like that:
default_environment: dev
project_id: fake-id
plugins:
extractors:
- name: tap-postgres
variant: transferwise
pip_url: pipelinewise-tap-postgres
- name: tap-google-analytics
variant: meltanolabs
pip_url: pip install git+https://github.com/MeltanoLabs/tap-google-analytics.git
loaders:
- name: target-postgres
variant: transferwise
pip_url: pipelinewise-target-postgres
config:
schema_mapping:
schema_1:
target_schema: replica_schema_1
default_target_schema: public
orchestrators:
- name: airflow
pip_url: psycopg2 apache-airflow==2.1.2 --constraint https://raw.githubusercontent.com/apache/airflow/constraints-2.1.2/constraints-${MELTANO__PYTHON_VERSION}.txt
files:
- name: airflow
pip_url: git+https://gitlab.com/meltano/files-airflow.git
schedules:
- name: schedule-name
extractor: tap-postgres
loader: target-postgres
transform: skip
interval: '*/30 * * * *'
environments:
- name: dev
config:
plugins:
extractors:
- name: tap-postgres
config:
dbname: $TAP_POSTGRES_DBNAME
user: $TAP_POSTGRES_USER
select:
- schema_name-table_name_1
- schema_name-table_name_2
loaders:
- name: target-postgres
config:
user: $TARGET_POSTGRES_USER
dbname: $TARGET_POSTGRES_DBNAME

Can we use template variables in helm3 values.yaml

Is it possible to use templating in helm values.yaml?
Use case -
values.yaml is as below -
myservice:
name: abc
namespace: abc
image:
ecr_uri: abc.dkr.ecr.us-east-2.amazonaws.com
repo_name: abc-123
version: 1.0.0
I want to pass these image metadata to my environment variable within values.yaml
env:
- name: POD_CONFIG
value: '{
"1234": {
"DEPLOYMENT": "xyz",
"IMAGE": "{{ .Values.myservice.image.ecr_uri }}/{{ .Values.myservice.image.repo_name }}:{{ .Values.myservice.image.version }}",
"REC_COUNTS_TO_POD_COUNTS": {"0": 0, "500": 1, "1000": 2, "1500": 3, "2000": 4, "2001": 5}
}
}'
Output -
$ helm template myservice does not render actual variable values but rather returns it as a string
- name: POD_CONFIG
value: '{ "1234": { "DEPLOYMENT": "xyz", "IMAGE": "{{ .Values.myservice.image.ecr_uri}}/{{ .Values.myservice.image.repo_name }}:{{ .Values.myservice.image.version }}", "REC_COUNTS_TO_POD_COUNTS":
{"0":0,"20":1, "40":2, "60":3} } }'
Is there any way to set this version for environment variables? as I will by passing the custom version with --set command from $ helm install and $ helm upgrade commands
e.g.
$ helm install myservice ./myservice --set myservice.image.version=1.0.1 -n mynamespace --debug
I am expecting this version to be reflected in my environment variables dynamically but that is not happening right now.

How to set the "required" value in templates from --set option?

How to force to specify --set option on helm install|upgrade?
in my case, some required environment variables. (e.g. "database.password")
Files
.
|-- Chart.yaml
|-- templates
| |-- NOTES.txt
| |-- _helpers.tpl
| |-- deployment.yaml
| |-- ingress.yaml
| |-- secret.yaml
| `-- service.yaml
`-- values.yaml
values.yaml (snip)
#...
database:
useExternal: no
host: "pgsql"
port: "5432"
name: "myapp"
userName: "myapp_user"
# password shouldn't write here.
# I want to be inject this value to secret.
password: ""
#...
templates/secrets.yaml
apiVersion: v1
kind: Secret
metadata:
name: myapp-secrets
type: Opaque
data:
app-database-password: {{required .Values.database.password | b64enc | quote }}
templates/deployment.yaml (snip)
#...
env:
- name: APP_DATABASE_HOST
value: {{ .Values.database.host | quote }}
- name: APP_DATABASE_PORT
value: {{ .Values.database.port | quote }}
- name: APP_DATABASE_NAME
value: {{ .Values.database.name | quote }}
- name: APP_DATABASE_USERNAME
value: {{ .Values.database.username | quote }}
- name: APP_DATABASE_PASSWORD
valueFrom:
secretKeyRef:
name: myapp-secrets
key: app-database-password
#...
command
# Retrieve from GCP KMS(prod) or define directly(dev)
DATABASE_PASSWORD=$( ... )
# Deploy.
helm upgrade --install \
-f ./values.yaml \
--set database.password=$DATABASE_PASSWORD \
myapp-dev ./ --dry-run --debug
It's failed with error.
Error: render error in "myapp/templates/secret.yaml": template: myapp/templates/secret.yaml:7:28: executing "myapp/templates/secret.yaml" at <required>: wrong number of args for required: want 2 got 1
It seems the required function is evaluate template file statically when parsing.
I need matters below:
database.password is switchable by env such as "prod" or "stage".
database.password should store to secret.
I want to set the actual database.password value using env vars on command execution.
Any ideas?
The Helm-specific required macro takes two parameters: the error message if the value isn't present, and the value that you're checking for. This syntax also lets it be used in pipeline form. In your example, the secret value could be
app-database-password: {{.Values.database.password | required "database password is required" | b64enc | quote }}