How to yq select contains on a field with an object with a value of tags and special characters? - aws-cloudformation

I need to add ManagedPolicyArns to many IAM roles in many cloudformation.yaml files. Some already have this policy added so I need to select, in order to add, only those that do not. I'm using yq (https://github.com/mikefarah/yq/) version 4.27.3, which is fantastic.
Command:
yq --from-file add_policy.yq cloudformation.yaml
cloudformation.yaml:
AWSTemplateFormatVersion: "2010-09-09"
Parameters:
MyPrefix:
Description: MyPrefix
Type: String
Resources:
MyRole:
Type: AWS::IAM::Role
Properties:
ManagedPolicyArns:
- Fn::ImportValue: !Sub "${MyPrefix}-my-policy-arn"
AssumeRolePolicyDocument:
Version: '2012-10-17'
Statement:
- Effect: Allow
Principal:
Service:
- ecs-tasks.amazonaws.com
- lambda.amazonaws.com
Action:
- sts:AssumeRole
- sts:TagSession
Path: /
Partially working in that it always adds the policy creating duplicates, add_policy.yq:
( .Resources[] |=
select(
(.Properties.AssumeRolePolicyDocument.Statement[].Principal.Service.[] == "ecs-tasks.amazonaws.com"
or .Properties.AssumeRolePolicyDocument.Statement[].Principal.Service.[] == "lambda.amazonaws.com")
and (.Properties.ManagedPolicyArns | contains([{"Fn::ImportValue": "${MyPrefix}-my-policy-arn"}]) | not ) )
.Properties.ManagedPolicyArns += {"Fn::ImportValue": "${MyPrefix}-my-policy-arn" | . tag = "!Sub" }
)
Does does not add the policy at all and fails silently, add_policy.yq
( .Resources[] |=
select(
(.Properties.AssumeRolePolicyDocument.Statement[].Principal.Service.[] == "ecs-tasks.amazonaws.com"
or .Properties.AssumeRolePolicyDocument.Statement[].Principal.Service.[] == "lambda.amazonaws.com")
and (.Properties.ManagedPolicyArns[]."Fn::ImportValue" | contains("${MyPrefix}-my-policy-arn" | . tag = "!Sub") | not) )
.Properties.ManagedPolicyArns += {"Fn::ImportValue": "${MyPrefix}-my-policy-arn" | . tag = "!Sub" }
)
I'm failing to see from the simpler examples in the docs, How do I keep ManagedPolicyArns unique and not add the new entry to it if it already exists?

Thanks to the question "https://stackoverflow.com/questions/42097410/how-to-check-for-presence-of-key-in-jq-before-iterating-over-the-values" and also trying out some various queries to see what output .Resources.Properties.ManagedPolicyArns[] is returning (ie. nulls), I figured it out.
There's a list of null values that it is iterating and that causes the expression to just break and not evaluate. I need to instead return true for items that are null or not matching, then all will be handled and the expression evaluated fully to true or false.
add_policy.yq:
( .Resources[] |=
select(
(.Properties.AssumeRolePolicyDocument.Statement[].Principal.Service.[] == "ecs-tasks.amazonaws.com"
or .Properties.AssumeRolePolicyDocument.Statement[].Principal.Service.[] == "lambda.amazonaws.com")
and (
.Properties.ManagedPolicyArns[] == null
or ( .Properties.ManagedPolicyArns[]."Fn::ImportValue" == "${MyPrefix}-my-policy-arn" | not )
)
)
.Properties.ManagedPolicyArns += {"Fn::ImportValue": "${MyPrefix}-my-policy-arn" | . tag = "!Sub" }
)

Related

Extracting specific value from stderr_lines

This is my ansible script
- name: show1
debug:
msg: "{{response.stderr_lines}}"
Here is the output
msg:
- Using endpoint [https://us-central1-aiplatform.googleapis.com/]
- CustomJob [projects/123456/locations/us-central1/customJobs/112233445566] is submitted successfully.
- ''
- Your job is still active. You may view the status of your job with the command
- ''
- ' $ gcloud ai custom-jobs describe projects/123456/locations/us-central1/customJobs/112233445566'
- ''
- or continue streaming the logs with the command
- ''
- ' $ gcloud ai custom-jobs stream-logs projects/123456/locations/us-central1/customJobs/112233445566'
Here I want to extract custom Job ID which is 112233445566
I used the select module like below
- name: show
debug:
msg: "{{train_custom_image_unmanaged_response.stderr_lines | select('search', 'describe') | list }}"
and it gives me this output
msg:
- ' $ gcloud ai custom-jobs describe projects/123456/locations/us-central1/customJobs/112233445566'
But I just want the job id as specified above. Any idea about that ?
Thanks.
You selected the line you are interested in. From that now you want to isolate the job id number in the end. You can do that using a regular expression like so:
- set_fact:
line: "{{train_custom_image_unmanaged_response.stderr_lines | select('search', 'describe') | list }}"
- debug:
msg: "{{ line | regex_search('.*/customJobs/(\\d+)', '\\1') }}"
This will give you all the digits in the end of the line after /customJobs/. See https://docs.ansible.com/ansible/latest/user_guide/playbooks_filters.html#searching-strings-with-regular-expressions

Create entry in helm template only if an entry exists in a map

Assuming this map exists in my values.yaml
something:
somethingElse:
variable1: value1
variable2: value2
variable3: value3
I want to create a helm template for a kubernetes Secret resource (although this is not of primary importance) if and only if say the key - value pair variable2: value2 exists. (I am only actually interest to match the variable2's existence, not what is the value of value2)
I know how to range to include all entries
{{- range $name, $value := .Values.something.somethingElse }}
{{indent 4 $name }}: {{ $value }}
{{- end }}
but in pseudocode, what I want is
if variable2 in .Values.something.somethingElse
variable2: value2
Is this somehow feasible using the helm templating language?
You can do:
{{if (index .Values.something.somethingElse "variable2")}}
Helm contain a flow control and you can create if / else conditional blocks, for your case this example can be helpful.
In your values.yaml
foo:
enabled: true
So, in your template:
{{- if .Values.foo.enabled }}
--- TEMPLATE CODE --
{{- end }}
Ref: https://helm.sh/docs/chart_template_guide/control_structures/

How to grab last two lines from ansible (register stdout) initialization of kubernetes cluster

This is the piece of my playbook file for the question:
- name: Initialize the Kubernetes cluster using kubeadm
command: kubeadm init --config /etc/kubernetes/kubeadminit.yaml
register: init_output
- name: Copy join command to local file
local_action: copy content={{ init_output.stdout }} dest="./join-command"
Currently join-command contains the entire stdout (30+ lines of text) for content. What I want to grab is just the last two lines of init_output.stdout instead of the entire output. I've looked into using index reference (ie. init_output.stdout[#]) but I don't know that the output will always be the same length and I don't know how to use indexes to grab more than one line, but i'm fairly certain that the last two lines will always be the join command. Any suggestions?
Select last 2 lines from the list stdout_lines
- local_action: copy content={{ init_output.stdout_lines[-2:] }} dest="./join-command"
It's possible to format the lines in a block. For example
- local_action:
module: copy
content: |
{{ init_output.stdout_lines[-2] }}
{{ init_output.stdout_lines[-1] }}
dest: "./join-command"
To append the lines in a loop try
- local_action:
module: lineinfile
path: "./join-command"
line: "{{ item }}"
insertafter: EOF
create: true
loop: "{{ init_output.stdout_lines[-2:] }}"
I encountered this kind of issue and did not want to copy the join command to a local file so I did a set_fact instead this way:
- set_fact:
join_cmd: '{{ init_output.stdout_lines[-2][:-2] }}{{ init_output.stdout_lines[-1] }}'
I did this...
- name: kubeadm init
shell: |
kubeadm init --control-plane-endpoint \
localhost \
--control-plane-endpoint kube-api.local >> /tmp/run_kube_init.sh
when: master == "yes"
- name: Get join from master
fetch:
src: "/tmp/run_kube_init.sh"
dest: "/tmp/run_kube_init.sh"
flat: yes
when: ansible_hostname == 'k-master'
- name: Add join file to nodes
copy:
src: "/tmp/run_kube_init.sh"
dest: "/tmp/run_kube_init.sh"
when: master == "no"
- name: Extract join token for nodes
shell: tail -n +2 /tmp/run_kube_init.sh | head -n -1 | awk '{print $5}' | tail -n 1
register: JOIN_TOKEN
- set_fact:
join_token: "{{ JOIN_TOKEN.stdout }}"
- name: join nodes
shell: |
kubeadm join kube-api.local:6443 \
--token {{ JOIN_TOKEN.stdout }} \
--discovery-token-unsafe-skip-ca-verification
when: master == "no"
- name: rm /tmp/run_kube_init.sh
ansible.builtin.file:
path: /tmp/run_kube_init.sh
state: absent

How to combine multiple dict variables into an array based on regex of dict variable names

I have multiple dict variables in my inventories that start with 'my_var_*'. I would like to combine these into an array of dicts named 'my_var'
In my playbook, I'm using 'set_fact:' to create the 'my_var' variable by attempting to pull the matching variables from "hostvars['localhost']" with a select filter and match regex, but join only works on strings.
variables.yml
my_var_1:
element1: value11
element2: value12
my_var_2:
element1: value21
element2: value22
playbook.yml
- hosts: localhost
connection: local
gather_facts: False
tasks:
- set_fact:
my_var: "{{ hostvars['localhost'] | select('match', '^my_var_*') | join(', ' }}"
- debug:
msg: "{{ my_var }}"
is it possible to join these 'dict' variables into an 'array' like this?
my_var:
- element1: value11
element2: value12
- element1: value21
element2: value22
or possibly even
my_var:
- name: 1
element1: value11
element2: value12
- name: 2
element1: value21
element2: value22
You're very close, but as you point out, the join method on a string is for joining strings. You want to append lists, which you accomplish with the + operator.
There are also a few other issues:
The expression:
hostvars['localhost'] | select('match', '^my_var_*')
Will produce a list that looks like:
[
"my_var_1",
"my_var_2"
]
...which isn't what you want. You want the values of these variables, not the key names. We can use the dict2items filter and the selectattr filter to generate the data we want:
---
- hosts: localhost
gather_facts: false
tasks:
- name: set facts on localhost
set_fact:
my_var_1:
element1: value11
element2: value12
my_var_2:
element1: value21
element2: value22
- hosts: localhost
gather_facts: false
tasks:
- name: merge vars into my_var
set_fact:
my_var: "{{ hostvars['localhost']|dict2items|selectattr('key', 'match', '^my_var_')|map(attribute='value')|list }}"
- name: show content of my_var
debug:
var: my_var
This will produce the following output:
TASK [show content of my_var] ************************************************************************************
ok: [localhost] => {
"my_var": [
{
"element1": "value11",
"element2": "value12"
},
{
"element1": "value21",
"element2": "value22"
}
]
}
If you get rid of the map(attribute='value') filter, you get:
TASK [show content of my_var] *****************************************************************************************
ok: [localhost] => {
"my_var": [
{
"key": "my_var_1",
"value": {
"element1": "value11",
"element2": "value12"
}
},
{
"key": "my_var_2",
"value": {
"element1": "value21",
"element2": "value22"
}
}
]
}
This isn't exactly what you ask for as the second option, but it does include both the key name and values.
Additional notes:
In the above, I've used a separate play running set_fact to set the values of these variables, because this solution will only work if the variables are host vars (aka "facts") rather than global variables. You don't show in your question how you're setting these variables so I don't know if this will all work as written.
In a regular expression, * means "the preceding character zero or more times", so the expression ^my_var_* would match my_var, my_var_1, my_var______________, my_varfoo, and so forth. You can simply write ^my_var_ to select the variable names in which you're interested (this will select anything that begins with the text my_var_).

Weird issue with hasMany in Grails using PostgreSQL. "ERROR: column <column_name> does not exist"

I'm having a hard time trying to get "hasMany" to work in Grails 2.0.1 using PostgreSQL 9.1. I got two tables:
CREATE TABLE "_QUESTIONS"
(
"QUESTION_ID" bigint NOT NULL,
"TEXT" text,
CONSTRAINT "PK" PRIMARY KEY ("QUESTION_ID" )
)
WITH (
OIDS=FALSE
);
ALTER TABLE "_QUESTIONS"
OWNER TO postgres;
CREATE TABLE "_ANSWERS"
(
"ANSWER_ID" bigint NOT NULL,
"TEXT" text,
"QUESTION_ID" bigint,
CONSTRAINT "PK1" PRIMARY KEY ("ANSWER_ID" ),
CONSTRAINT "FK" FOREIGN KEY ("QUESTION_ID")
REFERENCES "_QUESTIONS" ("QUESTION_ID") MATCH SIMPLE
ON UPDATE NO ACTION ON DELETE NO ACTION
)
WITH (
OIDS=FALSE
);
ALTER TABLE "_ANSWERS"
OWNER TO postgres;
and two domain classes:
class Question {
String text
String toString(){
text
}
static constraints = {
}
static hasMany = [answers: Answer]
static mapping = {
table '`_QUESTIONS`'
version false
id generator: 'identity'
id column: '`QUESTION_ID`'
text column: '`TEXT`'
}
}
class Answer {
String text
Question question
String toString(){
text
}
static constraints = {
}
static belongsTo = [question : Question]
static mapping = {
table '`_ANSWERS`'
version false
id generator: 'identity'
id column: '`ANSWER_ID`'
text column: '`TEXT`'
question column: '`QUESTION_ID`'
}
}
I've generated Views and Controllers for both of them and when I try to browse a particular Question I get the following error:
URI:/hasManyTest/question/show/1
Class:org.postgresql.util.PSQLException
Message:ERROR: column answers0_.question_id does not exist Position: 8
with stack trace:
Line | Method
->> 8 | runWorker in \grails-app\views\question\show.gsp
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Caused by SQLGrammarException: could not initialize a collection: [hasmanytest.Question.answers#1]
->> 26 | doCall in C__Users_root_IdeaProjects_hasManyTest_grails_app_views_question_show_gsp$_run_closure2
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
| 55 | run in C__Users_root_IdeaProjects_hasManyTest_grails_app_views_question_show_gsp
| 1110 | runWorker in java.util.concurrent.ThreadPoolExecutor
| 603 | run in java.util.concurrent.ThreadPoolExecutor$Worker
^ 722 | run . . . in java.lang.Thread
Caused by PSQLException: ERROR: column answers0_.question_id does not exist
Position: 8
->> 2103 | receiveErrorResponse in org.postgresql.core.v3.QueryExecutorImpl
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
| 1836 | processResults in ''
| 257 | execute . in ''
| 512 | execute in org.postgresql.jdbc2.AbstractJdbc2Statement
| 388 | executeWithFlags in ''
| 273 | executeQuery in ''
| 96 | executeQuery in org.apache.commons.dbcp.DelegatingPreparedStatement
| 26 | doCall in C__Users_root_IdeaProjects_hasManyTest_grails_app_views_question_show_gsp$_run_closure2
| 55 | run . . . in C__Users_root_IdeaProjects_hasManyTest_grails_app_views_question_show_gsp
| 1110 | runWorker in java.util.concurrent.ThreadPoolExecutor
| 603 | run . . . in java.util.concurrent.ThreadPoolExecutor$Worker
^ 722 | run in java.lang.Thread
I've done a lot of gymnastics for the past couple of days and nothing seems to help, when I remove association everything works fine though. Am I missing something obvious?
The issue appears to be a mismatch between assumptions made by Grails/GORM mapping and the SQL used to create the tables.
If you omit the quotes around table and column names in the above SQL, tables and columns will be case-insensitive. From http://wiki.postgresql.org/wiki/Things_to_find_out_about_when_moving_from_MySQL_to_PostgreSQL:
Database, table, field and columns names in PostgreSQL are case-independent, unless you created them with double-quotes around their name, in which case they are case-sensitive. In MySQL, table names can be case-sensitive or not, depending on which operating system you are using.
Taking Grails out of the equation, if you create the table using the following:
CREATE TABLE "_QUESTIONS"
(
"QUESTION_ID" bigint NOT NULL,
"TEXT" text,
CONSTRAINT "PK" PRIMARY KEY ("QUESTION_ID" )
)
WITH (
OIDS=FALSE
);
Then:
test=> select * from _questions;
ERROR: relation "_questions" does not exist
LINE 1: select * from _questions
^
test=> select * from _QUESTIONS;
ERROR: relation "_questions" does not exist
LINE 1: select * from _QUESTIONS;
^
test=> select * from "_QUESTIONS";
QUESTION_ID | TEXT
-------------+------
(0 rows)
However, if you create the table without the quotes around table and column names:
CREATE TABLE _QUESTIONS
(
QUESTION_ID bigint NOT NULL,
TEXT text,
CONSTRAINT PK PRIMARY KEY (QUESTION_ID)
)
WITH (
OIDS=FALSE
);
Then:
test=> select * from _questions;
question_id | text
-------------+------
(0 rows)
test=> select * from _QUESTIONS;
question_id | text
-------------+------
(0 rows)
test=> select * from "_QUESTIONS";
ERROR: relation "_QUESTIONS" does not exist
LINE 1: select * from "_QUESTIONS";
^
I've managed to figure this out. Turned out, somewhy ORM tries to access association column through its lowercased name, changing the name solved the problem.