What is the order of the directives in the cloud-config section of a cloud-init user-data object. This is important to avoid race type conditions.
I know bootcmd runs early and before runcmd, but is there a good list of the order of all the methods?
From https://git.launchpad.net/cloud-init/tree/config/cloud.cfg (thanks to garbelini)
(As of Sep 2017 the above link is incorrect and is now https://git.launchpad.net/cloud-init/tree/config/cloud.cfg.tmpl, see note below)
# The modules that run in the 'init' stage
cloud_init_modules:
- migrator
- ubuntu-init-switch
- seed_random
- bootcmd
- write-files
- growpart
- resizefs
- set_hostname
- update_hostname
- update_etc_hosts
- ca-certs
- rsyslog
- users-groups
- ssh
# The modules that run in the 'config' stage
cloud_config_modules:
# Emit the cloud config ready event
# this can be used by upstart jobs for 'start on cloud-config'.
- emit_upstart
- disk_setup
- mounts
- ssh-import-id
- locale
- set-passwords
- snappy
- grub-dpkg
- apt-pipelining
- apt-configure
- package-update-upgrade-install
- fan
- landscape
- timezone
- lxd
- puppet
- chef
- salt-minion
- mcollective
- disable-ec2-metadata
- runcmd
- byobu
# The modules that run in the 'final' stage
cloud_final_modules:
- rightscale_userdata
- scripts-vendor
- scripts-per-once
- scripts-per-boot
- scripts-per-instance
- scripts-user
- ssh-authkey-fingerprints
- keys-to-console
- phone-home
- final-message
- power-state-change
Additionally:
There is configuration merging "The order is the following:
- cli config files override environment config files
which override instance configs which override datasource
configs which override base configuration which overrides
default configuration." (see Changelog)
Within the individual generated script directories the scripts are run in the order given by the python "sorted()" builtin
NOTE: although this was correct at the time of answering, looking at the respository (as of Sept 2017) mentioned above there is now a cloud.cfg.tmpl template file which has slightly different settings for different distros
# The modules that run in the 'init' stage
cloud_init_modules:
- migrator
- seed_random
- bootcmd
- write-files
- growpart
- resizefs
{% if variant not in ["freebsd"] %}
- disk_setup
- mounts
{% endif %}
- set_hostname
- update_hostname
{% if variant not in ["freebsd"] %}
- update_etc_hosts
- ca-certs
- rsyslog
{% endif %}
- users-groups
- ssh
# The modules that run in the 'config' stage
cloud_config_modules:
{% if variant in ["ubuntu", "unknown", "debian"] %}
# Emit the cloud config ready event
# this can be used by upstart jobs for 'start on cloud-config'.
- emit_upstart
- snap_config
{% endif %}
- ssh-import-id
- locale
- set-passwords
{% if variant in ["rhel", "fedora"] %}
- spacewalk
- yum-add-repo
{% endif %}
{% if variant in ["ubuntu", "unknown", "debian"] %}
- grub-dpkg
- apt-pipelining
- apt-configure
{% endif %}
{% if variant not in ["freebsd"] %}
- ntp
{% endif %}
- timezone
- disable-ec2-metadata
- runcmd
{% if variant in ["ubuntu", "unknown", "debian"] %}
- byobu
{% endif %}
# The modules that run in the 'final' stage
cloud_final_modules:
{% if variant in ["ubuntu", "unknown", "debian"] %}
- snappy
{% endif %}
- package-update-upgrade-install
{% if variant in ["ubuntu", "unknown", "debian"] %}
- fan
- landscape
- lxd
{% endif %}
{% if variant not in ["freebsd"] %}
- puppet
- chef
- salt-minion
- mcollective
{% endif %}
- rightscale_userdata
- scripts-vendor
- scripts-per-once
- scripts-per-boot
- scripts-per-instance
- scripts-user
- ssh-authkey-fingerprints
- keys-to-console
- phone-home
- final-message
- power-state-change
Related
Question: How to correctly format {{ from_dttm }} and {{ to_dttm }} default Jinja variables so that they are parsed as datetime objects in Apache Superset Custom SQL metrics?
MWE: Say I want to show what is the time range covered by the data I use in my dashboards — what can be affected by the Time Range filter.
I use the public.birth_names demo dataset for the sake of the example.
So I create a BigNumber chart, with the following custom Metric:
age(
{% if from_dttm is not none %}
'{{ from_dttm }}'
{% else %}
min(ds)
{% endif %}
,
{% if to_dttm is not none %}
'{{ to_dttm }}'
{% else %}
max(ds)
{% endif %}
)
However, if I format the Jinja variables as:
{{ from_dttm }}, I get:
Error: syntax error at or near "{"
LINE 1: SELECT age({{ from_dttm }} , '{{ to_dttm }}') AS "age(
'{{ from_dttm }}', I get
Error: invalid input syntax for type timestamp with time zone: "{{ from_dttm }}"
LINE 1: SELECT age('{{ from_dttm }}'
"{{ from_dttm }}", I get
Error: column "{{ from_dttm }}" does not exist
LINE 1: SELECT age("{{ from_dttm }}" ,
I'm using Superset at 5ae7e5499 (Latest commit on Mar 25, 2022), with PostgreSQL as back-end db engine.
After offline discussion with #villebro, it turns out that:
'{{ from_dttm }}' is the valid syntax — one can try with a simpler example: MIN('{{ from_dttm }}'): this works, whereas both other syntaxes yield an error.
however, there is still a bug in current (version 1.4.*) versions, where {{ from_dttm }} is not rendered (what caused AGE() to yield an error in the particular example above). This issue has been raised in #19564, and a fix submitted in #19565.
I need to build a docker compose based on a yml file. In the next yml it will be the name, the image and the version of each service.
"services":
- "service": "front"
"image": "acalls-caselog-web-app"
"version": "latest"
- "service": "back"
"image": "acalls-caselog-web-service"
"version": "latest"
- "service": "vb"
"image": "acalls-caselog-vb-service"
"version": "latest"
- "service": "salesforce"
"image": "acalls-caselog-salesforce-app-service"
"version": "latest"
- "service": "tts"
"image": "ydilo-tts-service"
"version": "latest"
- "service": "ai classifier"
"image": "acalls-caselog-ai-classifier-service"
"version": "latest"
Up to now I had an array to set each image in the docker compose, like this
version: "3.3"
services:
front:
image: url/{{services[0].image}}:{{services[0].version}}
ports:
- "81:81"
extra_hosts:
- "backend:172.32.3.46"
environment:
profile: preproduction
back:
image: url/{{services[1].image}}:{{services[1].version}}
ports:
- "20101:20101"
environment:
profile: preproduction
saleforce:
image: url/{{services[2].image}}:{{services[2].version}}
ports:
- "20103:20103"
environment:
profile: preproduction
But I need to find a way to make this dynamically with a loop in the ansible task, for example, without the array position in the docker-compose file.
Main.yml
---
- name: stop container
ignore_errors: yes
become: True
shell:
cmd: "docker-compose down"
chdir: dir
- name: set docker-compose
template:
src: docker-compose-acalls.yml.j2
dest: dir/docker-compose.yml
mode: 0700
- name: Run container
become: True
shell:
cmd: "nohup docker-compose -f docker-compose.yml up -d"
chdir: dir
you create a template file: you have to play with whitespace , %- and -% to adjust the position, i have just given the general idea
version: "3.3"
services:
{% for item in services %}
{{ item.service }}:
{% if item.image is defined %}
image: url/{{item.image}}:{{item.version}}
{%- endif %}
{% if item.ports is defined %}
ports:
- "{{ item.ports[0] }}"
{%- endif %}
{% if item.extra_hosts is defined %}
extra_hosts:
- "{{ item.extra_hosts[0] }}"
{%- endif %}
{% if item.environment is defined %}
environment:
- profile: {{ item.environment.profile }}
{% endif %}
{% endfor %}
your playbook:
- name: test
hosts: localhost
vars_files:
- reference.yml
tasks:
template:
src: fileconf.j2
dest: composedocker.yml
and your reference.yml file:
services:
- service: "front"
image: "acalls-caselog-web-app"
ports:
- "81:81"
extra_hostss:
- "backend:172.32.3.46"
environment:
profile: preproduction
version: "latest"
- service: "back"
image: "acalls-caselog-web-service"
version: "latest"
ports:
- "20101:20101"
environment:
profile: preproduction
- service: "vb"
image: "acalls-caselog-vb-service"
version: "latest"
- service: "salesforce"
image: "acalls-caselog-salesforce-app-service"
version: "latest"
ports:
- "20103:20103"
environment:
profile: preproduction
- service: "tts"
image: "ydilo-tts-service"
version: "latest"
- service: "ai classifier"
image: "acalls-caselog-ai-classifier-service"
version: "latest"
result:
version: "3.3"
services:
front:
image: url/acalls-caselog-web-app:latest
ports:
- "81:81"
extra_hosts:
- "backend:172.32.3.46"
environment:
- profile: preproduction
back:
image: url/acalls-caselog-web-service:latest
ports:
- "20101:20101"
environment:
- profile: preproduction
vb:
image: url/acalls-caselog-vb-service:latest
salesforce:
image: url/acalls-caselog-salesforce-app-service:latest
ports:
- "20103:20103"
environment:
- profile: preproduction
tts:
image: url/ydilo-tts-service:latest
ai classifier:
image: url/acalls-caselog-ai-classifier-service:latest
I'm trying to filter or map this yml file. Up to now I was able to access it with this
{{services[0].version}}
But I need to access it with it name, without the position.
Yml file
"services":
- "service": "front"
"image": "acalls-caselog-web-app"
"version": "latest"
- "service": "back"
"image": "acalls-caselog-web-service"
"version": "latest"
docker-compose.yml.j2
version: "3.3"
services:
front:
image: url/{{services[0].image}}:{{services[0].version}}
ports:
- "81:81"
extra_hosts:
- "backend:172.32.3.46"
environment:
profile: preproduction
back:
image: url/ {{services[1].image}} : {{services[1].version}}
ports:
- "82:82"
extra_hosts:
- "backend:172.32.3.46"
environment:
profile: preproduction
I really don't now If I need to use map, filter or there is another form like service.service(front).image
You can do this using the selectattr filter:
{{services | selectattr("service", "equalto", "front")}}
In your YAML, you can apply it like this:
version: "3.3"
services:
front:
image: {% with front=services|selectattr("service", "equalto", "front") -%}
url/{{front.image}}:{{front.version}}
{%- endwith %}
ports:
- "81:81"
extra_hosts:
- "backend:172.32.3.46"
environment:
profile: preproduction
back:
image: {% with back = services|selectattr("service", "equalto", "back") -%}
url/ {{back.image}} : {{back.version}}
{%- endwith %}
ports:
- "82:82"
extra_hosts:
- "backend:172.32.3.46"
environment:
profile: preproduction
Need to set up mongo dB replica set in 3 instances ,one can be primary and rest two will be secondary.
Anyone can suggest me about how can I write the playbook.
Have started mongo shell in three servers and initiate the replication name
'''replication:
replSetName: "testingrs"'''
Ansible provides already plugin for it: community.mongodb.mongodb_replicaset
When I deployed my MongoDB sharded cluster, the plugin was still version 1.0 and had many limitations. We also had some problems with installing pymongo, so I developed the tasks manually. However, I think with current version there is no need anymore to write the tasks by your own.
Anyway, my playbook looks like this:
- name: Check if Replicaset is already initialized
shell:
cmd: "/usr/bin/mongo --norc --quiet localhost:{{ ports.config }}"
executable: /bin/bash
stdin: "rs.status().codeName"
register: result
changed_when: false
check_mode: no
- set_fact:
rs_initiate: |
{% set members = [] %}
{% for host in groups['config'] | sort %}
{% set m = {'_id': loop.index0 } %}
{% set _ = m.update({'host': host + '.' + domain + ':' + ports.config | string }) %}
{% set _ = members.append(m) %}
{% endfor %}
{% set init = {'_id': replica_set.conf} %}
{% set _ = init.update({'members': members}) %}
{{ init }}
rs: |
{% set i = (result.stdout == 'NotYetInitialized') %}
{% for host in ansible_play_hosts %}
{% set i = i and (hostvars[host].result.stdout == 'NotYetInitialized') %}
{% endfor %}
{{ {'NotYetInitialized': i} }}
- name: Init Replicaset
shell:
cmd: "/usr/bin/mongo --norc --quiet localhost:{{ ports.config }}"
executable: /bin/bash
stdin: |
rs.initiate({{ rs_initiate | to_json }})
rs.status()
while (! db.isMaster().ismaster ) sleep(1000)
when: rs.NotYetInitialized and inventory_hostname_short == (groups['config'] | sort | first)
One issue I had was to deal with authentication, because when you deploy a MongoDB from scratch then no user exist. Thus when you like to run the playbook multiple times, you have to distinct with and without authentication.
My playbook contains these tasks:
- name: Check if authentication is enabled
shell:
cmd: "/usr/bin/mongo --norc --quiet localhost:{{ ports.router }}"
executable: /bin/bash
stdin: "rs.status().codeName"
register: result
ignore_errors: yes
changed_when: false
when: inventory_hostname_short == (groups['application'] | sort | first)
- name: Authenticate if needed
set_fact:
authenticate: "{{ (result.stdout == 'Unauthorized') | ternary('-u admin -p ' + password[env].admin + ' --authenticationDatabase admin','') }}"
when: inventory_hostname_short == (groups['application'] | sort | first)
- name: Create users
shell:
cmd: "/usr/bin/mongo {{ authenticate }} --norc --quiet localhost:{{ ports.router }}"
executable: /bin/bash
stdin: |
admin = db.getSiblingDB("admin")
admin.createUser({ user: "admin", pwd: "{{ password[env].admin }}", roles: ["root"] })
admin.auth("admin", "{{ password[env].admin }}")
// create more users if needed
admin.createUser(...)
when: inventory_hostname_short == (groups['application'] | sort | first)
I'm working with Ansible 2.2.1.0 and I work on a old project made by someone else with errors.
I have the following variables in my code:
software_output:
- { file_input: 'Download_me.zip', file_output: 'download.zip' }
software_version:"0.5,0.6"
And I have this shell module instruction to download on a FTP:
- name: "MySoftware | get package on FTP"
shell: >
curl --ftp-ssl -k {{ ' --ssl-allow-beast ' if os == 'aix' else "" }} -# -f -u {{ ftp_user }}:{{ ftp_password }} -f "{{ ftp_url | replace('##software_version##',item[1]) }}{{ item[0].file_input }}"
-o {{ require_inst_dir }}/{{ item[0].file_output }} 2>/dev/null
with_nested:
- software_output
- "{{ software_version.split(',') }}"
when: software_version is defined
But it doesn't work at all, I have the following error:
'unicode object' has no attribute 'file_input'
It looks like with_nested is not used as it has to be used, did I missed something?
In:
with_nested:
- software_output
software_output is a string software_output.
To refer to the variable value, change to:
with_nested:
- "{{ software_output }}"
Long time ago the first syntax was valid, but it was long time ago.