Install all packages from a folder with Ansible 2.0 - find

I have a folder where I place unmaintained python packages, so I installed them from the zip rather than from their repository.
I am using Ansible 2.0, so the find command seems to be the way to do it
So far I was doing the following:
- name: Install unmaintained dependencies
pip:
name: "{{ my_project_app }}/requirements/{{ item }}"
virtualenv: "{{ my_project_venv }}"
with_items:
- django-hijack-2.0.0.zip
- django-image-cropping-django-19.zip
- pisa-3.0.33.zip
Now I'm playing with the find command
- name: Loading unmaintained dependencies
find:
paths: "{{ my_project_app }}/requirements"
patterns: "*.zip"
register: unmaintained_dependencies
- debug:
var: unmaintained_dependencies
If I run the playbook above, I get the following output
ok: [192.168.1.212] => {
"files_found": {
"changed": false,
"examined": 3,
"files": [
{
...
"path": "/data/my_project/requirements/pisa-3.0.33.zip",
...
},
...
],
"matched": 3,
"msg": ""
}
}
I guess that there must be a way to put everything together but here is where I'm stuck.

I still don't get what are you trying to achieve here yet.. Are you using the find module just because pip module doesn't install packages from zip files?
For your find workaround, you can create a task that iterates over the results of the find task, using with_items: files_found.files, and using {{ item.path }} whenever you need the path:
- name: Install unmaintained dependencies
pip:
name: "{{ item.path }}"
virtualenv: "{{ my_project_venv }}"
with_items: "{{files_found.files}}"
Also, instead of using file, you can try to make a loop using with_fileglob:
- pip:
name: {{ item }}
virtualenv: "{{ my_project_venv }}"
with_fileglob:
- "{{ my_project_app }}"/requirements/*.zip
Note, I didn't have time to test any of these solutions, or to ask more about what were you trying to achieve, but I hope they help with your problem.

Related

TYPO3: How to publish an extension to TER with Github actions and tailor and add third party library on the fly

I would like to publish an extension automatically to TER by using the Github actions and tailor, the CLI Tool for maintaining public TYPO3 Extensions . This works perfectly fine with the following workflow configuration:
name: TYPO3 Extension TER Release
on:
push:
tags:
- '*'
jobs:
publish:
name: Publish new version to TER
if: startsWith(github.ref, 'refs/tags/')
runs-on: ubuntu-20.04
env:
TYPO3_EXTENSION_KEY: ${{ secrets.TYPO3_EXTENSION_KEY }}
TYPO3_API_TOKEN: ${{ secrets.TYPO3_API_TOKEN }}
steps:
- name: Checkout repository
uses: actions/checkout#v2
- name: Check tag
run: |
if ! [[ ${{ github.ref }} =~ ^refs/tags/[0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3}$ ]]; then
exit 1
fi
- name: Get version
id: get-version
run: echo ::set-output name=version::${GITHUB_REF/refs\/tags\//}
- name: Get comment
id: get-comment
run: |
readonly local comment=$(git tag -n10 -l ${{ steps.get-version.outputs.version }} | sed "s/^[0-9.]*[ ]*//g")
if [[ -z "${comment// }" ]]; then
echo ::set-output name=comment::Released version ${{ steps.get-version.outputs.version }} of ${{ env.TYPO3_EXTENSION_KEY }}
else
echo ::set-output name=comment::$comment
fi
- name: Setup PHP
uses: shivammathur/setup-php#v2
with:
php-version: 7.4
extensions: intl, mbstring, json, zip, curl
tools: composer:v2
- name: Install tailor
run: composer global require typo3/tailor --prefer-dist --no-progress --no-suggest
- name: Publish to TER
run: php ~/.composer/vendor/bin/tailor ter:publish --comment "${{ steps.get-comment.outputs.comment }}" ${{ steps.get-version.outputs.version }}
Since my extension depends on a third party PHP library (which is loaded if the extension is installed with composer) I need to add this library on the fly, when the extension gets deployed to the TER. Therefore I added in Resources/Private/PHP/ a composer.json and a composer.lock.
Now I would like to tell tailor to execute composer install in Resources/Private/PHP/ and package the extension including the external library. Is this possible? If so, how?
Generally it is useful to put everything in Composer command scripts so that you are actually able to execute actions without depending on Github actions or a CI in general.
For example you could add a few commands like this:
{
"scripts": {
"build:cleanup": [
"git reset --hard",
"git clean -xfd"
],
"deploy:ter:setup": [
"#composer global require clue/phar-composer typo3/tailor"
],
"build:ter:vendors": [
"(mkdir -p /tmp/vendors && cd /tmp/vendors && composer require acme/foo:^1.0 acme/bar:^2.0 && composer global exec phar-composer build -v)",
"cp /tmp/vendors/vendors.phar ./Resources/Private/libraries.phar",
"echo \"require 'phar://' . \\TYPO3\\CMS\\Core\\Utility\\ExtensionManagementUtility::extPath('$(composer config extra.typo3/cms.extension-key)') . 'Resources/Private/libraries.phar/vendor/autoload.php';\" >> ext_localconf.php"
],
"deploy:ter:upload": [
"composer global exec -v -- tailor ter:publish --comment \"$(git tag -l --format='%(contents)' $TAG)\" $TAG"
],
"deploy:ter": [
"#build:cleanup",
"#deploy:ter:setup",
"#build:ter:vendors",
"#deploy:ter:upload"
]
}
}
The various scripts explained:
build:cleanup drops all pending files to ensure no undesired files are uploaded to the TER
build:ter:setup installs typo3/tailor for the TER upload and clue/phar-composer for building a Phar of your vendor dependencies
build:ter:vendors installs a manually maintained list of dependencies in a temporary directory and builds a Phar from that; it then copies that Phar to the current directory and adds a require call to your ext_localconf.php
deploy:ter:upload finally invokes Tailor to upload the current directory including the Phar and the ext_localconf.php adjustment and fetches the comment of the specified Git tag; notice that tagging should be done with --message here to have an annotated Git tag (e.g. git tag -a 1.2.3 -m "Bugfix release")
Now assuming you export/provide the environment variables TYPO3_API_USERNAME, TYPO3_API_PASSWORD and TAG you can instantly deploy the latest release like this:
# Provide username and password for TER, if not done yet
export TYPO3_API_USERNAME=YourName
export TYPO3_API_PASSWORD=YourSecretPassword
# Provide the tag to deploy
TAG=1.2.3 composer deploy:ter
Subsequently the related Github action becomes very simple:
jobs:
build:
# ...
release-ter:
name: TYPO3 TER release
if: startsWith(github.ref, 'refs/tags/')
needs: build
runs-on: ubuntu-latest
steps:
- uses: actions/checkout#v3
- name: Deploy to TER
env:
TYPO3_API_USERNAME: ${{secrets.TYPO3_API_USERNAME}}
TYPO3_API_PASSWORD: ${{secrets.TYPO3_API_PASSWORD}}
TAG: ${{github.ref_name}}
run: composer deploy:ter
Here is a live example:
.github/workflows/ci.yml
composer.json

Using nested values defined in environment

I have the following helmfile
releases:
- name: whoami-mn
namespace: whoami-mn-{{ .Environment.Name }}
chart: tons/whoami-mn
version: {{ .Values.chartVersions.whoami }}
installed: {{ .Values.installed }}
values:
- env/{{ .Environment.Name }}-values.yaml
environments:
dev:
values:
- installed: true
- chartVersions:
whoami: 0.3.0
otherApp: 0.2.0
prod:
values:
- installed: true
- chartVersions:
whoami: 0.2.0
otherApp: 0.1.0
But when running helmfile -e dev template I get the following error
in ./helmfile.yaml: error during helmfile.yaml.part.0 parsing: template: stringTemplate:5:23: executing "stringTemplate" at <.Values.chartVersions.whoami>: map has no entry for key "chartVersions"
I can do something like the following
environments:
dev:
values:
- installed: true
- chartVersionWhoami: 0.3.0
That works but I'd prefer having all chart version grouped under a parent key. Any suggestions about how to get the former working?
UPDATE:
The following works
...
version: {{ (index .Values "chartVersions").whoamiMn }}
...
With the environment defined like the following
environments:
dev:
values:
- installed: true
- chartVersions:
whoamiMn: 0.3.0
But I really feel like my first attempt should work as well!
After experimenting and discussing further here. I figured out that I can fix this issue by simply listing my environments before my releases. In my pedantic point of view I'd appreciate having my releases listed first but it works.

How to add a callback-plugin to AWX docker

Installed AWX docker from here - https://github.com/ansible/awx. I am trying to add a callback-plugin for a specific project as written here - https://docs.ansible.com/ansible-tower/latest/html/administration/tipsandtricks.html#using-callback-plugins-with-tower. Does not work. I add to Template-> EXTRA VARIABLES lines
---
bin_ansible_callbacks: true
callback_plugins: /callback_plugins
stdout_callback: selective
Does not work.
I add the directory /var/lib/awx/projects/test/callback_plugins/ to SETTINGS-> JOBS-> ANSIBLE CALLBACK PLUGINS - it doesn't work either.
Tell me, please, how to do it correctly, so that another (custom) plugin picks up and earns.
I'm issuing the same problem, after some debugs the problem I've open a issue on AWX project https://github.com/ansible/awx/issues/4149
In the meantime I've applied a workaround that consists in create a symlinks for each callback plugin you want to use in the callback_plugins folder of your roles project
For example, if you are using the ara project
- name: Research for callbacks in virtualenv libs
find:
path: '{{ ansible_playbook_python|dirname|dirname }}/{{ item }}'
file_type: file
depth: 1
patterns: '*.py'
excludes: '__init__*'
register: _internal__callbacks
with_items:
- lib/python3.6/site-packages/ara/plugins/callbacks
# TODO : prevent existing callbacks to be overwritten
- name: Create symlinks from virtualenv lib directory to local callback_plugins/
file:
src: '{{ item }}'
dest: '{{ playbook_dir }}/callback_plugins/{{ item|basename }}'
state: link
with_items: "{{ _internal__callbacks.results|map(attribute='files')|flatten|map(attribute='path')|list }}"
seems like you have to use callbacks_enabled instead of callback_plugins. put this example configuration in the /var/lib/awx/ansible.cfg file:
[defaults]
callback_whitelist = profile_tasks
--- works on AWX 17.x

setting environment variables in ansible permanently

i am using ansible to add permanent environment variables in ubuntu bashrc .
i have these settings defined in prod_vars file:
enviornment_variables:
PRODUCTION:
MONGO_IP: 0.0.0.0
MONGO_PORT: 27017
ELASTIC_IP: localhost
ELASTIC_PORT: 9200
how can i export it using a task? i kniow about lineinfile module but i do not want to repeat for every env var
- name: set env in the bashrc files
lineinfile: dest=/home/user/.bashrc line='export MONGO_IP=enviornment_variables[PRODUCTION][MONGO_IP]'
also above command gives synatx error?
Instead of using lineinfile module, use the blockinfile module.
So something like this should work:
- name: Adding to environment variables for user
blockinfile:
path: /home/user/.bashrc
insertafter: EOF
block: |
export {{ item.key }}={{ item.val }}
marker: "# {mark} {{ item.key }}"
with_dict:
"{{ enviornment_variables['PRODUCTION'] }}"
ps: The spelling error in "environment" literally took 20+ minutes for me to identify!

Ansible command from inside virtualenv?

This seems like it should be really simple:
tasks:
- name: install python packages
pip: name=${item} virtualenv=~/buildbot-env
with_items: [ buildbot ]
- name: create buildbot master
command: buildbot create-master ~/buildbot creates=~/buildbot/buildbot.tac
However, the command will not succeed unless the virtualenv's activate script is sourced first, and there doesn't seem to be provision to do that in the Ansible command module.
I've experimented with sourcing the activate script in various of .profile, .bashrc, .bash_login, etc, with no luck. Alternatively, there's the shell command, but it seems like kind of an awkward hack:
- name: create buildbot master
shell: source ~/buildbot-env/bin/activate && \
buildbot create-master ~/buildbot \
creates=~/buildbot/buildbot.tac executable=/bin/bash
Is there a better way?
The better way is to use the full path to installed script - it will run in its virtualenv automatically:
tasks:
- name: install python packages
pip: name={{ item }} virtualenv={{ venv }}
with_items: [ buildbot ]
- name: create buildbot master
command: "{{ venv }}/bin/buildbot create-master ~/buildbot
creates=~/buildbot/buildbot.tac"
This is a genericized version of the wrapper method.
venv_exec.j2:
#!/bin/bash
source {{ venv }}/bin/activate
$#
And then the playbook:
tasks:
- pip: name={{ item }} virtualenv={{ venv }}
with_items:
- buildbot
- template: src=venv_exec.j2 dest={{ venv }}/exec mode=755
- command: "{{ venv }}/exec buildbot create-master {{ buildbot_master }}"
Here's a way to enable the virtualenv for an entire play; this example builds the virtualenv in one play, then starts using it the next.
Not sure how clean it is, but it works. I'm just building a bit on what mikepurvis mentioned here.
---
# Build virtualenv
- hosts: all
vars:
PROJECT_HOME: "/tmp/my_test_home"
ansible_python_interpreter: "/usr/local/bin/python"
tasks:
- name: "Create virtualenv"
shell: virtualenv "{{ PROJECT_HOME }}/venv"
creates="{{ PROJECT_HOME }}/venv/bin/activate"
- name: "Copy virtualenv wrapper file"
synchronize: src=pyvenv
dest="{{ PROJECT_HOME }}/venv/bin/pyvenv"
# Use virtualenv
- hosts: all
vars:
PROJECT_HOME: "/tmp/my_test_home"
ansible_python_interpreter: "/tmp/my_test_home/venv/bin/pyvenv"
tasks:
- name: "Guard code, so we are more certain we are in a virtualenv"
shell: echo $VIRTUAL_ENV
register: command_result
failed_when: command_result.stdout == ""
pyenv wrapper file:
#!/bin/bash
source "$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )/activate"
python $#
Just run the virtualenvs pip in a shell:
shell: ~/buildbot-env/pip install ${item}
Works like a charm. I have no idea what the pip module does with virtualenvs, but it seems pretty useless.
As I commented above, I create a script, say it is called buildbot.sh:
source ~/buildbot-env/bin/activate
buildbot create-master [and more stuff]
Then run it on the remote with a task like this:
- name: Create buildbot master
script: buildbot.sh
To me this still seems unneccessary, but it maybe cleaner than running it in a shell command. Your playbook looks cleaner at the cost of not seeing immediately what the script does.
At least some modules do seem to use virtualenv, as both django_manage and rax_clb already have an inbuilt virtualenv parameter. It may not be such a big step for Ansible to include a command-in-virtenv sort of module.