I run airflow in docker and has volumes setting as
- ~/.aws:/usr/local/airflow/.aws:rw
I can see my local ~/.aws/credentials has:
[some_profile]
aws_access_key_id = xxxxx
aws_secret_access_key = xxxxx
and I can see the same content in airflow worker /usr/local/airflow/.aws/credentials, and aws configure list can return expected results like:
Name Value Type Location
---- ----- ---- --------
profile datalake env ['xxx_PROFILE', 'AWS_PROFILE']
access_key ****************xxxx shared-credentials-file
secret_key ****************xxxx shared-credentials-file
region <not set> None None
and cmd like aws s3 ls works locally. However for
sts_client = boto3.client('sts')
assumed_role = sts_client.assume_role(RoleArn=role_arn, RoleSessionName='xxx')
it throws:
botocore.exceptions.NoCredentialsError: Unable to locate credentials
What am I missing here? Thanks.
The problem could be here.
When you're declaring [some_profile] for the credentials file, it should be like this:
[default]
aws_access_key_id = xxxxx
aws_secret_access_key = xxxxx
There must be a default profile in your credentials.
If that doesn't work, you can always set the credentials manually
First, set the profile
[some_profile]
aws_access_key_id = xxxxx
aws_secret_access_key = xxxxx
Then, your Python code should look like this
os.environ['AWS_PROFILE'] = "some_profile"
os.environ['AWS_DEFAULT_REGION'] = "some_region"
s3 = boto3.client('s3', region_name='some_region')
Related
I have a secret being used as env var in another env var as follows:
- name: "PWD"
valueFrom:
secretKeyRef:
name: "credentials"
key: "password"
- name: HOST
value: "xyz.mongodb.net"
- name: MONGODB_URI
value: "mongodb+srv://user:$(PWD)#$(HOST)/db_name?"
When i exec into the container and run env command to see the values of env i see -
mongodb+srv://user:password123
#xyz.mongodb.net/db_name?
The container logs show error as authentication failure.
Is this something that is expected to work in kubernetes ? There docs talk about dependent env vars but do not give example using secrets. Did not find clear explanation on this after extensive search. Only found this one article doing something similar.
Some points to note -
The secret is a sealed secret.
This is the final manifest's contents, but all this is templated using helm.
The value is being used inside a spring boot application
Is the new line after 123 expected ?
If this evaluation of env from a secret in another env is possible then what am I doing wrong here ?
The issue was with the command used to encode the secret - echo "pasword" | base64. The echo adds a newline character at the end of the string. Using echo -n "password" | base64 fixed the secret.
Closing the issue.
I'm trying to receive a list of all my server names and their IPs on Hetzner using ansible and hcloud module however I'm receiving the following error;
ERROR! couldn't resolve module/action 'hcloud'. This often indicates a misspelling, missing collection, or incorrect module path.
The error appears to be in '/home/melvmagr/repos/ansible/server-content/server-content.yml': line 8, column 7, but may
be elsewhere in the file depending on the exact syntax problem.
The offending line appears to be:
tasks:
- name: List servers
^ here
Here is my YAML file;
---
- name: List Hetzner server names and IP addresses
hosts: servername_example
gather_facts: false
vars:
hcloud_token: "MY_HETZNER_API_TOKEN"
tasks:
- name: List servers
hcloud:
api_token: "MY_HETZNER_API_TOKEN"
state: present
command: server_list
register: server_list
- name: Print server names and IP addresses
debug:
msg: "Server {{ item.name }} has IP address {{ item.public_net.ipv4.ip }}"
loop: "{{ server_list.servers }}"
More info that could prove to be helpful:
❯ ansible --version
ansible [core 2.12.10]
config file = /home/melvmagr/repos/ansible/ansible.cfg
configured module search path = ['/home/melvmagr/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
ansible python module location = /usr/lib/python3/dist-packages/ansible
ansible collection location = /home/melvmagr/.ansible/collections:/usr/share/ansible/collections
executable location = /usr/bin/ansible
python version = 3.8.10 (default, Nov 14 2022, 12:59:47) [GCC 9.4.0]
jinja version = 2.10.1
libyaml = True
This is my ansible.cfg file;
[defaults]
inventory=./inventories
host_key_checking = False
log_path=/var/tmp/ansible_history
timeout=300
I've also installed the following collections;
ansible-galaxy collection install hetzner.hcloud
❯ ls ~/.ansible/collections/ansible_collections/hetzner/hcloud/
CHANGELOG.rst COPYING FILES.json MANIFEST.json README.md changelogs meta plugins tests
Any help would be greatly appreciated. Thank you!
Managed to solve it by simplifying my playbook the following way;
---
- name: Get server information from Hetzner Cloud
hosts: myservername
gather_facts: false
tasks:
- name: Print server names and IP addresses
debug:
msg: "{{ inventory_hostname }} IP is {{ ansible_ssh_host }}"
I'm installing Airflow on kind with the following command:
export RELEASE_NAME=first-release
export NAMESPACE=airflow
helm install $RELEASE_NAME apache-airflow/airflow --namespace $NAMESPACE \
--set images.airflow.repository=my-dags \
--set images.airflow.tag=0.0.1 \
--values env.yaml
Andthe file env.yaml looks like the below:
env:
- name: "AIRFLOW_VAR_KEY"
value: "value_1"
But from the Web UI (when I go to Admins --> Variables), these credentials don't appear there.
How do I pass these credentials during helm install? Thanks!
UPDATE: It turns out that the environment variable was set successfully. However it doesn't show up on the Web UI
i am not sure how your full env.yaml file is
but to set the environment variables in Airflow
## environment variables for the web/scheduler/worker Pods (for airflow configs)
##
## WARNING:
## - don't include sensitive variables in here, instead make use of `airflow.extraEnv` with Secrets
## - don't specify `AIRFLOW__CORE__SQL_ALCHEMY_CONN`, `AIRFLOW__CELERY__RESULT_BACKEND`,
## or `AIRFLOW__CELERY__BROKER_URL`, they are dynamically created from chart values
##
## NOTE:
## - airflow allows environment configs to be set as environment variables
## - they take the form: AIRFLOW__<section>__<key>
## - see the Airflow documentation: https://airflow.apache.org/docs/stable/howto/set-config.html
##
## EXAMPLE:
## config:
## ## Security
## AIRFLOW__CORE__SECURE_MODE: "True"
## AIRFLOW__API__AUTH_BACKEND: "airflow.api.auth.backend.deny_all"
Reference file
and after that you have to run your command and further your DAG will be able to access the variables.
Helm documentation : https://github.com/helm/charts/tree/master/stable/airflow#docs-airflow---configs
Make sure you are configuring section : airflow.config
Okay, so I've figured this one out: The environment variables are set just nicely on the pods. However, this will not appear on the Web UI.
Workaround: to make it appear in the web UI, I will have to go into the Scheduler pod and import the variables. It can be done with a bash script.
# Get the name of scheduler pod
export SCHEDULER_POD_NAME="$(kubectl get pods --no-headers -o custom-columns=":metadata.name" -n airflow | grep scheduler)"
# Copy variables to the scheduler pod
kubectl cp ./variables.json airflow/$SCHEDULER_POD_NAME:./
# Import variables to scheduler with airflow command
kubectl -n $NAMESPACE exec $SCHEDULER_POD_NAME -- airflow variables import variables.json
I need to use the "plugin/azure_rm.yml" version of azure_rm instead of the older/deprecated "script/azure_rm.ini" to gather dynamic inventory in Azure, in particular because we need VMs from scalesets to be included in inventory. How can I do this?
You can use an inventory script, which (again) calls ansible-inventory, and have it create the config file via a heredoc:
#!/usr/bin/env bash
cat > azure_rm.yml <<HEREDOC
---
plugin: azure_rm
include_vmss_resource_groups:
- '*'
hostvar_expressions:
ansible_host: private_ipv4_addresses | first
plain_host_names: true
keyed_groups:
# places each host in a group named 'tag_(tag name)_(tag value)' for each tag on a VM.
- prefix: tag
key: tags
# places each host in a group named 'azure_loc_(location name)', depending on the VM's location
- prefix: azure_loc
key: location
# group by platform (to copy prefix from ec2.py), eg: platform_windows
- prefix: platform
key: os_disk.operating_system_type
HEREDOC
ansible-inventory -i azure_rm.yml --list
rm azure_rm.yml
So it's literally having ansible-inventory call ansible-inventory, but with a different argument. Note that in order to get the inventory for the correct subscription, you have to create a copy the credential used, with the desired subscription id; it doesn't appear that you can override AZURE_SUBSCRIPTION_ID via the yml environment param.
In an Ansible role I generate the user's SSH key. After that I want to print it to the screen and pause so the user can copy and paste it somewhere else. So far I have something like this:
- name: Generate SSH keys for vagrant user
user: name=vagrant generate_ssh_key=yes ssh_key_bits=2048
- name: Show SSH public key
command: /bin/cat $home_directory/.ssh/id_rsa.pub
- name: Wait for user to copy SSH public key
pause: prompt="Please add the SSH public key above to your GitHub account"
The 'Show SSH public key' task completes but doesn't show the output.
TASK: [Show SSH public key] ***************************************************
changed: [default]
There may be a better way of going about this. I don't really like the fact that it will always show a 'changed' status. I did find this pull request for ansible - https://github.com/ansible/ansible/pull/2673 - but not sure if I can use it without writing my own module.
I'm not sure about the syntax of your specific commands (e.g., vagrant, etc), but in general...
Just register Ansible's (not-normally-shown) JSON output to a variable, then display each variable's stdout_lines attribute:
- name: Generate SSH keys for vagrant user
user: name=vagrant generate_ssh_key=yes ssh_key_bits=2048
register: vagrant
- debug: var=vagrant.stdout_lines
- name: Show SSH public key
command: /bin/cat $home_directory/.ssh/id_rsa.pub
register: cat
- debug: var=cat.stdout_lines
- name: Wait for user to copy SSH public key
pause: prompt="Please add the SSH public key above to your GitHub account"
register: pause
- debug: var=pause.stdout_lines
If you pass the -v flag to the ansible-playbook command, then ansible will show the output on your terminal.
For your use case, you may want to try using the fetch module to copy the public key from the server to your local machine. That way, it will only show a "changed" status when the file changes.
Prints pubkey and avoid the changed status by adding changed_when: False to cat task:
- name: Generate SSH keys for vagrant user
user: name=vagrant generate_ssh_key=yes ssh_key_bits=2048
- name: Check SSH public key
command: /bin/cat $home_directory/.ssh/id_rsa.pub
register: cat
changed_when: False
- name: Print SSH public key
debug: var=cat.stdout
- name: Wait for user to copy SSH public key
pause: prompt="Please add the SSH public key above to your GitHub account"