I writing an ansible role. I would like to stop a services on all host that belongs to a group.
How to manage this usecase with ansible?
Ansible Playbook will have scope for host group.
---
- hosts: host_group_name
Ansible role will have task for stopping service.
- name: stop service on host_group
become: yes
name: service_name
state: stopped
Ansible adhoc command:
ansible host_group_name -i inventory -a "service service_name stop"
Use a when condition something like below.
when: inventory_hostname in groups['groupname']
Related
I'm trying to execute a few simple commands on a kubernetes pod in Azure. I've successfully done so with the localhost + pod-as-module-parameter syntax:
---
- hosts: localhost
connection: kubectl
collections:
- kubernetes.core
gather_facts: False
tasks:
- name: Get pod
k8s_info:
kind: Pod
namespace: my-namespace
register: pod_list
- name: Run command
k8s_exec:
pod: "{{pod_list.resources[0].metadata.name}}"
namespace: my_namespace
command: "/bin/bash -c 'echo Hello world'"
However, I want to avoid the repetition of specifying pod and namespace for every kubernetes.core module call, as well as parsing the namespace explicitly in every playbook.
So I got the kubernetes dynamic inventory plugin to work, and can see the desired pod in a group label_app_some-predictable-name, as confirmed by output of ansible-inventory.
What I don't get is if at this point I should be able to run regular command module (I couldn't get that to work at all), or if I need to keep using k8s_exec, which still requires pod and namespace to be specified explicitly (albeit now I can refer to the guest facts populated by the inventory plugin), on top of now requiring delegate_to: localhost:
---
- name: Execute command
hosts: label_app_some-predicatable-name
connection: kubectl
gather_facts: false
collections:
- kubernetes.core
tasks:
- name: Execute command via kubectl
delegate_to: localhost
k8s_exec:
command: "/bin/sh -c 'echo Hello world'"
pod: "{{ansible_kubectl_pod}}"
namespace: "{{ansible_kubectl_namespace}}"
What am I missing? Is there a playbook example that makes use of the kubernetes dynamic inventory?
I am writing ansible scripts for deploying services using Kubernetes, I am stuck with a step that is for the post-deployment process:
I have deployed a service having "replicas: 3", and all the replicas are up and running now my problem is to I have to do a migration for which I have to get into the container and run a script already present there.
I can do it manually by getting into the container individually and then run the script but this will again require manual intervention.
What I want to achieve is once the deployment is done and all the replicas are up and running I want to run the scripts by getting into the containers and all these steps should be performed by ansible script and no manual effort required.
Is there a way to do this?
Take a look at k8s_exec module.
- name: Check RC status of command executed
community.kubernetes.k8s_exec:
namespace: myproject
pod: busybox-test
command: cmd_with_non_zero_exit_code
register: command_status
ignore_errors: True
- name: Check last command status
debug:
msg: "cmd failed"
when: command_status.return_code != 0
#Vasili Angapov is right - k8s_exec module is probably the best solution in this case but I would like to add some useful notes.
To use k8s_exec we need to know the exact Pod name (we need to pass it as pod parameter in ansible task). As you wrote, I assume that your Pods are managed by Deployment, so every Pod has random string in its name added by ReplicaSet. Therefore, you have to find the full names of the Pods somehow.
I've created simple playbook to illustrate how we can find Pod names for all Pods with label: app=web and then run sample touch file123456789 command on these Pods.
---
- hosts: localhost
collections:
- community.kubernetes
tasks:
- name: "Search for all Pods labelled app=web"
k8s_info:
kind: Pod
label_selectors:
- app = web
register: pod_names
- name: "Get Pod names"
set_fact:
pod_names: "{{ pod_names | json_query('resources[*].metadata.name') }}"
- name: "Run command on every Pod labelled app=web"
k8s_exec:
namespace: default
pod: "{{ item }}"
command: touch file123456789
with_items: "{{ pod_names }}"
NOTE: Instead of k8s_exec module you can use command module as well.
In our example instead of k8s_exec task we can have:
- name: "Run command on every Pod labelled app=web"
command: >
kubectl exec "{{ item }}" -n default -- touch file123456789
with_items: "{{ pod_names }}"
When using the "kubeconfig" option I get the error when I click on "verify connection"
Error: TFS.WebApi.Exception: No user credentials found for cluster in KubeConfig content. Make sure that the credentials exist and try again.
The kubeconfig I pasted in, and selected the correct context from, is a direct copy paste of what is in my ~/.kube./config file and this works fine w/ kubectl
apiVersion: v1
clusters:
- cluster:
certificate-authority-data: xxxxx
server: https://aks-my-stage-cluster-xxxxx.hcp.eastus.azmk8s.io:443
name: aks-my-stage-cluster-xxxxx
contexts:
- context:
cluster: aks-my-stage-cluster-xxxxx
user: clusterUser_aks-my-stage-cluster-xxxxx_aks-my-stage-cluster-xxxxx
name: aks-my-stage-cluster-xxxxx
current-context: aks-my-stage-cluster-xxxxx
kind: Config
preferences: {}
users:
- name: clusterUser_aks-my-stage-cluster-xxxxx_aks-my-stage-cluster-xxxxx
user:
auth-provider:
config:
access-token: xxxxx.xxx.xx-xx-xx-xx-xx
apiserver-id: xxxx
client-id: xxxxx
environment: AzurePublicCloud
expires-in: "3599"
expires-on: "1572377338"
refresh-token: xxxx
tenant-id: xxxxx
name: azure
Azure DevOps has an option to save the service connection without verification:
Even though the verification fails when editing the service connection, pipelines that use the service connection do work in my case.
Depending on the pasted KubeConfig you might encounter a 2nd problem where the Azure DevOps GUI for the service connection doesn't save or close, but also doesn't give you any error message. By inspecting the network traffic in e.g. Firefox' developer tools, I found out that the problem was the KubeConfig value being too long. Only ~ 20.000 characters are allowed. After removing irrelevant entries from the config, it worked.
PS: Another workaround is to run kubelogin in a script step in your pipeline.
It seems like it's not enough just to use converted with kubelogin kubeconfig. This plugin is required for kubectl to make a test connection and probably it's not used in the Azure DevOps service connection configuration.
As a workaround that can work for self-hosted build agent, you can install kubectl, kubelogin and whatever software you need to work with your AKS cluster and use shell scripts like:
export KUBECONFIG=~/.kube/config
kubectl apply -f deployment.yaml
You can try run below command to get the KubeConfig. And then copy the content of ~/.kube/config file the service connection to try again.
az aks get-credentials --resource-group myResourceGroup --name myAKSCluster
After run above command and copy the config from the ~/.kube/config on my local machine. i successfully add my kubernetes connection using kubeconfig option
You can also refer to the steps here.
I'm using Amazon EKS for Kubernetes deployment (initially created by an AWS admin user), and currently having difficulty to use the AWS credentials from AWS STS assume-role to execute kubectl commands to interact with the stack
I have 2 EKS stacks on 2 different AWS accounts (PROD & NONPROD), and I'm trying to get the CI/CD tool to deploy to both kubernetes stacks with the credentials provided by AWS STS assume-role but I'm constantly getting error such as error: You must be logged in to the server (the server has asked for the client to provide credentials).
I have followed the following link to add additional AWS IAM role to the config:
https://docs.aws.amazon.com/eks/latest/userguide/add-user-role.html
But I'm not sure what I'm not doing right.
I ran "aws eks update-kubeconfig" to update the local .kube/config file, contents populated as below:
apiVersion: v1
clusters:
- cluster:
certificate-authority-data: [hidden]
server: https://[hidden].eu-west-1.eks.amazonaws.com
name: arn:aws:eks:eu-west-1:[hidden]:cluster/demo-eks
contexts:
- context:
cluster: arn:aws:eks:eu-west-1:[hidden]:cluster/demo-eks
user: arn:aws:eks:eu-west-1:[hidden]:cluster/demo-eks
name: arn:aws:eks:eu-west-1:[hidden]:cluster/demo-eks
current-context: arn:aws:eks:eu-west-1:[hidden]:cluster/demo-eks
kind: Config
preferences: {}
users:
- name: arn:aws:eks:eu-west-1:[hidden]:cluster/demo-eks
user:
exec:
apiVersion: client.authentication.k8s.io/v1alpha1
args:
- token
- -i
- triage-eks
command: aws-iam-authenticator
and had previously updated Kubernetes aws-auth ConfigMap with an additional role as below:
data:
mapRoles: |
- rolearn: arn:aws:iam::[hidden]:role/ci_deployer
username: system:node:{{EC2PrivateDNSName}}
groups:
- system:masters
My CI/CD EC2 instance can assume the ci_deployer role for either AWS accounts.
Expected: I can call "kubectl version" to see both Client and Server versions
Actual: but I get "the server has asked for the client to provide credentials"
What is still missing?
After further testing, I can confirm kubectl will only work from an environment (e.g. my CI EC2 instance with an AWS instance role) of the same AWS account where the EKS stack is created. This means that my CI instance from account A will not be able to communicate with EKS from account B, even if the CI instance can assume a role from account B, and the account B role is included in the aws-auth of the kube config of account B EKS. I hope its due to missing configuration as I find this rather undesirable if a CI tool can't deploy to multiple EKS's from multiple AWS accounts using role assumption.
Look forward to further #Kubernetes support on this
Can kubectl work from an assumed role from AWS
Yes, it can work. A good way to troubleshoot it is to run from the same command line where you are running kubectl:
$ aws sts get-caller-identity
You can see the Arn for the role (or user) and then make sure there's a trust relationship in IAM between that and the role that you specify here in your kubeconfig:
command: aws-iam-authenticator
args:
- "token"
- "-i"
- "<cluster-name>"
- "-r"
- "<role-you-want-to-assume-arn>"
or with the newer option:
command: aws
args:
- eks
- get-token
- --cluster-name
- <cluster-name>
- --role
- <role-you-want-to-assume-arn>
Note that if you are using aws eks update-kubeconfig you can pass in the --role-arn flag to generate the above in your kubeconfig.
In your case, some things that you can look at:
The credential environment variables are not set in your CI?:
AWS_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY
Your ~/.aws/credentials file is not populated correctly in your CI. With something like this:
[default]
aws_access_key_id = xxxx
aws_secret_access_key = xxxx
Generally, the environment variables take precedence so it could be that you could have different credentials altogether in those environment variables too.
It could also be the AWS_PROFILE env variable or the AWS_PROFILE config in ~/.kube/config
users:
- name: aws
user:
exec:
apiVersion: client.authentication.k8s.io/v1alpha1
command: aws-iam-authenticator
args:
- "token"
- "-i"
- "<cluster-name>"
- "-r"
- "<role-arn>"
env:
- name: AWS_PROFILE <== is this value set
value: "<aws-profile>"
Is the profile set correctly under ~/.aws/config?
From Step 1: Create Your Amazon Cluster
When an Amazon EKS cluster is created, the IAM entity (user or role) that creates the cluster is added to the Kubernetes RBAC authorization table as the administrator (with system:master permissions. Initially, only that IAM user can make calls to the Kubernetes API server using kubectl.
As you have discovered you can only access the cluster with the same user/role that created the EKS cluster in the first place.
There is a way to add additional roles to the cluster after creation by editing the aws-auth ConfigMap that has been created.
Add User Role
By editing the aws-auth ConfigMap you can add different levels of access based on the role of the user.
First you MUST have the "system:node:{{EC2PrivateDNSName}}" user
apiVersion: v1
kind: ConfigMap
metadata:
name: aws-auth
namespace: kube-system
data:
mapRoles: |
- rolearn: <ARN of instance role (not instance profile)>
username: system:node:{{EC2PrivateDNSName}}
groups:
- system:bootstrappers
- system:nodes
This is required for Kubernetes to even work, giving the nodes the ability to join the cluster. The "ARN of instance role" is the role that includes the required policies AmazonEKSWorkerNodePolicy, AmazonEKS_CNI_Policy, AmazonEC2ContainerRegistryReadOnly etc.
Below that add your role
- rolearn: arn:aws:iam::[hidden]:role/ci_deployer
username: ci-deployer
groups:
- system:masters
The 'username' can actually be set to about anything. It appears to only be important if there are custom roles and bindings added to your EKS cluster.
Also, use the command 'aws sts get-caller-identity' to validate the environment/shell and the AWS credentials are properly configured. When correctly configured 'get-caller-identity' should return the same role ARN specified in aws-auth.
I deployed a squid proxy in each namespace cause I want to access the services from external via the squid proxy, thus I need to add the line below to the squid.conf so that I can access services just using service names:
append_domain .${namespace}.svc.cluster.local
Here is my problem:
I can get ${namespace} via metadata.namespace inside a pod, but how can I get the cluster domain ? Is it possible ?
I’ve tried this but it retruned an error when creating pod:
- name: POD_CLUSERDOMAIN
valueFrom:
fieldRef:
fieldPath: metadata.clusterName
Thanks for your help.
Alright, failed to get the current NAMESPACE inside a pod, but I find another way to reach the point -- retrieve the whole host domain from search domian in resolv.conf.
Here's the detail:
keep the Dockerfile unmodified
add a command item to deployment.yaml
image: squid:3.5.20
command: ["/bin/sh","-c"]
args: [ "echo append_domain .$(awk -v s=search '{if($1 == s)print $2}' /etc/resolv.conf) >> /etc/squid/squid.conf; /usr/sbin/squid -N" ]
This will add a line like append_domain .default.svc.cluster.local to the end of file /etc/squid/squid.conf then we can access the services from external via the squid proxy just using service name now.
The cluster domain is configured from the kubelet parameters, it must be the same through the whole cluster, so, you can't get it from pod's metadata, just use it as it is: svc.cluster.local