I am trying to validate that the pod is deleted.
- name: Verify whether the POD is deleted
command: "{{ local_kubectl }} get pod {{ pod_name}}"
but the task is failing as the command displays error as below
'Error from server (NotFound): pods ....'
But this is expected when i am checking for deletion of pod.
How to pass this task when it returns an error message?
You can just directly specify this using the k8s module
- name: Delete the POD
k8s:
api_version: v1
kind: Pod
namespace: "{{ k8s_namespace }}"
name: "{{ pod_name }}"
state: absent
Another path is to redefine "failure" to check for the expected result string.
- name: Verify whether the POD is deleted
command: "{{ local_kubectl }} get pod {{ pod_name}}"
register: verify
failed_when: "'NotFound' not in verify.stderr"
Try k8s_facts
- k8s_facts:
kind: Pod
name: "{{ pod_name}}"
register: result
- debug:
var: result
(not tested)
Related
I am using Kubernetes with Helm 3.8.0, with windows docker desktop configured on WSL2.
Sometime, after running: helm install, and retrieve a container, the container that is created behind sense, is an old container that created before (even after restarting the computer).
i.e: Now the yaml is declared with password: 12345, and database: test. before I tried to run the container yaml with password: 11111, and database: my_database.
Now when I do helm install mychart ./mychart --namespace test-chart --create-namespace for the current folder chart, the container is running with password: 11111 and database: my_datatbase, instead of the new parameters provided. There is no current yaml code with the old password, so I don't understand why the docker is run with the old one.
I did several actions, such as docker system prune, restarting Windows Docker Desktop, but still I get the old container, that cannot be seen, even in Windows Docker Desktop, I have checked the option in: Settings -> Kubernetes -> Show System Containers -> Show system containers.
After some investigations, I realized that that may be because of Kubernetes has it's own garbage collection handling of containers, and that is why I may refer to old container, even I didn't mean to.
In my case, I am creating a job template (I didn't put any line that reference this job in the _helpers.tpl file - I never changed that file, and I don't know whether that may cause a problem).
Here is my job template:
apiVersion: batch/v1
kind: Job
metadata:
name: {{ include "myChart.fullname" . }}-migration
labels:
name: {{ include "myChart.fullname" . }}-migration
annotations:
"helm.sh/hook": pre-install,pre-upgrade
"helm.sh/hook-weight": "-300"
"helm.sh/hook-delete-policy": before-hook-creation
spec:
parallelism: 1
completions: 1
backoffLimit: 1
template:
metadata:
labels:
app: {{ template "myChart.name" . }}
release: {{ .Release.Namespace }}
spec:
initContainers:
- name: wait-mysql
image: {{ .Values.mysql.image }}
imagePullPolicy: IfNotPresent
env:
- name: MYSQL_ROOT_PASSWORD
value: "12345"
- name: MYSQL_DATABASE
value: test
command:
- /bin/sh
- -c
- |
service mysql start &
until mysql -uroot -p12345 -e 'show databases'; do
echo `date +%H:%M:%S`' - Waiting for mysql...'
sleep 5
done
containers:
- name: migration
image: "{{ .Values.image.repository }}:{{ .Values.image.tag | default .Chart.AppVersion }}"
command: {{- toYaml .Values.image.entrypoint | nindent 12 }}
args: {{- toYaml .Values.image.cmd | nindent 12}}
restartPolicy: Never
In the job - there is a database, which is first created, and after that it has data that is populated with code.
Also, are the annotations (hooks) are necessary?
After running helm install myChart ./myChart --namespace my-namespace --create-namespace, I realized that I am using very old container, which I don't really need.
I didn't understand if I write the meta data, as the following example (in: Garbage Collection) really help, and what to put in uid, whether I don't know it, or don't have it.
metadata:
...
ownerReferences:
- apiVersion: extensions/v1beta1
controller: true
blockOwnerDeletion: true
kind: ReplicaSet
name: my-repset
uid: d9607e19-f88f-11e6-a518-42010a800195
Sometimes I really want to reference existing pod (or container) from several templates (use the same container, which is not stateless, such as database container - one template for the pod and the other for the job) - How can I do that, also?
Is there any command (in command line, or a kind of method) that clear all the cached in Garbage Collection, or not use Garbage Collection at all? (What are the main benefits for the GC of Kubernetes?)
I am trying to use the module community.kubernetes.k8s – Manage Kubernetes (K8s) objects with variables from the role (e.g. role/sampleRole/vars file).
I am failing when it comes to the integer point e.g.:
- name: sample
community.kubernetes.k8s:
state: present
definition:
apiVersion: apps/v1
kind: Deployment
metadata:
name: "{{ name }}"
namespace: "{{ namespace }}"
labels:
app: "{{ app }}"
spec:
replicas: 2
selector:
matchLabels:
app: "{{ app }}"
template:
metadata:
labels:
app: "{{ app }}"
spec:
containers:
- name: "{{ name }}"
image: "{{ image }}"
ports:
- containerPort: {{ containerPort }}
When I deploy with this format obviously it will fail at it can not parse the "reference" to the var.
Sample of error:
ERROR! We were unable to read either as JSON nor YAML, these are the errors we got from each:
JSON: Expecting value: line 1 column 1 (char 0)
Syntax Error while loading YAML.
found unacceptable key (unhashable type: 'AnsibleMapping')
The error appears to be in 'deploy.yml': line <some line>, column <some column>, but may
be elsewhere in the file depending on the exact syntax problem.
The offending line appears to be:
ports:
- containerPort: {{ containerPort }}
^ here
We could be wrong, but this one looks like it might be an issue with
missing quotes. Always quote template expression brackets when they
start a value. For instance:
with_items:
- {{ foo }}
Should be written as:
with_items:
- "{{ foo }}"
When I use quotes on the variable e.g. - containerPort: "{{ containerPort }}" then I get the following error (part of it):
v1.Deployment.Spec: v1.DeploymentSpec.Template: v1.PodTemplateSpec.Spec: v1.PodSpec.Containers: []v1.Container: v1.Container.Ports: []v1.ContainerPort: v1.ContainerPort.ContainerPort: readUint32: unexpected character: \\\\ufffd, error found in #10 byte of ...|nerPort\\\\\":\\\\\"80\\\\\"}]}],\\\\\"d|..., bigger context ...|\\\\\",\\\\\"name\\\\\":\\\\\"samplegreen\\\\\",\\\\\"ports\\\\\":[{\\\\\"containerPort\\\\\":\\\\\"80\\\\\"}]}],\\\\\"dnsPolicy\\\\\":\\\\\"ClusterFirst\\\\\",\\\\\"restartPolicy\\\\\"|...\",\"field\":\"patch\"}]},\"code\":422}\\n'", "reason": "Unprocessable Entity", "status": 422}
I tried to cast the string to int by using - containerPort: "{{ containerPort | int }}" but it did not worked. The problem seems to be coming from the quotes, independently how I define the var in my var file e.g. containerPort: 80 or containerPort: "80".
I found a similar question on the forum Ansible, k8s and variables but the user seems not to have the same problems that I am having.
I am running with the latest version of the module:
$ python3 -m pip show openshift
Name: openshift
Version: 0.11.2
Summary: OpenShift python client
Home-page: https://github.com/openshift/openshift-restclient-python
Author: OpenShift
Author-email: UNKNOWN
License: Apache License Version 2.0
Location: /usr/local/lib/python3.8/dist-packages
Requires: ruamel.yaml, python-string-utils, jinja2, six, kubernetes
Is there any workaround this problem or is it a bug?
Update (08-01-2020): The problem is fixed on version 0.17.0.
$ python3 -m pip show k8s
Name: k8s
Version: 0.17.0
Summary: Python client library for the Kubernetes API
Home-page: https://github.com/fiaas/k8s
Author: FiaaS developers
Author-email: fiaas#googlegroups.com
License: Apache License
Location: /usr/local/lib/python3.8/dist-packages
Requires: requests, pyrfc3339, six, cachetools
You could try the following as a workaround; in this example, we're creating a text template, and then using the from_yaml filter to transform this into our desired data structure:
- name: sample
community.kubernetes.k8s:
state: present
definition:
apiVersion: apps/v1
kind: Deployment
metadata:
name: "{{ name }}"
namespace: "{{ namespace }}"
labels:
app: "{{ app }}"
spec: "{{ spec|from_yaml }}"
vars:
spec: |
replicas: 2
selector:
matchLabels:
app: "{{ app }}"
template:
metadata:
labels:
app: "{{ app }}"
spec:
containers:
- name: "{{ name }}"
image: "{{ image }}"
ports:
- containerPort: {{ containerPort }}
The solution provided by larsks works perfectly. Although I got another problem on my case where I use templates with a bit more complex cases (e.g. loops etc) where I found my self having the same problem.
The only solution that I had before was to use ansible.builtin.template – Template a file out to a remote server and simply ssh the some_file.yml.j2 to one of my Master nodes and deploy through ansible.builtin.shell – Execute shell commands on targets (e.g. kubectl apply -f some_file.yml).
Thanks to community.kubernetes.k8s – Manage Kubernetes (K8s) objects I am able to do all this work with a single task e.g. (example taken from documentation):
- name: Read definition template file from the Ansible controller file system
community.kubernetes.k8s:
state: present
template: '/testing/deployment.j2'
The only requirement that the user needs to have in advance is to have the kubeconfig file placed in the default location (~/.kube/config) or use the kubeconfig flag to point to the location of the file.
As a last step I use it delegate_to to localhost command e.g.
- name: Read definition template file from the Ansible controller file system
community.kubernetes.k8s:
state: present
template: '/testing/deployment.j2'
delegate_to: localhost
The way that this task works is that the user ssh to himself and run kubectl apply -f some_file.yml.j2 towards the LB or Master node API and the API applies the request (if the user has the permissions).
I'm trying to write simple ansible playbook that would be able to execute some arbitrary command against the pod (container) running in kubernetes cluster.
I would like to utilise kubectl connection plugin: https://docs.ansible.com/ansible/latest/plugins/connection/kubectl.html but having struggle to figure out how to actually do that.
Couple of questions:
Do I need to first have inventory for k8s defined? Something like: https://docs.ansible.com/ansible/latest/plugins/inventory/k8s.html. My understanding is that I would define kube config via inventory which would be used by the kubectl plugin to actually connect to the pods to perform specific action.
If yes, is there any example of arbitrary command executed via kubectl plugin (but not via shell plugin that invokes kubectl on some remote machine - this is not what I'm looking for)
I'm assuming that, during the ansible-playbook invocation, I would point to k8s inventory.
Thanks.
I would like to utilise kubectl connection plugin: https://docs.ansible.com/ansible/latest/plugins/connection/kubectl.html but having struggle to figure out how to actually do that.
The fine manual describes how one uses connection plugins, and while it is possible to use in in tasks, that is unlikely to make any sense unless your inventory started with Pods.
The way I have seen that connection used is to start by identifying the Pods against which you might want to take action, and then run a playbook against a unique group for that purpose:
- hosts: all
tasks:
- set_fact:
# this is *just an example for brevity*
# in reality you would use `k8s:` or `kubectl get -o name pods -l my-selector=my-value` to get the pod names
pod_names:
- nginx-12345
- nginx-3456
- add_host:
name: '{{ item }}'
groups:
- my-pods
with_items: '{{ pod_names }}'
- hosts: my-pods
connection: kubectl
tasks:
# and now you are off to the races
- command: ps -ef
# watch out if the Pod doesn't have a working python installed
# as you will have to use raw: instead
# (and, of course, disable "gather_facts: no")
- raw: ps -ef
First install k8s collections
ansible-galaxy collection install community.kubernetes
and here is play-book, it will sort all pods and run a command in every pod
---
-
hosts: localhost
vars_files:
- vars/main.yaml
collections:
- community.kubernetes
tasks:
-
name: Get the pods in the specific namespace
k8s_info:
kubeconfig: '{{ k8s_kubeconfig }}'
kind: Pod
namespace: test
register: pod_list
-
name: Print pod names
debug:
msg: "pod_list: {{ pod_list | json_query('resources[*].status.podIP') }} "
- set_fact:
pod_names: "{{pod_list|json_query('resources[*].metadata.name')}}"
-
k8s_exec:
kubeconfig: '{{ k8s_kubeconfig }}'
namespace: "{{ namespace }}"
pod: "{{ item.metadata.name }}"
command: apt update
with_items: "{{ pod_list.resources }}"
register: exec
loop_control:
label: "{{ item.metadata.name }}"
Maybe you can use like this...
- shell: |
kubectl exec -i -n {{ namespace }} {{ pod_name }} -- bash -c 'clickhouse-client --query "INSERT INTO customer FORMAT CSV"
--user=test --password=test < /mnt/azure/azure/test/test.tbl'
As per the latest documentation you can use the following k8s modules
The following are some of the examples
- name: Create a k8s namespace
kubernetes.core.k8s:
name: testing
api_version: v1
kind: Namespace
state: present
- name: Create a Service object from an inline definition
kubernetes.core.k8s:
state: present
definition:
apiVersion: v1
kind: Service
metadata:
name: web
namespace: testing
labels:
app: galaxy
service: web
spec:
selector:
app: galaxy
service: web
ports:
- protocol: TCP
targetPort: 8000
name: port-8000-tcp
port: 8000
- name: Remove an existing Service object
kubernetes.core.k8s:
state: absent
api_version: v1
kind: Service
namespace: testing
name: web
I launch the GCP cluster with no problem but I do not know how to get k8s ansible module to work. I would prefer to get the api_key to authenticate into k8s module.
My playbook is the following.
- name: Hello k8s
hosts: all
tasks:
- name: Create a cluster
register: cluster
gcp_container_cluster:
name: thecluster
initial_node_count: 1
master_auth:
username: admin
password: TheRandomPassword
node_config:
machine_type: g1-small
disk_size_gb: 10
oauth_scopes:
- "https://www.googleapis.com/auth/compute"
- "https://www.googleapis.com/auth/devstorage.read_only"
- "https://www.googleapis.com/auth/logging.write"
- "https://www.googleapis.com/auth/monitoring"
zone: europe-west3-c
project: second-network-255214
auth_kind: serviceaccount
service_account_file: "{{ lookup('env', 'GOOGLE_CREDENTIALS') }}"
state: present
- name: Show results
debug: var=cluster
- name: Create temporary file for CA
tempfile:
state: file
suffix: build
register: ca_crt
- name: Save content to file
copy:
content: "{{ cluster.masterAuth.clusterCaCertificate |b64decode }}"
dest: "{{ ca_crt.path }}"
- name: Create a k8s namespace
k8s:
host: "https://{{ cluster.endpoint }}"
ca_cert: "{{ ca_crt.path }}"
api_key: "{{ cluster.HOW_I_GET_THE_API_KEY}}" <<<-- Here is what I want!!!
name: testing
api_version: v1
kind: Namespace
state: present
Any idea?
I founded a workaround that is to call gcloud directly:
- name: Get JWT
command: gcloud auth application-default print-access-token
register: api_key
Obviously, I needed to:
Install GCloud
Redefine the envvar with the auth.json to GOOGLE_APPLICATION_CREDENTIALS.
The task calls gcloud directly to obtain the token, so no need to generate the token. I will try to add to add this feature as a module into ansible for better interoperability with kubernetes.
Once obtained it is possible to call k8s module like this:
- name: Create ClusterRoleBinding
k8s:
state: present
host: "https://{{ cluster.endpoint }}"
ca_cert: "{{ ca_crt.path }}"
api_version: rbac.authorization.k8s.io/v1
api_key: "{{ api_key.stdout }}"
definition:
kind: ClusterRoleBinding
metadata:
name: kube-system_default_cluster-admin
subjects:
- kind: ServiceAccount
name: default # Name is case sensitive
namespace: kube-system
roleRef:
kind: ClusterRole
name: cluster-admin
apiGroup: rbac.authorization.k8s.io
According to the fine manual, masterAuth contains two other fields, clientCertificate and clientKey that correspond to the client_cert: and client_key: parameters, respectively. From that point, you can authenticate to your cluster's endpoint as cluster-admin using the very, very strong credentials of the private key, and from that point use the same k8s: task to provision yourself a cluster-admin ServiceAccount token if you wish to do that.
You can also apparently use masterAuth.username and masterAuth.password in the username: and password: parameters of k8s:, too, which should be just as safe since the credentials travel over HTTPS, but you seemed like you were more interested in a higher entropy authentication solution.
I'm using a script to run helm command which upgrades my k8s deployment.
Before I've used kubectl to directly deploy, as I've move to helm and started using charts, I see an error after deploying on the k8s pods:
MountVolume.SetUp failed for volume "secret" : invalid character '\r' in string literal
My script looks similar to:
value1="foo"
value2="bar"
helm upgrade deploymentName --debug --install --atomic --recreate-pods --reset-values --force --timeout 900 pathToChartDir --set value1 --set value2
The deployment.yaml is as following:
apiVersion: apps/v1
kind: Deployment
metadata:
name: deploymentName
spec:
selector:
matchLabels:
run: deploymentName
replicas: 2
template:
metadata:
labels:
run: deploymentName
app: appName
spec:
containers:
- name: deploymentName
image: {{ .Values.image.acr.registry }}/{{ .Values.image.name }}:{{ .Values.image.tag }}
volumeMounts:
- name: secret
mountPath: /secrets
readOnly: true
ports:
- containerPort: 1234
env:
- name: DOTENV_CONFIG_PATH
value: "/secrets/env"
volumes:
- name: secret
flexVolume:
driver: "azure/kv"
secretRef:
name: "kvcreds"
options:
usepodidentity: "false"
tenantid: {{ .Values.tenantid }}
subscriptionid: {{ .Values.subsid }}
resourcegroup: {{ .Values.rg }}
keyvaultname: {{ .Values.kvname }}
keyvaultobjecttype: secret
keyvaultobjectname: {{ .Values.objectname }}
As can be seen, the error relates to the secret volume and its values.
I've triple checked there is no line-break or anything like that in the values.
I've run helm lint - no errors found.
I've run helm template - nothing strange or missing in output.
Update:
I've copied the output of helm template and put in a deploy.yaml file.
Then used kubectl apply -f deploy.yaml to manually deploy the service, and... it works.
That makes me think it's actually some kind of a bug in helm? make sense?
Update 2:
I've also tried replacing the azure/kv volume with emptyDir volume and I was able to deploy using helm. It looks like a specific issue of helm with azure/kv volume?
Any ideas for a workaround?
A completely correct answer requires that I say the actual details of your \r problem might be different from mine.
I found the issue in my case by looking in the kv log of the AKS node (/var/log/kv-driver.log). In my case, the error was:
Original Error: autorest/azure: Service returned an error. Status=403 Code="Forbidden" Message="Access denied. Caller was not found on any access policy.\r\n
You can learn to SSH into the node on this page:
https://learn.microsoft.com/en-us/azure/aks/ssh
If you want to follow the solution, I opened an issue:
https://github.com/Azure/kubernetes-keyvault-flexvol/issues/121