Ansible: Failed to reload sysctl: sysctl: cannot stat /proc/sys/net/bridge/bridge-nf-call-iptables: No such file or directory - kubernetes

Question

I'm setting up kubernetes cluster with ansible. I get the following error when trying to enable kernel IP routing:
Failed to reload sysctl: sysctl: cannot stat /proc/sys/net/bridge/bridge-nf-call-iptables: No such file or directory
Is this a bug in ansible or is there something wrong with my playbook?
---
# file: site.yml
# description: Asentaa ja kaynnistaa kubernetes-klusterin riippuvuuksineen
#
# resources:
# - https://kubernetes.io/docs/setup/independent/install-kubeadm/
# - http://michele.sciabarra.com/2018/02/12/devops/Kubernetes-with-KubeAdm-Ansible-Vagrant/
# - https://docs.ansible.com/ansible/latest/modules/
# - https://github.com/geerlingguy/ansible-role-kubernetes/blob/master/tasks/setup-RedHat.yml
# - https://docs.docker.com/install/linux/docker-ce/centos/
#
# author: Tuomas Toivonen
# date: 30.12.2018
- name: Asenna docker ja kubernetes
hosts: k8s-machines
become: true
become_method: sudo
roles:
- common
vars:
ip_modules:
- ip_vs
- ip_vs_rr
- ip_vs_wrr
- ip_vs_sh
- nf_conntrack_ipv4
tasks:
- name: Poista swapfile
tags:
- os-settings
mount:
name: swap
fstype: swap
state: absent
- name: Disabloi swap-muisti
tags:
- os-settings
command: swapoff -a
when: ansible_swaptotal_mb > 0
- name: Konfiguroi verkkoasetukset
tags:
- os-settings
command: modprobe {{ item }}
loop: "{{ ip_modules }}"
- name: Modprobe
tags:
- os-settings
lineinfile:
path: "/etc/modules"
line: "{{ item }}"
create: yes
state: present
loop: "{{ ip_modules }}"
- name: Iptables
tags:
- os-settings
sysctl:
name: "{{ item }}"
value: 1
sysctl_set: yes
state: present
reload: yes
loop:
- 'net.bridge.bridge-nf-call-iptables'
- 'net.bridge.bridge-nf-call-ip6tables'
- name: Salli IP-reititys
sysctl:
name: net.ipv4.ip_forward
value: 1
state: present
reload: yes
sysctl_set: yes
- name: Lisaa docker-ce -repositorio
tags:
- repos
yum_repository:
name: docker-ce
description: docker-ce
baseurl: https://download.docker.com/linux/centos/7/x86_64/stable/
enabled: true
gpgcheck: true
repo_gpgcheck: true
gpgkey:
- https://download.docker.com/linux/centos/gpg
state: present
- name: Lisaa kubernetes -repositorio
tags:
- repos
yum_repository:
name: kubernetes
description: kubernetes
baseurl: https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64
enabled: true
gpgcheck: true
repo_gpgcheck: true
gpgkey:
- https://packages.cloud.google.com/yum/doc/yum-key.gpg
- https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
state: present
- name: Asenna docker-ce -paketti
tags:
- packages
yum:
name: docker-ce
state: present
- name: Asenna NTP -paketti
tags:
- packages
yum:
name: ntp
state: present
- name: Asenna kubernetes -paketit
tags:
- packages
yum:
name: "{{ item }}"
state: present
loop:
- kubelet
- kubeadm
- kubectl
- name: Kaynnista palvelut
tags:
- services
service: name={{ item }} state=started enabled=yes
loop:
- docker
- ntpd
- kubelet
- name: Alusta kubernetes masterit
become: true
become_method: sudo
hosts: k8s-masters
tags:
- cluster
tasks:
- name: kubeadm reset
shell: "kubeadm reset -f"
- name: kubeadm init
shell: "kubeadm init --token-ttl=0 --apiserver-advertise-address=10.0.0.101 --pod-network-cidr=20.0.0.0/8" # TODO
register: kubeadm_out
- set_fact:
kubeadm_join: "{{ kubeadm_out.stdout_lines[-1] }}"
when: kubeadm_out.stdout.find("kubeadm join") != -1
- debug:
var: kubeadm_join
- name: Aseta ymparistomuuttujat
shell: >
cp /etc/kubernetes/admin.conf /home/vagrant/ &&
chown vagrant:vagrant /home/vagrant/admin.conf &&
export KUBECONFIG=/home/vagrant/admin.conf &&
echo export KUBECONFIG=$KUBECONFIG >> /home/vagrant/.bashrc
- name: Konfiguroi CNI-verkko
become: true
become_method: sudo
hosts: k8s-masters
tags:
- cluster-network
tasks:
- sysctl: name=net.bridge.bridge-nf-call-iptables value=1 state=present reload=yes sysctl_set=yes
- sysctl: name=net.bridge.bridge-nf-call-ip6tables value=1 state=present reload=yes sysctl_set=yes
- name: Asenna Flannel-plugin
shell: >
export KUBECONFIG=/home/vagrant/admin.conf ;
kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
- shell: sleep 10
- name: Alusta kubernetes workerit
become: true
become_method: sudo
hosts: k8s-workers
tags:
- cluster
tasks:
- name: kubeadm reset
shell: "kubeadm reset -f"
- name: kubeadm join
tags:
- cluster
shell: "{{ hostvars['k8s-n1'].kubeadm_join }}" # TODO
Here is the full ansible log
ansible-controller: Running ansible-playbook...
cd /vagrant && PYTHONUNBUFFERED=1 ANSIBLE_NOCOLOR=true ANSIBLE_CONFIG='ansible/ansible.cfg' ansible-playbook --limit="all" --inventory-file=ansible/hosts -v ansible/site.yml
Using /vagrant/ansible/ansible.cfg as config file
/vagrant/ansible/hosts did not meet host_list requirements, check plugin documentation if this is unexpected
/vagrant/ansible/hosts did not meet script requirements, check plugin documentation if this is unexpected
PLAY [Asenna docker ja kubernetes] *********************************************
TASK [Gathering Facts] *********************************************************
ok: [k8s-n1]
ok: [k8s-n3]
ok: [k8s-n2]
TASK [common : Testaa] *********************************************************
changed: [k8s-n3] => {"changed": true, "checksum": "6920e1826e439962050ec0ab4221719b3a045f04", "dest": "/template.test", "gid": 0, "group": "root", "md5sum": "a4f61c365318c3e23d466914fbd02687", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:etc_runtime_t:s0", "size": 14, "src": "/home/vagrant/.ansible/tmp/ansible-tmp-1546760756.54-124542112178019/source", "state": "file", "uid": 0}
changed: [k8s-n2] => {"changed": true, "checksum": "6920e1826e439962050ec0ab4221719b3a045f04", "dest": "/template.test", "gid": 0, "group": "root", "md5sum": "a4f61c365318c3e23d466914fbd02687", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:etc_runtime_t:s0", "size": 14, "src": "/home/vagrant/.ansible/tmp/ansible-tmp-1546760756.51-240329169302936/source", "state": "file", "uid": 0}
changed: [k8s-n1] => {"changed": true, "checksum": "6920e1826e439962050ec0ab4221719b3a045f04", "dest": "/template.test", "gid": 0, "group": "root", "md5sum": "a4f61c365318c3e23d466914fbd02687", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:etc_runtime_t:s0", "size": 14, "src": "/home/vagrant/.ansible/tmp/ansible-tmp-1546760756.57-121244542660821/source", "state": "file", "uid": 0}
TASK [common : Asenna telnet] **************************************************
changed: [k8s-n2] => {"changed": true, "msg": "", "rc": 0, "results": ["Loaded plugins: fastestmirror\nLoading mirror speeds from cached hostfile\n * base: ftp.funet.fi\n * extras: ftp.funet.fi\n * updates: ftp.funet.fi\nResolving Dependencies\n--> Running transaction check\n---> Package telnet.x86_64 1:0.17-64.el7 will be installed\n--> Finished Dependency Resolution\n\nDependencies Resolved\n\n================================================================================\n Package Arch Version Repository Size\n================================================================================\nInstalling:\n telnet x86_64 1:0.17-64.el7 base 64 k\n\nTransaction Summary\n================================================================================\nInstall 1 Package\n\nTotal download size: 64 k\nInstalled size: 113 k\nDownloading packages:\nRunning transaction check\nRunning transaction test\nTransaction test succeeded\nRunning transaction\n Installing : 1:telnet-0.17-64.el7.x86_64 1/1 \n Verifying : 1:telnet-0.17-64.el7.x86_64 1/1 \n\nInstalled:\n telnet.x86_64 1:0.17-64.el7 \n\nComplete!\n"]}
changed: [k8s-n1] => {"changed": true, "msg": "", "rc": 0, "results": ["Loaded plugins: fastestmirror\nLoading mirror speeds from cached hostfile\n * base: centos.mirror.gnu.fi\n * extras: centos.mirror.gnu.fi\n * updates: centos.mirror.gnu.fi\nResolving Dependencies\n--> Running transaction check\n---> Package telnet.x86_64 1:0.17-64.el7 will be installed\n--> Finished Dependency Resolution\n\nDependencies Resolved\n\n================================================================================\n Package Arch Version Repository Size\n================================================================================\nInstalling:\n telnet x86_64 1:0.17-64.el7 base 64 k\n\nTransaction Summary\n================================================================================\nInstall 1 Package\n\nTotal download size: 64 k\nInstalled size: 113 k\nDownloading packages:\nRunning transaction check\nRunning transaction test\nTransaction test succeeded\nRunning transaction\n Installing : 1:telnet-0.17-64.el7.x86_64 1/1 \n Verifying : 1:telnet-0.17-64.el7.x86_64 1/1 \n\nInstalled:\n telnet.x86_64 1:0.17-64.el7 \n\nComplete!\n"]}
changed: [k8s-n3] => {"changed": true, "msg": "", "rc": 0, "results": ["Loaded plugins: fastestmirror\nLoading mirror speeds from cached hostfile\n * base: ftp.funet.fi\n * extras: ftp.funet.fi\n * updates: ftp.funet.fi\nResolving Dependencies\n--> Running transaction check\n---> Package telnet.x86_64 1:0.17-64.el7 will be installed\n--> Finished Dependency Resolution\n\nDependencies Resolved\n\n================================================================================\n Package Arch Version Repository Size\n================================================================================\nInstalling:\n telnet x86_64 1:0.17-64.el7 base 64 k\n\nTransaction Summary\n================================================================================\nInstall 1 Package\n\nTotal download size: 64 k\nInstalled size: 113 k\nDownloading packages:\nRunning transaction check\nRunning transaction test\nTransaction test succeeded\nRunning transaction\n Installing : 1:telnet-0.17-64.el7.x86_64 1/1 \n Verifying : 1:telnet-0.17-64.el7.x86_64 1/1 \n\nInstalled:\n telnet.x86_64 1:0.17-64.el7 \n\nComplete!\n"]}
TASK [Poista swapfile] *********************************************************
ok: [k8s-n1] => {"changed": false, "dump": "0", "fstab": "/etc/fstab", "fstype": "swap", "name": "swap", "opts": "defaults", "passno": "0"}
ok: [k8s-n2] => {"changed": false, "dump": "0", "fstab": "/etc/fstab", "fstype": "swap", "name": "swap", "opts": "defaults", "passno": "0"}
ok: [k8s-n3] => {"changed": false, "dump": "0", "fstab": "/etc/fstab", "fstype": "swap", "name": "swap", "opts": "defaults", "passno": "0"}
TASK [Disabloi swap-muisti] ****************************************************
changed: [k8s-n3] => {"changed": true, "cmd": ["swapoff", "-a"], "delta": "0:00:00.009581", "end": "2019-01-06 07:46:08.414842", "rc": 0, "start": "2019-01-06 07:46:08.405261", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []}
changed: [k8s-n1] => {"changed": true, "cmd": ["swapoff", "-a"], "delta": "0:00:00.119638", "end": "2019-01-06 07:46:08.484265", "rc": 0, "start": "2019-01-06 07:46:08.364627", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []}
changed: [k8s-n2] => {"changed": true, "cmd": ["swapoff", "-a"], "delta": "0:00:00.133924", "end": "2019-01-06 07:46:08.519646", "rc": 0, "start": "2019-01-06 07:46:08.385722", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []}
TASK [Konfiguroi verkkoasetukset] **********************************************
changed: [k8s-n2] => (item=ip_vs) => {"changed": true, "cmd": ["modprobe", "ip_vs"], "delta": "0:00:00.036881", "end": "2019-01-06 07:46:10.606797", "item": "ip_vs", "rc": 0, "start": "2019-01-06 07:46:10.569916", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []}
changed: [k8s-n3] => (item=ip_vs) => {"changed": true, "cmd": ["modprobe", "ip_vs"], "delta": "0:00:00.036141", "end": "2019-01-06 07:46:10.815043", "item": "ip_vs", "rc": 0, "start": "2019-01-06 07:46:10.778902", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []}
changed: [k8s-n1] => (item=ip_vs) => {"changed": true, "cmd": ["modprobe", "ip_vs"], "delta": "0:00:00.035888", "end": "2019-01-06 07:46:10.768267", "item": "ip_vs", "rc": 0, "start": "2019-01-06 07:46:10.732379", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []}
changed: [k8s-n2] => (item=ip_vs_rr) => {"changed": true, "cmd": ["modprobe", "ip_vs_rr"], "delta": "0:00:00.005942", "end": "2019-01-06 07:46:12.763004", "item": "ip_vs_rr", "rc": 0, "start": "2019-01-06 07:46:12.757062", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []}
changed: [k8s-n1] => (item=ip_vs_rr) => {"changed": true, "cmd": ["modprobe", "ip_vs_rr"], "delta": "0:00:00.006084", "end": "2019-01-06 07:46:12.896763", "item": "ip_vs_rr", "rc": 0, "start": "2019-01-06 07:46:12.890679", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []}
changed: [k8s-n3] => (item=ip_vs_rr) => {"changed": true, "cmd": ["modprobe", "ip_vs_rr"], "delta": "0:00:00.006325", "end": "2019-01-06 07:46:12.899750", "item": "ip_vs_rr", "rc": 0, "start": "2019-01-06 07:46:12.893425", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []}
changed: [k8s-n2] => (item=ip_vs_wrr) => {"changed": true, "cmd": ["modprobe", "ip_vs_wrr"], "delta": "0:00:00.006195", "end": "2019-01-06 07:46:14.795507", "item": "ip_vs_wrr", "rc": 0, "start": "2019-01-06 07:46:14.789312", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []}
changed: [k8s-n1] => (item=ip_vs_wrr) => {"changed": true, "cmd": ["modprobe", "ip_vs_wrr"], "delta": "0:00:00.007328", "end": "2019-01-06 07:46:14.819072", "item": "ip_vs_wrr", "rc": 0, "start": "2019-01-06 07:46:14.811744", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []}
changed: [k8s-n3] => (item=ip_vs_wrr) => {"changed": true, "cmd": ["modprobe", "ip_vs_wrr"], "delta": "0:00:00.007251", "end": "2019-01-06 07:46:14.863192", "item": "ip_vs_wrr", "rc": 0, "start": "2019-01-06 07:46:14.855941", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []}
changed: [k8s-n3] => (item=ip_vs_sh) => {"changed": true, "cmd": ["modprobe", "ip_vs_sh"], "delta": "0:00:00.007590", "end": "2019-01-06 07:46:16.815226", "item": "ip_vs_sh", "rc": 0, "start": "2019-01-06 07:46:16.807636", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []}
changed: [k8s-n1] => (item=ip_vs_sh) => {"changed": true, "cmd": ["modprobe", "ip_vs_sh"], "delta": "0:00:00.006380", "end": "2019-01-06 07:46:16.941470", "item": "ip_vs_sh", "rc": 0, "start": "2019-01-06 07:46:16.935090", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []}
changed: [k8s-n2] => (item=ip_vs_sh) => {"changed": true, "cmd": ["modprobe", "ip_vs_sh"], "delta": "0:00:00.006619", "end": "2019-01-06 07:46:16.808432", "item": "ip_vs_sh", "rc": 0, "start": "2019-01-06 07:46:16.801813", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []}
changed: [k8s-n3] => (item=nf_conntrack_ipv4) => {"changed": true, "cmd": ["modprobe", "nf_conntrack_ipv4"], "delta": "0:00:00.007618", "end": "2019-01-06 07:46:18.825593", "item": "nf_conntrack_ipv4", "rc": 0, "start": "2019-01-06 07:46:18.817975", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []}
changed: [k8s-n1] => (item=nf_conntrack_ipv4) => {"changed": true, "cmd": ["modprobe", "nf_conntrack_ipv4"], "delta": "0:00:00.008181", "end": "2019-01-06 07:46:18.910050", "item": "nf_conntrack_ipv4", "rc": 0, "start": "2019-01-06 07:46:18.901869", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []}
changed: [k8s-n2] => (item=nf_conntrack_ipv4) => {"changed": true, "cmd": ["modprobe", "nf_conntrack_ipv4"], "delta": "0:00:00.007427", "end": "2019-01-06 07:46:18.962850", "item": "nf_conntrack_ipv4", "rc": 0, "start": "2019-01-06 07:46:18.955423", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []}
TASK [Modprobe] ****************************************************************
changed: [k8s-n2] => (item=ip_vs) => {"backup": "", "changed": true, "item": "ip_vs", "msg": "line added"}
changed: [k8s-n1] => (item=ip_vs) => {"backup": "", "changed": true, "item": "ip_vs", "msg": "line added"}
changed: [k8s-n3] => (item=ip_vs) => {"backup": "", "changed": true, "item": "ip_vs", "msg": "line added"}
changed: [k8s-n2] => (item=ip_vs_rr) => {"backup": "", "changed": true, "item": "ip_vs_rr", "msg": "line added"}
changed: [k8s-n1] => (item=ip_vs_rr) => {"backup": "", "changed": true, "item": "ip_vs_rr", "msg": "line added"}
changed: [k8s-n3] => (item=ip_vs_rr) => {"backup": "", "changed": true, "item": "ip_vs_rr", "msg": "line added"}
changed: [k8s-n2] => (item=ip_vs_wrr) => {"backup": "", "changed": true, "item": "ip_vs_wrr", "msg": "line added"}
changed: [k8s-n1] => (item=ip_vs_wrr) => {"backup": "", "changed": true, "item": "ip_vs_wrr", "msg": "line added"}
changed: [k8s-n3] => (item=ip_vs_wrr) => {"backup": "", "changed": true, "item": "ip_vs_wrr", "msg": "line added"}
changed: [k8s-n2] => (item=ip_vs_sh) => {"backup": "", "changed": true, "item": "ip_vs_sh", "msg": "line added"}
changed: [k8s-n1] => (item=ip_vs_sh) => {"backup": "", "changed": true, "item": "ip_vs_sh", "msg": "line added"}
changed: [k8s-n3] => (item=ip_vs_sh) => {"backup": "", "changed": true, "item": "ip_vs_sh", "msg": "line added"}
changed: [k8s-n2] => (item=nf_conntrack_ipv4) => {"backup": "", "changed": true, "item": "nf_conntrack_ipv4", "msg": "line added"}
changed: [k8s-n1] => (item=nf_conntrack_ipv4) => {"backup": "", "changed": true, "item": "nf_conntrack_ipv4", "msg": "line added"}
changed: [k8s-n3] => (item=nf_conntrack_ipv4) => {"backup": "", "changed": true, "item": "nf_conntrack_ipv4", "msg": "line added"}
TASK [Iptables] ****************************************************************
failed: [k8s-n3] (item=net.bridge.bridge-nf-call-iptables) => {"changed": false, "item": "net.bridge.bridge-nf-call-iptables", "msg": "Failed to reload sysctl: sysctl: cannot stat /proc/sys/net/bridge/bridge-nf-call-iptables: No such file or directory\n"}
failed: [k8s-n1] (item=net.bridge.bridge-nf-call-iptables) => {"changed": false, "item": "net.bridge.bridge-nf-call-iptables", "msg": "Failed to reload sysctl: sysctl: cannot stat /proc/sys/net/bridge/bridge-nf-call-iptables: No such file or directory\n"}
failed: [k8s-n2] (item=net.bridge.bridge-nf-call-iptables) => {"changed": false, "item": "net.bridge.bridge-nf-call-iptables", "msg": "Failed to reload sysctl: sysctl: cannot stat /proc/sys/net/bridge/bridge-nf-call-iptables: No such file or directory\n"}
failed: [k8s-n3] (item=net.bridge.bridge-nf-call-ip6tables) => {"changed": false, "item": "net.bridge.bridge-nf-call-ip6tables", "msg": "Failed to reload sysctl: sysctl: cannot stat /proc/sys/net/bridge/bridge-nf-call-iptables: No such file or directory\nsysctl: cannot stat /proc/sys/net/bridge/bridge-nf-call-ip6tables: No such file or directory\n"}
failed: [k8s-n2] (item=net.bridge.bridge-nf-call-ip6tables) => {"changed": false, "item": "net.bridge.bridge-nf-call-ip6tables", "msg": "Failed to reload sysctl: sysctl: cannot stat /proc/sys/net/bridge/bridge-nf-call-iptables: No such file or directory\nsysctl: cannot stat /proc/sys/net/bridge/bridge-nf-call-ip6tables: No such file or directory\n"}
failed: [k8s-n1] (item=net.bridge.bridge-nf-call-ip6tables) => {"changed": false, "item": "net.bridge.bridge-nf-call-ip6tables", "msg": "Failed to reload sysctl: sysctl: cannot stat /proc/sys/net/bridge/bridge-nf-call-iptables: No such file or directory\nsysctl: cannot stat /proc/sys/net/bridge/bridge-nf-call-ip6tables: No such file or directory\n"}
to retry, use: --limit #/vagrant/ansible/site.retry
PLAY RECAP *********************************************************************
k8s-n1 : ok=7 changed=5 unreachable=0 failed=1
k8s-n2 : ok=7 changed=5 unreachable=0 failed=1
k8s-n3 : ok=7 changed=5 unreachable=0 failed=1
Ansible failed to complete successfully. Any error output should be
visible above. Please fix these errors and try again.

Answer 1

In the playbook, add the following task to load the br_netfilter module:
- name: Ensure br_netfilter is enabled.
modprobe:
name: br_netfilter
state: present

Answer 2

Loading br_netfilter kernel module fixed the problem. I simply appended it to the ip_modules list in playbook vars declaration. I'm using Centos 7.