Having trouble configuring static IPs to PODs attached with MACVLAN interface - kubernetes

Question

Here is the scenario.
There is a deployment set through which 2 PODs are created. I am attaching a MACVLAN interface to these PODs for external communication.
Macvlan definition
apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
name: test-macvlandef01
spec:
config: '{
"cniVersion": "0.3.0",
"name": "test-macvlandef01",
"type": "macvlan",
"master": "eth0",
"mode": "bridge",
"ipam": {
"type": "whereabouts",
"datastore": "kubernetes",
"kubernetes": { "kubeconfig": "/etc/cni/net.d/whereabouts.d/whereabouts.kubeconfig" },
"range": "192.168.0.0/24",
"range_start": "192.168.0.44",
"range_end": "192.168.0.45"
}
}'
Deployment Set
apiVersion: apps/v1
kind: Deployment
metadata:
name: centos-test
spec:
replicas: 2
selector:
matchLabels:
app: centos
template:
metadata:
labels:
app: centos
annotations:
k8s.v1.cni.cncf.io/networks: "test-macvlandef01"
spec:
nodeSelector:
test: "true"
containers:
- name: centos
image: centos
imagePullPolicy: IfNotPresent
command: ["bin/bash", "-c", "sleep 100000" ]
ports:
- containerPort: 80
Result. Both PODs have IPs from the allocated pool.
[master1 ~]# kubectl exec -it centos-test-64f8fbf47f-wrjr7 ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
2: tunl0#NONE: <NOARP> mtu 1480 qdisc noop state DOWN group default qlen 1000
link/ipip 0.0.0.0 brd 0.0.0.0
4: eth0#if61: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1440 qdisc noqueue state UP group default
link/ether 72:ef:ca:2c:31:3e brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet 10.20.14.176/32 scope global eth0
valid_lft forever preferred_lft forever
5: net1#if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UP group default
link/ether 52:2f:bd:f9:03:09 brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet 192.168.0.44/24 brd 192.168.0.255 scope global net1
valid_lft forever preferred_lft forever
[master1 ~]# kubectl exec -it centos-test-64f8fbf47f-vtkst ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
2: tunl0#NONE: <NOARP> mtu 1480 qdisc noop state DOWN group default qlen 1000
link/ipip 0.0.0.0 brd 0.0.0.0
4: eth0#if60: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1440 qdisc noqueue state UP group default
link/ether ae:e6:4e:95:2a:f2 brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet 10.20.14.175/32 scope global eth0
valid_lft forever preferred_lft forever
5: net1#if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UP group default
link/ether 72:fb:b5:90:d0:37 brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet 192.168.0.45/24 brd 192.168.0.255 scope global net1
valid_lft forever preferred_lft forever
Now what I need to configure is, a bigger allocation pool in macvlan definition file, but have only specific 2 IPs to be assigned to the PODs.
I tried below configuration.
Macvlan definition
apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
name: test-macvlandef01
spec:
config: '{
"cniVersion": "0.3.0",
"name": "test-macvlandef01",
"type": "macvlan",
"master": "eth0",
"mode": "bridge",
"ipam": {
"type": "whereabouts",
"datastore": "kubernetes",
"kubernetes": { "kubeconfig": "/etc/cni/net.d/whereabouts.d/whereabouts.kubeconfig" },
"range": "192.168.0.0/24",
"range_start": "192.168.0.40",
"range_end": "192.168.0.50"
}
}'
Deployment Set
apiVersion: apps/v1
kind: Deployment
metadata:
name: centos-test
spec:
replicas: 2
selector:
matchLabels:
app: centos
template:
metadata:
labels:
app: centos
annotations:
k8s.v1.cni.cncf.io/networks: '[{ "name": "test-macvlandef01","ips": "192.168.0.44"},{"name": "test-macvlandef01","ips": "192.168.0.45"}]'
spec:
nodeSelector:
test: "true"
containers:
- name: centos
image: centos
imagePullPolicy: IfNotPresent
command: ["bin/bash", "-c", "sleep 100000" ]
ports:
- containerPort: 80
PODs are coming up without MACVLAN interface and also I see no error associated with the POD.
[master1 ~]# kubectl exec -it centos-test-b59db89f7-2vvqx ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
2: tunl0#NONE: <NOARP> mtu 1480 qdisc noop state DOWN group default qlen 1000
link/ipip 0.0.0.0 brd 0.0.0.0
4: eth0#if65: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1440 qdisc noqueue state UP group default
link/ether 62:31:fc:64:8f:5b brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet 10.20.14.180/32 scope global eth0
valid_lft forever preferred_lft forever
[master1 ~]# kubectl exec -it centos-test-b59db89f7-6c75h ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
2: tunl0#NONE: <NOARP> mtu 1480 qdisc noop state DOWN group default qlen 1000
link/ipip 0.0.0.0 brd 0.0.0.0
4: eth0#if64: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1440 qdisc noqueue state UP group default
link/ether e6:23:30:ff:bf:c3 brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet 10.20.14.179/32 scope global eth0
valid_lft forever preferred_lft forever
Please suggest any modifications or additions that would help with the requirement.
Thanks in advance.

Answer 1

I want to pay your attention into 2 points below. Partial answer.
From your post I see that you want to use special IP addresses. To use such functionality, according to Extention convention from CNI you may need to use "capabilities": {"ips": true} capability in your Macvlan definition. Something like this:
spec:
config: '{
"cniVersion": "0.3.0",
"name": "test-macvlandef01",
"type": "macvlan",
"capabilities": {"ips": true}
"master": "eth0",
"mode": "bridge",
You can also find good explanation with examples in Attaching a pod to an additional network documentation.
I suppose that you use whereabouts plugin, since "type": "whereabouts" presents in your Macvlan definition. It supports exclusions:
You can also specify ranges to exclude from assignment, so if for
example you'd like to assign IP addresses within the range
192.168.2.0/24, you can exclude IP addresses within it by adding
them to an exclude list. For example, if you decide to exclude the
range 192.168.2.0/28, the first IP address assigned in the range
will be 192.168.2.16.
Knowing this fact, you can specify ranges of IPs to exclude from your configuration in accordance with Whereabouts IPAM Config example. Try to add exclude field in Macvlan definition with necessary IPs/subnets, which should be excluded.
Possible solution for your particular case:
spec:
config: '{
"cniVersion": "0.3.0",
"name": "test-macvlandef01",
"type": "macvlan",
"capabilities": {"ips": true}
"master": "eth0",
"mode": "bridge",
"ipam": {
"type": "whereabouts",
"range": "192.168.0.0/24",
"range_start": "192.168.0.40",
"range_end": "192.168.0.50"
"exclude": [
"192.168.0.40/32",
"192.168.0.41/32",
...
]
}
}'