linux bridge not work when attach more than one interface - kubernetes

Question

this question is continuation of this problem packets lost after postrouting snat
if i create a linux bridge by command，the bridge can work normally，but if i create bridge by code，the bridge not work, a lot of strange behavior occurs。 This problem can be reproduced from other linux machines
my script for create docker container attached to bridge:
new_sandbox.sh
#!/usr/bin/env bash
set -euo pipefail
container_name=$1
container_ip=$2
br_name=$3
gw_ip=$4
echo "ready to create container $container_name connect to br $br_name with ip $container_ip, gw $gw_ip"
docker run -d --network none --name $container_name schecter/test:v4
container_id=$(docker ps | grep $container_name | awk '{print $1}')
pid=$(docker inspect -f '{{.State.Pid}}' ${container_id})
ln -sfT /proc/$pid/ns/net /var/run/netns/${container_id}
echo "container_id=$container_id, pid=$pid"
host_veth="veth_"$pid"_0"
container_veth="veth_"$pid"_1"
ip link add dev $host_veth type veth peer name $container_veth
ip link set dev $host_veth master $br_name
ip link set dev $host_veth up
ip link set dev $container_veth netns ${container_id}
ip netns exec ${container_id} ip link set dev $container_veth up
ip netns exec ${container_id} ip address add $container_ip dev $container_veth
ip netns exec ${container_id} ip route add default via $gw_ip
my code for create bridge: br_init.go
package main
import (
"fmt"
"github.com/vishvananda/netlink"
"net"
)
var (
gwIp = "10.10.10.1/24"
bridgeName = "br1"
)
func main() {
br := &netlink.Bridge{
LinkAttrs: netlink.LinkAttrs{
Name: bridgeName,
MTU: 1500,
TxQLen: -1,
},
}
if err := netlink.LinkAdd(br); err != nil {
panic(err)
}
bridgeIf, err := netlink.LinkByName(bridgeName)
if err != nil {
panic(err)
}
err = netlink.LinkSetUp(bridgeIf)
if err != nil {
panic(err)
}
if err = netlink.SetPromiscOn(bridgeIf); err != nil {
panic(err)
}
gwip, gwnet, err := net.ParseCIDR(gwIp)
if err != nil {
panic(err)
}
gwnet.IP = gwip
addr := &netlink.Addr{IPNet: gwnet}
if err = netlink.AddrAdd(bridgeIf, addr); err != nil {
panic(err)
}
//// this line of code work fine
//exec.Command(fmt.Sprintf("ip link set br1 address %s", bridgeIf.Attrs().HardwareAddr))
//This line of code causes problems
if err = netlink.LinkSetHardwareAddr(bridgeIf, bridgeIf.Attrs().HardwareAddr); err != nil {
panic(err)
}
fmt.Println("success!!!!")
}
go version: 1.17，go.mod
module test
go 1.17
require github.com/vishvananda/netlink v1.2.1-beta.2
require (
github.com/vishvananda/netns v0.0.0-20210104183010-2eb08e3e575f // indirect
golang.org/x/sys v0.4.0 // indirect
)
my command for create bridge: br_init.sh
#!/usr/bin/env bash
set -euo pipefail
ip link add dev br2 type bridge
ip address add 10.10.11.1/24 dev br2
ip link set br2 promisc on
ip link set br2 up
ip link set br2 address ca:b6:90:1d:8f:cf
then i create two bridge for compare. one created by code, one create by command, and attache two container to each bridge。
go run br_init.go
./new_sandbox.sh wjj_test1 10.10.10.3/24 br1 10.10.10.1
./new_sandbox.sh wjj_test2 10.10.10.4/24 br1 10.10.10.1
./br_init.sh
./new_sandbox.sh wjj_test3 10.10.11.3/24 br2 10.10.11.1
./new_sandbox.sh wjj_test4 10.10.11.4/24 br2 10.10.11.1
we can see two bridge,br1 created by code, br2 created by command:
root#n225-066-136:~/test# ip l
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
link/ether fa:16:3e:2e:a0:69 brd ff:ff:ff:ff:ff:ff
808: br1: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
link/ether 36:84:ee:c7:e1:d1 brd ff:ff:ff:ff:ff:ff
809: br2: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
link/ether ca:b6:90:1d:8f:cf brd ff:ff:ff:ff:ff:ff
811: veth_1557963_0#if810: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br1 state UP mode DEFAULT group default qlen 1000
link/ether 76:82:f0:5b:0f:83 brd ff:ff:ff:ff:ff:ff link-netns 00feea33629f
813: veth_1558110_0#if812: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br1 state UP mode DEFAULT group default qlen 1000
link/ether 3e:d8:af:b0:35:68 brd ff:ff:ff:ff:ff:ff link-netns 99bff89a3221
815: veth_1558312_0#if814: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br2 state UP mode DEFAULT group default qlen 1000
link/ether 16:69:e3:4a:6c:dc brd ff:ff:ff:ff:ff:ff link-netns 68eab047136b
817: veth_1558482_0#if816: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br2 state UP mode DEFAULT group default qlen 1000
link/ether 4e:43:fa:0c:83:8f brd ff:ff:ff:ff:ff:ff link-netns c2cabb285996
then config iptables:
iptables --table filter --append FORWARD --in-interface br1 --jump ACCEPT
iptables --table filter --append FORWARD --out-interface br1 --jump ACCEPT
iptables --table filter --append FORWARD --in-interface br2 --jump ACCEPT
iptables --table filter --append FORWARD --out-interface br2 --jump ACCEPT
iptables --table nat --append POSTROUTING --source 10.10.10.0/24 --jump MASQUERADE
iptables --table nat --append POSTROUTING --source 10.10.11.0/24 --jump MASQUERADE
container wjj_test1、wjj_test2 attached to br1, wjj_test3、wjj_test4 attached to br2。in wjj_test1, ping 8.8.8.8 failed。in wjj_test3, ping 8.8.8.8 success。
for br1, if only one container attached to it, the container can work fine, ping 8.8.8.8, if attached another contaoner to br1, the previous container immediately failed to ping。
As the comments show, I basically located the problematic code is
if err = netlink.LinkSetHardwareAddr(bridgeIf, bridgeIf.Attrs().HardwareAddr); err != nil {
panic(err)
}
if i remove this line, the bridge can work fine.(but we need to fix bridge's mac, otherwise it will change)
if i repalce this line to
exec.Command(fmt.Sprintf("ip link set br1 address %s", bridgeIf.Attrs().HardwareAddr))
the bridge can work fine。so it seem netlink.LinkSetHardwareAddr not equal to ip link set $link address $hwaddr。There are many unknown side effects
why the code can't work