mDNS (Avahi) working TO RPis but not between them - raspberry-pi

Question

Avahi/mDNS is running by default on recent versions of Raspian. Great. Very convenient to just ssh pi#mypi.local.
I am doing development on a Mac and operating a local network of headless Raspberry Pis. Up until now, I was able to use mDNS to access the Pis, and the Pis used mDNS to connect to each other.
Today, I shifted the RPis to a private local network by setting them up on a wireless router unconnected to the internet. Once I join the private network, I am still able to access them via mDNS:
% ssh pi#scheduler.local
Linux scheduler 5.10.63-v7l+ #1459 SMP Wed Oct 6 16:41:57 BST 2021 armv7l
Last login: Mon Aug 1 09:07:43 2022
pi#scheduler:~ $
and
wes#macbook % ssh pi#crossing.local
Linux crossing 5.10.17-v7l+ #1414 SMP Fri Apr 30 13:20:47 BST 2021 armv7l
Last login: Mon Aug 1 09:07:46 2022
pi#crossing:~ $
But when they try to access each other, I get some results I don't understand:
pi#scheduler:~ $ ping crossing.local
PING crossing.local (10.0.0.1) 56(84) bytes of data.
From 192.168.0.1 (192.168.0.1) icmp_seq=1 Destination Net Unreachable
From 192.168.0.1 (192.168.0.1) icmp_seq=2 Destination Net Unreachable
From 192.168.0.1 (192.168.0.1) icmp_seq=3 Destination Net Unreachable
From 192.168.0.1 (192.168.0.1) icmp_seq=4 Destination Net Unreachable
Here's what Avahi reports:
pi#scheduler:~ $ service avahi-daemon status
● avahi-daemon.service - Avahi mDNS/DNS-SD Stack
Loaded: loaded (/lib/systemd/system/avahi-daemon.service; enabled; vendor preset: enabled)
Active: active (running) since Mon 2022-08-01 09:07:37 PDT; 41min ago
Main PID: 388 (avahi-daemon)
Status: "avahi-daemon 0.7 starting up."
Tasks: 2 (limit: 1438)
CGroup: /system.slice/avahi-daemon.service
├─388 avahi-daemon: running [scheduler.local]
└─414 avahi-daemon: chroot helper
Aug 01 09:08:08 scheduler avahi-daemon[388]: Leaving mDNS multicast group on interface wlan0.IPv4 with address 169.
Aug 01 09:08:08 scheduler avahi-daemon[388]: Joining mDNS multicast group on interface wlan0.IPv4 with address 192.
Aug 01 09:48:29 scheduler avahi-daemon[388]: Files changed, reloading.
Aug 01 09:48:29 scheduler avahi-daemon[388]: No service file found in /etc/avahi/services.
Here's my hosts and hostname files:
pi#scheduler:~ $ cat /etc/hosts
127.0.0.1 localhost
::1 localhost ip6-localhost ip6-loopback
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
127.0.1.1 scheduler
pi#scheduler:~ $ cat /etc/hostname
scheduler
What does avahi say about it? Let's see:
pi#brs-scheduler:~ $ avahi-resolve --name brs-crossing.local -4
brs-crossing.local 192.168.0.214
pi#brs-scheduler:~ $ ifconfig | grep "inet 192"
inet 192.168.0.109 netmask 255.255.255.0 broadcast 192.168.0.255
pi#brs-scheduler:~ $ ping brs-crossing.local
PING brs-crossing.local (10.0.0.1) 56(84) bytes of data.
From 192.168.0.1 (192.168.0.1) icmp_seq=1 Destination Net Unreachable
So for some reason, on this private network, mDNS is resolving correctly, but ping and ssh don't resolve properly?
What am I missing?

Answer 1

Unsurprisingly, since the pis worked fine on the local net and stopped working on a private net with a new router, it had to do with the configuration of the new router not mDNS.
mDNS was working fine:
pi#scheduler:~ $ avahi-resolve --name crossing.local -4
crossing.local 192.168.0.214
The new router on the private net had two operating modes "router" and "access point." In "router" mode, the router was pushing a DNS nameserver IP to clients which was somehow hosing ping and ssh and other services, despite mDNS working okay.
pi#scheduler:~ $ cat /etc/resolv.conf
# Generated by resolvconf
nameserver 192.168.0.1
Once the router was placed in "access point" mode, and DHCP was turned on manually, everything worked.
Obscure problem. Obscure solution.