how to configure kubespray DNS for bare-metal - kubernetes

Question

I am relatively new to kubernetes and have a project for my University class, to build a kubernetes Cluster on bare metal.
For this I have set up a PoC Environment, out of 6 Machines (of which 3 are KVM Machines on one Node) all the administration is done by MAAS, meaning DHCP, and DNS is administered by that one Machine. I have a DNS Zone delegated to the MAAS DNS server k8s.example.com where all the machines are inside. The whole network is in its own VLan 10.0.10.0/24, with the metallb IPRange reserved from DHCP.
This is a picture to illustrate the simple cluster:
software wise, all hosts are using ubuntu 20.04 and I use kubespray to deploy everything, meaning kubernetes, metallb and nginx-ingress-controller. My corresponding values for kubespray are:
dashboard_enabled: false
ingress_nginx_enabled: true
ingress_nginx_host_network: true
kube_proxy_strict_arp: true
metallb_enabled: true
metallb_speaker_enabled: true
metallb_ip_range:
- "10.0.10.100-10.0.10.120"
kubeconfig_localhost: true
My Problem is, that I am unable getting DNS out of the cluster to the Internet to work.
I had a wildcard A Record set for *.k8s.example.com to the nginx-Ingress external ip, which worked fine for every pod to be accessible from outside.
The Problem was, every container inside the Cluster could not reach the internet anymore. Every request was routed via the ingress. Meaning if I tried to reach www.google.net it would try to reach www.google.net.k8s.example.com which makes kind of sense. Only every .com domain could be reached without problems (example www.google.com) after removing the Wildcard A record it worked fine. All pods inside of the cluster have no problem reaching each other.
There are several configuration possibility I see, where it makes sense to tweak around, yet after 2 weeks I really would prefer a solution that is based on best practice and done right.
I would really love to be able to work with a wildcard A record, but I fear that might not be possible.
I hope I supplied every Information needed to give you enough overview to understand my Problem.
EDIT:
I used the standard kubespray DNS config as i was told it would suffice:
DNS configuration.
# Kubernetes cluster name, also will be used as DNS domain
cluster_name: cluster.local
# Subdomains of DNS domain to be resolved via /etc/resolv.conf for hostnet pods
ndots: 2
# Can be coredns, coredns_dual, manual or none
dns_mode: coredns
# Set manual server if using a custom cluster DNS server
# manual_dns_server: 10.x.x.x
# Enable nodelocal dns cache
enable_nodelocaldns: true
nodelocaldns_ip: 169.254.25.10
nodelocaldns_health_port: 9254
# nodelocaldns_external_zones:
# - zones:
# - example.com
# - example.io:1053
# nameservers:
# - 1.1.1.1
# - 2.2.2.2
# cache: 5
# - zones:
# - https://mycompany.local:4453
# nameservers:
# - 192.168.0.53
# cache: 0
# Enable k8s_external plugin for CoreDNS
enable_coredns_k8s_external: false
coredns_k8s_external_zone: k8s_external.local
# Enable endpoint_pod_names option for kubernetes plugin
enable_coredns_k8s_endpoint_pod_names: false
# Can be docker_dns, host_resolvconf or none
resolvconf_mode: docker_dns
# Deploy netchecker app to verify DNS resolve as an HTTP service
deploy_netchecker: false
# Ip address of the kubernetes skydns service
skydns_server: "{{ kube_service_addresses|ipaddr('net')|ipaddr(3)|ipaddr('address') }}"
skydns_server_secondary: "{{ kube_service_addresses|ipaddr('net')|ipaddr(4)|ipaddr('address') }}"
dns_domain: "{{ cluster_name }}"
What I noticed is, that the etc resolv.conf of pods looks like this:
/ $ cat /etc/resolv.conf
nameserver 169.254.25.10
search flux-system.svc.cluster.local svc.cluster.local cluster.local k8s.example.com maas
options ndots:5
for example on the node, which is managed by MAAS, it is:
# This file is managed by man:systemd-resolved(8). Do not edit.
#
# This is a dynamic resolv.conf file for connecting local clients to the
# internal DNS stub resolver of systemd-resolved. This file lists all
# configured search domains.
#
# Run "resolvectl status" to see details about the uplink DNS servers
# currently in use.
#
# Third party programs must not access this file directly, but only through the
# symlink at /etc/resolv.conf. To manage man:resolv.conf(5) in a different way,
# replace this symlink by a static file or a different symlink.
#
# See man:systemd-resolved.service(8) for details about the supported modes of
# operation for /etc/resolv.conf.
nameserver 127.0.0.53
options edns0 trust-ad
search k8s.example.com maas

Answer 1

As discussed in comments, the issue is with the resolv.conf on your Kubernetes nodes, and the fact that you are using a wildcard record, that matches one of the names in that resolv.conf search entries.
Any name you may call, from a Node or a Pod, would first be searched as ${input}.${search-entry}, while ${input} would only be queried if concatenation with your search did not return with some record already. Having a wildcard record in the domains search list would result in just any name resolving to that record.
Granted that in this case, the k8s.example.com record is pushed by MAAS, and that we can't really remove it persistently, the next best solution would be to use another name serving your Ingresses - either a subdomain, or something unrelated. Usually, changing an option in your DHCP server should be enough - or arguably better: don't use DHCP hosting Kubernetes nodes.