Connect Consul client on VM to Consul Server in Kubernetes - kubernetes

Question

I'm having trouble setting up this scenario. I have a server that's running 4 VMs:
k8s master
k8s worker 1
k8s worker 2
vm1
I've deployed a Consul cluster with Helm chart into k8s cluster, resulting in 1 server on each worker node, 1 client on each worker node. Here's current config (I'm trying all sorts of things so some might be commented/disabled)
global:
name: consul
enabled: true
datacenter: dc1
gossipEncryption:
autoGenerate: true
tls:
enabled: true
enableAutoEncrypt: true
verify: true
acls:
manageSystemACLs: true
# client:
# exposeGossipPorts: true
server:
replicas: 2
# exposeGossipAndRPCPorts: true
# ports:
# serflan:
# port: 9301
extraConfig: |
{ "log_level": "debug" }
exposeService:
enabled: true
type: NodePort
nodePort:
http: 31500 # 8500 + 23k
https: 31501 # 8501 + 23k
grpc: 31503 # 8503 + 23k
serf: 32301 # 9301 + 23k
rpc: 31300 # 8300 + 23k
securityContext:
runAsNonRoot: false
runAsUser: 0
storageClass: nfs-synology-test
connectInject:
enabled: true
controller:
enabled: true
syncCatalog:
enabled: true
dns:
enabled: true
This works mostly fine in the cluster itself (I can run a Job that does a dig to a Consul service name and I do get a response). Now I wanted to install another Consul client, this time on the vm1 VM and join it into my Consul in K8s cluster.
As you can see in the comments, I've tried exposing gossip and RPC ports as host ports, now instead exposing the server service as NodePort service on given ports.
From the VM I can verify with nmap that those ports are indeed open on TCP, but for the love of all that's holy I can't figure out what to configure in the vm1 client. Here's my current config:
{
"server": false,
"domain": "consul",
"datacenter": "dc1",
"data_dir": "/etc/consul/data",
"tls": {
"defaults": {
"ca_file": "/etc/consul/tls/ca/tls.crt",
"verify_incoming": false,
"verify_outgoing": true
},
"internal_rpc": {
"verify_server_hostname": true
}
},
"auto_encrypt": { "tls": true },
"encrypt": "redacted",
"log_level": "DEBUG",
"enable_syslog": true,
"leave_on_terminate": true,
"retry_join": [
"192.168.1.207:32301",
"192.168.1.208:32301",
"10.233.94.138:8300",
"10.233.119.94:8300"
],
"advertise_addr": "192.168.1.230",
"bind_addr": "0.0.0.0",
"ports": { "server": 31300 },
"acl": {
"tokens": {
"agent": "redacted",
"default": "redacted"
}
}
}
I've taken the value of encrypt from the secret in k8s, same as the tls.crt. I've tried to generate a token with the GUI assigned to client-policy, defined as:
node_prefix "" {
policy = "write"
}
service_prefix "" {
policy = "read"
}
But all to no avail. The client generally fails UDP connections, tries to connect to internal k8s cluster IPs (even without me adding them to retry_join, again just trying), overall geting timeouts and rpc error: lead thread didn't get connection.
I'm out of ideas and I'm at the stage of just trying random ports and configs until I hit jackpots. Can anyone help?

Answer 1

I somewhat figured out my answer so posting if anyone is stuck on the same thing. While I like exposing a service on node ports, it didn't quite work out. The servers were accessible from the 32301 port (so VM client was able to join), but the server themselves were advertising themselves on the pod IP which was inaccessible from outside.
The solution in my case was:
actually use the commented out client.exposeGossipPorts: true, server.exposeGossipAndRPCPorts: true and server.ports.serflan.port: 9301
retry_join in the config should use the node IP + 9301 port
the way I set up the TLS cert, tokens and encrypt was probably right
as far as RPC and failing UDP, the problem was in the fact that Kube cluster failed to deploy the servers and clients correctly (Serflan-UDP ports weren't exposed via UDP). When I kubectl patched it, it started to work (eventually we fixed the cluster itself instead)