What am I missing to set up an HA vault cluster using raft and autounseal with transit engine - hashicorp-vault

Question

I am setting up Vault with a Raft backend and I'm attempting to set up a cluster using this
guide https://learn.hashicorp.com/tutorials/vault/raft-storage
I got it working without TLS, however I am experiencing errors when trying to implment TLS. Setting up the transit engine and the first raft node is fine, however when trying to set up the 3rd node which would be the second in the cluster, I get the following errors.
> 2022-07-21T11:08:41.407Z [INFO] core: stored unseal keys supported,
> attempting fetch 2022-07-21T11:08:41.407Z [WARN] failed to unseal
> core: error="stored unseal keys are supported, but none were found"
> 2022-07-21T11:08:41.407Z [INFO] core: security barrier not
> initialized 2022-07-21T11:08:41.408Z [INFO] core: attempting to join
> possible raft leader node: leader_addr=https://10.20.30.42:8200
> 2022-07-21T11:08:41.462Z [WARN] core: join attempt failed:
> error="error during raft bootstrap init call: Put
> "https://10.20.30.42:8200/v1/sys/storage/raft/bootstrap/challenge":
> x509: certificate signed by unknown authority"
> 2022-07-21T11:08:41.462Z [INFO] core: security barrier not
> initialized 2022-07-21T11:08:41.462Z [INFO] core: attempting to join
> possible raft leader node: leader_addr=https://10.20.30.43:8200
> 2022-07-21T11:08:41.477Z [WARN] core: join attempt failed:
> error="error during raft bootstrap init call: Put
> "https://10.20.30.43:8200/v1/sys/storage/raft/bootstrap/challenge":
> x509: certificate signed by unknown authority"
> 2022-07-21T11:08:41.477Z [ERROR] core: failed to retry join raft
> cluster: retry=2s 2022-07-21T11:08:41.477Z [INFO] http: TLS handshake
> error from 172.17.0.1:56062: remote error: tls: bad certificate
> 2022-07-21T11:08:43.477Z [INFO] core: security barrier not
> initialized
I thought that setting the VAULT_CACERT env variable with the parth of the correct cert was enough to stop the unknown authority error, this has worked on setting up the original node but for some reason doesnt work on setting up the transit cluster.

Answer 1

Raft consensus occurs over the cluster port and uses a custom, Vault managed certificate. A valid TLS connection is required to call the join API. I don't know if Vault (when running as a server) honors the VAULT_SKIP_VERIFY environment variable, but even if it does, setting would lower the security of your installation.
The error log shows that Vault is trying to reach the leader with its IP address:
> error="error during raft bootstrap init call: Put
> "https://10.20.30.43:8200/v1/sys/storage/raft/bootstrap/challenge":
> x509: certificate signed by an unknown authority"
Make sure your configuration file sets the api_addr parameter to a name that matches the certificate you are using.

Answer 2

The issue was that I needed to pass
leader_ca_cert_file = "route/to/pem/file"
into the retry_join block in the config file. I thought that having declared it as an env variable was enough but apparently not