I installed vault locally and started, unsealed, and initialized the vault and added some secrets. After rebooting, I am unable to use the keys to unseal the vault. The first two unseal keys are accepted without issue, but after submitting the third key, I get an error response:
Error unsealing: Error making API request.
URL: PUT https://127.0.0.1:28200/v1/sys/unseal
Code: 500. Errors:
* failed to decrypt encrypted stored keys: cipher: message authentication failed
Any ideas what is going on? I am running vault version 1.4.2. The command I am using is vault operator unseal. The server config is:
vault_server.hcl
listener "tcp" {
address = "127.0.0.1:28200"
tls_cert_file = "/etc/vault/certs/vault_cert.crt"
tls_key_file = "/etc/vault/certs/vault_cert.key"
}
storage "file" {
path = "/etc/vault/mnt/data"
}
api_addr = "https://127.0.0.1:28200" # my $VAULT_ADDR is https://127.0.0.1:28200
disable_mlock = true
The relevant log output:
Jun 12 21:26:24 lambda vault[1147]: 2020-06-12T21:26:24.537-0500 [DEBUG] core: unseal key supplied
Jun 12 21:26:24 lambda vault[1147]: 2020-06-12T21:26:24.537-0500 [DEBUG] core: cannot unseal, not enough keys: keys=1 threshold=3 nonce=920f7d80-fdcc-3bc3-149e-8b069ef23acb
Jun 12 21:26:38 lambda vault[1147]: 2020-06-12T21:26:38.069-0500 [DEBUG] core: unseal key supplied
Jun 12 21:26:38 lambda vault[1147]: 2020-06-12T21:26:38.069-0500 [DEBUG] core: cannot unseal, not enough keys: keys=2 threshold=3 nonce=920f7d80-fdcc-3bc3-149e-8b069ef23acb
Jun 12 21:26:51 lambda vault[1147]: 2020-06-12T21:26:51.984-0500 [DEBUG] core: unseal key supplied
The most relevant issues I can find in web searches are for people who inadvertently corrupted their storage:
https://github.com/hashicorp/vault/issues/5498
https://groups.google.com/forum/#!msg/vault-tool/N9fc_dUejJw/OfovdNNHBwAJ
https://discuss.hashicorp.com/t/move-vault-installation-between-servers/6990/2
I'm not sure that applies here. I'm using filesystem storage, vault is the owner of everything in /etc/vault, and I can't tell that any data has been lost or corrupted.
I had the same issue with freshly installed vault 1.4.2 in HA mode on GKE using their official vault-k8s helm chart. I deployed it on 2 environments. The first one was OK, but the second one was failing exactly the same way as you described when I tried to join the 2nd vault instance to the HA cluster. I simply deleted and re-installed it a few times and eventually it worked.
TLDR Summary: vault will always accept keys until it hits the minimum count so it can attempt to assemble/use the resulting unseal key. Accepting keys is not an indicator of validity.
The keys distributed by the vault server are actually "shards" or "shares" (exact terminology changes between documenting sources) that are generated by splitting/sealing the master key using Shamir's secret Sharing. Because the master key cannot be decrypted without accepting the minimum number of shards (defaults to 3, but could be configured to a different value), the vault server has no way of determining if a provided shard is a valid until that minimum is provided so it can attempt to 1. generate an unseal key from shards, and 2. use resulting key against the master key.
Hashicorp provides a decent overview of the process here:
https://www.vaultproject.io/docs/concepts/seal
More information on shamir and the math behind it here:
https://en.wikipedia.org/wiki/Shamir%27s_Secret_Sharing
Now the bad news:
The error implies you are using an incorrect set of keys to unlock your vault: the resulting unseal key is incorrect. Either they have been changed or they are for a different vault. (maybe bad cut-n-paste newline injection?). While some have suggested a reinstall, I don't think this will solve anything. If you are desperate, you could try using a different version of vault in-case there is an unseal bug in one of your distros, but that's ... a reach.
I had the exact same issue. It turns out that I was not using the correct keys (I was using old keys copied elsewhere). After using the correct keys, I was able to unseal the vault without any issues.
https://discuss.hashicorp.com/t/not-able-to-unseal-vault-after-container-pod-restart/16797
Related
The answer on many forums is to simply "create the key". However, this also requires further maintenance because then the flow.xml.gz file cannot be easily migrated between deployments. I do not experience this on previous versions (v1.12.0 & 1.13.0).
Any thoughts on mitigating this issue?
Error:
ERROR [main] o.a.nifi.properties.NiFiPropertiesLoader Clustered Configuration Found: Shared Sensitive Properties Key [nifi.sensitive.props.key] required for cluster nodes
ERROR [main] org.apache.nifi.NiFi Failure to launch NiFi due to java.lang.IllegalArgumentException: There was an issue decrypting protected properties
Similar question: Docker - Nifi : 1.14.0 - Startup failure - Caused by: org.apache.nifi.properties.SensitivePropertyProtectionException
Create the key.
Don't let NiFi generate a random one, supply it from your deployment code along with all the other settings that go into nifi.properties. If you have the same key, you can copy/migrate the flow.xml.gz and share it within clusters.
This also works with an encrypted key if you provide the decryption hex key in bootstrap.conf when deploying.
The latest NiFi version has support for Hashicorp vaults. That might allow you to obtain the correct keys at runtime and share them among cluster nodes.
If you want to work without a key, you will need to use NiFi 1.13.2 or older. From the admin guide:
Starting with version 1.14.0, NiFi requires a value for 'nifi.sensitive.props.key' in nifi.properties.
The following command can be used to read an existing flow.xml.gz configuration and set a new sensitive properties key in nifi.properties:
$ ./bin/nifi.sh set-sensitive-properties-key [sensitivePropertiesKey]
The minimum required length for a new sensitive properties key is 12 characters.
Ignore this error.
Create a new deployment, either import the flow file via the GUI or copy and paste the XML flow file, then restart the deployment.
In my testing, I have not seen any evidence that the sensitive key property is required.
Been trying to figure out how to do this for awhile. Essentially, Vault does not have a secure option for its REST calls. I want to make these rest calls encrypted from as close between point a and b as possible. My thoughts have been the following:
Use an SSH tunnel
Use a TLS tunnel like Stunnel
I currently have Vault in a Docker container, so that’s something else to mention. Has anyone encountered this situation, and how did you deal with it?
UPDATE: So, using the Python API (HVAC), I am getting the following error:
requests.exceptions.SSLError: HTTPSConnectionPool(host='0.0.0.0',
port=8200): Max retries exceeded with url: /v1/secret (Caused by
SSLError(SSLError("bad handshake: Error([('SSL routines', 'ssl3_get_record',
'wrong version number')],)",),))
Using the following commands:
import os
import hvac
client = hvac.Client(url='https://0.0.0.0:8200', token='my-token-here')
Vault has TLS enabled by default, thus all your REST calls are encrypted already. If you are having trouble using https, have a look at the documentation of VAULT_CACERT and VAULT_CAPATH environment variables.
from vault's documentation.
VAULT_CACERT
Path to a PEM-encoded CA certificate file on the local disk. This file
is used to verify the Vault server's SSL certificate. This environment
variable takes precedence over VAULT_CAPATH.
VAULT_CAPATH Path to a directory of PEM-encoded CA certificate files
on the local disk. These certificates are used to verify the Vault
server's SSL certificate.
You can use tools like tcpdump or wireshark to make sure that your requests are indeed encrypted.
To elaborate for Vault running in a container, you need to create a configuration file for Vault that contains something similar to this this (Chef/Ruby code):
config_content = %(
"storage": {
...
},
"default_lease_ttl": "768h",
"max_lease_ttl": "8766h",
"listener": [
{"tcp": {
"address": "0.0.0.0:8200",
"tls_disable": 0,
"tls_cert_file": "/vault/certs/my-cert-combined.pem",
"tls_key_file": "/vault/certs/my-cert.key"
}}],
"log_level": "info"
)
Especially the listener portion. Make your backend storage whatever you want to use (not the Dev default of in-memory!).
Note you will need to get a valid certificate and its private key also in the volume bound into the container.
Store this configuration file in a directory that gets bound inside the container to the path /vault/config. I use /var/vault/config on my host. For example (more Ruby/Chef):
docker_container 'vault' do
container_name 'vault'
tag 'latest'
port '8200:8200'
cap_add ['IPC_LOCK']
restart_policy 'always'
volumes ['/var/vault:/vault']
command 'vault server -config /vault/config'
action :run_if_missing
end
That command tells Vault to look in /vault/config and it should find your config file there, with a .json extension. Note it is important to have the config file listener->tcp->address be 0.0.0.0, rather than 127.0.0.1, because Vault will not resolve external accesses properly.
Then Vault will startup with TLS encryption on all transactions. Define VAULT_ADDR to have https://your-host.com:8200 and away you go.
In my case, I've been testing it on my local environment. So instead of calling the secured https: https://localhost:8200, I called regular http: http://localhost:8200.
This solved the error.
We are getting the below error in Wildfly/Jboss when we are trying to encrypt the DB password using Vault. Can you provide a solution for this?
Caused by: org.jboss.security.vault.SecurityVaultException: PB00027: Vault Mismatch:Shared Key does not match for vault block:bea_interface and attributeName:password
There are three possible causes:
1). There is just a mismatch between the passwords. Check what you used when setting up the vault.
2). The encrypted password files are missing:
Aside of the keystore, you should not forget to put the two other files that vault.sh generates
vault.keystore
ENC.dat
Shared.dat
You need to copy all three files to the desired location, for example to the "standalone/configuration/" directory.
In the vault definition, these are the two paramaters that will tell JBoss where to find them:
<vault-option name="KEYSTORE_URL" value="${jboss.server.config.dir}/vault.keystore"/>
<vault-option name="ENC_FILE_DIR" value="${jboss.server.config.dir}/"/>
3). You are using a keystore alias name longer then 10 characters.
I reinstalled some nodes and a master. Now on the master I am getting:
Sep 15 04:53:58 master kube-apiserver[803]: I0915 04:53:58.413581 803 logs.go:41] http: TLS handshake error from $ip:54337: remote error: bad certificate
Where $ip is one of the nodes.
So I likely need to delete or recreate certificates. What would the location of those be? Any recommended commands to recreate or remove those or copy them from node to master or vice versa? Whatever gets me past this error message...
Take a look through the Creating Certificates section of authentication.md. It walks you through the certificates that you need to create and how to pass them to the system components, and you should be able to use that to re-generate certificates for your cluster.
I am not sure either what I am trying to do is possible or correct way.
One of my colleague spinup kubernetes gce cluster (with 1 master and 4 minions.) in a project which is shared with me as owner access.
After setup he shared his ~/.kubernetes_auth keys along with .kubecfg.crt, .kubecfg.ca.crt and .kubecfg.key. I copied all of the at my home folder and setup the kubernetes workspace.
I also set the project name as a default project in geconfig. and now I can connect to the master and slaves using 'gcutil ssh --zone us-central1-b kubernetes-master'
But when I try to list of existing pods using 'cluster/kubecfg.sh list pods'
I see
"F1017 21:05:31.037148 18021 kubecfg.go:422] Got request error: Get https://107.178.208.109/api/v1beta1/pods?namespace=default: x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "ChangeMe")
I tried to debug from my side but failed to come any conclusion. Any sort of clue will be helpful.
You can also copy the cert files off of the master again. They are located in /usr/share/nginx on the master.
It is probably due to a not implemented feature, see this issue:
https://github.com/GoogleCloudPlatform/kubernetes/issues/1886
you can copy the files from /usr/share/nginx/... on the master
into your home dir and try again.
I figured out a workaround: set the -insecure_skip_tls_verify option
In kubecfg.sh, change the code near the bottom to
else
auth_config=(
"-insecure_skip_tls_verify"
)
fi
Obviously this is insecure and you are putting yourself at risk of a man in the middle attack, etc.