I'm having troubles with Vault it returns permission denied 403 error, when I try to get secrets with my k8s AppRole.
I setup vault with kv version 2 engine.
Added policy for my AppRole:
Created secret under "dev/fra1/statement":
When I login with AppRole creds I have response with required policies:
When I try to execute get request with AppRole client_token I this error:
I tried different prefixes and so on (Since people on internet had problems with them).
But then was able to localize the problem, by performing that request with root token, so it went ok:
Now I'm our of ideas, I believe the only place where the problem can be is policy, what I'm doing wrong ?
Ok, so finally figured the right prefix our, it should be:
path "kv/data/dev/*" {
capabilities = ["read"]
}
Really, there is some hell with these prefixes in vault, they should describe it better in docs.
The "secret" prefix is used in v1 of Vault's KV API. v2 uses the mount name, which by default is "kv", but can be anything when you first create the mount for your KV secrets engine.
It is important to note that some tools which use Vault's API still use v1 of the KV API to access secrets, despite that your KV secrets engine may be v2. So you may need two different permissions in your policy.
I'm facing the same issue. I have a secrets engine called TestSecretsEngine and a single secret env. In my policy I add read to the path TestSecretsEngine/data/env to no avail. I'm using the node-vault npm module and it's failing at vault.approleLogin with a 403. It's got to be something with the policy because when I add a nonexistent path, I get a 404 instead.
Related
In deployment.yaml file, we don't add username and password but refer them using secret. This way,the username/password don't get stored in code repositories. In secret.yaml, the username and password are encoded in base64 (which can be decoded). From best practice persepective, doesn't secret.yaml also get source-controlled somewhere (thereby also storing the username/password in version control). If so, what is the benefit of using Secret ?
There are a few aspects to be considered when keeping k8s secrets secret.
Data encryption at rest
There's the configuration option --encryption-provider-config, which instructs the api server whether and how to encrypt data in etcd. There's lots more in the docs.
Authorization
Role Based Access Control is one authz possibility for k8s. Using it, access to secrets can be restricted, so not every user or service account can see secrets, already existing in the cluster, think of kubectl get secret others-secret -n some-ns -o yaml. With RBAC you can create roles with specific sets of permissions - allowing or not access to secrets per namespace - and then assign those roles to users, groups or even service accounts, as you see fit.
Secrets manifests and VCS
3.1 Encryption
There are quite a few tools allowing for encryption of files with sensitive data, which would allow you to commit the file with the secrets to a version control system, if that's what you need. A simpler one would be mozilla SOPS and one somewhat sophisticated and complex might be Vault for example. Whichever it is, it would definitely be nice if not necessary, to be possible to easily integrate it in any delivery pipeline.
3.2 Don't store secrets manifests
An alternative approach to the above would be to not store any files with secrets. Create the secret and with regular cluster backups (tools like velero for instance) you should have nothing to worry about.
As you said, secrets are not encrypted but only base64 encoded. Where secrets really add value is that they allow you to keep your passwords, keys, tokens out of your codebase/git repos. If you push your code to Github, you will not be pushing your secrets there. For this reason they are called secrets and add a layer of safety.
However, if someone gets access to your cluster, secrets will be mere plaintext to them.
The recommended way to store the secrets is to use a vault.
https://www.hashicorp.com/blog/injecting-vault-secrets-into-kubernetes-pods-via-a-sidecar
Good one. Some time ago I already answered Kubernetes secret is really secret? question, check there all the info.
If you configure the secret through a manifest (JSON or YAML) file
which has the secret data encoded as base64, sharing this file or
checking it in to a source repository means the secret is compromised.
Base64 encoding is not an encryption method and is considered the same
as plain text.
We have installed and configured Hashicorp Vault AppRole authentication for one server, by storing the role_id and secret_id in a local file on the server, and we're able to have code on the server read the values from file, authenticate to Vault, receive a token and then read the secrets it needs from Vault. So far so good. However, the secret_id expires after 31 days, and so the process fails.
I've read up on the concepts of using AppRoles, and they seem like the perfect fit for our use case, but for this expiration. We don't want to have to re-generate the secret_id every month.
From what I've read, if you create the role without setting secret_id_ttl it should be non-expiring, but that isn't the case. This may be due to how the AppRole auth method is configured, but I haven't seen anything solid on this.
So I found an article on the Hashicorp website where AppRoles are discussed in detail. The article gives good arguments for expiring secret_id's in a CI/CD environment, even illustrating how this works in 8 simple steps. I understand how this works, but the article fails to mention how the CI/CD and Orchestrator systems themselves are authenticated to Vault? Or am I missing something?
In the end, I want to have the secret_id not expire. Ever.
Without additional support from your environment you will have to write some logic in your installer, and have a service manager of some sort to start your services. In many cloud environments, you may already have the equivalent entities (Terraform, Cloud Formation, etc.) and you should leverage their secrets management capabilities where needed.
For custom installations, here is a workflow that I have used.
Have an installation manager process that can be invoked to perform installation / upgrade. Make sure installation / upgrade of services is always through this process.
Have a service manager process that is responsible for starting individual services and monitoring them / restarting them. Make sure service start-ups are always via this service manager.
During installation, generate self-signed certificates for Vault, installation manager and service manager. Vault certificates should trust the certs for the installation manager and the service manager. Store these with limited permission (600) in directories owned by the installation user or the service manager user as the case may be. Set up certificate-based authentication in Vault using these certs.
These credentials should have limited capabilities associated with them. The installation manager should only be able to create new roles and not delete anything. The service manager should only be able to create secrets for the named roles created by the installation manager, and delete nothing.
During installation / upgrade, the installation manager should connect to Vault and create all necessary service-specific roles. It should also be able to set role ids for individual services in per-service config files that the services may read on start-up.
During each service's start-up, the service manager should connect to Vault and create secret ids corresponding to each service's role. It should set the secret id in an environment variable and start the service. The secret id should have time-bound validity (by setting TTLs) so that they cannot be used for much beyond the creation of the auth token (see #7).
Each service should read the role id from the config file, and the secret id from the environment variable. It should then generate the auth token using these two, and use the token to authenticate itself with vault for its lifetime.
It is possible to create a Vault AppRole with a secret_id that essentially never expires. However, this should be limited to use on a Vault development server -- one that does not contain any production credentials -- and for use in a development environment.
That being said, here's the procedure I used based on several articles in the Vault documentation, but primarily AppRole Pull Authentication.
This assumes that the Vault approle authentication method is already installed at approle/ and that you are logged in to Vault, have root or admin privileges on the Vault server and have a valid, non-expired token.
Note: For the values supplied for the fields below, the maximum value that vault seems to accept is 999,999,999. For the TTL fields, that is the number of seconds which comes out to more than 31 years. That's not forever, but it is long enough that renewing the secret_id will probably be somebody else's problem (SEP).
# Vault server address to be used by the Vault CLI.
export VAULT_ADDR="https://vault-dev.example.com:8200/"
# Vault namespace to be used by the CLI.
# Required for Cloud and Enterprise editions
# Not applicable for Open Source edition
export VAULT_NAMESPACE="admin"
# The name of the Vault AppRole
export VAULT_ROLE=my-approle
# Override defaults on the approle authentication method
# NOTE: In this command, the field names, default-lease-ttl
# and max-lease-ttl contain dashes ('-'), NOT
# underscores ('_'), and are preceded by a single
# dash ('-').
vault auth tune \
-default-lease-ttl=999999999 \
-max-lease-ttl=999999999 approle/
# Override defaults on the approle
# NOTE: In this command, the field names, secret_id_ttl and
# secret_id_num contain underscores ('_'), NOT
# dashes ('-'), and are NOT preceded by a single
# dash ('-').
vault write auth/approle/role/my-approle \
secret_id_ttl=999999999 \
secret_id_num_uses=999999999
# Create a new secret_id for the approle which uses the new defaults
vault write -f auth/approle/role/my-approle/secret-id
Update the server config file to use the new secret_id and you are ready to go.
As the OP has noted, the Hashicorp Vault documentation assumes that the application is able to authenticate, somehow, to the vault and then retrieve the secret ID (possibly wrapped) from the vault and then, use that to authenticate and fetch a token used to actually work with secrets. The answers here are posing alternative approaches to retrieving that initial token.
Alan Thatcher wrote a blog article, Vault AppRole Authentication, that provides another well thought out approach:
Create a policy that allows the user to retrieve the secret-id and role-id, but nothing else.
Create a long lived, periodic/renewable token based on that policy.
Store the long lived token securely, e.g. as a Kubernetes secret
At runtime, use the long-lived token to:
acquire the secret-id and role-id,
authenticate to vault using these and acquire short-lived token
use current short-lived token to work with secrets
For Java applications, the Spring Vault project supports this approach if you configure the long-lived token as the "initial token" and the approle authencation name, e.g. chef-ro in the blog case.
My personal feeling is that this approach is about as secure but a bit simpler than the mutual TLS approach. I agree that using an infinite TTL for the secret-id is a less secure practice for Production environments.
Thanks to Mr. Thatcher for thinking this one through.
This is probably not the canonnical answer, but I found it empty so decided to add some pointers.
As per Hashicorp Vault AppRole: role-id and secret-id:
Additional brownie information: Ideally, it's best practice to keep
the TTL low, 30 minutes max - if your application is stateful, or
maybe even less if it's a stateless application. The secret key of
Vault approle should also be rotated every 90 days. Please note by
default, Vault approle backend has 31 days of TTL, so if you want to
set it to 90 days, you need to increase TTL of the approle backend as
well.
However (in the same question):
You can generate secret-id with indefinite validity. But doing so will
be as good as keeping your secrets in the configuration file.
For ephemeral instances you can use configuration management to pass in secrets via a third (broker) role. With regard to a server that exists indefinitely, i'm still working that out...
Ideas:
TLS certificates might work well on Windows, don't know about Linux.
GitHub Personal Access Tokens, but this is not org. friendly.
Review the other auth methods available to see if there's one that fits your requirements (e.g. AWS).
I set up vault backed by a consul cluster. I secured it with https and am trying to use the cli on a separate machine to get and set secrets in the kv engine. I am using version 1.0.2 of both the CLI and Vault server.
I have logged in with the root token so I should have access to everything. I have also set my VAULT_ADDR appropriately.
Here is my request:
vault kv put secret/my-secret my-value=yea
Here is the response:
Error making API request.
URL: GET https://{my-vault-address}/v1/sys/internal/ui/mounts/secret/my-secret
Code: 403. Errors:
* preflight capability check returned 403, please ensure client's policies grant access to path "secret/my-secret/"
I don't understand what is happening here. I am able to set and read secrets in the kv engine no problem from the vault ui. What am I missing?
This was a result of me not reading documentation.
The request was failing because there was no secret engine mounted at that path.
You can check your secret engine paths by running vault secrets list -detailed
This showed that my kv secret engine was mapped to path kv not secret as I was trying.
Therefore running vault kv put kv/my-secret my-value=yea worked as expected.
You can enable secret engine for specific path
vault secrets enable -path=kv kv
https://www.vaultproject.io/intro/getting-started/secrets-engines
You need to update secret/my-secret to whichever path you mounted when you enable the kv secret engine.
For example, if you enable the secret engine like this:
vault secrets enable -version=2 kv-v2
You should mount to kv-v2 instead of secret
vault kv put kv-v2/my-secret my-value=yea
I can't understand how secrets list works.
I have policy with path permission.
path "sys/mounts/*" {
capabilities = ["create", "read", "update", "delete", "list","sudo"]
}
I can run enable and disable flags
$ vault secrets enable -path=Test kv
Success! Enabled the kv secrets engine at: Test/
$ vault secrets disable Test
Success! Disabled the secrets engine (if it existed) at: Test/
But I can't run list or move
vault secrets list
Error listing secrets engines: Error making API request.
URL: GET http://localhost:8200/v1/sys/mounts
Code: 403. Errors:
* permission denied
vault secrets move Test Test2
Error moving secrets engine Test/ to Test2/: Error making API request.
URL: POST http://localhost/v1/sys/remount
Code: 403. Errors:
* permission denied
Its not a file system permission issue, after change admin-token to root-token everything work fine. So anyone can explain me this behavior ?
Try:
path "sys/mounts" {
capabilities = ["read"]
}
The command are performed on sys/mounts, not sys/mounts/*
Just to add. All the secret backend paths which can be allowed or disallowed for a particular token are listed on this web page https://www.vaultproject.io/api/system/index.html
When you get permission denied to any of the URL. You need to search that URL on this page. And you will get the whole information about that system backend. e.g. You can search both "sys/remount" and "sys/mounts"
I'm trying to get login/pass authentication working on Vault.
When I try the method given in the API documentation here: https://www.vaultproject.io/api/auth/userpass/index.html#login
I get this error:
$ curl --request POST --data #payload.json https://<myurl>:8200/v1/auth/userpass/login/<mylogin> -k
{"errors":["missing client token"]}
And I can't find information on this error. It makes me wonder what happens, because I want to authenticate with login/pass to get the token, so that's just normal to not have it.
Here is the content of the payload.json:
{
"password": "foo"
}
Is there any way to login with username/password? This is the only fallback method I have when the user does not know its token.
Thanks!
OK, so I figured it out by trials.
So the userpass AUTH was indeed disabled. I have to use LDAP auth. With the Vault-UI that is installed, I managed to find the URL to authenticate. If was the following : https://******:8200/v1/auth/<ldap>/login/<user>
And that way it's working.
Unfortunately, it does not help in the end. The idea was to synchronize Vault data locally, but the Vault API is really not built for that kind of access. It requires a LOT of requests, and end up being very slow for a few secrets synchronized.
Make sure you are logging in under the correct namespace. You will get this error if your authentication method is enabled under something other than the default namespace that your CLI tool is using.
You can specify the namespace with the -ns=my/namespace/ parameter or the VAULT_NAMESPACE environment variable.
For example, if your namespace is "desserts/icecream"
vault login -ns=desserts/icecream/ -method=userpass username=ian
# OR
export VAULT_NAMESPACE=desserts/icecream/
vault login -method=userpass username=ian
In my case, i was not setting the vault token to the right environment variable.
you have to set the value to VAULT_TOKEN so that it uses it in subsequent request my env variable was Vault_Token and due to this it was always saying missing client token.
By default, Vault checks for this environment variable to find the token.
vault kv get --field "ACCESS_KEY_ID" secret/my-secret