What if :
I want to replicate the data contained in dev vault to my local vault?
I want to export my local vault data to dev vault up to prod vault?
What is the ideal way of doing this securely and versioned? I am thinking of some export/import mechanism.. Fairly new to vault
Medusa is a open source cli tool that does exactly what you need.
The tool can handle a full tree structure in both import and export. It also supports end to end encryption of your secrets between export and import between Vault instances so that your secrets are always secure.
https://github.com/jonasvinther/medusa
export VAULT_ADDR=https://192.168.86.41:8201
export VAULT_SKIP_VERIFY=true
export VAULT_TOKEN=00000000-0000-0000-0000-000000000000
./medusa export kv/path/to/secret --format="yaml" --output="my-secrets.txt"
./medusa import kv/path/to/new/secret ./my-secrets.txt
Vault stores everything in the backends and encrypts them with the unseal keys.
If you wanted, you could copy the data else where, and then 'import' them to the next environment (and by copy, i mean db dump if you are using a database to store stuff, copy s3 buckets if you are using s3, etc).
That would require downtime as you would need to seal your cluster to make sure all the writes happen before you copy your data.
If you want something more automatic, you could upgrade to the enterprise version and use replication - there are various different replication options.
Related
I am currently working on a solution that centralizes connections to external databases.
For that, I initialize a database with flyway and connect to other postgresql sources with the postrgres_fdw extension to create my table projections (with foreign tables).
Everything works fine if I put my credentials, hosts, etc... in my application.yml configuration file (we are on spring), putting these values in placeholders and reusing them in my sql migration scripts. But we want to fetch this data from vault, where we store all this data.
However, although I have followed the flyway documentation on vault integration (https://flywaydb.org/blog/integrating-vault-to-secure-flyway-parameters), I cannot achieve my goal. I've tried putting my placeholders in vault (my secrets are of the form flyway.placeholders.[...]=), and connecting to my instance via my data in my configuration file
spring:
flyway:
vault:
url: https://localhost:8200/v1/
token: root
secrets: secret/data/...
but without success. Has anyone had this problem before? Is it possible to retrieve any secret value via placeholders to use in sql scripts or do we have to go through the java API to have a little more flexibility?
Is there a way to export vault secret data from one vault instance, and then import to another vault instance?
Example:
Export secrets from the source instance with the path secret/vault/path and import to the destination empty vault instance.
I am not aware of a 'native' way to do this. You will need to iterate over the requested secrets and export them to file (and import them in the same way).
You can try to use one of the following projects that attempt to do it:
Vault backup
Vault backup migrator
Medusa is a open source cli tool that can export and import your Vault secrets on different Vault instances.
The tool can handle a full tree structure in both import and export. It also supports end to end encryption of your secrets between export and import between Vault instances so that your secrets are always secure.
https://github.com/jonasvinther/medusa
export VAULT_ADDR=https://192.168.86.41:8201
export VAULT_SKIP_VERIFY=true
export VAULT_TOKEN=00000000-0000-0000-0000-000000000000
./medusa export kv/path/to/secret --format="yaml" --output="my-secrets.txt"
./medusa import kv/path/to/new/secret ./my-secrets.txt
Is it possible to use AWS KMS and a tool like credstash without the use of EC2 or equivalent or does it rely solely on IAM roles?
I've got a server elsewhere where I am testing some things out and ultimately I will be looking at migrating an app to EC2 etc. to make use of scaling. But for now whilst I'm setting up my deployment pipeline etc. I wondered if it was still possible to make use of KMS on my non-aws provisioned server?
The only possible way I can think of is by installing the AWS CLI tools on the server in question. Does this sounds like the right approach?
What #Viccari said is correct (in the comments). In terms of what you want to do (store passwords), the AWS Parameter Store would be a good fit for you. See https://docs.aws.amazon.com/systems-manager/latest/userguide/systems-manager-paramstore.html for more information. The guide explicitly calls out your use-case:
Parameter Store offers the following benefits and features.
Use a secure, scalable, hosted secrets management service (No servers to manage).
In the end, if you end up using Parameter Store or KMS, you will need some sort of credentials stored somewhere to grab an AWS STS token to use to call the underlying AWS services. If working outside of AWS EC2, you will need the AWS Access Key and AWS Secret Key from an IAM user. If you are in EC2, the IAM instance role will magically provide you the credentials and use that role to call those AWS services. The AWS SDK does this for you behind the scenes.
But, as you state, you don't want to run this in EC2 (to save money, or other reasons). The quickest way to store these credentials is to have them in a un-tracked file (added to your .gitignore) you can source from as environment variables, which your program will then read. This allows you to do local testing, and easily run it in EC2
with zero code changes. See https://docs.aws.amazon.com/cli/latest/userguide/cli-environment.html for what variables to set. Note that this doc talks about the CLI; the SDK's follow the same behavior.
Spring Boot - 2.0.0.M3
Spring cloud - Finchley.M1
I want to know if someone is using Spring Cloud config server with both vault and git support in a production setup using Database storage backend.
I have evaluated Spring cloud config using vault and contemplating whether to go for Oracle JCE to encrypt username/pwd or Vault and seek suggestions on the same. we are working on Springboot/microservices.
Following are my findings -
Vault will introduce an additional layer and thus will introduce additional usecases of security, auditing while communicating with Vault.
Spring cloud Config actuator endpoints are broken for the milestone release at this point for generation of encrypted values and /encrypt /decrypt may not work if we go for Oracle JCE support so we generate encrypted values through stable versions.
We do not wish to use consul server and are trying to use Cassandra as Storage backend.
I used Vault Authentication backend using AppRole and generated a Token (different from root token as it's unsafe to use the same) with read permissions. However, Spring Cloud config at the moment support only Token based authentication from client side. That means we first generate token from Vault and then pass it as commandline/env variable.
Some additional points of concern are expiry of token (though we can have non-expiry token not sure about pros/cons), restarts, safety issues, instantiating new microservices. There is no provision of dynamic tokens/authentication at cloud config side.
For milestone release i found that the client side encryption/decryption is not working as of now using recommended inclusion of RSA jar. Here is the ticket i opened.
https://github.com/spring-cloud/spring-cloud-config/issues/805#issuecomment-332491536
These are some of my observations, please share your thoughts if there is any case study/whitepaper that address spring cloud config vault usecases, setup and challenges for production micro-services environment.
Thanks
Thanks for reaching out to me. One think I would state is that the App Role backend utilizes two distinct tokens, and indeed spring-cloud-config-vault does indeed support this functionality, see: http://cloud.spring.io/spring-cloud-vault/single/spring-cloud-vault.html#_approle_authentication. I leverage vault in the same way I leverage config server, as per the documentation. I don't encrypt any values in my config, I just don't put them there. I put the secret values in vault and let it serve config. As long as keys don't collide, you don't have to mess with anything, otherwise you may need adjust the priority so vault wins, again see the documentation that I pointed to above. I wouldn't mess with encryption/decryption in spring-cloud-config personally. Because you have to check the keys into SCM or distribute them to your teams for local development, you lose the value of having these keys IMO.
Thanks Spring Cloud vault does support but not Spring cloud config with Vault. Only way seems to be passing X-Config-token from Microservice to Config Server. We are bit skeptical with this part of generating tokens manually or through script. Especially with containerization and when new MS instances will be spawn. Not sure about this approach especially in production setup.
I have a case where the RunDeck scripts do need some credentials in order to run. Obviously we do not want to store these in the job definitions because these are visible and also stored in SCM.
While I was able to use the Key Storage vault to put these secrets in, I was not able to find a way to access them from the job itself.
Rundeck 2.6.2 (released 2015-12-02) allows you to specify key storage secrets as default values for secure job options. See Secure Options using Key Storage