I am currently working on a solution that centralizes connections to external databases.
For that, I initialize a database with flyway and connect to other postgresql sources with the postrgres_fdw extension to create my table projections (with foreign tables).
Everything works fine if I put my credentials, hosts, etc... in my application.yml configuration file (we are on spring), putting these values in placeholders and reusing them in my sql migration scripts. But we want to fetch this data from vault, where we store all this data.
However, although I have followed the flyway documentation on vault integration (https://flywaydb.org/blog/integrating-vault-to-secure-flyway-parameters), I cannot achieve my goal. I've tried putting my placeholders in vault (my secrets are of the form flyway.placeholders.[...]=), and connecting to my instance via my data in my configuration file
spring:
flyway:
vault:
url: https://localhost:8200/v1/
token: root
secrets: secret/data/...
but without success. Has anyone had this problem before? Is it possible to retrieve any secret value via placeholders to use in sql scripts or do we have to go through the java API to have a little more flexibility?
Related
I have research it for several days but could not find a definitive answer for it.
My use case is I have a PostgreSQL database hosted on AWS EKS and I want to expose it using GraphQL, which generally leads to AWS AppSync.
I understand that AppSync can be auto imported from DynamoDB only however I am not using that. From several articles it suggested a Lambda Function to connect between AppSync and PostgreSQL. Which I tried, but I need the two feature which is:
Auto Generated Schema
Hot reload of the schema whenever there is changes in the database
Currently I am using PostGraphile for these two features however I am not sure AppSync can be connected to that, as I understand we need to push the schema generated by PostGraphile to AppSync, but I need it to be automatic.
Eg: I create a new table in PostgreSQL -> PostGraphile Lambda Function reload the schema -> Reflects on AppSync schema automatically -> User call new table via AppSync
Can this flow be achieved? Is there anything I can use as reference?
Thank you!
If anyone still wondering, I found a resource from AWS seems to be able to achieve this with some tweaks and changes:
https://github.com/aws-samples/appsync-with-postgraphile-rds
I deploy a web app to a local cloudfoundry environment. As a database service for my DEV environment I have chosen a Marketplace service google-cloudsql-postgres with the plan postgres-db-f1-micro. Using the Web UI I created an instance with the name myapp-test-database and mentioned it in the CF Manifest:
applications:
- name: myapp-test
services:
- myapp-test-database
At first, all is fine. I can even redeploy the existing artifact. However, when I build a new version of my app and push it to CF, the injected credentials are updated and the app can no longer access the tables:
PSQLException: ERROR: permission denied for table
The tables are still there, but they're owned by the previous user. They were automatically created by the ORM in the public schema.
While the -OLD application still exists I can retrieve the old username/password from the CF Web UI or $VCAP_SERVICES and drop the tables.
Is this all because of Rolling App Deployments? But then there should be a lot of complaints.
If you are strictly doing a cf push (or restart/restage), the broker isn't involved (Cloud Controller doesn't talk to it), and service credentials won't change.
The only action through cf commands that can modify your credentials is doing an unbind followed by a bind. Many, but not all, service brokers will throw away credentials on unbind and provide new, unique credentials for a bind. This is often desirable so that you can rotate credentials if credentials are compromised.
Where this can be a problem is if you have custom scripts or cf cli plugins to implement rolling deployments. Most tools like this will use two separate application instances, which means you'll have two separate bindings and two separate sets of credentials.
If you must have one set of credentials you can use a service key to work around this. Service keys are like bindings but not associated with an application in CloudFoundry.
The downside of the service key is that it's not automatically exposed to your application, like a binding, through $VCAP_SERVICES. To workaround this, you can pass the service key creds into a user-provided service and then bind that to your application, or you can pass them into your application through other environment variables, like DB_URL.
The other option is to switch away from using scripts and cf cli plugins for blue/green deployment and to use the support that is now built into Cloud Foundry. With cf cli version 7+, cf push has a --strategy option which can be set to rolling to perform a rolling deployment. This does not create multiple application instances and so there would only ever exist one service binding and one set of credentials.
Request a static username using the extra bind parameter "username":
cf bind-service my-app-test-CANDIDATE myapp-test-database -c "{\"username\":\"myuser\"}"
With cf7+ it's possible to add parameters to the manifest:
applications:
- name: myapp-test
services:
- name: myapp-test-database
parameters: { "username": "myuser" }
https://docs.cloudfoundry.org/devguide/services/application-binding.html#arbitrary-params-binding
Note: Arbitrary parameters are not supported in app manifests in cf CLI v6.x. Arbitrary parameters are supported in app manifests in cf CLI v7.0 and later.
However, I can't find the new syntax here: https://docs.cloudfoundry.org/devguide/deploy-apps/manifest-attributes.html#services-block . The syntax I use comes from some other SO question.
I’ve spent several hours scouring forums about how I can avoid the dreaded Your token is invalid. It might have expired or you might be using a token from a different project., but to no avail.
My setup:
Using prisma generate to make a Prisma client and connect it with
graphql-yoga Prisma service running on Heroku
Prisma service is pointed to an Amazon RDS instance
This setup works when I’m not using the secret property in my prisma.yml. However, when I add something like secret: mysecret and prisma deploy the service, then use something like prisma playground to play with the service, I get the above error. This also happens when I manually generate a token using prisma token and use it in an HTTP Authorization header.
What am I missing to make this work?
Make sure that you add the secret to your environment variables. E.g. via a .env file:
PRISMA_SECRET="mysecret"
When running prisma cli commands make sure to first set the environment variables. E.g. via something like dotenv:
dotenv -- prisma admin
For more information check out: https://www.prisma.io/docs/prisma-server/authentication-and-security-kke4/#prisma-services
Spring Boot - 2.0.0.M3
Spring cloud - Finchley.M1
I want to know if someone is using Spring Cloud config server with both vault and git support in a production setup using Database storage backend.
I have evaluated Spring cloud config using vault and contemplating whether to go for Oracle JCE to encrypt username/pwd or Vault and seek suggestions on the same. we are working on Springboot/microservices.
Following are my findings -
Vault will introduce an additional layer and thus will introduce additional usecases of security, auditing while communicating with Vault.
Spring cloud Config actuator endpoints are broken for the milestone release at this point for generation of encrypted values and /encrypt /decrypt may not work if we go for Oracle JCE support so we generate encrypted values through stable versions.
We do not wish to use consul server and are trying to use Cassandra as Storage backend.
I used Vault Authentication backend using AppRole and generated a Token (different from root token as it's unsafe to use the same) with read permissions. However, Spring Cloud config at the moment support only Token based authentication from client side. That means we first generate token from Vault and then pass it as commandline/env variable.
Some additional points of concern are expiry of token (though we can have non-expiry token not sure about pros/cons), restarts, safety issues, instantiating new microservices. There is no provision of dynamic tokens/authentication at cloud config side.
For milestone release i found that the client side encryption/decryption is not working as of now using recommended inclusion of RSA jar. Here is the ticket i opened.
https://github.com/spring-cloud/spring-cloud-config/issues/805#issuecomment-332491536
These are some of my observations, please share your thoughts if there is any case study/whitepaper that address spring cloud config vault usecases, setup and challenges for production micro-services environment.
Thanks
Thanks for reaching out to me. One think I would state is that the App Role backend utilizes two distinct tokens, and indeed spring-cloud-config-vault does indeed support this functionality, see: http://cloud.spring.io/spring-cloud-vault/single/spring-cloud-vault.html#_approle_authentication. I leverage vault in the same way I leverage config server, as per the documentation. I don't encrypt any values in my config, I just don't put them there. I put the secret values in vault and let it serve config. As long as keys don't collide, you don't have to mess with anything, otherwise you may need adjust the priority so vault wins, again see the documentation that I pointed to above. I wouldn't mess with encryption/decryption in spring-cloud-config personally. Because you have to check the keys into SCM or distribute them to your teams for local development, you lose the value of having these keys IMO.
Thanks Spring Cloud vault does support but not Spring cloud config with Vault. Only way seems to be passing X-Config-token from Microservice to Config Server. We are bit skeptical with this part of generating tokens manually or through script. Especially with containerization and when new MS instances will be spawn. Not sure about this approach especially in production setup.
I want to keep my smtp login information private. And its a hassle to edit application-prod.yml every time I have to deploy to production.
What is the correct method to avoid storing sensitive details in application-prod.yml on github ?
Many different ways depending on where and how you deploy to:
Don't store application-prod.yml in git and don't package it in your jar
Don't store secrets in application-prod.yml, use environment variables or command line options. See Spring Boot doc.
Encrypt secrets using git-crypt in application-prod.yml
Store secrets in external Spring Cloud Config Server (e.g. JHipster Registry) or HashiCorp Vault
and many other ways...