How to retrieve secret data from vault API using AppRole? - rest

My HashiCorp vault instance is runnning properly on CentOS7. I enabled AppRole authentication, created a policy and a role, enabled secret engine and created a secret for a client application.
I can retrieve the secret data using root CLI but I can't figure out how to get secret data from HTTP API with my application role using curl. I tried a few endpoint combinations without success. Retrieving the client token works, but I can't get secret data itself.
I wonder if the API endpoint is correct or if there is another setting in play.
Authentication method
vault auth enable approle
Policy
# File: my_app /etc/vault/my_app.hcl
path "kv/data/foo/*" {
capabilities = ["read", "list"]
}
# Command line
vault policy write my_app /etc/vault/my_app.hcl
Role
vault write auth/approle/role/my_app policies="my_app"
Secret creation
vault kv put kv/data/foo/user#domain.tld password=1234
API call token request
curl --request POST --data '{"role_id": "xxxxxxxxxxxxxxxxx", "secret_id": "xxxxxxxxxxxxxxxxxxxx"}' http://127.0.0.1:8200/v1/auth/approle/login | jq
Result: Token is properly retrieved
API call for secret data request
export VAULT_CLIENT_TOKEN=XXXXXXX
curl --header "X-Vault-Token: $VAULT_CLIENT_TOKEN" --request GET "http://127.0.0.1:8200/v1/kv/data/foo/user#domain.tld"
Result : No secret data retrieved
Output:
{"errors":[]}
CLI call for secret data
vault kv get -field=password kv/data/foo/user#domain.tld
Output:
1234
Global settings
vault secrets list
Path Type Accessor Description
---- ---- -------- -----------
cubbyhole/ cubbyhole cubbyhole_xxxxxxxx per-token private secret storage
identity/ identity identity_xxxxxxxx identity store
kv/ kv kv_xxxxxxxx n/a
sys/ system system_xxxxxxxx system endpoints used for control, policy and debugging

Related

Need to connect service principal with certificate to authorize using pyspark

I want to connect service principal with certificate to authorize using pyspark. I could see the code in scala in this link - https://github.com/Azure/azure-event-hubs-spark/blob/master/docs/use-aad-authentication-to-connect-eventhubs.md
I have client _id and tenant_id and certificate details. Could some please share me the code in pyspark for same?
You add the Azure AD service principal to the Azure Databricks workspace using the SCIM API 2.0. Authentication using Pyspark isn't available.
To authenticate using service principal, you need to follow below steps:
As you already have clientID and tenantID, so the service principal ID already created.
Create the Azure Databricks personal access token
You’ll use an Azure Databricks personal access token (PAT) to authenticate against the Databricks REST API. To create a PAT that can be used to make API requests:
Go to your Azure Databricks workspace.
Click the user icon in the top-right corner of the screen and click User Settings.
Click Access Tokens > Generate New Token.
Copy and save the token value.
Add the service principal to the Azure Databricks workspace
You add the Azure AD service principal to a workspace using the SCIM API 2.0. You must also give the service principal permission to launch automated job clusters. You can grant this through the allow-cluster-create permission. Open a terminal and run the following command to add the service principal and grant the required permissions:
curl -X POST 'https://<per-workspace-url>/api/2.0/preview/scim/v2/ServicePrincipals' \
--header 'Content-Type: application/scim+json' \
--header 'Authorization: Bearer <personal-access-token>' \
--data-raw '{
"schemas":[
"urn:ietf:params:scim:schemas:core:2.0:ServicePrincipal"
],
"applicationId":"<application-id>",
"displayName": "test-sp",
"entitlements":[
{
"value":"allow-cluster-create"
}
]
}'
Replace with the unique per-workspace URL for your Azure Databricks workspace.
Replace with the Azure Databricks personal access token.
Replace with the Application (client) ID for the Azure AD application registration.
Create an Azure Key Vault-backed secret scope in Azure Databricks
Secret scopes provide secure storage and management of secrets. You’ll store the secret associated with the service principal in a secret scope. You can store secrets in a Azure Databricks secret scope or an Azure Key Vault-backed secret scope. These instructions describe the Azure Key Vault-backed option:
Create an Azure Key Vault instance in the Azure portal.
Create the Azure Databricks secret scope backed by the Azure Key Vault instance.
Step 1: Create an Azure Key Vault instance
In the Azure portal, select Key Vaults > + Add and give the key vault a name.
Click Review + create.
After validation completes, click Create .
After creating the key vault, go to the Properties page for the new key vault.
Copy and save the Vault URI and Resource ID.
Step 2: Create An Azure Key Vault-backed secret scope
Azure Databricks resources can reference secrets stored in an Azure key vault by creating a Key Vault-backed secret scope. To create the Azure Databricks secret scope:
Go to the Azure Databricks Create Secret Scope page at https:///#secrets/createScope. Replace per-workspace-url with the unique per-workspace URL for your Azure Databricks workspace.
Enter a Scope Name.
Enter the Vault URI and Resource ID values for the Azure key vault you created in Step 1: Create an Azure Key Vault instance.
Click Create.
Save the client secret in Azure Key Vault
In the Azure portal, go to the Key vaults service.
Select the key vault created in Step 1: Create an Azure Key Vault instance.
Under Settings > Secrets, click Generate/Import.
Select the Manual upload option and enter the client secret in the Value field.
Click Create.
Grant the service principal read access to the secret scope
You’ve created a secret scope and stored the service principal’s client secret in that scope. Now you’ll give the service principal access to read the secret from the secret scope.
Open a terminal and run the following command:
curl -X POST 'https://<per-workspace-url/api/2.0/secrets/acls/put' \
--header 'Authorization: Bearer <personal-access-token>' \
--header 'Content-Type: application/json' \
--data-raw '{
"scope": "<scope-name>",
"principal": "<application-id>",
"permission": "READ"
}'
Replace with the unique per-workspace URL for your Azure Databricks workspace.
Replace with the Azure Databricks personal access token.
Replace with the name of the Azure Databricks secret scope that contains the client secret.
Replace with the Application (client) ID for the Azure AD application registration.
Create a job in Azure Databricks and configure the cluster to read secrets from the secret scope
You’re now ready to create a job that can run as the new service principal. You’ll use a notebook created in the Azure Databricks UI and add the configuration to allow the job cluster to retrieve the service principal’s secret.
Go to your Azure Databricks landing page and select Create Blank Notebook. Give your notebook a name and select SQL as the default language.
Enter SELECT 1 in the first cell of the notebook. This is a simple command that just displays 1 if it succeeds. If you have granted your service principal access to particular files or paths in Azure Data Lake Storage Gen 2, you can read from those paths instead.
Go to Workflows and click the + Create Job button. Give the job a name, click Select Notebook, and select the notebook you just created.
Click Edit next to the Cluster information.
On the Configure Cluster page, click Advanced Options.
On the Spark tab, enter the following Spark Config:
fs.azure.account.auth.type.acmeadls.dfs.core.windows.net OAuth
fs.azure.account.oauth.provider.type.acmeadls.dfs.core.windows.net org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider
fs.azure.account.oauth2.client.id.acmeadls.dfs.core.windows.net <application-id>
fs.azure.account.oauth2.client.secret.acmeadls.dfs.core.windows.net {{secrets/<secret-scope-name>/<secret-name>}}
fs.azure.account.oauth2.client.endpoint.acmeadls.dfs.core.windows.net https://login.microsoftonline.com/<directory-id>/oauth2/token
Replace with the name of the Azure Databricks secret scope that contains the client secret.
Replace with the Application (client) ID for the Azure AD application registration.
Replace with the name associated with the client secret value in the secret scope.
Replace with the Directory (tenant) ID for the Azure AD application registration.
Transfer ownership of the job to the service principal
A job can have exactly one owner, so you’ll need to transfer ownership of the job from yourself to the service principal. To ensure that other users can manage the job, you can also grant Can Manage permissions to a group. In this example, we use the Permissions API to set these permissions.
Open a terminal and run the following command:
curl -X PUT 'https://<per-workspace-url>/api/2.0/permissions/jobs/<job-id>' \
--header 'Authorization: Bearer <personal-access-token>' \
--header 'Content-Type: application/json' \
--data-raw '{
"access_control_list": [
{
"service_principal_name": "<application-id>",
"permission_level": "IS_OWNER"
},
{
"group_name": "admins",
"permission_level": "CAN_MANAGE"
}
]
}'
Replace with the unique per-workspace URL for your Azure Databricks workspace.
Replace with the Azure Databricks personal access token.
Replace with the Application (client) ID for the Azure AD application registration.
The job will also need read permissions to the notebook. Run the following command to grant the required permissions:
curl -X PUT 'https://<per-workspace-url>/api/2.0/permissions/notebooks/<notebook-id>' \
--header 'Authorization: Bearer <personal-access-token>' \
--header 'Content-Type: application/json' \
--data-raw '{
"access_control_list": [
{
"service_principal_name": "<application-id>",
"permission_level": "CAN_READ"
}
]
}'
Replace with the unique per-workspace URL for your Azure Databricks workspace.
Replace with the ID of the notebook associated with the job. To find the ID, go to the notebook in the Azure Databricks workspace and look for the numeric ID that follows notebook/ in the notebook’s URL.
Replace with the Azure Databricks personal access token.
Replace with the Application (client) ID for the Azure AD application registration.
You can now test the job. You run jobs with a service principal the same way you run jobs as a user, either through the UI, API, or CLI.

How can I represent authorization bearer token in YAML

I have generated the access token and placed in below mentioned mount path and this token need to be included in the Authorization header when making a request against the retrieve secret endpoint.
How can we achieve it in yaml scripting
volumeMounts:
- mountPath: /run/test
name: conjur-access-token
readOnly: true
This question is referencing CyberArk's Conjur Secrets Manager's Kubernetes authenticator. It uses a sidecar authenticator client to keep an authenticated session token for Conjur's API refreshed in a shared volume mount with an application container running within a Kubernetes pod. This allows the application container to request secret values Just-in-Time (JiT) from the Conjur API with a single API call.
There is a file located at /run/test/conjur-access-token (according to the manifest snippet you provided) that contains the authenticated session token to use to connect to the Conjur API. Your application container needs to read /run/test/conjur-access-token and use it in the Authorization header as a Token-based authorization. To use curl, this would look like:
curl -H "Authorization: Token token='$(cat /run/test/conjur-access-token)'" https://conjur.example.com/secrets/myorg/variable/prod%2Fdb%2Fpassword
Where:
/run/test/conjur-access-token is the path to the shared volume mount of the application container and sidecar Kubernetes authenticator client.
conjur.example.com is the Base URL for your Conjur Follower in the Kubernetes cluster (or outside, if that's the deployment method).
myorg is the organzation account configured at the time of Conjur deployment and configuration.
prod%2Fdb%2Fpassword is the URLified secret variable path in Conjur. This would be referenced otherwise as prod/db/password but since forward-slashes are part of URL/URI, we need this URLified to %2F.
If the file containing your token in the mount path is called token, then you can simply do (assuming that you use curl):
curl -H "Authorization: Bearer $(cat /run/test/token)" ...

Retrieval of secrets in Azure App Service from Hashicorp Vault using Managed Identity | Missing Role - Error

Hashicorp Vault is the native product of our organization and is a widely used and recommended approach for storing all the key-value pairs or any secrets. Any applications that are deployed on Azure too must store/retrieve the token from Hashicorp Vault and not from the Azure Key Vault. I provided this information just to add a bit of background to the requirement.
Now coming to the actual problem, I deployed the dotnet application on Azure App Service, enable the system-managed identity, and was able to successfully retrieve the JWT token.
As per the flow which I understood by reading the documentation, it says, first retrieve the application token deployed on Azure having System Managed Identity enabled. Once this is done, pass this token for validation to Vault which gets it validated using OIDC from AAD. On successful validation, I will be given back the Vault token which can be used to fetch the secrets from Vault.
To perform these steps configuration is required at the Vault side, for which, I performed all the below steps on the vault server installed on my windows local machine:-
Command line operation
Start the Vault server
Open the other command prompt and set the environment variables set
VAULT_ADDR=http://127.0.0.1:8200 set
VAULT_TOKEN=s.iDdVbLKPCzmqF2z0RiXPMxLk
vault auth enable jwt
vault write auth/jwt/config
oidc_discovery_url=https://sts.windows.net/4a95f16f-35ba-4a52-9cb3-7f300cdc0c60/
bound_issuer=https://sts.windows.net/4a95f16f-35ba-4a52-9cb3-7f300cdc0c60/
vault read auth/jwt/config
Policy associated with the sqlconnection:-
create a role (webapp-role) by using the command
curl --header “X-Vault-Token: %VAULT_TOKEN%” --insecure --request POST
--data #C:\Users\48013\source\repos\HashVaultAzure\Vault-files\payload.json
%VAULT_ADDR%/v1/auth/jwt/role/webapp-role
–payload.json { “bound_audiences”: “https://management.azure.com/”,
“bound_claims”: { “idp”:
“https://sts.windows.net/4a95f16f-35ba-4a52-9cb3-7f300cdc0c60/”,
“oid”: “8d2b99fb-f4f4-4afb-9ee3-276891f40a65”, “tid”:
“4a95f16f-35ba-4a52-9cb3-7f300cdc0c60/” }, “bound_subject”:
“8d2b99fb-f4f4-4afb-9ee3-276891f40a65”, “claim_mappings”: { “appid”:
“application_id”, “xms_mirid”: “resource_id” }, “policies”:
[“sqlconnection”], “role_type”: “jwt”, “token_bound_cidrs”:
[“10.0.0.0/16”], “token_max_ttl”: “24h”, “user_claim”: “sub” }
Vault read auth/jwt/role/webapp-role
Run the command below with the JWT token retrieved from the application (having the managed identity enabled) deployed on Azure
AAD and pass it as “your_jwt”. This command should return the vault
token as shown in the link https://www.vaultproject.io/docs/auth/jwt
curl --request POST --data '{"jwt": "your_jwt", "role":
"webapp-role"}' http://127.0.0.1:8200/v1/auth/jwt/login
At this point I receive an error – “Missing Role”,
I am stuck here and not able to find any solution.
Expected response should be a vault token/client_token as shown:-
JWT Token decoded information
{
"aud": "https://management.azure.com",
"iss": "https://sts.windows.net/4a95f16f-35ba-4a52-9cb3-7f300cdc0c60/",
"iat": 1631172032,
"nbf": 1631172032,
"exp": 1631258732,
"aio": "E2ZgYNBN4JVfle92Tsl1b8m8pc9jAA==",
"appid": "cf5c734c-a4fd-4d85-8049-53de46db4ec0",
"appidacr": "2",
"idp": "https://sts.windows.net/4a95f16f-35ba-4a52-9cb3-7f300cdc0c60/",
"oid": "8d2b99fb-f4f4-4afb-9ee3-276891f40a65",
"rh": "0.AVMAb_GVSro1Ukqcs38wDNwMYExzXM_9pIVNgElT3kbbTsBTAAA.",
"sub": "8d2b99fb-f4f4-4afb-9ee3-276891f40a65",
"tid": "4a95f16f-35ba-4a52-9cb3-7f300cdc0c60",
"uti": "LDjkUZdlKUS4paEleUUFAA",
"ver": "1.0",
"xms_mirid": "/subscriptions/0edeaa4a-d371-4fa8-acbd-3675861b0ac8/resourcegroups/AzureAADResource/providers/Microsoft.Web/sites/hashvault-test",
"xms_tcdt": "1600006540"
}
The issue was with the missing configuration both at the Azure Cloud and Vault side.
These were the addition steps done further to make it work.
Create an Azure SPN (which is equal to creating an app registration with client secret)
az ad sp create-for-rbac --name "Hashicorp Vault Prod AzureSPN"
--skip-assignment Assign as Reader on subscription
Create Vault config
vault auth enable azure vault write auth/jwt/config
tenant_id=lg240e12-76g1-748b-cd9c-je6f29562476
resource=https://management.azure.com/ client_id=34906a49-
9a8f-462b-9d68-33ae40hgf8ug client_secret=123456ABCDEF

How to know the validity of a vault token that is being used for connecting to vault?

Currently, I am connecting to a corporate vault service where I am using a vault token and passing it through below header in my spring cloud config service where properties of all microservices are kept.
curl -X "GET" "http://localhost:8080/my-client-microservice/dev" -H "X-Config-Token: s.myvaulttoken"
where http://localhost:8080 is my spring cloud config service and s.myvaulttoken is my vault token. This is working absolutely fine.
I want to know the validity of this token. What I have read the documentation that token can be of two type: service or batch. I want to know whether this token can be used infinitely (as root tokens validity is infinite).
Since the client microservices require the vault token, I want to figure out the way to know the validity of a token. Can you guys help me to tell more about this?
I followed this link: https://learn.hashicorp.com/vault/getting-started/authentication
Every non-root token has a time-to-live (TTL) associated with it.
For example:
with a root token, the ttl is 0
vault token lookup -format json | jq .data.ttl
0
with a regular user, the ttl is non-zero
VAULT_TOKEN=$(vault token create -policy default -field token) vault token
lookup -format json | jq .data.ttl
2764799
This check is possible through the API as well.

Hashicorp Vault cli return 403 when trying to use kv

I set up vault backed by a consul cluster. I secured it with https and am trying to use the cli on a separate machine to get and set secrets in the kv engine. I am using version 1.0.2 of both the CLI and Vault server.
I have logged in with the root token so I should have access to everything. I have also set my VAULT_ADDR appropriately.
Here is my request:
vault kv put secret/my-secret my-value=yea
Here is the response:
Error making API request.
URL: GET https://{my-vault-address}/v1/sys/internal/ui/mounts/secret/my-secret
Code: 403. Errors:
* preflight capability check returned 403, please ensure client's policies grant access to path "secret/my-secret/"
I don't understand what is happening here. I am able to set and read secrets in the kv engine no problem from the vault ui. What am I missing?
This was a result of me not reading documentation.
The request was failing because there was no secret engine mounted at that path.
You can check your secret engine paths by running vault secrets list -detailed
This showed that my kv secret engine was mapped to path kv not secret as I was trying.
Therefore running vault kv put kv/my-secret my-value=yea worked as expected.
You can enable secret engine for specific path
vault secrets enable -path=kv kv
https://www.vaultproject.io/intro/getting-started/secrets-engines
You need to update secret/my-secret to whichever path you mounted when you enable the kv secret engine.
For example, if you enable the secret engine like this:
vault secrets enable -version=2 kv-v2
You should mount to kv-v2 instead of secret
vault kv put kv-v2/my-secret my-value=yea