In my current project I'm working with StreamSets and I would like to use Hashicorp Vault as my credentials store, however I'm not able to use credential:get() function wherever I want to. E.g. in Shared Access Key in Azure IoT Hub Producer block. I know that I could use Runtime Properties but I don't think it solves my problem.
Am I missing something or I can use credential:get() only in fields marked with a key icon?
You can only use credential:get() in fields marked with a key icon. This is by design, to minimize the chance of leaking credentials. For example, if credential:get() was allowed in URL parameters, a pipeline designer could send a request to a web server under their control to discover the credential. It may make sense to allow Shared Access Key to receive credentials. Please file an issue at https://issues.streamsets.com with your enhancement request.
Related
I am looking for the system that allows to create and store symmetric master keys in a safe manner. One of such systems is Amazon KMS, where I can create master private key per user and use it to encrypt some data (e.g. user's private keys).
But I need to support several platforms and so I have a question about Vault project (https://www.vaultproject.io). Is it appropriate tool for this task ?
I have found that Vault supports authorization functionality ( https://www.vaultproject.io/docs/auth/userpass.html) and I am wondering is it okay to use this API intensively and store 50k users or so ?
Said that, it looks like these services solve different problems, and Vault is not supposed to be used like Amazon KMS service. But I need to discuss this idea with someone in order to be completely sure.
Many thanks!
You may look into Cubbyhole backend for Vault. This backend works like a unique space for each token. Destroying the access token deletes all the data stored in its cubbyhole space.
From Cubbyhole authentication principles:
The cubbyhole backend is a simple filesystem abstraction similar to the generic backend (which is mounted by default at secret/) with one important twist: the entire filesystem is scoped to a single token and is completely inaccessible to any other token.
In other words, it does not matter, what policies attached to the token, but matter what the token is themselves. And only a single token can be used to set or retrieve values in its cubbyhole.
The Problem
MS Azure provides an extensive REST API. However, there is a significant amount of complexity when trying to get that API to work. From outdated and incomplete documentation to simple examples not working, performing what should be an easy task is instead nightmarish.
The Task
What are the exact, precise, detailed steps necessary to list the available VMs for someone who has logged in using Oauth2? For instance, this can be done using the azure-cli.
azure vm list
What are the steps to accomplish the same thing using REST and Oauth2?
The Requirements
The answer must not use Visual Studio, PowerShell, C#, an SDK, or any other such tool to accomplish this task. Only the portal is allowed for setup, and only Oauth2 is allowed for authorization, and only the REST API is allowed for actual information retrieval.
The answer must not simply link to external sources, although external references are encouraged for completeness.
The answer may assume the user has an Azure account. It must include information about creating the Oauth2 client, credentials, and any step necessary to get the appropriate token.
The answer must be detailed.
The answer must provide examples. Images, too are strongly encouraged.
The answer should include information about possible errors, their meanings, and what too look for to fix them.
First, we can find this rest API in azure resource portal. It is the same with Azure CLI command azure vm list.
I have tested it on my local with http request. here is my tested screenshot:
Request URL: https://management.azure.com/subscriptions/<subscription ID>/providers/Microsoft.Compute/virtualMachines?api-version=2016-03-30
Header:Authorization: bearer eyJ0eXAiOiJKV1QiLCJhbGciOiJSUzI.....
So It is very import if we get the access token. The following demonstrate us how to get the access token.
Get Token(POST):
Request URL: https://login.microsoftonline.com/<tenant id>/oauth2/token
Body: grant_type=client_credentials&client_id=<client id>&resource=https%3A%2F%2Fmanagement.core.windows.net%2F&client_secret=<client secret>
Here is my screenshot in fildder:
We need to get client id and client secret in azure ad application. For how to regist an application in Azure AD. Please have a look at this article: https://learn.microsoft.com/en-us/azure/azure-resource-manager/resource-group-create-service-principal-portal.
Please note:
1) we need to add "Windows Azure Service Management API" in portal "Required permissions" like the following screenshot:
2) We need assign "Contributor" for this service principal. click subscription-> Access Control-> click Add -> click "Select a Role" -> click Contributor->click Add User-> Find the application you created above-> click OK.
Overview
Making requests against the Azure Rest API is a bit more complicated than perhaps you would think at first glance. In particular, there are a number of esoteric and not-so-helpful error messages you may run into while getting the nobs tweaked just right.
Introduction and Terms
Setting up The Application
Getting the access_token
Making the API request
Common errors thus far
Introduction and Terms
One of the pieces of this process that can make it so confusing and difficult is the terminology. Until you understand that, knowing how to deal with errors is very difficult. We'll go over some of the more common ones here.
Subscription - This is basically what you'd expect. It refers to the Microsoft Azure Services subscription. It basically acts as the top-level umbrella for the organization.
Tenant - This is like a sub-organization, maybe a department or group. There can be multiple tenants under a single subscription.
User - As expected, a user is a single individual. Users are scoped to tenants.
Application - The Application is the software program trying to use the API. It must be registered and configured to do so.
Service Principal - This is essentially The Application. It is the entity making API Service requests.
Setting Up the Application
Although you might not guess it, this is probably the most complicated part of the process. Let's start by creating The Application in the portal.
Create The Application
Follow this click path: Azure Active Directory -> App Registrations -> New
There should be a form for application creation, with the following fields:
Name
This is simple the name of The Application. When authenticating, it will be shown to users. For the purposes of this "tutorial", we'll call it API Tutorial. This can be edited after creation.
Application Type
The type of The Application. For our purposes, we should choose "Web app/API". This cannot be edited after creation.
Sign-on URL
The is the redirect that will be used if we go the "authorization_code" route for sign on. This can be useful as the response will include an "id_token". We'll get into that a bit later. For now, let's make this http://123AzureApiTutorial.com/code. This can be edited after creation.
Once the Application has been created, you should see a property, Application ID. This is the client_id used in the OAuth2 flow. Take a note of its value.
Create the Client Secret
The OAuth2 flow requires a client secret value for authentication.
To generate it, follow this click path: Azure Active Directory -> App Registrations -> API Tutorial -> All Settings -> Keys
Enter the key description: API Tutorial Key, and the Duration: In 1 year.
Click Save. This will generate the Key Value. You must copy the value here and save it somewhere. You will not have another opportunity to do so.
This value is the client_secret in the OAuth2 flow.
Add the correct permissions
To get to the permissions, follow this click path: Azure Active Directory -> App Registrations -> API Tutorial -> All Settings -> Required Permissions -> Add
Here you will see the list of possible APIs. The one we care about for Azure is Windows Azure Service Management API. There is currently only one permission: Access Azure Service Management as organization users (preview). Select it, click Select, and then Done.
Getting the access_token
The access_token is what allows us to make requests against the API. There are two primary ways to do this. I suggest reading about both before trying to implement them.
Authorization Code
The authorization code is a two-step process. First we obtain the authorization code, and then we use that to get the access_token. A benefit of this route is that we get back an id_token as well, with a variety of useful claims like the user's name, email address, etc.
The request format is as follows: (GET) https://login.microsoftonline.com/<tenant-id>/oauth2/authorize?client_id=<client-id>&scope=api&redirect_uri=<redirect-uri>&response_type=code&prompt=consent. Let's go over the parameters here really quick.
Tenant ID
This can be obtained be using the click path Azure Active Directory -> Properties and copy the Directory ID. This is, in fact the Tenant ID value. It just has a different name to help with the overall confusion.
Client ID
This is the Application ID we retrieved previously.
Scope
This is the scope of the code. We just want to use the API.
Redirect URI
This is the sign-on URL you specified when creating API Tutorial. After the user logs in, they will be redirected to this URL with a "code" parameter in the query string.
Response Type
This is what we want the response to be. We want an authorization code, so we just use the value code.
Prompt
This specifies whether or not to prompt the user to consent to the permissions. If we did not have this, and changed permissions, the request would just unexpectedly fail. Very frustrating. But it can be removed once permission has been granted as long as you don't change the permissions. If the application is accessing an API that requires admin permission, this value should be admin_consent.
Alright, so once we shoot off this request we will be redirected to the login page. We login, accept the permissions, and then we should be redirected to here: http://123AzureApiTutorial.com/code?code=SOME_REALLY_LONG_STRING_OF_CHARACTERS. That string of characters is the code.
Getting the Access Token
Next, we take the code and use it to get the access_token. To do so, we need to make another request.
(POST) https://login.microsoftonline.com/<tenant-id>/oauth2/token
In addition to the url, we need to add parameters. These should be consistent with the content type application/x-www-form-urlencoded. This means they are submitted as form parameters. They are as follows:
client_id
This is again the client id (Application ID) we already have.
client_secret
This is the Application Key we generated earlier. I hope you saved it! If not, go back to that step and generate another one.
code
This is the value of the code we just received: SOME_REALLY_LONG_STRING_OF_CHARACTERS.
`grant_type
Because we're going the authorization code route, this value should be authorization_code
redirect_uri
This is the redirect uri we specified for the API Tutorial. The value from our example should be http://123AzureApiTutorial.com/code.
resource
This is very important. It is the resource API we want to access. For the Azure API, this value is https://management.azure.com.
Our response will be a json object with a variety of fields. Of these, the one we care about is access_token. Yay!
Client Credentials
This methodology skips getting the code (and thus needing the redirect_uri) at the expense of not getting an id_token.
The request is the same as in the Getting the Access Token section, with a few small differences.
We do not need to specify redirect_uri.
The value of grant_type should change to client_credentials.
Alright, we have an access_token! Now we're cooking!
Making the API request
With all the prep work thus far, this is the easiest part of the process.
The API URL we are requesting against is:
https://management.azure.com/subscriptions/<subscription-id>/providers/Microsoft.Compute/virtualMachines?api-version=2016-03-30
Add the following header to the request:
Authorization: Bearer <access-token>. Yes, the access_token value must be prefaced with the word "Bearer" in the header value.
"But wait!" You exclaim. "How do I get the subscription id?"
Excellent question! To find it through the portal, click Subscriptions -> -> Overview and copy the Subscription Id value.
Use that value, and run the request. You should see all the vms listed!
Common errors thus far
InvalidAuthenticationToken
When making the API request, you get an error response that says something like this: InvalidAuthenticationToken: The access token is invalid. This means you haven't added the API permissions to the API Tutorial. Go back to the Add the correct permissions step and double-check you have the right permissions. Also, when requesting the token make sure you use the prompt=consent, otherwise the you will not be prompted with the new permissions and the token will fail.
InvalidAuthenticationTokenTenant
Make sure that the tenant-id used when requesting the token belongs to the subscriber used when making the API call.
Lets assume i have REST api with products and i want it to be accessible only for specified users.
The list of users is known for me and i'm looking for some way to give them safe access to my API.
I don't want to create authentication with username and password, generate token and this stuff.
What i can imagine is that i'm giving each of my users some secret string
and they use it in every request.
And what i need is some example/tutorial/name of this kind of solution,
i'm sure there are some standards for that but i don't know it.
I know it's kind of nooby question - sorry for that, i'm just asking ;).
Thanks in advance!!
You are looking for a simple shared-secret authentication. A simple solution in this case is just to check for the secret as a param (or it could be in the request header). For example, the client can call:
https://example.com/valuable-stuff?secret=2Hard2Gue$$
You implement this in your web request handler as:
SECRET = '2Hard2Gue$$'
function showValuableStuff() {
if (params['secret'] != SECRET)
return NotFounderError;
// now retrieve and output the resource
}
Some practical considerations are:
Use a secure connection for this to prevent the secret being leaked (ie a secure HTTPS exposure).
Be careful where you store the source code if you're hard-coding it. A fancier solution is use an environment variable which is set on the server, so you keep this out of the source code. Or at least to encrypt the part of the source that contains the secret.
While this is fine for simple solutions, it violates the basic security principle of accountability because you are sharing the secret with multiple people. You might want to consider allocating each individual their own random string, in which case, you may as well use HTTP Basic Authentication as it's well supported by Apache and other web servers, and still quite a lightweight approach.
I'm building an app with Phonegap. It simply reads xml feeds to display latest articles uploaded by a school for parents to read.
There will be an option where each user of the app can decide whether they want to receive Push Notifications or not. I have a simple database table where the device's registration i.d. from Google Cloud Console is stored. When the user clicks "yes", the registration i.d. is generated and stored on the server. If they then click "no", it's deleted. I want to secure these call to the server with basic HTTP authentification.
Any documentation I have seen on basic authentification describes the sending of username and passwords. But with my application, there is no username or password as the users do not need to sign up. What do I send to authenticate the client? Would there be a key hard-coded on the client side and sent with each request? Couln't this be easily exposed by unpacking the .apk?
I object to the premise of the question. I actually see this as less a security issue and more a preferences issue. Understanding the distinction makes the development of your feature much easier.
Simply modify your application to allow the user to configure what he or she wants to see in the settings and then store the preferences wherever you want on the client (like local storage). The server can then push as before, but the app should simply refuse to render those pushes that the user doesn't want to see.
If you want to simply disseminate non-sensitive content to the users who want to see it, this is a preferences issue and/or a publish/subscribe issue. But it is not a security issue.
Since you have access to server side, you have the control of the whole process. So, in order to handle that, you may think about something like session cookies that a web server creates in case of normal authentication.
I have done something similar and what I've done is to generate a kind of token server side that is stored in the cookies of the device or the localStorage.
So the process flow should be something like this :
Generate a token and store it on the device (cookies or local storage).
For each request, send this value in a http header
From server side, you may identify the user from that token.
For example : you maintain a table that identifies device/token.
That's it
In addition to what the other answers said you can pass a custom useragent string as part of the requests, and validate it matches what you expect. It's not a sure way to 'secure' requests, but along with a (simple) token system and HTTPS this could be enough for your needs.
I have a PHP application that uses the Github API to check if the local system of the user is up to date with the lastest version on Github.
By default you can only query the API 60 times per hour. But i can increase this when i authenticate first. But, you can also increase the rate limit without authenticating. Which is what i want.
The example that i found says i only have to call this URL through CUrl:
https://api.github.com/users/whatever?client_id=xxxxxxxxxxxxxx&client_secret=yyyyyyyyyyyyyyyyyyyyy
Not sure if this will work, also no idea what they mean with whatever in the URL.
In anyway this should do trick for increasing the rate limit. But they also say the following:
This method should only be used for server-to-server calls. You should
never share your client secret with anyone or include it in
client-side browser code.
Since my application is an "open source" PHP application, then that basically means i'm going to share my secret key with others who use my application...
Is there any other way to increase the rate limit, without worrying that i'm sharing sensitive data with others?
You have to contact GitHub support to have your ratelimit increased, contrary to what spuder claims. They'll raise it for you.
As for not sharing your client_id or client_secret, spuder has the right answer there. Use an environment variable in your production system and get that and set the secret/id pair from there. Alternatively, use a configuration file that will be added to .gitignore so you never accidentally commit it.
Why not set the client secret as a variable, and purposely put a fake client secret in your code. That will prevent anyone who pulls your code from accidentally or maliciously using your key. To actually use the program, the end user will have to change the key variable.
The number of API requests per hour can not be increased {Correction: unless you contact github}