Security of cloudant query from OpenWhisk - ibm-cloud

I'm building an Angular SPA with a Cloudant data store on Bluemix.
Since the Bluemix implementation of OpenWhisk doesn't use VCAP services, I see 3 options to use OpenWhisk as my api provider for cloudant queries for my Angular app:
Follow the pattern of passing credentials as seen here: https://github.com/IBM-Bluemix/openwhisk-visionapp (very interesting approach btw)
Include the credentials as though I'm running locally as seen here: https://github.com/IBM-Bluemix/nodejs-cloudant/blob/master/app.js
Use the http API as seen here: https://docs.cloudant.com/api.html (which highlights the security problem passing credentials.
Since my service is not intended for publishing (it's intended for my own app) I'm thinking option 2 is my "least of all evils" choice. Am I missing something? My thinking is such that while fragile to changes it would be the most secure since credentials aren't passed in the open. The serverless infrastructure would have to be hacked...
Thanks in advance!
(lengthy) Update: (apologies in advance)
I've gotten a little farther along but still no answer - stuck in execution right now.
To clarify, my objective is for the app to flow from Angular Client -> OpenWhisk -> Cloudant.
In this simplest use case, I want to pass a startTime parameter and an endTime parameter, have OpenWhisk fetch all the records in that time range with all fields, and passing back selected fields. In my example, I have USGS earthquake data in a modified GeoJSON format.
Following information from the following articles below, I've concluded that I can invoke the wsk command line actions and use the bindings I've setup from within my Javascript function and therefore not pass my credentials to the database. This gives me a measure of security (still question the rest endpoint of my OpenWhisk action) but I figure once I get my sample running I think through that part of it.
My command line (that works):
wsk action invoke /my#orgname.com_mybluemixspace/mycfAppName/exec-query-find --blocking --result --param dbname perils --param query {\"selector\":{\"_id\":{\"$gt\":0},\"properties.time\":{\"$gt\":1484190609500,\"$lt\":1484190609700}}}
This successfully returns the following:
{
"docs": [
{
"_id": "eq1484190609589",
"_rev": "1-b4fe3de75d9c5efc0eb05df38f056a65",
"dbSaveTime": 1.484191201099e+12,
"fipsalpha": "AK",
"fipsnumer": "02",
"geometry": {
"coordinates": [
-149.3691,
62.5456,
0
],
"type": "Point"
},
"id": "ak15062242",
"properties": {
"alert": null,
"cdi": null,
"code": "15062242",
"detail": "http://earthquake.usgs.gov/earthquakes/feed/v1.0/detail/ak15062242.geojson",
"dmin": null,
"felt": null,
"gap": null,
"ids": ",ak15062242,",
"mag": 1.4,
"magType": "ml",
"mmi": null,
"net": "ak",
"nst": null,
"place": "45km ENE of Talkeetna, Alaska",
"rms": 0.5,
"sig": 30,
"sources": ",ak,",
"status": "automatic",
"time": 1.484190609589e+12,
"title": "M 1.4 - 45km ENE of Talkeetna, Alaska",
"tsunami": 0,
"type": "earthquake",
"types": ",geoserve,origin,",
"tz": -540,
"updated": 1.484191127265e+12,
"url": "http://earthquake.usgs.gov/earthquakes/eventpage/ak15062242"
},
"type": "Feature"
}
]
}
The action I created in OpenWhisk (below) returns an Internal Server Error. I'm passing the input value as
{
"startTime": "1484161200000",
"endTime": "1484190000000"
}
Here's the code for my action:
`var openWhisk = require('openwhisk');
var ow = openWhisk({
api_key:'im really a host'
});
function main(params) {
return new Promise(function(resolve, reject) {
ow.actions.invoke({
actionName:'/my#orgname.com_mybluemixspace/mycfAppName/exec-query-find',
blocking:true,
parameters:{
dbname: 'perils',
query: {
"selector": {
"_id": {
"$gt": 0
},
"properties.time": {
"$gt": params.startTime,
"$lt": params.endTime
}
}
}
}
}).then(function(res) {
//get the raw result
var raw = res.response.result.rows;
//lets make a new one
var result = [];
raw.forEach(function(c) {
result.push({id:c.docs._id, time:c.docs.properties.time, title:c.docs.properties.title});
});
resolve({result:result});
});
});
}`
Here are the links to my research:
http://infrastructuredevops.com/08-17-2016/news-openwhisk-uniq.html
Useful because of the use of the exec-query-find and selector syntax usage but also cool for the update function I need to build for populating my data!
https://www.raymondcamden.com/2016/12/23/going-serverless-with-openwhisk
The article referenced by #csantanapr
Am I overlooking something?
Thanks!

I'm assuming what you are trying to do is to access your Cloudant DB directly from your angular client side code from the Browser.
If you don't need any business logic, or you can get away by using Cloudant features (design docs, views, map, reduce, etc..) and you are generating Cloudant API keys with certain access (i.e. write vs. read), then you don't need a server or serveless middlewear/tier.
But now let's get real, most people need that tier, and if you are looking a OpenWhisk, then you are in good luck this is very easy to do.
OpenWhisk on Bluemix support VCAP service credentials, but in a different way.
Let's name you have a Bluemix Org carlos#example.com and space dev that would translate to OpenWhisk namespace carlos#example.com_dev
If you add a Cloudant service under the space dev in Bluemix, this will generate service key credentials for this Cloudant Account. This credentials give you super power access meaning you are admin.
If you want to use this Cloudant credentials in OpenWhisk, you can use the automatic binding generated with the cloudant package.
To do this using the OpenWhisk CLI run wsk package refresh this will pull the Cloudant credentials and create you a new package with the credentials binded as default parameter for all the cloudant actions under that package. This is modified version of #1 above
Another alternative is to bind the credentials manually to a package or an action as default parameters, this makes sense when you don't want to use the super power admin credentials, and you generated a Cloudant API key for a specific database. This is option #1 above.
I would not recommend to put the credentials in source code #2
For option #3, what's insecure is to pass your credentials as part of the URL like https://username:password#user.cloudant.com, but passing the username and password in the Authorization header over https is secured.
This is because even if you are using secure transport https everything in the URI/URL is not encrypted anyone can see that value, but passing secrets in body or header is standard practice as this is transfer after secure connection is established.
Then you create actions that use the credentials as parameters in your OpenWhisk actions to build your business logic for your backend.
Then how to do you access this backend from the Browser, well OpenWhisk has a API Gateway feature in experimental that allows your to expose your actions as public APIs with CORS enable.
Only a url is expose, your credentials as default parameters are never expose.
If you want to see an example on check out Raymond Camden Blog posts where he show Ionic/Angular App accessing his Cloudant Database of Cats
https://www.raymondcamden.com/2016/12/23/going-serverless-with-openwhisk

Related

Azure DevOps - Unable to Create Var Group using Azure DevOps API and Auth Token

Requirements: We would like to create a Variable Group (along with some variables) in a given Project.
Option1: We are able to create a new Variable Group successfully
when we create a request via PostMan using PAT Token which has FULL access.
Option2: Our end goal is to invoke the ADO Rest API in the Web App which uses
OAuth. When the end user logs in and make a call (pls see the input
details below) we are getting '401 Un Authorized - The user is not authorized to access this resource.' error. The Web App's application has the Variable Groups manage scope as shown below.
TroubleShooting: As part of troubleshooting, for Option1 which uses PAT (with full access) in Postman, we have updated the permissions of the PAT to just have Create, Read and Manage Var Groups as shown below.
Now, even the Option1 is not working after making the PAT to have Custom Defined access.
Are we missing something?
Postman Details:
URL: https://dev.azure.com/myorgname/_apis/distributedtask/variablegroups?api-version=6.0-preview.2
Verb: Post
Headers: Authorization: Basic
Body:
{
"name": "This is ignored",
"description": "This is ignored",
"type": "Vsts",
"variables": {
"BuildConfiguration": {
"value": "Release"
}
},
"variableGroupProjectReferences": [
{
"name": "VarGroup",
"description": "The variable group to store the information about the variables using in the Pipeline",
"projectReference": {
"id": "#ProjectId#",
"name": "#ProjectName#"
}
}
]
}
I can also reproduce your issue with option 1, not only Read, create, & manage for Variable Groups, even I select all the scopes via Custom defined, it still does not work.
According to this doc - https://learn.microsoft.com/en-us/azure/devops/organizations/accounts/manage-pats-with-policies-for-administrators?view=azure-devops#restrict-creation-of-full-scoped-pats
Some of our public APIs are currently unassociated with a PAT scope, and can therefore only be used with “full-scoped” PATs. Because of this, restricting the creation of full-scoped PATs might block some workflows. We're working to identify and document the affected APIs and eventually associate them with the appropriate scope. For now, these workflows can be unblocked by using the allow list.
I believe this should be the reason for this issue, there may be some additional permissions to create variable groups. For option 2, there may be a similar cause.
So in this case, you may need to use the Full access PAT temporarily, as mentioned in the doc We're working to identify and document the affected APIs and eventually associate them with the appropriate scope.

Watson Assistant API V2 "manages the context automatically" - more details?

This is a question about Watson Assistant API V1/V2 difference. The doc says like this:
Note that if your app uses the v1 API, it communicates directly with the dialog skill, bypassing the orchestration and state-management capabilities of the assistant. This means that your application is responsible for maintaining state information. This is done using the context, an object that is passed back and forth between your application and the Watson Assistant service. Your application must maintain the context by saving the context received with each response and sending it back to the service with each new message request.
An application using the v2 API can also use the context to access and store persistent information, but the context is maintained automatically (on a per-session basis) by the assistant.
It seems that in V2, "the context is maintained automatically by the assistant". What does this mean exactly ? If I'd like to pass some data to the dialog flow, I might use context on "/message". Is is allowed in V2?( yes, it seems.) Then in V1 days, I have to receive context from responses and send it back on every request. Does assistant also send the context back in V2? What should my client-app do in V2? Any detailed info is welcomed ..Thanks.
Answering your second question first - If you check the API docs for Watson Assistant V2 here, there is a MessageContext object in the response with Global Context and skill specific context values.
You also have a sample Request where you can manually pass the context (both global and user-defined)
curl -u "apikey:{apikey}" -X POST -H "Content-Type:application/json" -d "{\"input\": {\"text\": \"Hello\"}, \"context\": {\"global\": {\"system\": {\"user_id\": \"my_user_id\"}}}}" "https://gateway.watsonplatform.net/conversation/api/v2/assistants/{assistant_id}/sessions/{session_id}/message?version=2018-11-08"
From the client side, you can use this code
service.message({
assistant_id: '{assistant_id}',
session_id: '{session_id}',
input: {
'message_type': 'text',
'text': 'Hello'
},
context: {
'global': {
'system': {
'user_id': 'my_user_id'
}
},
"skills": {
"main skill": {
"user_defined": {
"my_result_variable": "result_value"
}
}
}
}
}
Reference link
Check the API Methods Summary to understand what is supported in V2 as of today.
There is a new concept in V2 called Session. A session is used to send user input to a skill and receive responses. It also maintains the state of the conversation which is Context automatically for you.
V2 API as of today supports Runtime Methods, methods that enable a client application to interact with (but not modify) an existing assistant or skill. You can use these methods to develop a user-facing client that can be deployed for production use, an application that brokers communication between an assistant and another service (such as a chat service or back-end system), or a testing application.
The complete cURL example that works for me
curl -u "apikey:<API_KEY>" -X POST -H "Content-Type:application/json" -d "{
\"input\": {
\"text\": \"What's the time?\",
\"options\": {
\"alternate_intents\": true,
\"debug\": true,\"return_context\": true
}
},
\"context\": {
\"global\": {
\"system\": {
\"user_id\": \"derp\",\"turn_count\":1
}
}
},
\"skills\": {
\"main_skill\":{\"user_defined\": {
\"chosen_service\": \"dental\"
}}
}
}" "https://gateway.watsonplatform.net/assistant/api/v2/assistants/{ASSISTANT_ID}/sessions/{SESSION_ID}/message?version=2018-11-08"
for session ID, run this command
curl -u "apikey:<API_KEY>" -X POST "https://gateway.watsonplatform.net/assistant/api/v2/assistants/{ASSISTANT_ID}/sessions?version=2018-11-08"
The response includes user_defined skill context
{"output":{"generic":[{"response_type":"text","text":"Hey ! how are you today? Let me know if you need any help or get stuck looking for information."}],"debug":{"nodes_visited":[{"dialog_node":"node_6_1475611948267","title":null,"conditions":"conversation_start"}],"log_messages":[],"branch_exited":true,"branch_exited_reason":"completed"},"intents":[{"intent":"General_Greetings","confidence":0.32179955244064334},{"intent":"General_Jokes","confidence":0.296911633014679},{"intent":"goodbye","confidence":0.2852578103542328},{"intent":"General_Ending","confidence":0.2513303637504578},{"intent":"off_topic","confidence":0.24435781836509707},{"intent":"select_detail","confidence":0.24206179082393647},{"intent":"list_options","confidence":0.22829059958457948},{"intent":"still-here","confidence":0.22606439888477325},{"intent":"select_service","confidence":0.22488142400979996},{"intent":"General_Security_Assurance","confidence":0.2210852071642876}],"entities":[]},"context":{"global":{"system":{"turn_count":1,"user_id":"derp"}},"skills":{"main skill":{"user_defined":{"chosen_service":"dental"}}}}}
Thanks, #Vidyasagar Machupalli #data_henrik.
(I created "answer" section to paste the images below .)
1) Now it worked fine. I set the variable reference in dialog editor like this.
Then I post user-variable in context like this.
context={
"skills": {
"main skill": {
"user_defined": {
"myname": "ishida"
}
}
}
}
then the response was:
{'output': {'generic': [{'response_type': 'text', 'text': 'Hi, ishida .'}], 'intents': [], 'entities': []}}
It seems that "skills"->"main skill"->"user_defined" section is a case-sensitive / FIXED keyword.
From what I see, if I change one of them, I cannot read the variable on dialog editor.
2) I also found the documentation issue. The starting point of my question was : "API V2 doc says that I should use "skills" entry to handle the user variables, but no detailed info more than that .. ". #Vidyasagar Machupalli says "As per the V2 API documentation, user_defined is arbitrary variables ".. but I cannot find the paragraph in the API doc.
Then I found that
a) when I select "curl" tab, "user_defined" explanation appears.
b) when I select other(Java/Node/Python) tab, no explanation appears.
I knew that the info was not there as I refered the "Python" tab. I'd appreciate this be fixed soon to avoid same confusion. And I think that the fact of "skills->main skill->user_defined section is a case-sensitive / FIXED keyword" should be noted, as it is not clear in the current description. Thanks!

How do add an apitoken as part of a custom VSTS service endpoint datasource?

I'm am trying to implement a VSTS extension which adds a new service endpoint. Crucially, the authentication method for this service includes the API as part of the querystring.
I am using the "type": "ms.vss-endpoint.endpoint-auth-scheme-token" for AuthenticationScheme.
I've defined the dataSources like so:
"dataSources": [
{
"name": "TestConnection",
"endpointUrl": "{{endpoint.url}}projects?token={{endpoint.apitoken}}"
}
]
However, in performing a test to Verify Connection:
Failed to query service endpoint api: https://myserver.com/projects?token=.
endpoint.apitoken is always blank.
Is there a placeholder/replacement value that can be used to get access to this value or another way of achieving the same end result?
I've tried using different authentication schemes (such as 'none') and included a inputDescriptor to capture my apitoken, but I have the same result. There doesn't seem to be a way to reference these values?
No, it is not supported. This article may benefit you: Service endpoint authentication schemes

Watson Conversation making programmatic calls to client

I have a node app that uses the Watson Conversation service. I am able to successfully trigger a call to another API via a dialogue node using the JSON that it uses for the reply. However after reading up it seems I am doing it wrong. I am triggering my client server to make a REST call by adding an action property to the context.
{
"context": {
"action": "lookup"
},
"output": {}
}
When I get my result I add it onto the context object and pass it back to the conversation service. This seems to work ok, but it causes some issues.
1) having to manually delete these props after I trigger the thing I want
2) In conversation I must wait for user input even though I am not actually requesting user input on the front end but rather my client app is sending a message with no input text and the results of the REST call on the context object. This message which is returned to the conversation at the node where the action was triggered is what triggers the child nodes. It seems like there is a standardized way IBM wants you to make these programmatic calls regardless of if it's to an IBM cloud function, or your own client app. https://console.bluemix.net/docs/services/conversation/dialog-actions.html#dialog-actions
docs method:
{
"context": {
"variable_name" : "variable_value"
},
"actions": [
{
"name":"<actionName>",
"type":"client | server",
"parameters": {
"<parameter_name>":"<parameter_value>",
"<parameter_name>":"<parameter_value>"
},
"result_variable": "<result_variable_name>",
"credentials": "<reference_to_credentials>"
}
],
"output": {
"text": "response text"
}
}
Is this a new feature? I was referencing sample projects for my own app and I didn't see this pattern. By using this format will it tell the parent node to wait for a response to come back before trying to process the children? Will it prevent me from needing to delete properties off the context object so that I'm not calling the same action over and over with the same parameters in further turns of the conversation?

What types are available for profile variables in Watson Dialog Service?

The Watson Dialog Service on IBM Bluemix allows to create profile variables and to pass values to them. In the examples these variables always seem to have the type "TEXT" (see "myVariable" below). Are there any other types available? How would I pass a JSON object and how would I access specific values inside a dialog?
<variables>
<var_folder name="Home">
<var name="myVariable" type="TEXT"/>
</var_folder>
</variables>
I talked with the service team, and the recommendation is to use the new Watson Conversation service as it is actually possible to pass an array of name:value pairs. Below is a sample that the team came up with on the fly. Hopefully it's helpful.
Sample:
{
"client_id": 4435,
"name_values": [
{
"name": "string",
"value": "string"
}
]
}
For example, if you want to post to a context variable named JSON_object, the PUT context payload would be:
{
"client_id": 4435,
"name_values": [
{
"name": "JSON_object",
"value": "{"sample":"data"}"
}
]
}
HOWEVER, they highly recommend converting the JSON to flat XML before posting as context, since dialog has much more versatility to parse XML, using {variable_name.xmlElementName}.
More info found on the API explorer - https://watson-api-explorer.mybluemix.net/apis/dialog-v1#!/Profile/setProfile under PUT CONTEXT method.
Best to go to the documentation.
variables can be objects - consisting of many vars
http://www.ibm.com/watson/developercloud/doc/dialog/reference_elements.shtml#reference_variables
vars can be of types
http://www.ibm.com/watson/developercloud/doc/dialog/reference_elements.shtml#reference_var