Implementing Conversation Webhook as Azure Function App - actions-on-google

I have an azure function app that I am using as a webhook for my google assistant action. I've tried to follow the documentation for the proper response but I continually get the following error in the simulator when testing my webhook. Is there anything in my response message that looks wrong?
Failed to parse SDKResponse from http_response:
HTTP/1.1 200 OK
Cache-Control: no-cache
Pragma: no-cache
Content-Length: 451
Content-Type: application/json; charset=utf-8
Content-Encoding: gzip
Expires: -1
Vary: Accept-Encoding
Server: Microsoft-IIS/8.0
X-AspNet-Version: 4.0.30319
X-Powered-By: ASP.NET
Date: Sun, 28 May 2017 19:00:13 GMT
{"conversationToken":"cee44ab4-97dd-4e18-99c7-2b8613eb7584","expectUserResponse":true,"expectedInputs":[{"inputPrompt":{"richInitialPrompt":{"items":[{"simpleResponse":{"textToSpeech":"So, you want to become a great swordsman? First, you must learn the proper technique of insult sword fighting. The current difficulty level is Easy. Say 'Tutorial' for some quick instructions. Say 'Start Game' to start the game. Say 'Options' for more options. "}}]}}}]}
Here is the json formatted for readability:
{
"conversationToken": "cee44ab4-97dd-4e18-99c7-2b8613eb7584",
"expectUserResponse": true,
"expectedInputs": [
{
"inputPrompt": {
"richInitialPrompt": {
"items": [
{
"simpleResponse": {
"textToSpeech": "So, you want to become a great swordsman? ... "
}
}
]
}
}
}
]
}
With my latest test, I tried sending the exact example response given on the Fulfillment page instructions and it still fails: https://developers.google.com/actions/components/fulfillment
{
"conversationToken": "{\"state\":null,\"data\":{}}",
"expectUserResponse": true,
"expectedInputs": [
{
"inputPrompt": {
"richInitialPrompt": {
"items": [
{
"simpleResponse": {
"textToSpeech": "Howdy! I can tell you fun facts about almost any number, like 42. What do you have in mind?",
"displayText": "Howdy! I can tell you fun facts about almost any number. What do you have in mind?"
}
}
],
"suggestions": []
}
},
"possibleIntents": [ { "intent": "actions.intent.TEXT" } ]
}
]
}

It looks like you have a slightly invalid entry for your items. The Item object is defined as a union field, indicating that one of the three attributes (simpleResponse, basicCard, or structuredResponse) and its respective value must be set.
So the textToSpeech attribute should not be under the richInitialPrompt.item directly, instead you should have a simpleResponse attribute and, under this, a textToSpeech attribute (or one of the other attributes that make sense for a SimpleResponse object. You must have at least one SimpleResponse (and it must be first), and you may have no more than two.
But the text attached to your second response doesn't make sense in this context. Both responses will be said/shown - not one in case there is a delay in user action.
The v1 protocol had a way to support details of the re-prompt, but I don't see the same thing in v2.
So the JSON should probably look more like:
{
"conversationToken": "fa3bfc17-de0a-4df8-900d-44dbb17b86c6",
"expectUserResponse": true,
"expectedInputs": [
{
"inputPrompt": {
"richInitialPrompt": {
"items": [
"simpleResponse": {
"textToSpeech": "Text for my response"
}
]
}
}
}
]
}

Related

POST parameters on REST api - links or inline?

I am trying to design a rest api and i have a resource with links to another resource. say
GET /resource1/id1
{
"property1": "value1",
...
"links" : [
{
"rel": "res1to2rel",
"href": "/resource2/r2"
}
]
now on a post to create resource1, i want to pass a relationship to resource2. should the POST call look like
POST /resource1
{
"res1to2rel": "/resource/r2"
}
or should it look like
POST /resource1
{
"links" : [
{
"rel": "res1to2rel",
"href": "/resource2/r2"
}
]
}
What is the best practice, should the POST match the GET or can it have properties that you dont see on GET?
There're few absolute rules when it comes to REST design, but I think that it'd be easier for the client if a resource has a consistent set of representations. When you POST a representation, you typically create a new resource. This can be made even more explicit by:
Returning 201 Created.
Returning a Location header in the response.
Returning a copy of the representation as the server sees it.
Thus, a POST might look like this:
POST /resource1 HTTP/1.1
Content-Type: application/json
{
"property1": "value1",
...
"links" : [
{
"rel": "res1to2rel",
"href": "/resource2/r2"
}
]
}
The response might then be:
HTTP/1.1 201 Created
Content-Type: application/json; charset=utf-8
Location: /resource1/id1
{
"property1": "value1",
...
"links" : [
{
"rel": "self",
"href": "/resource1/id1"
},
{
"rel": "res1to2rel",
"href": "/resource2/r2"
}
]
}
Notice that the server is allowed to enrich the resource, in the above example by adding a "self" link.
If a client later makes a GET request against the address supplied in that LOCATION header, it'll receive the same response as the echo it received from the POST request:
HTTP/1.1 200 OK
Content-Type: application/json; charset=utf-8
{
"property1": "value1",
...
"links" : [
{
"rel": "self",
"href": "/resource1/id1"
},
{
"rel": "res1to2rel",
"href": "/resource2/r2"
}
]
}
This makes it easier for clients to interact with the API because it can use the same internal model for reading and writing (i.e. serialising and deserialising) representations of a resource.

Media Response in Custom Payload (Dialogflow)

I'm trying to code a Media Response in a Custom Payload with no luck. I'm surely doing something wrong but I have no idea :) The Media Response does not show up when testing. (Please note that I'm trying this in an english action). Here's the JSON code:
{
"platform": "google",
"type": "custom_payload",
"payload": {
"google": {
"richResponse": {
"items": [
{
"mediaResponse": {
"mediaType": "AUDIO",
"mediaObjects": [
{
"name": "Exercises",
"description": "ex",
"largeImage": {
"url": "https://firebasestorage.googleapis.com/...",
"accessibilityText": "image..."
},
"contentUrl": "https://firebasestorage.googleapis.com/..."
}
]
}
}
]
}
}
}
}
UPDATE:
I've updated the JSON to something like this. But I get an error :"API Version 2: Failed to parse JSON response string with 'INVALID_ARGUMENT' error: ": Cannot find field."
{
"platform":"google",
"type":"custom_payload",
"payload":{
"google":{
"richResponse":{
"items":[
{
"simpleResponse":{
"textToSpeech":"Hey! Good to see you."
}
},
{
"mediaResponse":{
"mediaType":"AUDIO",
"mediaObjects":[
{
"name":"Exercises",
"description":"ex",
"largeImage":{
"url":"https://firebasestorage...",
"accessibilityText":"..."
},
"contentUrl":"https://firebasestorage.googleapis.com/..."
}
]
}
}
],
"suggestions":[
{
"title":"chips"
}
]
}
}
And here's the debug information:
{
"audioResponse": "//NExAARWG...",
"conversationToken": "GidzaW11bG...",
"debugInfo": {
"agentToAssistantDebug": {
"agentToAssistantJson": "{\"message\":\"Failed to parse Dialogflow response into AppResponse, exception thrown with message: Empty speech response\",\"apiResponse\":{\"id\":\"cd7204ac-ab80-42aa-9755-6123cbb938a6\",\"timestamp\":\"2018-03-11T09:02:35.827Z\",\"lang\":\"en-us\",\"result\":{},\"status\":{\"code\":200,\"errorType\":\"success\"},\"sessionId\":\"1520758955600\"}}"
},
"assistantToAgentDebug": {
"assistantToAgentJson": "{\"user\":{\"userId\":\"AA9douaa4XGkqtmcU_EDjPy7PQ_9\",\"locale\":\"en-US\",\"lastSeen\":\"2018-03-11T09:02:09Z\"},\"conversation\":{\"conversationId\":\"1520758955600\",\"type\":\"NEW\"},\"inputs\":[{\"intent\":\"actions.intent.MAIN\",\"rawInputs\":[{\"inputType\":\"VOICE\",\"query\":\"Talk to Zen Coach\"}]}],\"surface\":{\"capabilities\":[{\"name\":\"actions.capability.SCREEN_OUTPUT\"},{\"name\":\"actions.capability.MEDIA_RESPONSE_AUDIO\"},{\"name\":\"actions.capability.WEB_BROWSER\"},{\"name\":\"actions.capability.AUDIO_OUTPUT\"}]},\"isInSandbox\":true,\"availableSurfaces\":[{\"capabilities\":[{\"name\":\"actions.capability.SCREEN_OUTPUT\"},{\"name\":\"actions.capability.AUDIO_OUTPUT\"}]}]}",
"curlCommand": "curl -v 'https://api.api.ai/api/integrations/google?token=0def1bb6be4b4bf2810ec68bf6f37a6a' -H 'Content-Type: application/json;charset=UTF-8' -H 'Google-Actions-API-Version: 2' -H 'Authorization: eyJhbGciOiJSUzI1NiIsImtpZCI6ImFjMmI2M2ZhZWZjZjgzNjJmNGM1MjhlN2M3ODQzMzg3OTM4NzAxNmIifQ.eyJhdWQiOiJ6ZW4tY29hY2giLCJhenAiOiI0OTYwOTIwOTE1NzEtMGNhY3VtczVkZ3F1OWpkM2k0dHZpOGFiOTVydXQ2NnQuYXBwcy5nb29nbGV1c2VyY29udGVudC5jb20iLCJleHAiOjE1MjA3NTkwNzUsImlzcyI6Imh0dHBzOi8vYWNjb3VudHMuZ29vZ2xlLmNvbSIsImp0aSI6IjY4NDc0NThhNTNhZGExODAxZjMwMjAyYjkxZGIyODZhMjk1NzA2YmIiLCJpYXQiOjE1MjA3NTg5NTUsIm5iZiI6MTUyMDc1ODY1NX0.e1cqg96F5L-BvD0yJz3UFgsnX_0TRox0Lu8R9K5NhhXcQVfC7mq1QwCqs2DGrUJGquGdW2GhzBU2lzf4ro2TUeieg4ozak1OmiYAMqtiCH0EodeHy59AXXqzb3a35YuD7CmSDu6qVQRfEp8uaaH2t-Sq9lUchudNOgjucip3ex9Rr2XacHm0qWtV69H1o-Yq5INl5HHR0kNqtEIsxUox961imKvDLN5s--F35yTbAhIWibr6OmaACyzSQW5X7OjrJ2781DSmEdYn73poDbuwMS9E2l9B-QTUHAIpUM5b4WqrFkD6XKALdf2pQFwZlRRhDzRiDKWLA-i1w-mcak0LWw' -A 'Mozilla/5.0 (compatible; Google-Cloud-Functions/2.1; +http://www.google.com/bot.html)' -X POST -d '{\"user\":{\"userId\":\"AA9douaa4XGkqtmcU_EDjPy7PQ_9\",\"locale\":\"en-US\",\"lastSeen\":\"2018-03-11T09:02:09Z\"},\"conversation\":{\"conversationId\":\"1520758955600\",\"type\":\"NEW\"},\"inputs\":[{\"intent\":\"actions.intent.MAIN\",\"rawInputs\":[{\"inputType\":\"VOICE\",\"query\":\"Talk to Zen Coach\"}]}],\"surface\":{\"capabilities\":[{\"name\":\"actions.capability.SCREEN_OUTPUT\"},{\"name\":\"actions.capability.MEDIA_RESPONSE_AUDIO\"},{\"name\":\"actions.capability.WEB_BROWSER\"},{\"name\":\"actions.capability.AUDIO_OUTPUT\"}]},\"isInSandbox\":true,\"availableSurfaces\":[{\"capabilities\":[{\"name\":\"actions.capability.SCREEN_OUTPUT\"},{\"name\":\"actions.capability.AUDIO_OUTPUT\"}]}]}'"
},
"sharedDebugInfo": [
{
"name": "ResponseValidation",
"subDebugEntry": [
{
"debugInfo": "API Version 2: Failed to parse JSON response string with 'INVALID_ARGUMENT' error: \": Cannot find field.\".",
"name": "UnparseableJsonResponse"
}
]
}
]
},
"response": "Zen coach isn't responding right now. Try again soon.",
"visualResponse": {
"visualElements": []
}
}
Are you adding "platform":"google" and "type":"custom_payload" in the custom payload? If so, try removing that.
I made the following work with my Voice Metronome application:
{
"google":{
"richResponse":{
"items":[
{
"simpleResponse":{
"textToSpeech":"Hey! Good to see you."
}
},
{
"mediaResponse":{
"mediaType":"AUDIO",
"mediaObjects":[
{
"name":"Exercises",
"description":"ex",
"largeImage":{
"url":"http://res.freestockphotos.biz/pictures/17/17903-balloons-pv.jpg",
"accessibilityText":"..."
},
"contentUrl":"https://freepd.com/Chill/Chill Air.mp3"
}
]
}
}
],
"suggestions":[
{
"title":"chips"
}
]
}
}
}
The issue is that the richResponse property still needs to follow the rules of the RichResponse object. The first item in it must be a SimpleResponse object. (I haven't tested, but you can probably have that say nothing, but it is a good spot to have an introduction to your audio.)
The error message Failed to parse Dialogflow response into AppResponse, exception thrown with message: Empty speech response suggests that it might also be looking for a speech parameter on the top-level object in the response, which is what Dialogflow v1 expects to duplicate either the simpleResponse ssml or textToSpeech parameters. I'm not sure why that would appear if you're set to v2, but it sounds like something might be confused there. I would make sure you're using v1 and that you have a speech parameter.
Also keep in mind that the reviewers will look for suggestion chips about how to move the conversation forward during or after the audio if this isn't a final response.

Simulator keeps replying that endpoint could not be called, while I see a POST request in the logs

I am currently writing an integration into Actions on Google using PHP. I have generated a action.json file with my test endpoint as the fulfillment. I use ngrok to expose my local development machine publicly.
Unfortunately the simulator keeps insisting that the app isn't responding. In the access logs, and the ngrok Inspector I do see that a request came in, and it has been neatly answered with a JSON reply.
In an act of pure desperation I even upload a JSON response, directly taken from the Fulfillment documentation page to a server and set that as the fulfillment URL. The result is the same, same error.
I do not see a way to get a more detailed error message from Actions on Google, explaining why it does not work.
My action.json:
{
"actions": [
{
"name": "MAIN",
"intent": {
"name": "actions.intent.MAIN"
},
"fulfillment": {
"conversationName": "development4"
}
},
{
"name": "TEXT",
"intent": {
"name": "actions.intent.TEXT"
},
"fulfillment": {
"conversationName": "development4"
}
}
],
"conversations": {
"development4": {
"name": "development4",
"url": "https:\/\/02c085c0.ngrok.io\/actionsongoogle\/process\/development4"
}
}
}
The json I response with:
{
"expectUserResponse": false,
"expectedInputs": [{
"inputPrompt": {
"richInitialPrompt": {
"items": [{
"simpleResponse": {
"textToSpeech": "hello"
}
}]
}
},
"possibleIntents": [{
"intent": ["actions.intent.TEXT"]
}]
}]
}
The result output in the Simulator then displays:
{
"response": "my test app isn’t responding right now. Try again soon.\n",
"audioResponse": "//NExAARq...",
"debugInfo": {
"sharedDebugInfo": [
{
"name": "ExecutionResponse",
"debugInfo": "Failed to call your endpoint."
}
]
},
"visualResponse": {}
}
Two things to investigate:
Check how long it is taking for the reply to be sent. Agents need to send a response back within about 5 seconds or the Assistant will time out and give the "not responding" message.
Check what the reply code being issued is. If you think you're sending something back, but your server actually crashes before an HTTP 200 response code is sent back with the body, the Assistant never gets the response. I've also seen it where people are thinking they're sending a response - but no response is actually sent at all.
Testing with curl or wget to your ngrok URL (so it makes a full round trip) might be able to help diagnose both of these.
The issue was in the returned JSON. I have set up the nodejs SDK and reply in the same way as the PHP cod will (a simple hello). The response generated by the NodeJS SDK is very different. Apparently the example that I used from the docs earlier is also invalid.
I advice anyone building their own implementation using Webhooks to first generate the desired response via de NodeJS sdk, the reference in the manual is not always clear (at least for me).
Response by NodeJS SDK:
{
"expectUserResponse": false,
"finalResponse": {
"speechResponse": {
"textToSpeech": "hello"
}
}
}
Implementation in SDK:
var express = require('express')
var bodyParser = require('body-parser')
var app = express()
// parse application/x-www-form-urlencoded
app.use(bodyParser.urlencoded({ extended: false }))
// parse application/json
app.use(bodyParser.json())
app.use(function (req, res) {
let ActionsSdkApp = require('actions-on-google').ActionsSdkApp;
const actionsApp = new ActionsSdkApp({request: req, response: res});
//const inputPrompt = actionsApp.buildInputPrompt(false, 'hello')
actionsApp.tell('hello');
})
app.listen(80);

REST API Design - add / remove an item in an array field of a resource

Let's assume we have the following REST resource collection endpoint:
/products
The resource looks like this:
GET /products/123
Accept: application/json
{
"id": "123",
"name": "Shampoo",
...
"ingredients": [
"Sodium Lauryl Sulfate",
"Sodium Laureth Sulfate",
"Hydrochloric Acid",
...
]
}
Now, let's say I want to update the resource, and add or remove an ingredient to / from the ingredients field.
One way is to GET the resource, manipulate the array, and PUT or PATCH back. However, this is pretty chatty protocol, and it also poses scaleability problem, if the array is too big.
What is the best way to achieve this add / remove operation without having the source array?
I already explored the following: JSON Patch Specification
But it seems the spec is not REST. It uses the same resource with different "operation" like representation, which is entirely different from the resource's representation.
I would like to keep as much as possible the same representation of resource for PUT / PATCH method as it is appearing in GET.
Here are my thoughts on the matter.
Change the string type item to object
Add meta field to the object called $op
So in this design, here is how the resource looks like on GET
GET /products/123
Accept: application/json
{
"id": "123",
"name": "Shampoo",
...
"ingredients": [
{
"name": "Sodium Lauryl Sulfate"
},
{
"name": "Sodium Laureth Sulfate"
},
{
"name": "Hydrochloric Acid"
},
...
]
}
And now if I want to remove an item from the ingredients array:
PATCH /products/123
Content-Type: application/json
Accept: application/json
{
"ingredients": [
{
"$op": "remove",
"name": "Hydrochloric Acid"
}
]
}
Or add it back:
PATCH /products/123
Content-Type: application/json
Accept: application/json
{
"ingredients": [
{
"$op": "add",
"name": "Hydrochloric Acid"
}
]
}
Does anyone has a better way? Is there some standard on the subject?
In my opinion the best solution would be to provide another REST endpoint for product's ingredients management. When you in addition provide links in a HATEOAS manner, you will be able to fully controll that array right after getting a product resource.
Consider this representation:
GET /products/123
Accept: application/json
{
"id": "123",
"name": "Shampoo",
...
"ingredients": [
{
"name": "Sodium Lauryl Sulfate"
"links": [
{
"rel": "self",
"href": "/products/123/ingredients/1"
}
]
},
{
"name": "Sodium Laureth Sulfate",
"links": [
{
"rel": "self",
"href": "/products/123/ingredients/2"
}
]
},
...
],
"links": [
{
"rel": "self",
"href": "/products/123"
},
{
"rel": "ingredients",
"href": "/products/123/ingredients"
}
]
}
Having this you could add new ingredient to a product...
POST /products/123/ingredients
Content-Type: application/json
Accept: application/json
{
"name": "Hydrochloric Acid"
}
...or delete existing one.
DELETE /products/123/ingredients/2
What is the best way to achieve this add / remove operation without having the source array?
If you remove ingredients array from product's JSON representation and leave only link to it you are still able to add new ingredient right away.
Remove operation still need source array but you could download it asynchronously. In fact i see no point in deleting anything from array without having it first. How do you know that ingredient you try to remove is inside?
Best regards

Loopback get multipart form-data parameters

I have a file container hosting my images. I did create some additional properties for that container using slc loopback:property and put in a string types such as: title, description, date(date). I am able to upload my image via api/containers/{container}/upload
Using Advanced REST Client, I add an HTTP Header: application/x-www-form-urlencoded
I then input data into my forms. I add my image and send, here is my response:
```
Vary: Origin, Accept-Encoding
Access-Control-Allow-Credentials: true
X-Xss-Protection: 1; mode=block
X-Frame-Options: DENY
X-Download-Options: noopen
X-Content-Type-Options: nosniff
Content-Type: application/json; charset=utf-8
Content-Length: 205
Etag: W/"cd-dUHU2bNp6fDC61813wVPRw"
Date: Fri, 24 Jun 2016 20:52:13 GMT
Connection: keep-alive
```
```
{
"result": {
"files": {
"fileUpload2": [
{
"container": "images",
"name": "pnw.png",
"type": "image/png",
"size": 269360
}
]
},
"fields": {
"created": [
"04/20/2016"
],
"title": [
"title"
],
"description": [
"this is a description"
]
}
}
}
```
I am curious, is the fields object stored somewhere? (I do have mongo connector good to go) or is this in memory? I can't seem to find it anywhere and not sure how to do a GET req. My ultimate goal is to do one image post with additional parameters/properties and retrieve the same in one instance. Thanks!
I will go another route as I do not think I am understanding content-disposition correctly.