Validation errors when linking authentication with Dialogflow / Google Action - actions-on-google

We are trying to get DialogFlow / Google Actions to link to an external OAuth account within a conversation. Upon following the examples we are consistently getting request validation errors that we don't know how to resolve. As far as I can tell we are doing everything by the book.
Our questions:
Why won't DialogFlow kick off the authentication process?
The documented webhook response for requesting the sign-in helper not have the same schema as a standard DialogFlow webhook response. Is the sign-in helper response supposed to be used for something else?...
Things we have tried:
We have configured our app and OAuth credentials per the documentation
We have verified that we can fetch an OAuth token from the service via the OAuth 2.0 playground
We have enabled transactions on the Action.
Our intent code is as follows:
const { DialogflowApp } = require('actions-on-google');
const { ActionsSdkApp } = require('actions-on-google');
function processV2Request (request, response) {
...
let app = new DialogflowApp({request: request, response: response});
let sdkapp = new ActionsSdkApp({request: request, response: response});
const actionHandlers = {
'pto': () => {
app.askForSignIn();
}
...
}
...
}
When we execute this intent, we receive the following debug error message about a speech field.
"agentToAssistantJson": {
"message": "Unexpected apiai response format: Empty speech response",
"apiResponse": {
"id": "751e4716-a94d-4d11-908d-3ee193092d9c",
"timestamp": "2017-12-05T19:52:01.962Z",
"lang": "en-us",
"result": {},
"status": {
"code": 206,
"errorType": "partial_content",
"errorDetails": "Webhook call failed. Error: Failed to parse webhook JSON response: Cannot find field: speech in message google.cloud.dialogflow.v2beta1.WebhookResponse."
}
}
}
If instead we try using the ActionsSdkApp, sdkapp.askForSignIn(), we get an error message about a conversation_token field.
"agentToAssistantJson": {
"message": "Unexpected apiai response format: Empty speech response",
"apiResponse": {
"id": "7de79093-6db8-44ee-8717-16b0f317174c",
"timestamp": "2017-12-05T19:54:29.709Z",
"lang": "en-us",
"result": {},
"status": {
"code": 206,
"errorType": "partial_content",
"errorDetails": "Webhook call failed. Error: Failed to parse webhook JSON response: Cannot find field: conversation_token in message google.cloud.dialogflow.v2beta1.WebhookResponse."
},
}
}
And if it's helpful, here's our package.json. It's the DialogFlow default as of recently, with a bumped up version of actions-on-google.
{
"name": "dialogflowFirebaseFulfillment",
"description": "This is the default fulfillment for a Dialogflow agents using Cloud Functions for Firebase",
"version": "0.0.1",
"private": true,
"license": "Apache Version 2.0",
"author": "Google Inc.",
"engines": {
"node": "~6.0"
},
"scripts": {
"start": "firebase serve --only functions:dialogflowFirebaseFulfillment",
"deploy": "firebase deploy --only functions:dialogflowFirebaseFulfillment"
},
"dependencies": {
"actions-on-google": "^1.6.x",
"firebase-admin": "^4.2.1",
"firebase-functions": "^0.5.7",
"apiai": "^4.0.3"
}
}

We determined that in addition to having Transactions enabled for the app, you must also change the Development Sandbox toggled to Active:

Related

Messenger bot with dialogflow does not respond to user

I have a messenger bot built with google's dialogflow engine, to respond the user I have a webhook. when I send a message with a "hello", the default welcome intent its triggered and the webhook it its called and it responds to me.
After that I need the products of my business so I write "Comprar" (Buy), the Products intent it is triggered and the webhook responds with a fulfillment message with a list of available products.
In the history section in google dialogflow, I can see the json respond of the wekhook with the message that the user will receive.
But facebook does not show the message. I use the same webhook for many intents, and it is not working in this case.
this is the respond I got in the History section for this case:
{
"id": "f75002cd-2f8a-422a-960d-dd2c9e00b490-74fe87bc",
"fulfillmentText": "",
"language_code": "es",
"queryText": "Comprar",
"webhookPayload": {},
"intentDetectionConfidence": 1,
"action": "GET_CATEGORIES",
"webhookSource": "",
"parameters": {},
"fulfillmentMessages": [
{
"quickReplies": {
"title": "Escoge alguna de estas Categorías:",
"quickReplies": [
"Volver",
"",
"FIRMA ELECTRONICA"
]
}
}
],
"diagnosticInfo": {
"webhook_latency_ms": "732.0"
},
"webhookStatus": {
"webhookStatus": {
"message": "Webhook execution successful"
},
"webhookUsed": true
},
"outputContexts": [
{
"lifespanCount": 1,
"name": "vercatalogo-followup",
"parameters": {}
}
],
"intent": {
"isFallback": false,
"displayName": "ver.catalogo",
"id": "41dcecf2-11b6-4e50-8294-b86d849093e1"
}
}
I solved lol
Apparently quickReplies must not contain a empty string ("") as a quick reply, that option is not valid. So I removed it and it worked.

Error trying to write data on google's fit web aplication via rest api

I'm trying to write data on google's fit via rest api. The idea is to write data on google's rest api, and the goal is to show that information on both google fit's web and app.
The information that i'm trying to write is a session (equals to workout)
Request:
GET https://www.googleapis.com/fitness/v1/users/me/sessions/11xxxx
Body:
{
"id": "11xxxx",
"name": "My example workout",
"description": "A very intense workout",
"startTimeMillis": 1396710000000,
"endTimeMillis": 1396713600000,
"version": 1,
"lastModifiedToken": "exampleToken",
"application": {
"packageName": "com.google.android.apps.fitness",
"version": "web"
},
"activityType": 1
}
Response:
{
"error": {
"code": 403,
"message": "Application package name (com.google.android.apps.fitness) provided by un-trusted source.",
"errors": [
{
"domain": "global",
"message": "Application package name (com.google.android.apps.fitness) provided by un-trusted source.",
"reason": "forbidden"
}
]
}
}

Error "Empty speech response"

I tried to connect DialogFlow and Actions on Google, so I created some intents, connected the services, added explicit and implicit invocations etc, but when I try the bot in the simulator https://console.actions.google.com/project/[projectId]/simulator/ it always gives me the error:
"Failed to parse Dialogflow response into AppResponse, exception
thrown with message: Empty speech response"
even tough inputType was "KEYBOARD".
What I tried so far:
I did set "Response from this tab will be sent to the Google Assistant integration" in Dialog Flow (do you have to set it for every single intent?), but I don't see any extra setting for speech.
I disabled the second language, first I had also intents in German
I also turned off the Fullfillment Webhook (implemented in API v1 and then also v2) with no change
I only found this user with the same problem https://productforums.google.com/forum/#!topic/dialogflow/xYjKlz31yW0;context-place=topicsearchin/dialogflow/Empty$20speech$20response but no resolution.
the fulfillment checkbox is checked at the intents
The bot works fine when I use it through "Try it now" on the very right in Dialog Flow or in the Web Demo https://bot.dialogflow.com/994dda8b-4849-4a8a-ab24-c0cd03b5f420
Unfortunately the docs don't say anything about this error. Any ideas?
Here a screenshot of the error on the Actions integration:
This is the full debug output:
{
"agentToAssistantDebug": {
"agentToAssistantJson": {
"message": "Failed to parse Dialogflow response into AppResponse, exception thrown with message: Empty speech response",
"apiResponse": {
"id": "c12e1389-e887-49d4-b399-a332188ca946",
"timestamp": "2018-01-27T03:55:30.931Z",
"lang": "en-us",
"result": {},
"status": {
"code": 200,
"errorType": "success"
},
"sessionId": "1517025330705"
}
}
},
"assistantToAgentDebug": {
"assistantToAgentJson": {
"user": {
"userId": "USER_ID",
"locale": "en-US",
"lastSeen": "2018-01-27T03:55:03Z"
},
"conversation": {
"conversationId": "1517025330705",
"type": "NEW"
},
"inputs": [
{
"intent": "actions.intent.MAIN",
"rawInputs": [
{
"inputType": "KEYBOARD",
"query": "Talk to Mica, the Hipster Cat Bot"
}
]
}
],
"surface": {
"capabilities": [
{
"name": "actions.capability.MEDIA_RESPONSE_AUDIO"
},
{
"name": "actions.capability.WEB_BROWSER"
},
{
"name": "actions.capability.AUDIO_OUTPUT"
},
{
"name": "actions.capability.SCREEN_OUTPUT"
}
]
},
"isInSandbox": true,
"availableSurfaces": [
{
"capabilities": [
{
"name": "actions.capability.AUDIO_OUTPUT"
},
{
"name": "actions.capability.SCREEN_OUTPUT"
}
]
}
]
},
"curlCommand": "curl -v 'https://api.api.ai/api/integrations/google?token=TOKEN' -H 'Content-Type: application/json;charset=UTF-8' -H 'Google-Actions-API-Version: 2' -H 'Authorization: AUTH_TOKEN' -A 'Mozilla/5.0 (compatible; Google-Cloud-Functions/2.1; +http://www.google.com/bot.html)' -X POST -d '{\"user\":{\"userId\":\"USER_ID\",\"locale\":\"en-US\",\"lastSeen\":\"2018-01-27T03:55:03Z\"},\"conversation\":{\"conversationId\":\"1517025330705\",\"type\":\"NEW\"},\"inputs\":[{\"intent\":\"actions.intent.MAIN\",\"rawInputs\":[{\"inputType\":\"KEYBOARD\",\"query\":\"Talk to Mica, the Hipster Cat Bot\"}]}],\"surface\":{\"capabilities\":[{\"name\":\"actions.capability.MEDIA_RESPONSE_AUDIO\"},{\"name\":\"actions.capability.WEB_BROWSER\"},{\"name\":\"actions.capability.AUDIO_OUTPUT\"},{\"name\":\"actions.capability.SCREEN_OUTPUT\"}]},\"isInSandbox\":true,\"availableSurfaces\":[{\"capabilities\":[{\"name\":\"actions.capability.AUDIO_OUTPUT\"},{\"name\":\"actions.capability.SCREEN_OUTPUT\"}]}]}'"
},
"sharedDebugInfo": [
{
"name": "ResponseValidation",
"subDebugEntry": [
{
"debugInfo": "API Version 2: Failed to parse JSON response string with 'INVALID_ARGUMENT' error: \": Cannot find field.\".",
"name": "UnparseableJsonResponse"
}
]
}
]
}
Also "debugInfo" sounds like an internal problem:
"API Version 2: Failed to parse JSON response string with
'INVALID_ARGUMENT' error: \": Cannot find field.\"."
Here a screenshot of the welcome intent:
ps.
It took me AGES to figure out, what
"Query pattern is missing for custom intent"
means - so I just document it here: In Dialog Flow - Intent - "User says" you have to DOUBLE CLICK on a word in the text input field when you want to set it as query parameter - which seems to be required for Actions on Google.
This happened to me. If this happens for an Intent you just added in the Dialogflow console and you are using Webhook fulfillment for the action, check the intent's fulfillment settings and ensure that the Webhook fulfillment slider is on. Evidently new intents don't automatically get webhook fulfillment: you have to opt each one in piecemeal (or at least, that was my experience).
I experienced this situation too.
My problem was that I used a SimpleResponse in my fulfillment index.js without referencing to it. So the solution for me was to add SimpleResponse like this in index.js:
const {dialogflow, SimpleResponse} = require('actions-on-google');
So, always check if you aren't using any dependencies without including it in your js-file.
Probably not the most common cause of the problem, but it can be.
I got this when running through the codelabs tutorial (https://codelabs.developers.google.com/codelabs/actions-1/index.html#4) and didn't name my intent the same name as it is referenced in the webhook script:
I came across this error when trying to develop my own WebHook. I first verified that my code was called by looking into the Nginx log, after which I knew there was a problem in my JSON output because I based my output on outdated examples.
The (up-to-date) documentation for both V1 and V2 of the API can be found here:
https://dialogflow.com/docs/fulfillment/how-it-works
This example response for v2 of the dialogflow webhook API helped me to resolve this error:
{
"payload": {
"google": {
"expectUserResponse": true,
"richResponse": {
"items": [
{
"simpleResponse": {
"textToSpeech": "this is a simple response"
}
}
]
}
}
}
}
Source: https://github.com/dialogflow/fulfillment-webhook-json/blob/master/responses/v2/ActionsOnGoogle/RichResponses/SimpleResponse.json
You can find more examples in the official github repository linked above.
Another possibility is if you have a text response (even an empty one) like so:
Then you need to click the trash can next to the response to clear it out to use the webhook.
The Actions on Google support helped me fix this problem:
I needed to add a text as Default Response to the intent used for Explicit Invocation.

Error Response with Google Smart Home (Thermostat)

I am unable to understand the syntax of Json for errors
I have tried
return {
"requestId": self.request_id,
"payload": {
"commands": [{
"ids": [self.device_id],
"status": "OFFLINE",
"errorCode": "deviceTurnedOff"
}]
}
}
and
return {
"requestId": self.request_id,
"payload": {
"errorCode": "authFailure",
"commands": [{
"devices": [{
"ids": [self.device_id],
"status": "ERROR",
"errorCode": "deviceTurnedOff"
}]
}]
}
}
and
return {
"requestId": self.request_id,
"payload": {
"errorCode": self.error_code
}
}
none of above syntax are working, most of the time google home says that your actions is performed but I am returning error Json there.
I have read google actions documentation but failed to understand.
I have come up with a few payloads and have managed to get them successfully triggering errors using the AoG simulator after consulting the docs.
Node.js snippet:
let resBody = {
requestId: request.requestId,
payload: {
"errorCode": "notSupported"
}
}
response.status(200).json(resBody);
Are you sure that the requestId is the same from the smart home request? How are you testing these errors?

unable to get my emails via graph API

I'm using the graph api to fetch my emails. and I'm using a postman client, I'm getting the authorize token and sending it to the api as header.
When I run the below query.
https://graph.microsoft.com/v1.0/me
the response that I get is
{
"#odata.context": "https://graph.microsoft.com/v1.0/$metadata#users/$entity",
"id": "someId",
"businessPhones": [],
"displayName": "MyName",
"givenName": "MyFirstName",
"jobTitle": "MyDesignation",
"mail": "myEmailAddress",
"mobilePhone": "MyMobileNumber",
"officeLocation": "MyWorkLocation",
"preferredLanguage": null,
"surname": "MySurName",
"userPrincipalName": "MyEmailAddress"
}
and when I try to run the below query.
https://graph.microsoft.com/v1.0/me/messages
The response that I get is as below.
{
"error": {
"code": "ResourceNotFound",
"message": "Resource could not be discovered.",
"innerError": {
"request-id": "59a0a25b-67c4-4fe8-8aff-eb035b974a4b",
"date": "2017-03-10T14:09:13"
}
}
}
Also tried the below URL to get the emails.
https://graph.microsoft.com/v1.0/me/mailFolders/Inbox/messages
I get the same response as above.
Finally I tried
https://graph.microsoft.com/v1.0/me/mail
Here the response that I got is
{
"#odata.context": "https://graph.microsoft.com/v1.0/$metadata#users('someRandomId')/mail",
"value": "myEmailAddress"
}
please let me know how can I get my emails shown.
In the portal(https://apps.dev.microsoft.com), I've the below things enabled for this application.
Thanks