Actions-On-Google[Permission Intent] Get User Location / Name - actions-on-google

const agent = new WebhookClient({request, response});
const {WebhookClient} = require('dialogflow-fulfillment');
const {Text, Card, Image, Suggestion, Payload} = require('dialogflow-fulfillment');
let payload = {
"systemIntent": {
"intent": "actions.intent.PERMISSION",
"data": {
"#type": "type.googleapis.com/google.actions.v2.PermissionValueSpec",
"optContext": "To deliver your order",
"permissions": [
"NAME",
"DEVICE_PRECISE_LOCATION"
]
}
}
};
agent.add('PLACEHOLDER_FOR_PERMISSION');
agent.add(new Payload(PLATFORMS.ACTIONS_ON_GOOGLE, payload));
Simple payload to get UserLocation and Name using the PERMISSION intent
The response to the above I get it
To deliver your order, I'll need to get your name and street address
from Google. Is that ok?
Follow intent to this intent is also set up with event actions_intent_PERMISSION in it
I have been trying to solve this for 2 days by trying to fire actions_intent_PERMISSION using suggestion chips etc but nothing happens post this ?
Where am I going wrong I am not able to comprehend. There is some silly mistake somewhere - please if someone can point it out - would help a lot.
Thanks
===========EDIT============IMAGES ATTACHED FOR THE INTENTS============
permissions intent
permissions response with event actions_intent_PERMISSION
edit: cant embed images because of points. above links are there. thanks
=================================request-response json===================
when the intent permissions is triggered below is the request
{
"responseId": "54a4be35-3d0b-4cc8-b036-46fab0d09361",
"queryResult": {
"queryText": "permissions",
"action": "permissions",
"parameters": {},
"allRequiredParamsPresent": true,
"fulfillmentMessages": [
{
"text": {
"text": [
""
]
}
}
],
"intent": {
"name": "projects/projectid/agent/intents/95237653-0af0-4d0c-9101-0cd8ee0db186",
"displayName": "permissions"
},
"intentDetectionConfidence": 1,
"diagnosticInfo": {},
"languageCode": "en"
},
"originalDetectIntentRequest": {
"payload": {}
},
"session": "projects/projectid/agent/sessions/13213e7f-dba5-4d0c-979a-f626f7ac4691"
}
fulfillment response
{
"conversationToken": "[]",
"expectUserResponse": true,
"expectedInputs": [
{
"inputPrompt": {},
"possibleIntents": [
{
"intent": "actions.intent.PERMISSION",
"inputValueData": {
"#type": "type.googleapis.com/google.actions.v2.PermissionValueSpec",
"permissions": [
"NAME",
"DEVICE_PRECISE_LOCATION"
],
"optContext": "To locate you"
}
}
],
"speechBiasingHints": [
"$name-type",
"$sports",
"$gender"
]
}
],
"responseMetadata": {
"status": {
"message": "Success (200)"
},
"queryMatchInfo": {
"queryMatched": true,
"intent": "95237653-0af0-4d0c-9101-0cd8ee0db186"
}
}
}
The response from simulator for permissions intent
=======================================================================
Issue has been resolved
Points to note:
Actions-on-Google simulator works very weirdly to be trusted upon
whether your webhook is working or not
Promise resolution was an
issue - agent was waiting for a promise to be resolved before that
it was getting passed to next time
Correct way to test your bot
is to publish your bot in ALPHA on the assistant directory rather
than testing on the Simulator because it is very unstable in terms
of you can never predict its behavior. Will never tell you the
correct error to be debugged and will abruptly stop working for no
reason whatsoever

Related

Messenger bot with dialogflow does not respond to user

I have a messenger bot built with google's dialogflow engine, to respond the user I have a webhook. when I send a message with a "hello", the default welcome intent its triggered and the webhook it its called and it responds to me.
After that I need the products of my business so I write "Comprar" (Buy), the Products intent it is triggered and the webhook responds with a fulfillment message with a list of available products.
In the history section in google dialogflow, I can see the json respond of the wekhook with the message that the user will receive.
But facebook does not show the message. I use the same webhook for many intents, and it is not working in this case.
this is the respond I got in the History section for this case:
{
"id": "f75002cd-2f8a-422a-960d-dd2c9e00b490-74fe87bc",
"fulfillmentText": "",
"language_code": "es",
"queryText": "Comprar",
"webhookPayload": {},
"intentDetectionConfidence": 1,
"action": "GET_CATEGORIES",
"webhookSource": "",
"parameters": {},
"fulfillmentMessages": [
{
"quickReplies": {
"title": "Escoge alguna de estas Categorías:",
"quickReplies": [
"Volver",
"",
"FIRMA ELECTRONICA"
]
}
}
],
"diagnosticInfo": {
"webhook_latency_ms": "732.0"
},
"webhookStatus": {
"webhookStatus": {
"message": "Webhook execution successful"
},
"webhookUsed": true
},
"outputContexts": [
{
"lifespanCount": 1,
"name": "vercatalogo-followup",
"parameters": {}
}
],
"intent": {
"isFallback": false,
"displayName": "ver.catalogo",
"id": "41dcecf2-11b6-4e50-8294-b86d849093e1"
}
}
I solved lol
Apparently quickReplies must not contain a empty string ("") as a quick reply, that option is not valid. So I removed it and it worked.

No token in request - Account Linking with new Actions SDK

I am building an action with the new Actions Builder and everything is going pretty smoothly. I just setup account linking and can successfully link my account, however, once I do link my account there is no token included in the subsequent requests for me to use, even though the account linking status is in the request as "LINKED". Can anyone shed any light on why I am not seeing a token?
For reference, here is a version of one of my requests.
{
"requestJson": {
"handler": {
"name": "main"
},
"intent": {
"name": "actions.intent.MAIN",
"params": {},
"query": "Talk to my new app"
},
"scene": {
"name": "actions.scene.START_CONVERSATION",
"slotFillingStatus": "UNSPECIFIED",
"slots": {},
"next": {
"name": "ListPrompt"
}
},
"session": {
"id": "ABwppHE7M6NS8KdyjljEptrtZZ5GkE3qDdaiwjYbL9ehrA-t_c-ZsCrZ_WhN0ZTG5lXXXXXXhU6Im5vgeSwow",
"params": {},
"typeOverrides": [],
"languageCode": ""
},
"user": {
"locale": "en-US",
"params": {},
"accountLinkingStatus": "LINKED",
"verificationStatus": "VERIFIED",
"packageEntitlements": [],
"lastSeenTime": "2020-07-13T12:02:42Z"
},
"home": {
"params": {}
},
"device": {
"capabilities": [
"SPEECH",
"RICH_RESPONSE",
"LONG_FORM_AUDIO"
]
}
}
}
The Google docs for the Account Linking with the new Actions Builder have now been updated with additional information. The token is now provided within the headers of the incoming request. Details of how to find and decode this can be found at https://developers.google.com/assistant/identity/google-sign-in#handle_data_access_requests

Google Action / Dialogflow : how to ask for geolocation

I'm trying to implement a simple app for Google Assistant.
All works fine, but now I have a problem with the "permission" helper :
https://developers.google.com/actions/assistant/helpers#helper_intents
I have an intent connected with webhook to my java application. When an user types a sentence similar to "near to me", I want to ask to him his location and then use lat/lon to perform a search.
es: Brazilian restaurant near to me
my intent "searchRestaurant" is fired
I receive the webhook request and I parse it
if I find a parameter that is connected to a sentence like "near to me", so instead to response with a "Card" or a "List" I return a JSON that represent the helper request :
{
"conversationToken": "[]",
"expectUserResponse": true,
"expectedInputs": [
{
"inputPrompt": {
"initialPrompts": [
{
"textToSpeech": "PLACEHOLDER_FOR_PERMISSION"
}
],
"noInputPrompts": []
},
"possibileIntents": [
{
"intent": "actions.intent.PERMISSION",
"inputValueData": {
"#type": "type.googleapis.com/google.actions.v2.PermissionValueSpec",
"optContext": "Posso accedere alla tua posizione?",
"permission": [
"NAME",
"DEVICE_PRECISE_LOCATION"
]
}
}
]
}
]
}
but something seems to be wrong, and I receive an error:
"{\n \"responseMetadata\": {\n \"status\": {\n \"code\": 10,\n \"message\": \"Failed to parse Dialogflow response into AppResponse because of empty speech response\",\n \"details\": [{\n \"#type\": \"type.googleapis.com/google.protobuf.Value\",\n \"value\": \"{\\"id\\":\\"1cc45c5e-c398-4ea7-98a5-408f31ce142d\\",\\"timestamp\\":\\"2018-08-02T14:45:05.752Z\\",\\"lang\\":\\"it\\",\\"result\\":{},\\"alternateResult\\":{},\\"status\\":{\\"code\\":206,\\"errorType\\":\\"partial_content\\",\\"errorDetails\\":\\"Webhook call failed. Error: Failed to parse webhook JSON response: Cannot find field: conversationToken in message google.cloud.dialogflow.v2.WebhookResponse.\\"},\\"sessionId\\":\\"1533221100163\\"}\"\n }]\n }\n }\n}"
The "conversationToken" is filled, so I don't understand the error message.
Maybe I'm trying to perform the operation in a wrong way.
So, which is the correct way to call this helper?
--> I've created a second intent "askGeolocation" that have "actions_intent_PERMISSION" as "Event", and ... if I understand correctly the documentation, should be trigger if the request for helper is correct.
How can I get this working?
UPDATE :
I find some example of the json response for ask permission and seems that it should be different from the one above that i'm using :
https://github.com/dialogflow/fulfillment-webhook-json/blob/master/responses/v2/ActionsOnGoogle/AskForPermission.json
{
"payload": {
"google": {
"expectUserResponse": true,
"systemIntent": {
"intent": "actions.intent.PERMISSION",
"data": {
"#type": "type.googleapis.com/google.actions.v2.PermissionValueSpec",
"optContext": "To deliver your order",
"permissions": [
"NAME",
"DEVICE_PRECISE_LOCATION"
]
}
}
}
}
}
so, i've implemented it and now the response seems to be good (no more error on parsing it), but i still receive an error on it validation :
UnparseableJsonResponse
API Version 2: Failed to parse JSON response string with 'INVALID_ARGUMENT' error: "permission: Cannot find field."
so, a problem still persist.
Anyone know the cause?
Thanks
After some tests i found the problem.
I was returning a wrong json repsonse with "permission" instead of "permissions":
"permission**s**": [
"NAME",
"DEVICE_PRECISE_LOCATION"
]
So the steps to ask for location are correct. I report them here as a little tutorial in order to help who is facing on it for the first time:
1) In DialogFlow, add some logic to your Intent, in order to understand when is ok to ask to user his location. In my case, i've added a "parameter" that identify sentences like "nearby" and so on.
2) When my Intent is fired i receive to my java application a request like this :
...
"queryResult": {
"queryText": "ristorante argentino qui vicino",
"action": "bycategory",
"parameters": {
"askgeolocation": "qui vicino",
"TipoRistorante": ["ristorante", "argentino"]
},
...
3) If "askgeolocation" parameter is filled, instead to return a "simple message" o other type of message, i return a json for ask the permission to geolocation :
{
"payload": {
"google": {
"expectUserResponse": true,
"systemIntent": {
"intent": "actions.intent.PERMISSION",
"data": {
"#type": "type.googleapis.com/google.actions.v2.PermissionValueSpec",
"optContext": "To deliver your order",
"permissions": [
"NAME",
"DEVICE_PRECISE_LOCATION"
]
}
}
}
}
}
4) You MUST have a second Intent that is configured with "actions_intent_PERMISSION " event :
No training phrases
No Action and params
No Responses
But with Fulfillment active :
5) Once your response arrive to Google Assistant this is the message that appear :
6) Now, if user answer "ok" you receive this json on your webhook :
{
"responseId": "32cf46cf-80d8-xxxxxxxxxxxxxxxxxxxxx",
"queryResult": {
"queryText": "actions_intent_PERMISSION",
"action": "geoposition",
"parameters": {
},
"allRequiredParamsPresent": true,
"fulfillmentMessages": [{
"text": {
"text": [""]
}
}],
"outputContexts": [{
"name": "projects/xxxxxxxxxxxxxxxxxxxxx"
}, {
"name": "projects/xxxxxxxxxxxxxxxxxxxxx",
"parameters": {
"PERMISSION": true
}
}, {
"name": "projects/xxxxxxxxxxxxxxxxxxxxx"
}, {
"name": "projects/xxxxxxxxxxxxxxxxxxxxx"
}, {
"name": "projects/xxxxxxxxxxxxxxxxxxxxx"
}, {
"name": "projects/xxxxxxxxxxxxxxxxxxxxx"
}],
"intent": {
"name": "projects/xxxxxxxxxxxxxxxxxxxxx",
"displayName": "geoposition"
},
"intentDetectionConfidence": 1.0,
"languageCode": "it"
},
"originalDetectIntentRequest": {
"source": "google",
"version": "2",
"payload": {
"isInSandbox": true,
"surface": {
"capabilities": [{
"name": "actions.capability.MEDIA_RESPONSE_AUDIO"
}, {
"name": "actions.capability.SCREEN_OUTPUT"
}, {
"name": "actions.capability.AUDIO_OUTPUT"
}, {
"name": "actions.capability.WEB_BROWSER"
}]
},
"requestType": "SIMULATOR",
"inputs": [{
"rawInputs": [{
"inputType": "KEYBOARD"
}],
"arguments": [{
"textValue": "true",
"name": "PERMISSION",
"boolValue": true
}],
"intent": "actions.intent.PERMISSION"
}],
"user": {
"lastSeen": "2018-08-03T08:55:20Z",
"permissions": ["NAME", "DEVICE_PRECISE_LOCATION"],
"profile": {
"displayName": ".... full name of the user ...",
"givenName": "... name ...",
"familyName": "... surname ..."
},
"locale": "it-IT",
"userId": "xxxxxxxxxxxxxxxxxxxxx"
},
"device": {
"location": {
"coordinates": {
"latitude": 45.xxxxxx,
"longitude": 9.xxxxxx
}
}
},
"conversation": {
"conversationId": "xxxxxxxxxxxxxxxxxxxxx",
"type": "ACTIVE",
"conversationToken": "[]"
},
"availableSurfaces": [{
"capabilities": [{
"name": "actions.capability.SCREEN_OUTPUT"
}, {
"name": "actions.capability.AUDIO_OUTPUT"
}, {
"name": "actions.capability.WEB_BROWSER"
}]
}]
}
},
"session": "projects/xxxxxxxxxxxxxxxxxxxxx"
}
that contains, name/surname and latitude/longitude. This information can be saved in your application, in order to not perform again this steps.
I hope this helps.
Davide
In your intent, you can ask for a parameter with a custom Entity. This you can do like this:
entity you can define as "near"
put all the synonyms for near for which you want to trigger location permission in this entity
do not mark this parameter as "required"
do not put any prompt
in the training phrases, add some phrases with this parameter
in your webhook, keep a check on the parameter, if present ask for permission if not continue.
add a permission event to another intent
do your post permission processing in that intent
Entity
Intent
I hope you get it.
There are samples on this topic specifically that guide you through exactly what's needed for requesting permissions in Node and Java.
Note: There are helper intents samples available in Node and Java as well.

How can I trigger a `action.intent.INTENT_NAME` intent from my webhook?

I want to create a chatbot with Dialogflow and Google Assistant along with Google Transactions API for enabling a user to order a chocolate box. For now my agent contains the following four intents:
Default Welcome Intent (text response: Hello, do you want to buy a chocolate box?)
Default Fallback Intent
Int1 (training phrase: Yes, I want, fulfilment: enabled webhook call)
Int2 (event: actions_intent_TRANSACTION_REQUIREMENTS_CHECK )
I am using Dialogflow Json instead of Node.js to connect my agent with Transactions API. I want to test that the user meets the transaction requirements (when ordering the chocolate box) by using the actions.intent.TRANSACTION_REQUIREMENTS_CHECK action of Google actions. For this reason, following Google docs, when Int1 is triggered I am using a webhook which connect Google Assistant to the following python script (back-end):
from flask import Flask, render_template, request, jsonify
from flask_cors import CORS
import requests
app = Flask(__name__)
CORS(app)
#app.route("/", methods=['POST'])
def index():
data = request.get_json()
intent = data["queryResult"]["intent"]["displayName"]
if (intent == 'Int1'):
return jsonify({ "data": {
"google": {
"expectUserResponse": True,
"isSsml": False,
"noInputPrompts": [],
"systemIntent": {
"data": {
"#type": "type.googleapis.com/google.actions.v2.TransactionRequirementsCheckSpec",
"paymentOptions": {
"actionProvidedOptions": {
"displayName": "VISA-1234",
"paymentType": "PAYMENT_CARD"
}
}
},
"intent": "actions.intent.TRANSACTION_REQUIREMENTS_CHECK"
}
}
}
})
else:
return jsonify({'message': 'HERE'})
if __name__== "__main__":
app.run(debug=True)
The json which I return above when intent = 'Int1' is the one specified at Google docs for "Checking requirements with your own payment method".
According to Google docs, this must be done next:
Receiving the result of a requirements check
After the Assistant fulfills the intent, it sends your fulfillment a
request with the actions.intent.TRANSACTION_REQUIREMENTS_CHECK intent
with the result of the check.
To properly handle this request, declare a Dialogflow intent that's
triggered by the actions_intent_TRANSACTION_REQUIREMENTS_CHECK event.
For this reason, I defined Int2 and as its event the actions_intent_TRANSACTION_REQUIREMENTS_CHECK.
However, I do not receive anything at my back-end like a result of the check and therefore I do not know if the action actions.intent.TRANSACTION_REQUIREMENTS_CHECK is really triggered. Why is this happening?
In general, how can I trigger one actions.intent.INTENT_NAME intent from my webhook/back-end?
When I am using the v2 version of Dialogflow, I am getting the following info/message about the webhook on Dialogflow when Int1 is triggered:
"webhookStatus": {
"code": 3,
"message": "Webhook call failed. Error: Failed to parse webhook JSON response: Cannot find field: data in message google.cloud.dialogflow.v2.WebhookResponse."
}
In the same case, I am getting the following info/message about the webhook on Google Assistant simulator when Int1 is triggered:
"responseMetadata": {
"status": {
"code": 14,
"message": "Webhook error (206)"
}
Finally, let me mention that I am testing all this with Python and ngrok at my local computer so perhaps this poses a problem because at the beginning of Google docs the following is mentioned:
Warning: The Actions Web Simulator should not be used to test an app
with transactions. Please use an Assistant-enabled Android or iOS
device to accurately test your app during development.
I finally solved this problem.
I had to replace the key "data" in the json which I was sending back when Int1 was triggered with the key "payload". In other words, I had to adjust my fulfilment response to the v2 version of Dialogflow.
Therefore, now I do get a second post request at my back-end which is sent because of the trigger of actions.intent.TRANSACTION_REQUIREMENTS_CHECK and of Int2.
Specifically, I get the following:
{
"responseId": "*****************************",
"queryResult": {
"queryText": "actions_intent_TRANSACTION_REQUIREMENTS_CHECK",
"parameters": {},
"allRequiredParamsPresent": true,
"fulfillmentText": "HERE",
"fulfillmentMessages": [
{
"text": {
"text": [
"HERE"
]
}
}
],
"outputContexts": [
{
"name": "*****************************"
},
{
"name": "*****************************"
},
{
"name": "*****************************"
},
{
"name": "*****************************"
},
{
"name": "*****************************"
},
{
"name": "*****************************",
"parameters": {
"TRANSACTION_REQUIREMENTS_CHECK_RESULT": {
"#type": "type.googleapis.com/google.actions.v2.TransactionRequirementsCheckResult",
"resultType": "OK"
}
}
}
],
"intent": {
"name": "*****************************",
"displayName": "Int2"
},
"intentDetectionConfidence": 1,
"diagnosticInfo": {},
"languageCode": "en-us"
},
"originalDetectIntentRequest": {
"source": "google",
"version": "2",
"payload": {
"isInSandbox": true,
"surface": {
"capabilities": [
{
"name": "actions.capability.WEB_BROWSER"
},
{
"name": "actions.capability.MEDIA_RESPONSE_AUDIO"
},
{
"name": "actions.capability.SCREEN_OUTPUT"
},
{
"name": "actions.capability.AUDIO_OUTPUT"
}
]
},
"inputs": [
{
"rawInputs": [
{
"inputType": "KEYBOARD"
}
],
"arguments": [
{
"extension": {
"#type": "type.googleapis.com/google.actions.v2.TransactionRequirementsCheckResult",
"resultType": "OK"
},
"name": "TRANSACTION_REQUIREMENTS_CHECK_RESULT"
}
],
"intent": "actions.intent.TRANSACTION_REQUIREMENTS_CHECK"
}
],
"user": {
"lastSeen": "2018-05-16T11:15:14Z",
"locale": "en-US",
"userId": "*****************************"
},
"conversation": {
"conversationId": "1526470000479",
"type": "ACTIVE",
"conversationToken": "[]"
},
"availableSurfaces": [
{
"capabilities": [
{
"name": "actions.capability.SCREEN_OUTPUT"
},
{
"name": "actions.capability.AUDIO_OUTPUT"
}
]
}
]
}
},
"session": "*****************************"
}
I think your response object is incorrect. the intent attribute should be inside the systemIntent object
"data": {
"google": {
"expectUserResponse": true,
"isSsml": false,
"noInputPrompts": [],
"systemIntent": {
"intent": "actions.intent.TRANSACTION_REQUIREMENTS_CHECK",
"data": {
"#type": "type.googleapis.com/google.actions.v2.TransactionRequirementsCheckSpec",
"paymentOptions": {
"actionProvidedOptions": {
"displayName": "VISA-1234",
"paymentType": "PAYMENT_CARD"
}
}
}
}
}
}

Google Home dialogFlow V2 API mediaResponse not working

I decided to upgrade my Google Assistant action to use "dialogFlow V2 API" and my webhook returns an object like this
{
"fulfillmentText": "Testing",
"fulfillmentMessages": [
{
"text": {
"text": [
"fulfillmentMessages text attribute"
]
}
}
],
"payload": {
"google": {
"richResponse": {
"items": [
{
"mediaResponse": {
"mediaType": "AUDIO",
"mediaObjects": [
{
"name": "mediaResponse name",
"description": "mediaResponse description",
"largeImage": {
"url": "https://.../640x480.jpg"
},
"contentUrl": "https://.../20183832714.mp3"
}
]
},
"simpleResponse": {
"textToSpeech": "simpleResponse: testing",
"ssml": "simpleResponse: ssml",
"displayText": "simpleResponse displayText"
}
}
]
}
}
},
"source": "webhook-play-sample"
}
But I get an error message saying my action it is not available, is mediaResponse supported by V2?, should I format my object differently?, also, when I remove "mediaResponse" object works just fine and the assistant will speak the simpleResponse part.
This action was re-created this Mid March 2018 and I read about May deadline and that is why I decide to upgrade to V2, do you think I should go back to V1, I know I will have to delete it and re-created but that is fine. This is a link to the JSON object I see in the debug tab. Thanks once again
I set "API V2" in my action dialogFlow console, this is a screenshot of that setting
Here is an screenshoot of my action's integration -> Google Assistant
Thanks Allen, Yes I do have "expectUserResponse": false, I added the suggestion object you recommended but, unfortunately nothing changed, I am still getting this error
Simulator debug tag details
First of all - this is not a problem with Dialogflow V2. You also seem to be confusing the sunset of Actions on Google V1 with the release of Dialogflow V2 - they are two different creatures completely. If your project was using AoG V1, there would be a setting on the Actions integration screen, and thee isn't.
It is fine if you want to move to Dialogflow V2, but it isn't required. Media definitely works under Dialogflow V2.
The array of items must include a simpleResponse item first, before any of the other items in the RichResponse. (You also shouldn't include both ssml and textToSpeech - just one of them.) You also don't need the fulfillmentText and fulfillmentMessages components, since those are provided by the richResponse.
You also need to include suggestions chips unless you have set expectUserResponse to false. Somewhere in the simulator debug is probably a block that says
{
"name": "MalformedResponse",
"debugInfo": "expected_inputs[0].input_prompt.rich_initial_prompt: Suggestions must be provided if media_response is used..",
"subDebugEntryList": []
}
So something more like this should work:
{
"payload": {
"google": {
"richResponse": {
"items": [
{
"simpleResponse": {
"textToSpeech": "simpleResponse: testing",
"displayText": "simpleResponse displayText"
},
"mediaResponse": {
"mediaType": "AUDIO",
"mediaObjects": [
{
"name": "mediaResponse name",
"description": "mediaResponse description",
"largeImage": {
"url": "https://.../640x480.jpg"
},
"contentUrl": "https://.../20183832714.mp3"
}
]
}
}
]
"suggestions": [
{
"title": "This"
},
{
"title": "That"
}
]
}
}
},
"source": "webhook-play-sample"
}