Related
For Basic Card Issue: We are sending one Button but on assistant we are
seeing two button one without any text.
Here's sample Request to Google:
{
"expectUserResponse": true,
"expectedInputs": [
{
"possibleIntents": [
{
"intent": "actions.intent.TEXT"
}
],
"inputPrompt": {
"richInitialPrompt": {
"items": [
{
"simpleResponse": {
"textToSpeech": "<speak>The NAV for Franklin India Bluechip Fund as of 24 Jan 2022 is: \nDirect-Growth: 744.3406 \nDirect-Idcw: 48.0136 \nGrowth: 691.6646 \nIdcw: 42.6334 \n\nFor more information on the historical NAV of Franklin India Bluechip Fund, please visit our website at www.franklintempletonindia.com. Is there anything else, I can help you with?</speak>"
}
},
{
"basicCard": {
"buttons": [
{
"title": "Historical NAV",
"openUrlAction": {
"url": "https://www.franklintempletonindia.com/investor/fund-details/fund-historicalnavs/-4614"
}
}
]
}
}
]
}
}
}
]
}
Here's how it looks on smartphone, with one extra button with no text.
Can any body why is that so, and what's the issue here.
Check The Screenshot Here
I was connected my chatbot to google assistant action. They give only the userID, how to get username by using this userID?
You can get username without knowing userid, by the permissions document here. You can take a look at this sample code.
Or you can use account linking feature.
Tip! for userID, you can check out this doc
For Python:
There is no official library for developing google action using Python but,
You can add permission intent in possibleIntent array. So your Action SDK JSON will be,
{
"expectUserResponse": true,
"expectedInputs": [
{
"inputPrompt": {
"richInitialPrompt": {
"items": [
{
"simpleResponse": {
"textToSpeech": "PLACEHOLDER"
}
}
]
}
},
"possibleIntents": [
{
"intent": "actions.intent.PERMISSION",
"inputValueData": {
"#type": "type.googleapis.com/google.actions.v2.PermissionValueSpec",
"optContext": "To address you by name and know your location",
"permissions": [
"NAME",
"DEVICE_PRECISE_LOCATION"
]
}
}
]
}
],
"conversationToken": "{\"data\":{}}",
"userStorage": "{\"data\":{}}"
}
{`"actions": [
{
"description": "Default Welcome Intent",
"name": "MAIN",
"fulfillment": {
"conversationName": "welcome"
},
"intent": {
"name": "actions.intent.MAIN",
"trigger": {
"queryPatterns":["talk to Mr Bot"]
}
}
},
{
"description": "Rasa Intent",
"name": "TEXT",
"fulfillment": {
"conversationName": "rasa_intent"
},
"intent": {
"name": "actions.intent.TEXT",
"trigger": {
"queryPatterns":[]
}
}
}],
"conversations": {
"welcome": {
"name": "welcome",
"url": "https://ac752bb0.ngrok.io/webhooks/google_home/webhook",
"fulfillmentApiVersion": 2
},
"rasa_intent": {
"name": "rasa_intent",
"url": "https://ac752bb0.ngrok.io/webhooks/google_home/webhook",
"fulfillmentApiVersion": 2
}
} }
this is my action.json,
class GoogleConnector(InputChannel):
#classmethod
def name(cls):
return "google_home"
#def __init__(self):
# self.out_channel = CustomOutput(url, access_token)
def blueprint(self, on_new_message):
google_webhook = Blueprint('google_webhook', __name__)
#google_webhook.route("/", methods=['GET'])
def health():
return jsonify({"status": "ok"})
#google_webhook.route("/webhook", methods=['POST'])
def receive():
payload = json.loads(request.data)
sender_id = payload['user']['userId']
intent = payload['inputs'][0]['intent']
text = payload['inputs'][0]['rawInputs'][0]['query']
if intent == 'actions.intent.MAIN':
message = "<speak>Hello! <break time=\"1\"/> Welcome to the Rasa-powered Google Assistant skill. You can start by saying hi."
else:
out = CollectingOutputChannel()
on_new_message(UserMessage(text, out, sender_id))
responses = [m["text"] for m in out.messages]
message = responses[0]
r = json.dumps(
{
"conversationToken": "{\"state\":null,\"data\":{}}",
"expectUserResponse": 'true',
"expectedInputs": [
{
"inputPrompt": {
"initialPrompts": [
{
"ssml": message
}
]
},
"possibleIntents": [
{
"intent": "actions.intent.TEXT"
}
]
}
]
})
return r
return google_webhook
this my google connector python code,
how to modified this for account signin
I am connecting to Dialogflow REST API v2beta1 to the method: projects.agent.sessions.detectIntent. In the first request I send a text and the response is returning the expected result which contains outputContexts; when I made the 2nd request I send the context and it should find the intent which is linked to that context, but instead of that it is returning the Default Fallback Intent.
What may be the problem on the 2nd request?
Here are the URL and requests with their respective responses, and below I added the screenshots of the intents expected to match.
URL
https://dialogflow.googleapis.com/v2beta1/projects/project-name/agent/sessions/12343:detectIntent
1st request
{
"queryInput":{
"text":{
"text":"play a video about love",
"languageCode":"en"
}
}
}
1st response
{
"responseId": "15a3b767-52fe-4fc2-8ffd-9d7bb9c6961a",
"queryResult": {
"queryText": "play a video about love",
"action": "video.play",
"parameters": {
"organization": "",
"tag": "Love",
"item": ""
},
"allRequiredParamsPresent": true,
"fulfillmentText": "Here is a video about Love!",
"fulfillmentMessages": [
{
"platform": "ACTIONS_ON_GOOGLE",
"simpleResponses": {
"simpleResponses": [
{
"textToSpeech": "Here is a video about Love!"
}
]
}
},
{
"text": {
"text": [
"Here is a video about Love!"
]
}
}
],
"outputContexts": [
{
"name": "projects/project-name/agent/sessions/12343/contexts/play-video",
"lifespanCount": 5,
"parameters": {
"tag": "Love",
"organization": "",
"tag.original": "love",
"item": "",
"organization.original": "",
"item.original": ""
}
}
],
"intent": {
"name": "projects/project-name/agent/intents/9e5d2bbc-81f3-4700-8740-01504b05443f",
"displayName": "video-play"
},
"intentDetectionConfidence": 1,
"languageCode": "en"
}
}
2nd request (where the problem should be)
{
"queryParams":{
"contexts":[
{
"name":"projects/project-name/agent/sessions/12342/contexts/play-video"
}
]
},
"queryInput":{
"text":{
"text":"that video matters a lot for me",
"languageCode":"en"
}
}
}
2nd response
{
"responseId": "40d1f94f-4673-4644-aa53-99c854ff2596",
"queryResult": {
"queryText": "that video matters a lot for me",
"action": "input.unknown",
"parameters": {},
"allRequiredParamsPresent": true,
"fulfillmentText": "Can you say that again?",
"fulfillmentMessages": [
{
"text": {
"text": [
"Sorry, what was that?"
]
}
}
],
"intent": {
"name": "projects/project-name/agent/intents/10c88e8d-f16a-4905-b829-f596d3b3c588",
"displayName": "Default Fallback Intent",
"isFallback": true
},
"intentDetectionConfidence": 1,
"languageCode": "en"
}
}
Screenshots of the intents expected to match
1st intent
2nd intent
Useful documentation
Doc of the method: https://dialogflow.com/docs/reference/api-v2/rest/v2beta1/projects.agent.sessions/detectIntent
Doc of the Context object: https://dialogflow.com/docs/reference/api-v2/rest/Shared.Types/Context
Doc of the Params object to be sent: https://dialogflow.com/docs/reference/api-v2/rest/v2beta1/QueryParameters
It looks like your second request has an incomplete context. Although you're specifying the name, you're not including the lifespanCount parameter. Since you're not providing a parameter, it defaults to 0, which means that it has timed out.
You should send back exactly what you received from the outputContext attribute in the previous reply.
{
"queryParams":{
"contexts":[
{
"name": "projects/project-name/agent/sessions/12343/contexts/play-video",
"lifespanCount": 5,
"parameters": {
"tag": "Love",
"organization": "",
"tag.original": "love",
"item": "",
"organization.original": "",
"item.original": ""
}
}
]
},
"queryInput":{
"text":{
"text":"that video matters a lot for me",
"languageCode":"en"
}
}
}
I want to create a chatbot with Dialogflow and Google Assistant along with Google Transactions API for enabling a user to order a chocolate box. For now my agent contains the following four intents:
Default Welcome Intent (text response: Hello, do you want to buy a chocolate box?)
Default Fallback Intent
Int1 (training phrase: Yes, I want, fulfilment: enabled webhook call)
Int2 (event: actions_intent_TRANSACTION_REQUIREMENTS_CHECK )
I am using Dialogflow Json instead of Node.js to connect my agent with Transactions API. I want to test that the user meets the transaction requirements (when ordering the chocolate box) by using the actions.intent.TRANSACTION_REQUIREMENTS_CHECK action of Google actions. For this reason, following Google docs, when Int1 is triggered I am using a webhook which connect Google Assistant to the following python script (back-end):
from flask import Flask, render_template, request, jsonify
from flask_cors import CORS
import requests
app = Flask(__name__)
CORS(app)
#app.route("/", methods=['POST'])
def index():
data = request.get_json()
intent = data["queryResult"]["intent"]["displayName"]
if (intent == 'Int1'):
return jsonify({ "data": {
"google": {
"expectUserResponse": True,
"isSsml": False,
"noInputPrompts": [],
"systemIntent": {
"data": {
"#type": "type.googleapis.com/google.actions.v2.TransactionRequirementsCheckSpec",
"paymentOptions": {
"actionProvidedOptions": {
"displayName": "VISA-1234",
"paymentType": "PAYMENT_CARD"
}
}
},
"intent": "actions.intent.TRANSACTION_REQUIREMENTS_CHECK"
}
}
}
})
else:
return jsonify({'message': 'HERE'})
if __name__== "__main__":
app.run(debug=True)
The json which I return above when intent = 'Int1' is the one specified at Google docs for "Checking requirements with your own payment method".
According to Google docs, this must be done next:
Receiving the result of a requirements check
After the Assistant fulfills the intent, it sends your fulfillment a
request with the actions.intent.TRANSACTION_REQUIREMENTS_CHECK intent
with the result of the check.
To properly handle this request, declare a Dialogflow intent that's
triggered by the actions_intent_TRANSACTION_REQUIREMENTS_CHECK event.
For this reason, I defined Int2 and as its event the actions_intent_TRANSACTION_REQUIREMENTS_CHECK.
However, I do not receive anything at my back-end like a result of the check and therefore I do not know if the action actions.intent.TRANSACTION_REQUIREMENTS_CHECK is really triggered. Why is this happening?
In general, how can I trigger one actions.intent.INTENT_NAME intent from my webhook/back-end?
When I am using the v2 version of Dialogflow, I am getting the following info/message about the webhook on Dialogflow when Int1 is triggered:
"webhookStatus": {
"code": 3,
"message": "Webhook call failed. Error: Failed to parse webhook JSON response: Cannot find field: data in message google.cloud.dialogflow.v2.WebhookResponse."
}
In the same case, I am getting the following info/message about the webhook on Google Assistant simulator when Int1 is triggered:
"responseMetadata": {
"status": {
"code": 14,
"message": "Webhook error (206)"
}
Finally, let me mention that I am testing all this with Python and ngrok at my local computer so perhaps this poses a problem because at the beginning of Google docs the following is mentioned:
Warning: The Actions Web Simulator should not be used to test an app
with transactions. Please use an Assistant-enabled Android or iOS
device to accurately test your app during development.
I finally solved this problem.
I had to replace the key "data" in the json which I was sending back when Int1 was triggered with the key "payload". In other words, I had to adjust my fulfilment response to the v2 version of Dialogflow.
Therefore, now I do get a second post request at my back-end which is sent because of the trigger of actions.intent.TRANSACTION_REQUIREMENTS_CHECK and of Int2.
Specifically, I get the following:
{
"responseId": "*****************************",
"queryResult": {
"queryText": "actions_intent_TRANSACTION_REQUIREMENTS_CHECK",
"parameters": {},
"allRequiredParamsPresent": true,
"fulfillmentText": "HERE",
"fulfillmentMessages": [
{
"text": {
"text": [
"HERE"
]
}
}
],
"outputContexts": [
{
"name": "*****************************"
},
{
"name": "*****************************"
},
{
"name": "*****************************"
},
{
"name": "*****************************"
},
{
"name": "*****************************"
},
{
"name": "*****************************",
"parameters": {
"TRANSACTION_REQUIREMENTS_CHECK_RESULT": {
"#type": "type.googleapis.com/google.actions.v2.TransactionRequirementsCheckResult",
"resultType": "OK"
}
}
}
],
"intent": {
"name": "*****************************",
"displayName": "Int2"
},
"intentDetectionConfidence": 1,
"diagnosticInfo": {},
"languageCode": "en-us"
},
"originalDetectIntentRequest": {
"source": "google",
"version": "2",
"payload": {
"isInSandbox": true,
"surface": {
"capabilities": [
{
"name": "actions.capability.WEB_BROWSER"
},
{
"name": "actions.capability.MEDIA_RESPONSE_AUDIO"
},
{
"name": "actions.capability.SCREEN_OUTPUT"
},
{
"name": "actions.capability.AUDIO_OUTPUT"
}
]
},
"inputs": [
{
"rawInputs": [
{
"inputType": "KEYBOARD"
}
],
"arguments": [
{
"extension": {
"#type": "type.googleapis.com/google.actions.v2.TransactionRequirementsCheckResult",
"resultType": "OK"
},
"name": "TRANSACTION_REQUIREMENTS_CHECK_RESULT"
}
],
"intent": "actions.intent.TRANSACTION_REQUIREMENTS_CHECK"
}
],
"user": {
"lastSeen": "2018-05-16T11:15:14Z",
"locale": "en-US",
"userId": "*****************************"
},
"conversation": {
"conversationId": "1526470000479",
"type": "ACTIVE",
"conversationToken": "[]"
},
"availableSurfaces": [
{
"capabilities": [
{
"name": "actions.capability.SCREEN_OUTPUT"
},
{
"name": "actions.capability.AUDIO_OUTPUT"
}
]
}
]
}
},
"session": "*****************************"
}
I think your response object is incorrect. the intent attribute should be inside the systemIntent object
"data": {
"google": {
"expectUserResponse": true,
"isSsml": false,
"noInputPrompts": [],
"systemIntent": {
"intent": "actions.intent.TRANSACTION_REQUIREMENTS_CHECK",
"data": {
"#type": "type.googleapis.com/google.actions.v2.TransactionRequirementsCheckSpec",
"paymentOptions": {
"actionProvidedOptions": {
"displayName": "VISA-1234",
"paymentType": "PAYMENT_CARD"
}
}
}
}
}
}
I have been unable to get google-actions sdk for node.js to pass arguments. I installed the https://github.com/actions-on-google/actionssdk-eliza-nodejs sample project and noticed arguments are not working for that project either. Any insight?
In the web simulator I entered "i am feeling sad"
Here is the request I get
{
"query": "i am feeling sad",
"accessToken": "**masked**",
"expectUserResponse": true,
"conversationToken": "CiZDIzU4O..."content_copy,
"debugInfo": {
"assistantToAgentDebug": {
"assistantToAgentJson": {
"user": {
"user_id": "**masked**"
},
"conversation": {
"conversation_id": "1484523995718",
"type": 2,
"conversation_token": "{\"elizaInstance\":{\"noRandom\":false,\"capitalizeFirstLetter\":true,\"debug\":false,\"memSize\":20,\"version\":\"1.1 (original)\",\"quit\":false,\"mem\":[],\"lastchoice\":[[-1],[-1],[-1],[-1],[-1,-1,-1],[-1,-1],[-1],[-1],[-1],[-1,-1,-1],[-1,-1,-1],[-1],[-1],[-1],[-1],[-1],[-1],[-1],[-1],[-1],[-1],[-1],[-1],[-1],[-1],[-1],[-1,0,-1],[-1,-1,-1],[-1],[-1,0,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1],[-1,-1,-1,-1],[-1],[-1,-1],[-1,-1],[-1],[-1],[-1],[-1],[-1],[-1],[-1,-1,-1],[-1]],\"sentence\":\"i am feeling sad\"}}"
},
"inputs": [
{
"intent": "assistant.intent.action.TEXT",
"raw_inputs": [
{
"input_type": 2,
"query": "i am feeling sad"
}
],
"arguments": [
{
"name": "text",
"raw_text": "i am feeling sad",
"text_value": "i am feeling sad"
}
]
}
]
}
}
}
}
text_value should = "sad", not "i am feeling sad" based on eliza.json which has this:
{
"versionLabel": "Eliza v1",
"agentInfo": {
"languageCode": "en-US",
"projectId": "**masked**",
"invocationNames": [
"eliza"
],
"voiceName": "female_1"
},
"actions": [
{
"description": "Start an Eliza consultation",
"initialTrigger": {
"intent": "assistant.intent.action.MAIN"
},
"httpExecution": {
"url": "https://**masked**"
}
},
{
"description": "Deep link to Eliza consultation",
"initialTrigger": {
"intent": "raw.input",
"queryPatterns": [
{
"queryPattern": "my emotional state is $SchemaOrg_Text:text"
},
{
"queryPattern": "I am concerned about $SchemaOrg_Text:text"
},
{
"queryPattern": "I am feeling $SchemaOrg_Text:text"
},
{
"queryPattern": "I need to talk about my feelings"
}
]
},
"httpExecution": {
"url": "**masked**"
}
}
],
"deploymentStatus": {
"state": "NEW"
},
"versionId": "1"
}
The Google Actions SDK will parse patterns and correctly bind arguments for intents that activate your action. For example, if you say
At my action I am feeling sad
You will recieve arguments that look like this
"arguments": [
{
"name": "text",
"raw_text": "sad",
"text_value": "sad"
},
{
"name": "trigger_query",
"raw_text": "i am feeling sad",
"text_value": "i am feeling sad"
}
]
However, if you request input from the action by setting expect_user_response to true, the arguments are always returned in raw form using the intent "assistant.intent.action.TEXT". It is up to you to parse those raw arguments as indicated in the other answer to this question.
Also note that as far as I can tell, the Schema_Org types are not returned per the documentation. More information here.
If you are using the Actions SDK, then you have to use your own NLP for understanding the raw user query and for extracting arguments. If you don't have you own NLP, then we recommend that you use API.AI.