I have created an actions SDK app which I can view in my google assistant. On clicking account linkage is complete with Auth2.0, but SYNC intent is not getting triggered and app just shows a message "Something went wrong".
My action.json :
{
"actions": [{
"name": "actions.devices",
"fulfillment": {
"conversationName": "automation"
}
}],
"conversations": {
"automation" :
{
"name": "automation",
"url": "MY_APPLICATION_ENDPOINT"
}
}
}
Related
I am building an action with the new Actions Builder and everything is going pretty smoothly. I just setup account linking and can successfully link my account, however, once I do link my account there is no token included in the subsequent requests for me to use, even though the account linking status is in the request as "LINKED". Can anyone shed any light on why I am not seeing a token?
For reference, here is a version of one of my requests.
{
"requestJson": {
"handler": {
"name": "main"
},
"intent": {
"name": "actions.intent.MAIN",
"params": {},
"query": "Talk to my new app"
},
"scene": {
"name": "actions.scene.START_CONVERSATION",
"slotFillingStatus": "UNSPECIFIED",
"slots": {},
"next": {
"name": "ListPrompt"
}
},
"session": {
"id": "ABwppHE7M6NS8KdyjljEptrtZZ5GkE3qDdaiwjYbL9ehrA-t_c-ZsCrZ_WhN0ZTG5lXXXXXXhU6Im5vgeSwow",
"params": {},
"typeOverrides": [],
"languageCode": ""
},
"user": {
"locale": "en-US",
"params": {},
"accountLinkingStatus": "LINKED",
"verificationStatus": "VERIFIED",
"packageEntitlements": [],
"lastSeenTime": "2020-07-13T12:02:42Z"
},
"home": {
"params": {}
},
"device": {
"capabilities": [
"SPEECH",
"RICH_RESPONSE",
"LONG_FORM_AUDIO"
]
}
}
}
The Google docs for the Account Linking with the new Actions Builder have now been updated with additional information. The token is now provided within the headers of the incoming request. Details of how to find and decode this can be found at https://developers.google.com/assistant/identity/google-sign-in#handle_data_access_requests
I am trying to integrate camera to my smarthome with google assistant. I followed CameraStream in actions on google. i have synced my camera and tried execute and responding with an URL(cameraStreamAccessUrl).
Command: "show camera"
google home reply: "Sure, streaming camera" <-- but it is not streaming the video.
Command: "show camera on my phone"
google home reply: "Sorry, i don't know where to play the video. Please tell me the exact name of the screen"
My question is how can i stream it on my phone or on the app?
execution request google json:
{
"inputs": [
"intent": "action.devices.EXECUTE",
"payload": {
"commands": [
{
"devices": [
{
"customData": {
"barValue": true,
"bazValue": "lambtwirl",
"fooValue": 74
},
"id": "id"
}
],
"execution": [
{
"command": "action.devices.commands.GetCameraStream",
"params": {
"StreamToChromecast": false,
"SupportedStreamProtocols": [
"hls"
]
}
}
]
}
]
}
}], "requestId": "requestId"
}
my json response:
{
"requestId": "requestId",
"payload": {
"commands": [
{
"ids": [
"requestId"
],
"status": "SUCCESS",
"states": {
"cameraStreamAccessUrl": "https://url.url"
}
}
]
}
}
If you have a Chromecast in your home, the Google Home will forward the video stream to the Chromecast. I don't know where the stream would go if you don't have a display.
Saying "on my phone" is not a valid target from a Google Home. To get it on your phone, you'd need to use the Assistant on your phone or the Google Home App.
I want to create a chatbot with Dialogflow and Google Assistant along with Google Transactions API for enabling a user to order a chocolate box. For now my agent contains the following four intents:
Default Welcome Intent (text response: Hello, do you want to buy a chocolate box?)
Default Fallback Intent
Int1 (training phrase: Yes, I want, fulfilment: enabled webhook call)
Int2 (event: actions_intent_TRANSACTION_REQUIREMENTS_CHECK )
I am using Dialogflow Json instead of Node.js to connect my agent with Transactions API. I want to test that the user meets the transaction requirements (when ordering the chocolate box) by using the actions.intent.TRANSACTION_REQUIREMENTS_CHECK action of Google actions. For this reason, following Google docs, when Int1 is triggered I am using a webhook which connect Google Assistant to the following python script (back-end):
from flask import Flask, render_template, request, jsonify
from flask_cors import CORS
import requests
app = Flask(__name__)
CORS(app)
#app.route("/", methods=['POST'])
def index():
data = request.get_json()
intent = data["queryResult"]["intent"]["displayName"]
if (intent == 'Int1'):
return jsonify({ "data": {
"google": {
"expectUserResponse": True,
"isSsml": False,
"noInputPrompts": [],
"systemIntent": {
"data": {
"#type": "type.googleapis.com/google.actions.v2.TransactionRequirementsCheckSpec",
"paymentOptions": {
"actionProvidedOptions": {
"displayName": "VISA-1234",
"paymentType": "PAYMENT_CARD"
}
}
},
"intent": "actions.intent.TRANSACTION_REQUIREMENTS_CHECK"
}
}
}
})
else:
return jsonify({'message': 'HERE'})
if __name__== "__main__":
app.run(debug=True)
The json which I return above when intent = 'Int1' is the one specified at Google docs for "Checking requirements with your own payment method".
According to Google docs, this must be done next:
Receiving the result of a requirements check
After the Assistant fulfills the intent, it sends your fulfillment a
request with the actions.intent.TRANSACTION_REQUIREMENTS_CHECK intent
with the result of the check.
To properly handle this request, declare a Dialogflow intent that's
triggered by the actions_intent_TRANSACTION_REQUIREMENTS_CHECK event.
For this reason, I defined Int2 and as its event the actions_intent_TRANSACTION_REQUIREMENTS_CHECK.
However, I do not receive anything at my back-end like a result of the check and therefore I do not know if the action actions.intent.TRANSACTION_REQUIREMENTS_CHECK is really triggered. Why is this happening?
In general, how can I trigger one actions.intent.INTENT_NAME intent from my webhook/back-end?
When I am using the v2 version of Dialogflow, I am getting the following info/message about the webhook on Dialogflow when Int1 is triggered:
"webhookStatus": {
"code": 3,
"message": "Webhook call failed. Error: Failed to parse webhook JSON response: Cannot find field: data in message google.cloud.dialogflow.v2.WebhookResponse."
}
In the same case, I am getting the following info/message about the webhook on Google Assistant simulator when Int1 is triggered:
"responseMetadata": {
"status": {
"code": 14,
"message": "Webhook error (206)"
}
Finally, let me mention that I am testing all this with Python and ngrok at my local computer so perhaps this poses a problem because at the beginning of Google docs the following is mentioned:
Warning: The Actions Web Simulator should not be used to test an app
with transactions. Please use an Assistant-enabled Android or iOS
device to accurately test your app during development.
I finally solved this problem.
I had to replace the key "data" in the json which I was sending back when Int1 was triggered with the key "payload". In other words, I had to adjust my fulfilment response to the v2 version of Dialogflow.
Therefore, now I do get a second post request at my back-end which is sent because of the trigger of actions.intent.TRANSACTION_REQUIREMENTS_CHECK and of Int2.
Specifically, I get the following:
{
"responseId": "*****************************",
"queryResult": {
"queryText": "actions_intent_TRANSACTION_REQUIREMENTS_CHECK",
"parameters": {},
"allRequiredParamsPresent": true,
"fulfillmentText": "HERE",
"fulfillmentMessages": [
{
"text": {
"text": [
"HERE"
]
}
}
],
"outputContexts": [
{
"name": "*****************************"
},
{
"name": "*****************************"
},
{
"name": "*****************************"
},
{
"name": "*****************************"
},
{
"name": "*****************************"
},
{
"name": "*****************************",
"parameters": {
"TRANSACTION_REQUIREMENTS_CHECK_RESULT": {
"#type": "type.googleapis.com/google.actions.v2.TransactionRequirementsCheckResult",
"resultType": "OK"
}
}
}
],
"intent": {
"name": "*****************************",
"displayName": "Int2"
},
"intentDetectionConfidence": 1,
"diagnosticInfo": {},
"languageCode": "en-us"
},
"originalDetectIntentRequest": {
"source": "google",
"version": "2",
"payload": {
"isInSandbox": true,
"surface": {
"capabilities": [
{
"name": "actions.capability.WEB_BROWSER"
},
{
"name": "actions.capability.MEDIA_RESPONSE_AUDIO"
},
{
"name": "actions.capability.SCREEN_OUTPUT"
},
{
"name": "actions.capability.AUDIO_OUTPUT"
}
]
},
"inputs": [
{
"rawInputs": [
{
"inputType": "KEYBOARD"
}
],
"arguments": [
{
"extension": {
"#type": "type.googleapis.com/google.actions.v2.TransactionRequirementsCheckResult",
"resultType": "OK"
},
"name": "TRANSACTION_REQUIREMENTS_CHECK_RESULT"
}
],
"intent": "actions.intent.TRANSACTION_REQUIREMENTS_CHECK"
}
],
"user": {
"lastSeen": "2018-05-16T11:15:14Z",
"locale": "en-US",
"userId": "*****************************"
},
"conversation": {
"conversationId": "1526470000479",
"type": "ACTIVE",
"conversationToken": "[]"
},
"availableSurfaces": [
{
"capabilities": [
{
"name": "actions.capability.SCREEN_OUTPUT"
},
{
"name": "actions.capability.AUDIO_OUTPUT"
}
]
}
]
}
},
"session": "*****************************"
}
I think your response object is incorrect. the intent attribute should be inside the systemIntent object
"data": {
"google": {
"expectUserResponse": true,
"isSsml": false,
"noInputPrompts": [],
"systemIntent": {
"intent": "actions.intent.TRANSACTION_REQUIREMENTS_CHECK",
"data": {
"#type": "type.googleapis.com/google.actions.v2.TransactionRequirementsCheckSpec",
"paymentOptions": {
"actionProvidedOptions": {
"displayName": "VISA-1234",
"paymentType": "PAYMENT_CARD"
}
}
}
}
}
}
I decided to upgrade my Google Assistant action to use "dialogFlow V2 API" and my webhook returns an object like this
{
"fulfillmentText": "Testing",
"fulfillmentMessages": [
{
"text": {
"text": [
"fulfillmentMessages text attribute"
]
}
}
],
"payload": {
"google": {
"richResponse": {
"items": [
{
"mediaResponse": {
"mediaType": "AUDIO",
"mediaObjects": [
{
"name": "mediaResponse name",
"description": "mediaResponse description",
"largeImage": {
"url": "https://.../640x480.jpg"
},
"contentUrl": "https://.../20183832714.mp3"
}
]
},
"simpleResponse": {
"textToSpeech": "simpleResponse: testing",
"ssml": "simpleResponse: ssml",
"displayText": "simpleResponse displayText"
}
}
]
}
}
},
"source": "webhook-play-sample"
}
But I get an error message saying my action it is not available, is mediaResponse supported by V2?, should I format my object differently?, also, when I remove "mediaResponse" object works just fine and the assistant will speak the simpleResponse part.
This action was re-created this Mid March 2018 and I read about May deadline and that is why I decide to upgrade to V2, do you think I should go back to V1, I know I will have to delete it and re-created but that is fine. This is a link to the JSON object I see in the debug tab. Thanks once again
I set "API V2" in my action dialogFlow console, this is a screenshot of that setting
Here is an screenshoot of my action's integration -> Google Assistant
Thanks Allen, Yes I do have "expectUserResponse": false, I added the suggestion object you recommended but, unfortunately nothing changed, I am still getting this error
Simulator debug tag details
First of all - this is not a problem with Dialogflow V2. You also seem to be confusing the sunset of Actions on Google V1 with the release of Dialogflow V2 - they are two different creatures completely. If your project was using AoG V1, there would be a setting on the Actions integration screen, and thee isn't.
It is fine if you want to move to Dialogflow V2, but it isn't required. Media definitely works under Dialogflow V2.
The array of items must include a simpleResponse item first, before any of the other items in the RichResponse. (You also shouldn't include both ssml and textToSpeech - just one of them.) You also don't need the fulfillmentText and fulfillmentMessages components, since those are provided by the richResponse.
You also need to include suggestions chips unless you have set expectUserResponse to false. Somewhere in the simulator debug is probably a block that says
{
"name": "MalformedResponse",
"debugInfo": "expected_inputs[0].input_prompt.rich_initial_prompt: Suggestions must be provided if media_response is used..",
"subDebugEntryList": []
}
So something more like this should work:
{
"payload": {
"google": {
"richResponse": {
"items": [
{
"simpleResponse": {
"textToSpeech": "simpleResponse: testing",
"displayText": "simpleResponse displayText"
},
"mediaResponse": {
"mediaType": "AUDIO",
"mediaObjects": [
{
"name": "mediaResponse name",
"description": "mediaResponse description",
"largeImage": {
"url": "https://.../640x480.jpg"
},
"contentUrl": "https://.../20183832714.mp3"
}
]
}
}
]
"suggestions": [
{
"title": "This"
},
{
"title": "That"
}
]
}
}
},
"source": "webhook-play-sample"
}
I have the problem that the google home/assitant action console simulators response and request tab are not working. At least when I am using a custom action.json.
For me I am not sure if all have this problem or only some. That are using an custom action sdk. Or is it a problem only because something of my action.json is maybe not 100% correct configured.
here is the action.json:
{
"actions": [
{
"description": "Default Welcome Intent",
"name": "MAIN",
"fulfillment": {
"conversationName": "testApp"
},
"intent": {
"name": "actions.intent.MAIN",
"trigger": {
"queryPatterns": [
"open special manager",
"open s p m"
]
}
}
}
],
"types": [],
"conversations": {
"testApp": {
"name": "testApp",
"url": "https://572e66a2.ngrok.io/",
"fulfillmentApiVersion": 2,
"in_dialog_intents": [
{
"name": "actions.intent.NO_INPUT"
},
{
"name": "actions.intent.SIGN_IN"
}
]
}
}
here is a picture of the request:
As you can see it is only the dummy content in request tab anyway if the chat is working.
The response tab is completly empty. But the messages and voice is correctly working. Also on my google home.
Does anybody have an idea? I will add of course more debug informations if necessary. Can it be trouble with the response or request messages from my server?
But actually the messages, they are working...