When calling the webhook multiple times in one scene and sending simple responses there is a bug at merging the simple responses.
prompt from the first webhook call
{
"override": false,
"firstSimple": {
"speech": "<speak><audio src=\"https://www.example.com/audio/file1.mp3\"></speak>",
"text": "Text 1"
}
}
prompt from the second webhook call
{
"override": false,
"firstSimple": {
"speech": "<speak><audio src=\"https://www.example.com/audio/file2.mp3\"></audio> <audio src=\"https://www.example.com/audio/file3.mp3\"></audio></speak>",
"text": " Text 2"
}
}
merged prompt in the response send to the user
{
"firstSimple": {
"speech": "<speak><speak><audio src=\"https://www.example.com/audio/file1.mp3\"></speak> <audio src=\"https://www.example.com/audio/file2.mp3\"/> <audio src=\"https://www.example.com/audio/file3.mp3\"/></speak>",
"text": "Text 1 Text2"
}
}
So with the two speak tags the SSML is invalide and is not spoken out.
Sometimes the speech object is completely missing.
I already created an Github issue for that.
So found out that the merging Bug is related to invalid SSML. Unfortunately there is no error message from Google for SSML errors.
And as a workaround for the problem that the speech object is completely missing I changed conv.add(new Simple('Text')) to conv.prompt.firstSimple = new Simple('Text') or conv.prompt.lastSimple = new Simple('Text').
Related
Trying to implement the action.devices.commands.mediaClosedCaptioningOn command for the TransportControl trait.. I have included 'CAPTION_CONTROL' in the transportControlSupportedCommands array I send on the SYNC intent for the device, but when I actually ask Assistant to turn on subtitles on the device it replies "That device cannot play video" and my code never actually gets an EXECUTE Intent from Assistant.
I've also successfully implemented the mediaShuffle command but noticed that when I issue it, Assistant responds with "Music will shuffle after the current item" when my device actually plays videos.
Anyone have any clue how I tell Assistant that my device plays video? I can't seem to find it in the documentation for TransportControl -- is it under a different trait maybe?
Edit 10/13/2020:
As requested, the complete SYNC response I'm currently sending for this device:
{
"requestId": "11724082196092058876",
"payload": {
"agentUserId": "xxxxxx",
"devices": [
{
"id": "10.0.6.252",
"type": "action.devices.types.STREAMING_STICK",
"traits": [
"action.devices.traits.TransportControl",
"action.devices.traits.Volume",
"action.devices.traits.MediaState"
],
"name": {
"name": "Bedroom Stick"
},
"willReportState": true,
"roomHint": "bedroom",
"attributes": {
"transportControlSupportedCommands": [
"NEXT",
"PREVIOUS",
"PAUSE",
"STOP",
"RESUME",
"CAPTION_CONTROL",
"SEEK_TO_POSITION",
"SEEK_RELATIVE",
"SET_REPEAT",
"SHUFFLE"
],
"volumeMaxLevel": 100,
"volumeCanMuteAndUnmute": true,
"levelStepSize": 10,
"commandOnlyVolume": false,
"volumeDefaultPercentage": 100,
"supportActivityState": false,
"supportPlaybackState": true
}
}
]
}
}
Exact assistant queries that produce this issue include "OK Google, turn on subtitles on Bedroom Stick" (Response: "That device can't play videos.") and "OK Google, shuffle on Bedroom Stick." (Response: "Alright, music will start shuffling after this track.")
There are certain changes you need to make to your sync response that you have posted which might make your device function properly .
In the sync response I do see that you are making use of a STREAMING_STICK as your device type. If you go through the documentation you will see that this device has the following required traits :
action.devices.traits.AppSelector
action.devices.traits.MediaState
action.devices.traits.TransportControl
action.devices.traits.Volume
Your sync response is missing the AppSelector trait.
It would be good to add the deviceInfo attribute at the end of the sync response.
More information about how to work around with Streaming stick can be found here :
https://developers.google.com/assistant/smarthome/guides/streamingstick#response
The Problem
I have a Google Actions SDK project and I am using the nodejs client library for building fulfillment. I am facing some problem trying to use the Confirmation Helper intent. It is overriding previous responses given before it. Let me give you a simplified example of the problem:
First, Action says, "Hi, hope you are having a great day"
Then, Action asks for a Confirmation, "Do you want today's weather report?"
But in the simulator, all I hear is the second question. I am targeting a Voice Only situation, so I really need all the responses. Interestingly, I can see all the responses in the AUDIO tab of simulator. How can I hear both of these phrases?
I am copy/pasting the response JSON as shown in the RESPONSE tab.
{
"expectUserResponse": true,
"expectedInputs": [
{
"possibleIntents": [
{
"intent": "actions.intent.CONFIRMATION",
"inputValueData": {
"#type": "type.googleapis.com/google.actions.v2.ConfirmationValueSpec",
"dialogSpec": {
"requestConfirmationText": "Do you want today's weather report?"
}
}
}
],
"inputPrompt": {
"richInitialPrompt": {
"items": [
{
"simpleResponse": {
"textToSpeech": "Hope you are doing well!"
}
}
]
}
}
}
],
"conversationToken": ""
}
How can I fix this situation. Thanks!
This is intentional behavior as using one of the intents like Confirmation will be the only response, and other responses will be ignored.
There are two potential ways to get around this.
Combine all of your responses to be in the Confirmation
Create your own Yes/No intent and use simple responses for everything.
I am trying to send a payload from my service with a Basic Card and the payload I am sending looks like such with the following fulfillmentMessages:
{
"fulfillmentMessages": [
{
"simpleResponses": {
"simpleResponses": [
{
"displayText": "A 13-inch laptop with advanced color, sound and streaming for an immersive viewing experience. Featuring Dell Cinema and next-generation InfinityEdge.",
"ssml": "<speak>Here's the first item I found. XPS 13 on sale for $849.99. To navigate you can say Next, Previous, or First. You can also say 'More Details' or 'Text me the product link'.</speak>"
}
]
}
},
{
"basicCard": {
"buttons": [
{
"openUriAction": {
"uri": "https://www.dell.com/en-us/shop/productdetails/xps-13-9370-laptop/dycwi622h"
},
"title": "See on Dell.com"
}
],
"formattedText": "A 13-inch laptop with advanced color, sound and streaming for an immersive viewing experience. Featuring Dell Cinema and next-generation InfinityEdge.",
"image": {
"accessibilityText": "product image",
"imageUri": "https://i.dell.com/is/image/DellContent//content/dam/global-site-design/product_images/dell_client_products/notebooks/xps_notebooks/13_9370/global_spi/rose_gold/notebook-xps-13-9370-best-of-500-rosegold-ng.psd?fmt=png-alpha"
},
"subtitle": "$849.99",
"title": "XPS 13"
}
}
]
}
I am running into issues with Actions On Google throwing errors with my payload defined this way:
MalformedResponse
Failed to parse Dialogflow response into AppResponse because of empty speech response.
I am unsure what the issue is here, or why Actions on Google throws this exception when the Dialogflow tester shows our card as expected.
How do I correctly format my payload so that both Actions on Google and Dialogflow display as expected?
Figured it out.
Needed to assign the platform value to each message object as "ACTIONS_ON_GOOGLE".
I have a messenger bot webview with a form with option to select, I am having an issue with message back to the thread using beginShareFlow. I have added Messenger Extensions JS SDK for the Webview. How do i implement this? Any link real world example is helpful.
Compose a JSON message to send back to the thread, and then on user share button click or whatever event you wish, run MessengerExtensions.startShareFlow, passing in functions to handle success and error events, the message you created, and the message share type. Find more at the docs
<script>
var messageToShare = {
"attachment":{
"type":"template",
"payload":{
"template_type":"generic",
"elements": [{
"title":"I took Peter's 'Which Hat Are You?' Quiz",
"image_url": "https://bot.peters-hats.com/img/hats/fez.jpg",
"subtitle": "My result: Fez",
"default_action":{
"type":"web_url",
"url": "https://bot.peters-hats.com/view_quiz_results.php?user=24601"
},
"buttons":[{
"type":"web_url",
"url":"https://bot.peters-hats.com/hatquiz.php?referer=24601",
"title":"Take the Quiz"
}]
}]
}
}
};
MessengerExtensions.beginShareFlow(function success(response) {
// Share successful
}, function error(errorCode, errorMessage) {
// The user was not able to share
},
messageToShare,
"broadcast");
</script>
Introduction
I've got a problem with my extension for Chrome. It supposed to show a small overlay popup window (created in jQuery) with search results from google based on your text selection. Basically you should be able to highlight a text on any page, right click on it (context menu), click on "Search for 'selected keyword'" and a small window pops up in the same tab as an overlay with all search results from google or different search engine.
The problem
Now the extension works really well and without problems, HOWEVER after extensive using of this extension (let's say 1 hour) when I'll highlight another keyword and search for it, extension REMEMBERS last keyword and shows wrong search results. Then again, when I'll highlight another keywords it rememebrs the keyword I've just highlighted but previously didn't get the results for it. It's like a chaining process and I'm always one keyword behind.
My thoughts
I think that the cause of this problem might be event listeners, because when this problem occurs I've got a log from console.log(chrome.extension.onRequest.hasListeners()); which says true. It means that there are 2 or more event listeners attached and they shouldn't as I'm removing them by chrome.extension.onRequest.removeListener(listener1);
Maybe it happens when the actual web page is still loading and I'm highlighting keyword, clicking on context menu and extension can't load yet but somehow event listeners firing... I really can't fully replicate this problem... and what causes this problem.
Manifest File
{
"name": "Search Accelerator",
"version": "1.0",
"manifest_version": 2,
"description": "Search Accelerator extension",
"icons": {
"16": "icon19.png",
"48": "icon48.png",
"128": "icon128.png"
},
"background": {
"scripts": ["content_script.js"]
},
"minimum_chrome_version": "18",
"permissions": [
"contextMenus",
"tabs",
"http://*/*",
"https://*/*",
"management",
"webRequest"
]
}
Content Script
chrome.contextMenus.create({ "title": 'Search for "%s"',
"contexts":['selection'], "onclick": getClickHandler() });
function getClickHandler() { return function(info, tab) {
console.log(chrome.extension.onRequest.hasListeners());
chrome.extension.onRequest.addListener(function listener1 (request, sender, sendResponse) {
var url = "http://www.google.com/?s=" + info.selectionText;
console.log(url); var keywordObj = {keyword: url};
if (request.keywordRequest == "Yes") {
console.log(keywordObj);
sendResponse(keywordObj);
};
chrome.extension.onRequest.removeListener(listener1); } );
chrome.tabs.executeScript(null, { file: "jquery.js" }, function() {
chrome.tabs.executeScript(null, { file: "popup.js" });
}); }; };
Popup js
chrome.extension.sendRequest({keywordRequest: "Yes"}, function(response) {
$(document).ready(function() {
if($("#e14_accelerator")) {
$("#e14_accelerator").remove();
}
var rkeyword = response.keyword;
$("body").append("<div id=\"e14_accelerator\" style=\"position: fixed;top: 30px;right: -330px;z-index: 999999; \"><iframe style=\"border:5px solid #c9c9c9;-webkit-box-shadow: 1px 1px 15px 1px rgba(0, 0, 0, 0.2);\" src=\""+ rkeyword +"\" width=\"328\" height=\"240\"></iframe></div>");
$("#e14_accelerator").animate({right:-13},500);
$(document).click(function() {
$("#e14_accelerator").remove();
});
$("#e14_accelerator").click(function() {
return false;
});
});
});
Error messages that appear when this problem occur:
Error during tabs.executeScript: Cannot access contents of url "chrome-devtools://devtools/devtools.html?docked=true&dockSide=bottom&toolbarColor=rgba(230,230,230,1)&textColor=rgba(0,0,0,1)". Extension manifest must request permission to access this host. sendRequest:21
chromeHidden.handleResponse sendRequest:21
Error during tabs.executeScript: Cannot access contents of url "chrome-devtools://devtools/devtools.html?docked=true&dockSide=bottom&toolbarColor=rgba(230,230,230,1)&textColor=rgba(0,0,0,1)". Extension manifest must request permission to access this host. sendRequest:21
chromeHidden.handleResponse