Issue in make a phone call within Google Assistant App - actions-on-google

I try to implement button in basic card i got an error
API Version 2: Failed to parse JSON response string with 'INVALID_ARGUMENT' error: "(expected_inputs[0].input_prompt.rich_initial_prompt.items[1].basic_card.buttons[0].open_url_action) versions: Cannot find field." HTTP Status Code: 200.
conv.ask(new BasicCard({
text: `This is a basic card. Text in a basic card can include "quotes" and`,
subtitle: 'This is a subtitle',
buttons: [
{
"title": "Call",
"openUrlAction": {
"url": "tel:+91123456789",
"androidApp": {
"packageName": "com.android.phone"
},
"versions": []
}
},
],
"formattedText": "Some text",
image: new Image({
url: 'https://storage.googleapis.com/actionsresources/logo_assistant_2x_64dp.png',
alt: 'Image alternate text',
}),
"title": "Card Title"
}));
and I tried all the ways add another empty object in button array also same error:
buttons: [
{
"title": "Call",
"openUrlAction": {
"url": "tel:+91123456789",
"androidApp": {
"packageName": "com.android.phone"
},
"versions": []
}
},
{}
],
Tried code below got error:
expected_inputs[0].input_prompt.rich_initial_prompt.items[1].basic_card.buttons[0].open_url_action: the protocol must be http or https.
buttons: new Button({
"title": "button text",
'url': "tel:+91123456789",
"androidApp": {
"packageName": "com.android.phone"
}
})
got response in action console simulator:
"buttons": [
{
"title": "button text",
"openUrlAction": {
"url": "tel:+919177723773"
}
}
]

You cannot open a tel: URL through the Assistant.
As the error noted:
expected_inputs[0].input_prompt.rich_initial_prompt.items[1].basic_card.buttons[0].open_url_action: the protocol must be http or https.
Keep in mind that the Assistant runs on devices that do not support tel: URLs, even if they support https: URLs.

The URL property that you are using includes a phonenumber. The link property can only use links to a webpage. This also comes back in the error message that you are getting:
"the protocol must be http or https.".
So if you want to fix the error, you need to include a http or https link in your button instead of a phonenumber.
As for calling a phonenumber from the Google Assistant App, I'm not sure if this feature is supported at the moment.

Related

Messenger bot with dialogflow does not respond to user

I have a messenger bot built with google's dialogflow engine, to respond the user I have a webhook. when I send a message with a "hello", the default welcome intent its triggered and the webhook it its called and it responds to me.
After that I need the products of my business so I write "Comprar" (Buy), the Products intent it is triggered and the webhook responds with a fulfillment message with a list of available products.
In the history section in google dialogflow, I can see the json respond of the wekhook with the message that the user will receive.
But facebook does not show the message. I use the same webhook for many intents, and it is not working in this case.
this is the respond I got in the History section for this case:
{
"id": "f75002cd-2f8a-422a-960d-dd2c9e00b490-74fe87bc",
"fulfillmentText": "",
"language_code": "es",
"queryText": "Comprar",
"webhookPayload": {},
"intentDetectionConfidence": 1,
"action": "GET_CATEGORIES",
"webhookSource": "",
"parameters": {},
"fulfillmentMessages": [
{
"quickReplies": {
"title": "Escoge alguna de estas Categorías:",
"quickReplies": [
"Volver",
"",
"FIRMA ELECTRONICA"
]
}
}
],
"diagnosticInfo": {
"webhook_latency_ms": "732.0"
},
"webhookStatus": {
"webhookStatus": {
"message": "Webhook execution successful"
},
"webhookUsed": true
},
"outputContexts": [
{
"lifespanCount": 1,
"name": "vercatalogo-followup",
"parameters": {}
}
],
"intent": {
"isFallback": false,
"displayName": "ver.catalogo",
"id": "41dcecf2-11b6-4e50-8294-b86d849093e1"
}
}
I solved lol
Apparently quickReplies must not contain a empty string ("") as a quick reply, that option is not valid. So I removed it and it worked.

The custom payload for the facebook messenger integration only shows up the first time it's called and not thereafter

I built a winebot that can suggest food pairings or make recommendations.
The integration with FB Messenger works fine, and it's supposed to call a carousel after a search to show you the various options. I found that when I call on it multiple times it will only display the carousel the first time and not a new one each time.
This is the function that calls the carousel
// CREATE RICH MESSAGES FOR FACEBOOK
function create_carousel(agent, carousel_params) {
agent.context.set({
name: 'carousel_output',
lifespan: 2,
parameters: carousel_params,
});
agent.setFollowupEvent('createCarousel');
agent.add(''); //add a dummy payload
return;
}
IntentMap is set up like this:
// Run the proper function handler based on the matched Dialogflow intent name
let intentMap = new Map();
intentMap.set('createCarousel', create_carousel);
agent.handleRequest(intentMap);
The createCarousel intent is set up with an event createCarousel and an output context called carousel_output
Under the messenger tab it has a text message followed by a payload. The text message and payload both use the content of carousel_output to make a recommendation.
{
"facebook": {
"attachment": {
"type": "template",
"payload": {
"elements": [
{
"image_url": "#carousel_output.product1.image",
"default_action": {
"url": "https://google.com",
"webview_height_ratio": "tall",
"type": "web_url"
},
"subtitle": "#carousel_output.product1.subtitle",
"title": "#carousel_output.product1.title",
"buttons": [
{
"url": "https://www.winemag.com/buying-guide/brovia-2013-garblet-sue-barolo/",
"title": "#carousel_output.product1.price",
"type": "web_url"
},
{
"title": "Some Text",
"payload": "Some Text",
"type": "postback"
}
]
},
{
"image_url": "#carousel_output.product2.image",
"subtitle": "#carousel_output.product2.subtitle",
"default_action": {
"type": "web_url",
"webview_height_ratio": "tall",
"url": "https://google.com"
},
"title": "#carousel_output.product2.title",
"buttons": [
{
"type": "web_url",
"url": "https://www.wine-searcher.com/find/limerick+lane+cellars+rocky+knoll+zinfandel+russian+river+valley+sonoma+county+north+coast/2013",
"title": "#carousel_output.product2.price"
},
{
"title": "Some Text",
"payload": "Some Text",
"type": "postback"
}
]
},
{
"default_action": {
"webview_height_ratio": "tall",
"type": "web_url",
"url": "https://google.com"
},
"image_url": "#carousel_output.product3.image",
"buttons": [
{
"title": "#carousel_output.product3.price",
"type": "web_url",
"url": "https://www.wine-searcher.com/find/charles+smith+k+vintner+the+hidden+sra+wahluke+slope+washington+usa/2009"
},
{
"title": "Some Text",
"type": "postback",
"payload": "Some Text"
}
],
"title": "#carousel_output.product3.title",
"subtitle": "#carousel_output.product3.subtitle"
}
],
"template_type": "generic"
}
}
}
}
What I noticed is that when I call the intent a second or third time , the message appears but the payload does not.
I 've already tried clearing output_context and deleting it but this does not seem to affect anything. I looked through the documentation of how payloads work but nothing specifically referencing the issue.
I was wondering if this may be a limitation on facebook messenger but don't see anything in the logs that is a flag. The function runs and the context is updated each time with the right content but just doesn't display in the bot.
The logs didn't show any errors, but the problem appeared in the data provided to the payload.
The code that creates the payload pulls images 3 at a time from a list of urls. There were only 5 urls provided in the list so on the second run it killed the carousel and did not show an undefined error message or a missing data error. Works fine after adding more urls to the list.

Adding Videos to a Magento 2 product using REST

Magento version 2.2.5
Endpoint in question /V1/products/{sku}/media METHOD: POST
I am trying to add a video under a product sku using Magento's provided API endpoint.
{
"entry": {
"media_type": "external-video",
"label": "Video 2",
"position": 2,
"disabled": false,
"types": ["thumbnail"],
"content" : {
"base64_encoded_data": "encoded image data ",
"type": "image/jpeg",
"name": "0.jpg"
},
"extension_attributes": {
"video_content": {
"media_type": "external-video",
"video_provider": "youtube",
"video_url": "some youtube video url",
"video_title": "some title",
"video_description": "",
"video_metadata": ""
}
}
}
The response I get from this call is "invalid option value", I had debugged this in a local environment which leads me to failure during ProductRepository->save() within Magento\Catalog\Model\Product\GalleryGalleryManagement.php on line 70.
Is this an issue with my payload or actual magento bug?
Resolved this issue by adding videos directly using POST /V1/products or PUT /V1/products/{sku} endpoints.
post method example:
{
"product": {
"sku": "some-sku",
..... other product data,
"media_gallery_entries": {
"media_type": "external-video",
"label": "Video 2",
"position": 2,
"disabled": false,
"types": [],
"content" : {
"base64_encoded_data": "encoded image data ",
"type": "image/jpeg",
"name": "0.jpg"
},
"extension_attributes": {
"video_content": {
"media_type": "external-video",
"video_provider": "youtube",
"video_url": "some youtube video url",
"video_title": "some title",
"video_description": "",
"video_metadata": ""
}
}
}
}
I had no luck with Magento's MediaGallery POST endpoint (/V1/products/{sku}/media.
If you want to add a video to the product page then why you did not try using YouTube API?
You just need to follow these steps:
Generate a YouTube API Key by going goodle developer console
Then add Youtube API key to your Magento 2 by going to STORES --> Configuration
Then Add Video URL

Continue a conversation after Browsing carousel in DialogFlow/Actions

So I have been experimenting with different Response types for DialogFlow through Actions: Actions Responses and Webhook/Fulfillment.
And so far, I have been able to generate proper responses for types like List, Basic Card, Suggestion Chips successfully. What I need now is a list-based response that lets the user open a link in a browser when touched as well as "not" generate a chat bubble. "Browsing carousel" fits the criteria: Browsing Carousel.
I have successfully created and simulated the output with 2 sample items. The issue is when the user wants to continue the conversation. As per the Guidance section in the help above, the browsing carousel:
By default, the mic remains closed after a browse carousel is sent. If you want to continue the conversation afterwards, we strongly recommend adding suggestion chips below the carousel.
From this what I understood is that the user has to invoke the App again by saying "Ok Google, talk to [app]". This doesn't seem very user-friendly as the user expects to return back to the conversation she was having with the agent after she has looked through the links from the carousel. Please note, I have simulated the flow using the Google Actions Simulator on Console.Actions page.
As soon as I invoke the intent with the Browsing Carousel, it is shown to me with the sample Items. But when I enter/say the next command to continue the conversation, the agent simply returns with:
We're sorry, but something went wrong. Please try again.
And the REQUEST/REQUEST window as well as the ERRORS/DEBUG are empty. I have logged calls to the Webhook and there is no call received.
The question: Is there a way to give the user the ability to browse an informative link from a response "list" (not Basic Card) and return to the conversation without ending it.
Here is the response for Browsing Carousel from RESPONSE window in Actions > Simulator (note I've removed non-relevant parts):
{
"conversationToken": "[token info]",
"expectUserResponse": true,
"expectedInputs": [
{
"inputPrompt": {
"richInitialPrompt": {
"items": [
{
"simpleResponse": {
"textToSpeech": "You have the following 2 options:",
"displayText": "You have the following 2 options:"
}
},
{
"carouselBrowse": {
"items": [
{
"title": "Test 1",
"description": "Desc 2",
"image": {
"url": "[some url]"
},
"openUrlAction": {
"url": "[some url]"
}
},
{
"title": "Test 2",
"description": "Desc 2",
"image": {
"url": "[some url]"
},
"openUrlAction": {
"url": "[some url]"
}
}
]
}
}
],
"suggestions": [
{
"title": "Continue"
},
{
"title": "End"
}
]
}
}
}
],
"responseMetadata": {
"status": {
"message": "Success (200)"
},
"queryMatchInfo": {
"queryMatched": true
}
}
}
For all those who are using the Simulator to test the Browsing Carousel and seeing that it stops responding after the output, please use a device instead to test it. When you use a device to render the output and use the Carousel response, it doesn't kill the conversation but turns off the mic. This is the intended behavior. One can introduce Suggestion chips to assist the user to continue the conversation.
Updated: Also make sure each Webhook Intent call has a proper response with Simple Response. As long as the response has required textual and audio information correctly setup, the Simulator will not fail.

Error "Empty speech response"

I tried to connect DialogFlow and Actions on Google, so I created some intents, connected the services, added explicit and implicit invocations etc, but when I try the bot in the simulator https://console.actions.google.com/project/[projectId]/simulator/ it always gives me the error:
"Failed to parse Dialogflow response into AppResponse, exception
thrown with message: Empty speech response"
even tough inputType was "KEYBOARD".
What I tried so far:
I did set "Response from this tab will be sent to the Google Assistant integration" in Dialog Flow (do you have to set it for every single intent?), but I don't see any extra setting for speech.
I disabled the second language, first I had also intents in German
I also turned off the Fullfillment Webhook (implemented in API v1 and then also v2) with no change
I only found this user with the same problem https://productforums.google.com/forum/#!topic/dialogflow/xYjKlz31yW0;context-place=topicsearchin/dialogflow/Empty$20speech$20response but no resolution.
the fulfillment checkbox is checked at the intents
The bot works fine when I use it through "Try it now" on the very right in Dialog Flow or in the Web Demo https://bot.dialogflow.com/994dda8b-4849-4a8a-ab24-c0cd03b5f420
Unfortunately the docs don't say anything about this error. Any ideas?
Here a screenshot of the error on the Actions integration:
This is the full debug output:
{
"agentToAssistantDebug": {
"agentToAssistantJson": {
"message": "Failed to parse Dialogflow response into AppResponse, exception thrown with message: Empty speech response",
"apiResponse": {
"id": "c12e1389-e887-49d4-b399-a332188ca946",
"timestamp": "2018-01-27T03:55:30.931Z",
"lang": "en-us",
"result": {},
"status": {
"code": 200,
"errorType": "success"
},
"sessionId": "1517025330705"
}
}
},
"assistantToAgentDebug": {
"assistantToAgentJson": {
"user": {
"userId": "USER_ID",
"locale": "en-US",
"lastSeen": "2018-01-27T03:55:03Z"
},
"conversation": {
"conversationId": "1517025330705",
"type": "NEW"
},
"inputs": [
{
"intent": "actions.intent.MAIN",
"rawInputs": [
{
"inputType": "KEYBOARD",
"query": "Talk to Mica, the Hipster Cat Bot"
}
]
}
],
"surface": {
"capabilities": [
{
"name": "actions.capability.MEDIA_RESPONSE_AUDIO"
},
{
"name": "actions.capability.WEB_BROWSER"
},
{
"name": "actions.capability.AUDIO_OUTPUT"
},
{
"name": "actions.capability.SCREEN_OUTPUT"
}
]
},
"isInSandbox": true,
"availableSurfaces": [
{
"capabilities": [
{
"name": "actions.capability.AUDIO_OUTPUT"
},
{
"name": "actions.capability.SCREEN_OUTPUT"
}
]
}
]
},
"curlCommand": "curl -v 'https://api.api.ai/api/integrations/google?token=TOKEN' -H 'Content-Type: application/json;charset=UTF-8' -H 'Google-Actions-API-Version: 2' -H 'Authorization: AUTH_TOKEN' -A 'Mozilla/5.0 (compatible; Google-Cloud-Functions/2.1; +http://www.google.com/bot.html)' -X POST -d '{\"user\":{\"userId\":\"USER_ID\",\"locale\":\"en-US\",\"lastSeen\":\"2018-01-27T03:55:03Z\"},\"conversation\":{\"conversationId\":\"1517025330705\",\"type\":\"NEW\"},\"inputs\":[{\"intent\":\"actions.intent.MAIN\",\"rawInputs\":[{\"inputType\":\"KEYBOARD\",\"query\":\"Talk to Mica, the Hipster Cat Bot\"}]}],\"surface\":{\"capabilities\":[{\"name\":\"actions.capability.MEDIA_RESPONSE_AUDIO\"},{\"name\":\"actions.capability.WEB_BROWSER\"},{\"name\":\"actions.capability.AUDIO_OUTPUT\"},{\"name\":\"actions.capability.SCREEN_OUTPUT\"}]},\"isInSandbox\":true,\"availableSurfaces\":[{\"capabilities\":[{\"name\":\"actions.capability.AUDIO_OUTPUT\"},{\"name\":\"actions.capability.SCREEN_OUTPUT\"}]}]}'"
},
"sharedDebugInfo": [
{
"name": "ResponseValidation",
"subDebugEntry": [
{
"debugInfo": "API Version 2: Failed to parse JSON response string with 'INVALID_ARGUMENT' error: \": Cannot find field.\".",
"name": "UnparseableJsonResponse"
}
]
}
]
}
Also "debugInfo" sounds like an internal problem:
"API Version 2: Failed to parse JSON response string with
'INVALID_ARGUMENT' error: \": Cannot find field.\"."
Here a screenshot of the welcome intent:
ps.
It took me AGES to figure out, what
"Query pattern is missing for custom intent"
means - so I just document it here: In Dialog Flow - Intent - "User says" you have to DOUBLE CLICK on a word in the text input field when you want to set it as query parameter - which seems to be required for Actions on Google.
This happened to me. If this happens for an Intent you just added in the Dialogflow console and you are using Webhook fulfillment for the action, check the intent's fulfillment settings and ensure that the Webhook fulfillment slider is on. Evidently new intents don't automatically get webhook fulfillment: you have to opt each one in piecemeal (or at least, that was my experience).
I experienced this situation too.
My problem was that I used a SimpleResponse in my fulfillment index.js without referencing to it. So the solution for me was to add SimpleResponse like this in index.js:
const {dialogflow, SimpleResponse} = require('actions-on-google');
So, always check if you aren't using any dependencies without including it in your js-file.
Probably not the most common cause of the problem, but it can be.
I got this when running through the codelabs tutorial (https://codelabs.developers.google.com/codelabs/actions-1/index.html#4) and didn't name my intent the same name as it is referenced in the webhook script:
I came across this error when trying to develop my own WebHook. I first verified that my code was called by looking into the Nginx log, after which I knew there was a problem in my JSON output because I based my output on outdated examples.
The (up-to-date) documentation for both V1 and V2 of the API can be found here:
https://dialogflow.com/docs/fulfillment/how-it-works
This example response for v2 of the dialogflow webhook API helped me to resolve this error:
{
"payload": {
"google": {
"expectUserResponse": true,
"richResponse": {
"items": [
{
"simpleResponse": {
"textToSpeech": "this is a simple response"
}
}
]
}
}
}
}
Source: https://github.com/dialogflow/fulfillment-webhook-json/blob/master/responses/v2/ActionsOnGoogle/RichResponses/SimpleResponse.json
You can find more examples in the official github repository linked above.
Another possibility is if you have a text response (even an empty one) like so:
Then you need to click the trash can next to the response to clear it out to use the webhook.
The Actions on Google support helped me fix this problem:
I needed to add a text as Default Response to the intent used for Explicit Invocation.