I'm developing an action for Google Assistant and I was wondering if I could continue an action which I have started on a device without a screen (e.g. Google Home) on my mobile phone.
E.g. I ask Google Home to search for a good hotel in New York and when it tells me the result I maybe want to visit the corresponding website on my phone. Another example would be: I'm ordering pizza and for the payment process I would have to authenticate via fingerprint / password on my mobile.
Does anybody know if that is / will be possible?
Yes, I guess it is possible. It depends on how better you write the webhook for the application.
For this example:
When you ask Google Home to search for a good hotel in New York, keep in mind to make a follow-up intent if the user is asking to check for more details of that hotel, like phone number, or email or other contact details.
In webhook, make an action for this follow-up intent which is to be triggered.
And in the response, call the getHotelDetail() function, which would be a user defined function.
Here is a demo code to understand:
def getHotelDetail(hotel):
# get all the hotel details using an API or JSON
# return hotel-details
def processRequest(req):
# if req.get("result").get("action") == "follow-up-hotel-details":
# data = getHotelDetail()
# else:
# return {}
# res = makeWebhookResult(data)
# return res
def makeWebhookResult(data):
# return whatever you want to display/speak as per the
# processRequest(req)
Related
I have integrated google assistant with my dialogflow agent. I need to get the user who is invoking the intent.
For eg, If an user account "ABC" have access to invoke my agent via Google assistant app, on the welcome intent I have to send a response like "Welcome ABC". How do I achieve this with google assistant app is my endpoint.
Thanks in Advance.
You have two questions here: How to get the user's name and how to get their id.
The first thing to realize is that this information is considered personally identifiable information (PII), so Google doesn't give it to you without the permission of the user. How you ask for that permission, and how it is delivered to you, depends on some of your exact needs.
User ID
Historically, you could get an anonymous user ID for the Assistant account. This would be different than the Google User ID that is available below and was meant to be a persistent identifier so you could keep track of returning users.
This has been deprecated, and if this is all you need, then you can create your own identifier and save it as part of the userStorage.
Requesting user information
The traditional way of getting their name is to request the user for permission to access their information. If you're using the actions-on-google library, you do this using the Permission object with something like this:
const options = {
// We just want permission to get their name
permissions: ['NAME'],
// Prompt them why we want the information
context: 'To address you by name'
};
conv.ask(new Permission(options));
If the user grants permission, the results will be available in conv.user.name. You should save this in the userStorage, since the permission is not persistent. So this might look something like:
var userStorageStr = conv.user.userStorage || '{}';
var userStorage = JSON.parse( userStorageStr );
var name = conv.user.name || userStorage.name;
userStorage.name = name;
// ...
conv.user.userStorage = JSON.stringify( userStorage );
With the multivocal library, you would indicate that the User/Name environment property is one of the Requirements for the action or intent you want. So this might be in your configuration as
Local: {
en: {
Requirements: {
"Action.multivocal.welcome": "User/Name"
}
}
}
The name will be available in the environment under User/Name.
If you're using JSON, then you need to use the user information helper. For Dialogflow, this would be under the payload.google.systemIntent property, while for the Actions SDK this would be in expectedInputs[0].possibleIntents[0]. You might specify something like this:
{
"intent": "actions.intent.PERMISSION",
"inputValueData": {
"#type": "type.googleapis.com/google.actions.v2.PermissionValueSpec",
"optContext": "To address you by name",
"permissions": [
"NAME"
]
}
}
The name will be under the originalDetectIntentRequest.payload.user.profile field if you are using Dialogflow and user.profile for the Action SDK.
All of this seems like a lot, just to get a name. And you can't get the email address if you want that in addition. But there are other options.
Requesting their Google Profile
Their Google Profile contains both their unique Google ID, their full name (in the "name" field, given_name, last_name, and typically some other information such as their email address (the email address isn't guaranteed since they can omit this from their profile, but is typically there). You would use Google Sign-In for the Assistant to request this information. There is some configuration required in the Action console, and then you would request permission to get it using the sign-in helper.
With the actions-on-google library, the line would be something like:
conv.ask(new SignIn());
Once the user granted it, you can get their profile in
conv.user.profile.payload
their name in
conv.user.profile.payload.name
and their email in, you guessed it,
conv.user.profile.payload.email
Note that unlike asking for the user information, the profile will be available in all future activity with you. You don't need to store it.
With multivocal, you would say that the User/IsAuthenticated environment setting is one of the Requirements for the action or intent you want. So this might be in your configuration as
Local: {
en: {
Requirements: {
"Action.multivocal.welcome": "User/IsAuthenticated"
}
}
}
The profile will be available in the environment under User/Profile, the name would be in User/Profile/name, and the email in User/Profile/email.
If you're using JSON, then you need to use the sign-in helper. For Dialogflow, this would be under the payload.google.systemIntent property, while for the Actions SDK this would be in expectedInputs[0].possibleIntents[0]. You might specify something like this:
{
"intent": "actions.intent.SIGN_IN",
"inputValueData": {}
}
You will get an identity token for the user in the originalDetectIntentRequest.payload.user.idToken field if you are using Dialogflow and user.idToken for the Action SDK. You will need to validate and decode this JWT. (The actions-on-google and multivocal libraries handle this step for you.)
The easiest would be to use Google Sign-In for the Assistant: https://developers.google.com/actions/identity/google-sign-in
I am starting in Dialogflow and I would like to know if the information I keep from a user in a database, be it Firebase, that this information could occupy it to put it in a web page.
An example is the user's email stored in the database to be able to put it in a page where I am asked to enter with mail so that the user does not have to manually enter it that Dialogflow does.
What I am looking for with Dialogflow is to make the user able to pay from the created chat of Dialogflow in a web page only entering once his data and that the application does the rest collecting the saved data.
If you're using Actions on Google, you can use a feature to store user information:
function simpleResponse(conv) {
conv.user.storage.count = 1;
conv.ask(new SimpleResponse({
speech: 'Howdy! I can tell you fun facts about ' +
'almost any number, like 42. What do you have in mind?',
text: 'Howdy! I can tell you fun facts about almost any ' +
'number. What do you have in mind?',
}));
}
I've implemented the implicit flow for Actions on Google account linking, and am using Dialogflow (previously API.AI) to define intents.
The full flow works in the device simulator (from AOG). The first intent gets a "It looks like your account isn't linked yet..." response, and the debug pane includes a URL to initiate linking:
https://assistant.google.com/services/auth/handoffs/auth/start?account_name=[account]#gmail.com&provider=[project_id]_dev&scopes=email&return_url=https://www.google.com/
If I follow this URI in a cache-less window:
I'm redirected to my app's authentication page
I choose to sign in with my Google account (same as [account] above)
I'm redirected to google.com with a success message in the URI bar
The simulator now accepts actions via my app and responds correctly
However, if I follow the same flow using a physical Google Home & the gH app for Android.
Device tells me account not yet linked
Open Google home and follow 'Link to [my app]' link
Browser opens to authentication page
Sign in as user
Redirected to a white page with a single link "Return to app", which has an href: about:invalid#zClosurez
Linking was unsuccessful, so additional attempts to run intents on the Google Home get the same "Account not yet linked" response.
I've inspected the intermediate access_token and state variables at length, and they all match and look to be correctly formatted:
Authentication URL (app sign in page): https://flowdash.co/auth/google?response_type=token&client_id=[client_id]&redirect_uri=https://oauth-redirect.googleusercontent.com/r/[project_id]&scope=email&state=[state]
After authenticating, redirected to (this is the white screen with 'return to app' broken link): https://oauth-redirect.googleusercontent.com/r/genzai-app#access_token=[token]&token_type=bearer&state=[state]
So, it seems there's something non-parallel about the way the simulator and physical devices work in terms of implicit flow account linking.
I've been struggling with this, and with the AOG support team for a very long time to no avail. Anyone else see a similar issue?
Updated with response redirect code:
Login handled by react-google-login component with profile & email scopes. On success we call:
finish_auth(id_token) {
let provider = {
uri: '/api/auth/google_auth',
params: ['client_id', 'redirect_uri', 'state', 'response_type'],
name: "Google Assistant"
}
if (provider) {
let data = {};
provider.params.forEach((p) => {
data[p] = this.props.location.query[p];
});
if (id_token) data.id_token = id_token;
api.post(provider.uri, data, (res) => {
if (res.redirect) window.location = res.redirect;
else if (res.error) toastr.error(res.error);
});
} else {
toastr.error("Provider not found");
}
}
provider.uri hits this API endpoint:
def google_auth(self):
client_id = self.request.get('client_id')
redirect_uri = self.request.get('redirect_uri')
state = self.request.get('state')
id_token = self.request.get('id_token')
redir_url = user = None
if client_id == DF_CLIENT_ID:
# Part of Google Home / API.AI auth flow
if redirect_uri == "https://oauth-redirect.googleusercontent.com/r/%s" % secrets.GOOGLE_PROJECT_ID:
if not user:
ok, _email, name = self.validate_google_id_token(id_token)
if ok:
user = User.GetByEmail(_email, create_if_missing=True, name=name)
if user:
access_token = user.aes_access_token(client_id=DF_CLIENT_ID)
redir_url = 'https://oauth-redirect.googleusercontent.com/r/%s#' % secrets.GOOGLE_PROJECT_ID
redir_url += urllib.urlencode({
'access_token': access_token,
'token_type': 'bearer',
'state': state
})
self.success = True
else:
self.message = "Malformed"
else:
self.message = "Malformed"
self.set_response({'redirect': redir_url}, debug=True)
I am able to make it work after a long time. We have to enable the webhook first and we can see how to enable the webhook in the dialog flow fulfillment docs If we are going to use Google Assistant, then we have to enable the Google Assistant Integration in the integrations first. Then follow the steps mentioned below for the Account Linking in actions on google:-
Go to google cloud console -> APIsand Services -> Credentials -> OAuth 2.0 client IDs -> Web client -> Note the client ID, client secret from there -> Download JSON - from json note down the project id, auth_uri, token_uri -> Authorised Redirect URIs -> White list our app's URL -> in this URL fixed part is https://oauth-redirect.googleusercontent.com/r/ and append the project id in the URL -> Save the changes
Actions on Google -> Account linking setup 1. Grant type = Authorisation code 2. Client info 1. Fill up client id,client secrtet, auth_uri, token_uri 2. Enter the auth uri as https://www.googleapis.com/auth and token_uri as https://www.googleapis.com/token 3. Save and run 4. It will show an error while running on the google assistant, but dont worry 5. Come back to the account linking section in the assistant settings and enter auth_uri as https://accounts.google.com/o/oauth2/auth and token_uri as https://accounts.google.com/o/oauth2/token 6. Put the scopes as https://www.googleapis.com/auth/userinfo.profile and https://www.googleapis.com/auth/userinfo.email and weare good to go. 7. Save the changes.
In the hosting server(heroku)logs, we can see the access token value and through access token, we can get the details regarding the email address.
Append the access token to this link "https://www.googleapis.com/oauth2/v1/userinfo?access_token=" and we can get the required details in the resulting json page.
`accessToken = req.get("originalRequest").get("data").get("user").get("accessToken")
r = requests.get(link)
print("Email Id= " + r.json()["email"])
print("Name= " + r.json()["name"])`
Not sure which python middleware or modules you are using but
self.set_response({'redirect': redir_url}, debug=True)
seems to be setting parameters for a returning a response which isn't correct. Instead you should redirect your response to the redirect_url. For example importing the redirect module in Flask or Django like:
from flask import redirect or from django.shortcuts import redirect
then redirect like:
return redirect(redirect_url)
It appears Google has made a change that has partially solved this problem in that it is now possible to complete the implicit account linking flow outside of the simulator, in the way outlined in my question.
It seems the problem stemmed from an odd handling (on the AOG side) of the client-side redirect case used after sign in with the Google sign-in button.
From Jeff Craig in this thread:
The current workaround, where we provide the "Return to app" link
currently what we're able to provide. The issue is with the way that
redirecting to custom-scheme URIs is handled in Chrome, specifically,
with regard to the redirect happening in the context of a user action.
XHR will break that context, so what is happening is that you click
the Google Sign-In Button, which triggers an XHR to Google's servers,
and then you (most likely) do a client-side redirect back to the
redirect_url we supply, our handler executes, and isn't able to do a
JS redirect to the custom scheme URI of the app, because were outside
of the context of a direct user click.
This is more of a problem with the Implicit (response_type=token) flow
than with the authorization code (response_type=code) flow, and the
"Return to app" link is the best fallback case we currently have,
though we are always looking for better solutions here as well.
The current behavior shows the 'Return to app' link, but as of last week, this link's href is no longer about:invalid#zClosurez, but instead successfully completes the sign-in and linking process. It's an odd and confusing UX that I hope Google will improve in the future, but it was sufficient to get my app approved by the AOG team without any changes to my flow.
We have a chatbot for our website today, that is not build using Google technology. The bot has a JSON REST API where you can send the question to and which replies with the corresponding answers. So all the intents and entities are being resolved by the existing chatbot.
What is the best way to wrap this functionality in Google Assistant / for Google Home?
To me it seems I need to extract the "original" question from the JSON that is send to our webservice (when I enable fullfilment).
But since context is used to exchange "state" I have to find a way to exchange the context between the dialogflow and our own chatbot (see above).
But maybe there are other ways ? Can it (invoke our chatbot) be done directly (without DialogFlow as man in the middle) ?
This is one of the those responses that may not be enough for someone who doesn't know what I am talking about and too much for someone who does. Here goes:
It sounds to me as if you need to build an Action with the Actions SDK rather than with Dialog flow. Then you implement a text "intent" in your Action - i.e. one that runs every time the user speaks something. In that text intent you ask the AoG platform for the text - see getRawInput(). Now you do two things. One, you take that raw input and pass it to your bot. Two, you return a promise to tell AoG that you are working on a reply but you don't have it yet. Once the promise is fulfilled - i.e. when your bot replies - you reply with the text you got from your bot.
I have a sample Action called the French Parrot here https://github.com/unclewill/french_parrot. As far as speech goes it simply speaks back whatever it hears as a parrot would. It also goes to a translation service to translate the text and return the (loose) French equivalent.
Your mission, should you choose to accept it, is to take the sample, rip out the code that goes to the translation service and insert the code that goes to your bot. :-)
Two things I should mention. One, it is not "idiomatic" Node or JavaScript you'll find in my sample. What can I say - I think the rest of the world is confused. Really. Two, I have a minimal sample of about 50 lines that eschews the translation here https://github.com/unclewill/parrot. Another option is to use that as a base and add code to call your bot and the Promise-y code to wait on it to it.
If you go the latter route remove the trigger phrases from the action package (action.json).
So you already have a Backend that process user inputs and sends responses back and you want to use it to process a new input flow (coming from Google Assistant)?
That actually my case, I've a service as a Facebook Messenger ChatBot and recently started developing a Google Home Action for it.
It's quite simple. You just need to:
Create an action here https://console.actions.google.com
Download GActions-Cli from here https://developers.google.com/actions/tools/gactions-cli
Create a JSON file action.[fr/en/de/it].json (choose a language). The file is your mean to define your intents and the URL to your webhook (a middleware between your backend and google assistant). It may look like this:
{
"locale": "en",
"actions": [
{
"name": "MAIN",
"description": "Default Welcome Intent",
"fulfillment": {
"conversationName": "app name"
},
"intent": {
"name": "actions.intent.MAIN",
"trigger": {
"queryPatterns": [
"Talk to app name"
]
}
}
}
],
"conversations": {
"app name": {
"name": "app name",
"url": "https://your_nodejs_middleware.com/"
}
}
}
Upload the JSON file using gactions update --action_package action.en.json --project PROJECT_ID
AFAIK, there only a Node.js client library for Actions-on-google https://github.com/actions-on-google/actions-on-google-nodejs that why you need a Node.js middleware before hitting your backend
Now, user inputs will be sent to your Node.js middleware (app.js) hosted at https://your_nodejs_middleware.com/ which may look like:
//require express and all required staff to build a Node.js server,
//look on internet how to build a simple web server in Node.js
//if you a new to this domain. const {
ActionsSdkApp } = require('actions-on-google');
app.post('/', (req, res) => {
req.body = JSON.parse(req.body);
const app = new ActionsSdkApp({
request: req,
response: res
});
// Create functions to handle requests here
function mainIntent(app) {
let inputPrompt = app.buildInputPrompt(false,
'Hey! Welcome to app name!');
app.ask(inputPrompt);
}
function respond(app) {
let userInput = app.getRawInput();
//HERE you get what user typed/said to Google Assistant.
//NOW you can send the input to your BACKEND, process it, get the response_from_your_backend and send it back
app.ask(response_from_your_backend);
}
let actionMap = new Map();
actionMap.set('actions.intent.MAIN', mainIntent);
actionMap.set('actions.intent.TEXT', respond);
app.handleRequest(actionMap); });
Hope that helped!
Thanks for all the help, the main parts of the solution are already given, but I summarize them here
action.json that passes on everything to fullfilment service
man in the middle (in my case IBM Cloud Function) to map JSON between services
Share context/state through the conversationToken property
You can find the demo here: Hey Google talk to Watson
I'm a newbie to app development. I am building a Windows Phone 8.1 app and have followed the tutorial here: http://azure.microsoft.com/en-us/documentation/articles/app-service-mobile-dotnet-backend-windows-store-dotnet-get-started-users-preview/ to add authentication using Facebook. Everything seems to work fine, except that every now and again it appears to stop bringing back any data from my Azure database. Further investigation revealed that the UserId that is being shown from the code below, changes periodically (although I can't quite work out how often it changes).
// Define a member variable for storing the signed-in user.
private MobileServiceUser user;
...
var provider = "Facebook";
...
// Login with the identity provider.
user = await App.MobileService.LoginAsync(provider);
// Create and store the user credentials.
credential = new PasswordCredential(provider,
user.UserId, user.MobileServiceAuthenticationToken);
vault.Add(credential);
...
message = string.Format("You are now logged in - {0}", user.UserId);
var dialog = new MessageDialog(message);
dialog.Commands.Add(new UICommand("OK"));
await dialog.ShowAsync();
This code is identical to the code in the tutorial. The Facebook app settings (on the Facebook developers site) confirm that I am using v2.3 of their API so I should be getting app-scoped UserIds back. I have only ever logged in with one Facebook account, so I would expect the UserId to be the same each time, but they're not. The UserId is prefaced with 'sid:', which someone on the Facebook developers group on Facebook itself says stands for Session ID, which they would expect to change, but if that's the case, I can't work out where to get the actual UserId from that I can then store in my database and do useful things with. I'm sure I must be doing something basic wrong, but I have spent hours Googling this and cannot (unusually) find an answer.
Any help would be greatly appreciated.
Thanks!
So dug deeper. This is how Mobile Apps work (I was thinking from a Mobile Services perspective). The issue here is that the Gateway doesn't provide static SIDs, which is what User.userId provides. The work around to this is listed in the migration doc.
You can only get the Facebook AppId on the server.
ServiceUser user = (ServiceUser) this.User;
FacebookCredentials creds = (await user.GetIdentitiesAsync()).OfType< FacebookCredentials >().FirstOrDefault();
string mobileServicesUserId = creds.Provider + ":" + creds.UserId;
You should note, that this Id is directly connected with your Facebook App registration. If you ever want to migrate your App to a new Facebook App, you'd have to migrate them. You can also use the Facebook AppId to look up the user's global facebook Id via the Facebook Graph API, which you could use between applications. If you don't see yourself using multiple apps, etc., you can use the Facebook AppId just fine.
Hard to tell what's going on to cause you to use a SID instead of the Faceboook token (which like Facebook:10153...).
It may be faster to rip out the code and reimplement the Auth GetStarted. Maybe you missed a step or misconfigured something along the way. If you have the code hosted on github, I can try to take a look.
Another thing you can do is to not trust the user to give you their User id when you save it to a table. On your insert function, you can add it there.
function insert(item, user, request) {
item.id = user.userId;
request.execute();
}
That should, theoretically, be a valid Facebook token. Let me know if that doesn't work; can dig deeper.