Question
Is it possible to resume the content of a media object once a user has tried to interact with a Google Home device during the playing of that media?
The problem
Say you have started playing an mp3 file using conv.ask. Your call to conv.ask will look something along the lines of this:
conv.ask(`<speak><audio src="${someUrl1}"><desc>${someDescription}</desc></audio></speak>`)
.add(new MediaObject({
url: someUrl2
}))
.add(new Suggestions(['suggestion1', 'suggestion2']));
This plays all fine and well. But then say a user says something along the lines of 'Ok Google, Gobbledygoop', you then might want to tell the user that their request was nonsensical, and then continue the playing of the media in the media object.
What I have tried already
app.fallback(): This does not seem compatible with the actions SDK. It does not seem possible under any circumstance to get the callback (the one provided to app.fallback) to be called.
Providing conv.ask with null/empty string responses: This was a desperate attempt to see what would happen if you provided nothing to conv.ask. There was a hope that it would see the empty response and just keep playing the media.
There is nothing that is part of Actions on Google itself that will do this for you.
The best you can probably do requires a lot of effort on your part:
You can include as part of the session state (or in a context) when you replied to the user with the Media result.
If you get another request before the Media Status event, you can determine the difference between the two, and this will roughly be how long the audio had been playing.
You can then return a URL for audio that includes starting at this point in the audio. However, the audio offset isn't something that the Assistant does, you'll have to support this on the server that contains the audio as well.
As for the two things you attempted - app.fallback() should have worked to handle any intent that you had set to go to your webhook that didn't have any other handler defined, but that still wouldn't just be able to "resume" the audio. conv.ask() requires you to ask something - null replies aren't allowed.
In this case, you at least want to tell the user that what they said made no sense... before resuming the audio.
Related
I have an NFC tag that has integrated environmental sensors inside (MLX90129 to be exact). I would like to make an iPhone app that can read the realtime data from the tag multiple times per second and graph them. I'm not looking for background tag reading, and you can assume that the app will be open and the phone is near the tag at all times.
From what I can see on Apple documentation and other sources, the Swift support for NFC tags is mostly built for single session interrogation. Has anyone succeeded in getting continuous and repeated NFC tag reading for this type of purpose?
As you pointed out: "to make continuous and repeated NFC readings" it's not the intended functionality.
While I think that you can sort this out, there's another thing that could be a headache... to make multiple readings per second it's directly confronted to the current implementation of NFC tag reading in iOS.
Every time you start a reading, it shows the native window which informs the user that you are making a NFC Reading. A part of this process is the interaction of the user, and is exactly that part the one that imposes a time constraint. Even if the interaction with the user is not needed, there is an animation, and that animation has its lifecycle's events (start reading, reading, OK, KO, close...).
Afaik you can't bypass that animation which definitely could represent a couple seconds in the best case.
With that said, you should have a few things in mind, if you still want to try:
NFCTagReaderSession can only have one active reading at a time, and when that reading ends (OK/KO), it should be invalidated. So if you want to make another reading, you'll need to create and configure a new instance.
For the google actions that i am developing some responses are complex and take more than 5 seconds to process.
Could someone please suggest how can this be handled.
Generally i would consider using loading dots and then replacing that message with the result but i don't see any Google Action API for it. Also is there any endpoint to which we could async send back the result later ?
Thanks
PS: I am using Conversation API.
We don't really have a good way to handle this right now, but we have a couple of approaches that sorta work based on your needs.
Notifications are currently available for the Assistant on smartphones, and they're coming for speakers. In some cases, it might make sense to say that you're working on the problem and you'll send a notification when you have it, and then resume the conversation from the notification.
Another approach is to use the Media Response to play a bit of "hold music". At the end of the segment of music, your webhook will get a notice that the music has completed. If you have the result available, you can report it at that time.
In my conversation dialogflow, I would like to add some progress messages like hang in with me, I'm looking up for that data or similar in the conversation. Is there any guidance or best practice to do this?
Unfortunately, there is no good way to do this at this time. If your webhook takes longer than about 5 seconds, Dialogflow will return one of the default responses it is set with. If you're not using Dialogflow, the Action SDK will say your webhook isn't responding and will close the conversation.
There is currently no way to send a reply, and then send another reply without the user saying something first.
One workaround might be to have the default response be something like "I'm looking that information up. Ask me again in a few seconds." When your lookup finally completes, cache the information so when/if the user asks the question again, you can return it more quickly.
Depending how long it takes, you may also wish to register a dynamic reprompt. This will send an event to your webhook if the user doesn't say anything. In a situation like this, they may say nothing for a few seconds, but that may be long enough for you to have computed the reply. So after a few seconds of silence you can suddenly announce "I've figured it out, the answer you were looking for is..." or something similar. This has some limitations - you can only reprompt twice like this before Google sends you a final reprompt and closes the conversation.
Although the platform does support notifications, these are still in developer preview and don't work with all devices. They also don't quite continue the conversation (it doesn't just start talking) - they just send a notification to a phone that there is a message and that they can restart the conversation. Depending on your use case, this may be useful combined with the above.
Update
The Media Response includes a feature that we can take advantage of to handle this. Similar to the dynamic reprompt method above, you'll get a call automatically when the media you're playing ends. So you can play a short "hold music" and your webhook will be called when it is finished. You can then either give the result or say you're still working on it and play more hold music.
I'm pretty sure the answer to this is "no" but I figured I'd share the question anyway in case others have a clever workaround :)
I'm building a recipe action so the user could say "Let's make tortilla soup" and then say "next step" to move on to the next part of the recipe. Between each step there might be a long pause as the user is cutting vegetables, etc. Is it possible to have Home either indefinitely wait for a user response or wait for several minutes? Currently it'll wait a few seconds and say something like "Sorry I didn't understand that response" and eventually quit the action.
Forcing the user to go through the "OK, Google, let me talk to the Chef" action over and over is pretty annoying and, I assume, would require immediately ending the conversation after every step, otherwise the device will hang and say "Sorry I didn't understand".
Update
I've found a pretty hacky way of doing this by abusing SSML. There are two options, you can stack up <break/>s
assistant.ask(`<speak><break time="120s"/><break time="120s"/></speak>`);
This actually causes the Home to play a really weird droning noise. Something the devs might wanna look at :D
Another option (which avoids the drone) is to play a 2 minute silent audio clip. According to the docs, 2 minutes is the limit for <audio> but you can stack them up. I just verified that you can make it sit there for at least ten minutes.
assistant.ask(`<speak><audio src="https://.../pause.mp3">hello</audio><audio src="https://.../pause.mp3"></audio></speak>`)
You cannot speak directly to the agent while its paused like this but you can say "OK Google, [whatever command]" and that command will actually get passed to the agent.
There's not currently a way to have the Assistant wait indefinitely for a response.
While technically possible, an app using the suggested workaround to delay responding wouldn't pass the review process - the policies document mentions avoiding playing a silent sound file and communicating for a period in excess of 120 seconds.
A different approach could be to have your app remember the current position in a recipe, but end the conversation after each step. You could then use action invocation to allow the user to say something like "OK Google, ask [your app name] to continue the recipe", jumping back into the conversation and hearing the next step.
There is not, but there is a recently announced feature that might help you do what you're trying.
When answering, you can give the reply and play an audio file using the Media Control. This has several advantages over using the SSML approach you give:
You'll get an event when the audio finishes, so you can prompt again with the guide, or a tip, or a reminder that your action is still there... and then play more audio while you wait.
At any time, the user can say "Hey Google, next step" and your server will get the message and you can send the next step. Or they can say "hey Google, repeat that" and you'll get that message and can repeat the instructions.
They can also ask other questions of your Action that you can answer.
I've decided to integrate OpenFeint into my new game to have achievements and leaderboards.
The game is dynamic and I would like user to be rewarded immediately for some successful results, but as it seems for me, OpenFeint's achievements are a bit sluggish and it shows visual notification only when it receives confirmation from the server.
Is it possible to change something in settings or hack it a little bit to show notification immediately as soon as it checks only local database if the achievement has not been unlocked it?
Not sure if this relates to the Android version of the SDK (which seems even slower), but we couldn't figure out how to make it faster. It was so unacceptably slow that we started developing our own framework that fixes most of open feint's shortcomings and then some. Check out Swarm, it might fit your needs better.
There are several things you can do to more tightly control the timing of these notifications. I'll explain one approach and you can use this as a starting point to explore further on your own. These suggestions apply specifically to iOS apps. One caveat is that these suggestions refer to internal APIs in OFSDK 2.8 for iOS and not ordinarily recommended for high level use and subject to change in future versions.
The first thing I recommend is that you build the sample app with your own product key. Use the standard sample app to experiment before applying the result to your own code.
You are going to get the snappiest response by separating the notification pop-up UI from the process of submitting the achievement. This way you don't have to worry about getting wrapped up in the logic for deciding whether the submission is going just to the local db or is doing the full confirmation on an async network transaction.
See the declaration of "showAchievementNotice" in "OFNotification.h". Performing a search in the sample app, you will see that this is the internal API used for displaying the achievement pop-up when an achievement is earned. It does not actually submit the achievement. You can call this method directly as it is called from "OFAchievementService.mm" to directly control when the message appears. You can then use the following article to disable the pop-up from being called when the actual submission occurs:
http://support.openfeint.com/dev/notification-pop-ups-in-ios/
This gives you complete freedom to call the submission at a later time provided you keep track of the need to do so. For example, you could locally serialize a flag to take care of the actual submission either after the level is done or the next time the app starts up. Don't forget that the user could quit out of a game without cleanly finishing a level.