I am writing an app to work with Google Actions. The only bummer is that I can't find any information about how to form my response so that Google will stream audio from a given URL. Does Google even support this yet?
I have written the same app on Alexa already, and on Alexa all you have to do is Return an audio item (token, URL, play command) and Alexa will start streaming it.
I should mention that I am NOT using API.AI, but am simply using the Actions SDK and am hosting my web service on Asure using C#.
So, bottom line... How can I format a response via the Actions SDK to stream an MP3 file to Google Home?
UPDATE: The first answer works only with the V1 of Dialogflow. As for the V2, you can create the mediaResponse this way (from Google's doc):
conv.ask(new MediaObject({
name: 'Jazz in Paris',
url: 'http://storage.googleapis.com/automotive-media/Jazz_In_Paris.mp3',
description: 'A funky Jazz tune',
icon: new Image({
url: 'http://storage.googleapis.com/automotive-media/album_art.jpg',
alt: 'Media icon',
}),
}));
========================================================================
I posted an answer over here.
Basically you can create a mediaResponse object that will play your audio file. I can play a 50 mins audio file just fine.
A code example in Node.js could be (with the current documentation):
const richResponse = app.buildRichResponse()
.addSimpleResponse("Here's song one.")
.addMediaResponse(app.buildMediaResponse()
.addMediaObjects([
app.buildMediaObject("Song One", "https://....mp3")
.setDescription("Song One with description and large image.") // Optional
.setImage("https://....jpg", app.Media.ImageType.LARGE)
// Optional. Use app.Media.ImageType.ICON if displaying icon.
])
)
.addSuggestions(["other songs"]);
According to the documentation you can embed elements in SSML. https://developers.google.com/actions/reference/ssml includes the following example:
<speak>
Here are <say-as interpet-as="characters">SSML</say-as> samples.
I can pause <break time="3s"/>.
I can play a sound
<audio src="https://www.example.com/MY_MP3_FILE.mp3">didn't get your MP3 audio file</audio>.
I can speak in cardinals. Your number is <say-as interpret-as="cardinal">10</say-as>.
Or I can speak in ordinals. You are <say-as interpret-as="ordinal">10</say-as> in line.
Or I can even speak in digits. The digits for ten are <say-as interpret-as="characters">10</say-as>.
I can also substitute phrases, like the <sub alias="World Wide Web Consortium">W3C</sub>.
Finally, I can speak a paragraph with two sentences.
<p><s>This is sentence one.</s><s>This is sentence two.</s></p>
</speak>
EDIT
p/s : SSML in Documents has these limitations :
Single channel is preferred, but stereo is acceptable.
120 seconds maximum duration. If you want to play audio with a longer duration, consider implementing a media response. 5 megabyte file size limit.
Source URL must use HTTPS protocol.
Our UserAgent when fetching the
audio is "Google-Speech-Actions".
Related
I have a form application where I use the System voice to read words. However, it sounds robotic, how to make it more human-like?
I am using this:
using namespace System::Speech::Synthesis;
AND this:
SpeechSynthesizer^speaker=gcnew SpeechSynthesizer();
speaker->SpeakAsync(textBox1->Text);
The program works though, but I want it to sound like a human.
System.Speech.Synthesis is very VERY old (15+ years old). Thus, it sounds robotic.
You could try using the latest speech platform from Microsoft. NOTE: I worked on System.Speech and on this latest speech platform (Cognitive Services Speech). You can find samples and instructions on how to use this platform here: https://aka.ms/speech/sdk.
Additionally, if you'd like to hear what these voices sound like, you can listen to samples of them here: https://azure.microsoft.com/en-us/services/cognitive-services/text-to-speech/
--robch
when responding to a user query using actions sdk I am able to create a basic card using:
conv.ask(new BasicCard({
text: 'Text with card display',
title: 'Title:',
display: 'CROPPED',
}));
However, if I wish to provide the user with some audio (different from the display text) how do I do it?
I tried to add a conv.ask('<speak>' + 'Hello' + '</speak>'); but it throws a error
MalformedResponse
expected_inputs[0].input_prompt.rich_initial_prompt.items[0].simple_response: 'display_text' must be set or 'ssml' must have a valid display rendering.
What is the best way to include an audio in a google actions project? Thanks
If you want to play the audio in the background, I'd suggest using SSML, but if your actual goal is to just deliver the audio to the user (like if it's a podcast or something) you can use a Media Response.
If, however, you want the text displayed on a device with a screen to be different from the text that's spoken, you could add a Simple Response (which has the option to add different text and speech).
The Basic Card does not have audio attached to it. As the name suggests, it is a visual cue rather than an audible one. It is meant to supplement the text that is spoken and displayed - not replace it.
While you can create a SimpleResponse that has different text that is spoken vs displayed, you should make sure that both responses are substantially the same. You can use a SimpleResponse with something like this:
conv.ask(new SimpleResponse({
speech: '<speak>Here are the results, showing our sunny recreational facilities. <audio src="https://actions.google.com/sounds/v1/animals/cicada_chirp.ogg">And the sounds of nature.</audio></speak>',
text: 'Here is the result',
}));
I'm using https://github.com/gilesvangruisen/Swift-YouTube-Player to read youtube videos in my app such as this one : https://www.youtube.com/watch?v=Bw0tjdBgKBY
With a WEB browser it works well, but using the library, a message appears saying :
This video is unavailable
I can see the video preview before playing the video, but when the video starts the message appears in my app.
Is there video contents locked by youtube when using this kind of library ?
Every video published by FIFATV seems to be unavailable using this library...
I'm setting the video in the youtubePlayer like that :
youtubePlayerView.loadVideoURL(URL(string: "https://www.youtube.com/watch?v=PBTFz-9bozc"))
Other videos not from FIFATV are working.
Any idea concerning the problem ?
To understand what happened to your streaming, you should add another parameter, so following the Swift-YouTube-Player example correct this part adding the origin parameter:
Swift-YouTube-Player/YouTubePlayerExample/YouTubePlayerExample/ViewController.swift
#IBAction func loadVideo(sender: UIButton) {
playerView.playerVars = [
"playsinline": "1",
"controls": "0",
"showinfo": "0",
"origin": "https://www.youtube.com"
]
playerView.loadVideoID("wQg3bXrVLtg")
}
If you add this parameter the framework avoid to use a blank page as source origin of the streaming and use your value, infact if you take a look to the official sources:
YTPlayerView.m
if (![playerVars objectForKey:#"origin"]) {
self.originURL = [NSURL URLWithString:#"about:blank"];
} else {
self.originURL = [NSURL URLWithString: [playerVars objectForKey:#"origin"]];
}
Now you should have a message like:
"This video contains content from FIFA, who has blocked it from display on this website or application. Watch on YouTube"
Reading the Youtube API docs you can put also http://example.com as value:
where searching for enablejsapi you will find:
Unfortunately,speaking about the YT iOS framework you can search this parameter but you'll find that it's already setted to 1 inside the file player.js and if you'll try to set it inside the playerVars nothing it will happen.
So it's remain (for now..) an Youtube iOS framework bug, probably not properly a bug but YT staff should add to their policies another rule for the iOS content flux, same thing maded for VEVO and a part of UMG content if I remember (and there are also other bugs yet unresolved..)
Does anyone know if the Web Audio API provides the ability to save audio played using the WebAudioContext?
I actually wrote a small utility called RecorderJS that might help.
There is a startRendering function in Chrome at least (haven't checked Safari). I think it's undergoing some rework and thus isn't included in the spec, but might be added at a later stage (or not). If you want to check out the current implementation, have a look at the answer at Is there a way to use the Web Audio API to sample audio faster than real-time?
There is a W3C specification for a recording API http://www.w3.org/TR/mediastream-recording/ , but as of now it is being implemented only in Firefox.
Client side there is available only the ScriptProcessorNode hack (which is what Record.js is based on).
Alternatively, for some use cases it might make sense to stream the audio to a server using WebRTC and write a server side recorder using Libjingle.
This library work fine, web audio api only (meaning no i.e users):
https://github.com/higuma/web-audio-recorder-js
But we can fairly use it now:
http://caniuse.com/#feat=audio-api
Anyway like you said your sound is already in an audiocontext, so I think you are looking for how to use the AudioDestinationNode, the final node of the web audio api. As soon as you can playing your audio through a regular html audio player, you will gain the record function on right click, like playDataUri do. You need to add the attribute "controls" to the player, or you can make a special link with download attribute.
I made a small enhancement of the Mdn script to send the data to an player, it should give you a good idea:
var audioCtx = new AudioContext();
var source = audioCtx.createMediaElementSource(myMediaElement);
myMediaElement = document.createElement("audio");
myMediaElement.setAttribute("autoplay", true);
myMediaElement.setAttribute("src", uri);
myMediaElement.setAttribute("controls", "controls");
document.getElementById('player').appendChild(myMediaElement);
source.connect(audioCtx.destination);
The AudioDestinationNode interface represents the end destination of
an audio graph in a given context — usually the speakers of your
device. It can also be the node that will "record" the audio data when
used with an OfflineAudioContext.
https://developer.mozilla.org/en-US/docs/Web/API/AudioDestinationNode
This is now available in the latest browsers, its called media recorder you can find more information here
Easiest way is to create a stream as
var dest = audioCtx.createMediaStreamDestination();
var options = {
audioBitsPerSecond : 320000,
sampleSize: 16,
channelCount: 2,
mimeType : 'audio/ogg'
}
var mediaRecorder = new MediaRecorder(dest.stream, options);
var chunks = [];
var isrecording = "Not Recording";
function rec(){
mediaRecorder.start();// start record
dataavailable = true;
isrecording = mediaRecorder.state;
}
you can can check out an example of my soundrec
App here. Although its also a full on Multiband compressor and 5 band paragraphic here.
Oh and if you check out my link . the most important thing to get your head around is the ondataavilable method. And Saving as a blob is a bit of ahead bash too.
Ps if anyone wants to help me get this working on chrome sen me an Email. Thanx.
It will only work in Firefox at the moment.
MultiBand Compressor
I am trying to create an aggregator that pulls song links to a central site from many different blogs using youtube, soundcloud, etc. I was wondering where I can find more info about the js streaming api. I'd like to be able to seek with the js streaming api, along with have a callback be called when that song is done playing.
Two things you'll want to check out:
Our HTML5 Widget has an API that allows you to bind code to events and control the player widget (e.g. seek to a specific point in a track). This'll work if you want to use our player widget anyway.
The JavaScript SDK uses SoundManager 2 under the hood. Certain methods, such as SC.stream will return a soundObject which you can use to seek and bind code to events.
Hope that helps. Comment if you have any questions and I'll edit my answer.
This doesn't really help me. I can't find a single example of stopping a stream anywhere. In fact I can't seem to bind to any of the methods described in sound manager. Neither of these work.
+++
SC.initialize({
client_id: "xxx"
});
// Stream the audio
sound = SC.stream("/tracks/293");
sound.play();
+++
SC.stream("/tracks/293", {
autoPlay: true,
ontimedcomments: function(comments){ console.log(comments[0].body);
},
onplay: function(sound){
alert(sound);
}
});
+++
Really, I just want to create a custom streaming player where I can change the music, and stop and hit play (not in jQuery). I haven't found much on this topic.
use SC object to stop the playback. more information at
http://developers.soundcloud.com/docs/api/sdks#methods
SC.recordStop();
top play the song again
'SC.recordPlay();'
hope this helps!
Jesse, Use this to play songs.
SC.get('/tracks/294', function(track){
console.log(track)
// SC.oEmbed(track.permalink_url, document.getElementById('player'))
})
Make sure you have a div#player in your html somewhere. Then, if you uncomment that line you'll see SoundCloud's branded music player.