I'm designing a web app with Electron to play back pipe organ sample files. Whenever concurrent note polyphony nears ~1024, the sound completely drops out, including subsequent reverb nodes. After the sounds would theoretically stop playing in the background (because I have released the key), the audio eventually comes back in.
Is this a hard limit on the Web Audio API? I also notice high CPU usage for that tab when it seems to be jammed. Is there a way to enable more concurrent audio sources? Ideally, I need to have tens of thousands for proper polyphony (although many of them are the same audio files being repeated)
I'm currently looping the samples with Tone.js, if that makes a difference.
Since you mentioned that you're building an Electron app I think it should be possible to run the same code in Chrome as well. Which means you could also profile it like any other code running in Chrome.
Here is the official guide from the Chrome team on profiling: Profiling Web Audio apps in Chrome.
Doing that hopefully allows you to spot any bottlenecks or performance problems in your app.
Related
I feel like I have searched and tried every audio package out there, but can't find a way to control the microphone gain. I'm building a recording app and theres a big issue when it comes to the volume of the recorded sounds. I don't want to increase the output volume, I want to increase the input volume.
If I understand it correctly, the iOS native API is the AVAudioSession. I found this package that I thought would enable this control, but doesn't seem to offer this functionality. Do I need to write some bridge code to deal with the native APIs for this or is it not even possible?
I'd like to develop a game/skill on Google Assistant that requires the following, once the user has entered the game/session (“hey Google, start game123”)
playing an audio file that is a few minutes long
playing a second audio file while the first clip is still playing
always listening. While the files are playing, the game needs to listen and respond for specific voice phrases without the “Hey Google” keyword.
Are these capabilities supported? Thanks in advance.
"Maybe." A lot of it depends what devices on the Actions on Google platform you're looking to support and how necessary some of the requirements are. Depending on your needs, you may be able to play some tricks.
Playing an audio file that is "a few minutes" long.
You can play audio using SSML that is up to 120 seconds long. But that will be played before the microphone is opened to accept a response.
For longer files, you can use a Media Response. This has the interesting feature that when the audio finishes, an event will be sent to your server, so you have some limited way to handle timed responses and looping. On the downside - users have to say "Hey Google" to interrupt it. (And there are currently some bugs when using it.)
Since you're doing a game, you can take advantage of the Interactive Canvas. This will let you use things such as the HTML <audio> tag and the Web Audio API. The big downside is that this is only available on Smart Displays and Android devices - you can't use it on Smart Speakers.
Playing multiple audio tracks
Google has an extension to SSML that allows parallel audio tracks for multiple spoken and audio output. But you can't layer these on top of a Media Response.
If you're using the Web Audio API with the Interactive Canvas, I believe it supports multiple simultaneous inputs.
Can I leave the microphone open so they don't have to say "Hey Google" every time.
Probably not, but this may not be a good idea in some cases, anyway.
For Smart Speakers, you can't do this. People are used to something conversational, so they're waiting for the silence to know when they should be saying something. If you are constantly providing audio, they don't necessarily know when it is their "turn".
With the Interactive Canvas devices, we have a display that we can work with that cues them. And we can keep the microphone open during this time... at least to a point. The downside is that we don't know when the microphone is open and closed, so we can't duck the audio during this time. (At least not yet.)
Can I do what I want?
You're the only judge of that. It sounds like the Interactive Canvas might work well for your needs - but won't work everywhere. In some cases, you might be able to determine the capabilities of the device the user is playing with and present slightly different games depending on the features you have. Google does this, for example, with their "Lucky Trivia" game.
I ran into this problem in Safari where it appears that WebRTC is not fully supported. So when I call
navigator.webkitGetuserMedia()
I get an undefined error.
So my question to the community is what is the best way to write a Meteor app that captures Video on a mobile device and saves it on the said device.
If you have done this, I would appreciate it very much if you could share with me and the community how you went about this.
Specific Answer
The modern API is: navigator.mediaDevices.getUserMedia(constraints). See the docs here.
In the past, I've been unsuccessful with getUserMedia on iOS, but according to this post it can be done on iOS 11.
As for saving it, you can write to the browser's file system, but that API is only supported in Chrome. If you want to write to the camera roll, you'd need native code in the mix.
General Advice
I've spent several years of my life dealing with recording, uploading, and processing video using meteor. If you are doing anything more than trivial web recording, these observations may save you some time:
Chrome (on everything but iOS) has the best API for web recording. If you can require chrome for recording, that's ideal. Firefox is a close second, only because it doesn't support the file system API.
If you need to record and upload long videos on iOS, build a native app. Don't consider any kind of hybrid - that's a serious trap. The number of corner cases and things you need to check is pretty astounding, and the only way to get over those hurdles is with native code.
I'd like to get real-time video from the iPhone to another device (either desktop browser or another iPhone, e.g. point-to-point).
NOTE: It's not one-to-many, just one-to-one at the moment. Audio can be part of stream or via telephone call on iphone.
There are four ways I can think of...
Capture frames on iPhone, send
frames to mediaserver, have
mediaserver publish realtime video
using host webserver.
Capture frames on iPhone, convert to
images, send to httpserver, have
javascript/AJAX in browser reload
images from server as fast as
possible.
Run httpServer on iPhone, Capture 1 second duration movies on
iPhone, create M3U8 files on iPhone, have the other
user connect directly to httpServer on iPhone for
liveStreaming.
Capture 1 second duration movies on
iPhone, create M3U8 files on iPhone,
send to httpServer, have the other
user connected to the httpServer
for liveStreaming. This is a good answer, has anyone gotten it to work?
Is there a better, more efficient option?
What's the fastest way to get data off the iPhone? Is it ASIHTTPRequest?
Thanks, everyone.
Sending raw frames or individual images will never work well enough for you (because of the amount of data and number of frames). Nor can you reasonably serve anything from the phone (WWAN networks have all sorts of firewalls). You'll need to encode the video, and stream it to a server, most likely over a standard streaming format (RTSP, RTMP). There is an H.264 encoder chip on the iPhone >= 3GS. The problem is that it is not stream oriented. That is, it outputs the metadata required to parse the video last. This leaves you with a few options.
Get the raw data and use FFmpeg to encode on the phone (will use a ton of CPU and battery).
Write your own parser for the H.264/AAC output (very hard)
Record and process in chunks (will add latency equal to the length of the chunks, and drop around 1/4 second of video between each chunk as you start and stop the sessions).
"Record and process in chunks (will add latency equal to the length of the chunks, and drop around 1/4 second of video between each chunk as you start and stop the sessions)."
I have just wrote such a code, but it is quite possible to eliminate such a gap by overlapping two AVAssetWriters. Since it uses the hardware encoder, I strongly recommend this approach.
We have similar needs; to be more specific, we want to implement streaming video & audio between an iOS device and a web UI. The goal is to enable high-quality video discussions between participants using these platforms. We did some research on how to implement this:
We decided to use OpenTok and managed to pretty quickly implement a proof-of-concept style video chat between an iPad and a website using the OpenTok getting started guide. There's also a PhoneGap plugin for OpenTok, which is handy for us as we are not doing native iOS.
Liblinphone also seemed to be a potential solution, but we didn't investigate further.
iDoubs also came up, but again, we felt OpenTok was the most promising one for our needs and thus didn't look at iDoubs in more detail.
I'm trying to build a web application for iPhone and Android that deals with audio input.
Is this possible?
Apparently ... yes, or it should be able to when it's finished at least. It will supposedly become possible using the device API which is due to be part of HTML5 when it's finished and released (HTML5 isn't finalised yet however, and information is subject to potential for change).
W3C Device API Requirements (camera section)
Sony Erricson community blog posting, with examples (pre-final API)
While it isn't explicitly mentioned in the W3C spec, audio recording as part of (web)camera interactions is, so it's definitely hopeful. There seems to be a shortage of good information at this stage though. I'd expect to see more as HTML5 comes closer to being finalised.
As of now, HTML5 Can't record Audio. but in future, it will be able to, by using the Device's native features.
HTML 5 can not record audio (at least currently). HTML basically is a markup language and therefore only declares how a browser should display certain content. Although HTML 5 introduces new features that make some interaction possible, you can't record audio straight into.. HTML (even saying that sounds wrong). When the HTML5 spec is finished, it might become reality, until then, no way.
Web applications that record audio normally require a plugin like Flash or Silverlight, because those can access system resources like audio hardware. Both are a no-go on iOS, although Flash is theoretically possible on Android, I don't know if it supports audio input.
I would suggest you write a native app (for iOS and Android) that can access the audio hardware and connects to your web application in the background, so that the recording takes place natively and the recorded audio will be transmitted to your servers (think of Shazam, for example).
Here are the basic developer guides on recording audio in:
Android
OS X, iOS
A new MediaStream Recording API is being worked on. It is currently availble only in the Firefox Nightly build for demo purposes
Here's the draft with the latest updates directly form W3C site:
https://dvcs.w3.org/hg/dap/raw-file/default/media-stream-capture/MediaRecorder.html
Also the following article covers up other attempts on recording audio and video directly in the browser:
http://hdfvr.com/html5-video-recording