I am making a video chatting web application using C# socket programming to transfer data. I want to use the Web Audio API to capture audio and video in my view page, but I dont know how to transfer the audio using sockets (which are defined in controller class.) Can the API be used for socket programming if I can capture the raw bits from the API?
(I've also tried using WEB RTC, but I am unable to create multiple peer connections. As my application involves multiple peers, I prefer normal socket programming.)
If you mean, can you just get access to the raw audio/video bits from getUserMedia - yes, you can. (For audio, check out any of the input demos on webaudiodemos.appspot.com - particularly AudioRecorder shows how to get the bits from a ScriptProcessor node.) But I would caution that streaming audio and video over the net is not a trivial task. You can't really just push the bits over the wire with no thought to buffering (or adaptive capabilities, unless you can guarantee high-speed local network only).
Related
I am building an application that gets real-time audio from our organization's VoIP system, records the call and transcribe the real-time voice. The transcription then passed to our analytics engine and get the insights.
We are able to transcribe the recorded audio and get the insights from the transcription. We have a solution for real-time transcription also. It will transcribe the voice from the microphone and even an RTSP stream also. We are having trouble finding a solution for getting the real-time audio from SIP/VoIP systems. I read that SIP Trunking and option and also WebRTC is also another option. But I don't know how to and where to start with.
I am experienced in Java and Python, I requesting experts to give me suggestions or examples on how to get the real-time audio stream from a SIP/VoIP conversation.
I am not familiar with SIP/VoIP and never written VoIP application.
A solution that might suit your needs is Oreka, which is the open source version of Orecx, a call recording software for VoIP.
I used it in the past and it works perfectly well with SIP calls that use open audio codecs like g711 (alaw,ulaw) or speex but it may have problems decoding the audio of calls that use the propietary g729 codec (I had to work out my own codecs at that time).
The paid version might support more codecs and protocols like Avaya's H323.
Have in mind that this app works by sniffing the network, so the setup is not trivial. Anyway, I suggest you give it a try.
Link: https://www.orecx.com/open-source/
For anyone out there. if you want to have access to live/realtime audio data from a VoIP call I suggest you use Twilio Streams.
If you're just looking to get realtime transcriptions without access to the actual audio data Twilio and Plivo also provide that.
How can i stream a live video call between 2 people, to thousands of people. I prefer to use webRTC but I can't find the answer to my question. The viewers should be able to watch the stream in a web app.
Streaming to thousands of people is not trivial! It's not as hard as it used to be 10 years ago but is still pretty hard.
WebRTC supports direct browser to browser (peer to peer) connections. This means that WebRTC is primarily targeted at 1:1 conversation. If you want the same conversation (video or audio) to be shared among more than 2 people you have the following options:
Connect any user to any other user. This creates a fully connected graph between the viewers. This is easy to do because all you need is webrtc. No special hardware/software. However it is also very inefficient in thems of trafic and distribution and doesn't scale boyound 5-6 people.
Use A WebRTC Video relay like Jitsi VideoBridge. According to the official performance claims VideoBridge can scale to 500-1000 people given fast and wide enough internet connection.
Direct the Webrtc stream between the two participants to a WebRTC enabled streaming server. If needed, transcode the input stream to a suitalbe codex - x264/VP8/VP9. Convert the input stream to a sutable protocl - RTMP/HLS/DASH. Distribute the content using the buildin functionality of the media server or by the use of a CDN. Play the video on the client side with a player - Flowplayer/JwPlayer/ViblastPlayer/VideoJs/your own Custome Player or a combination of the above. This is the hardest solution but it is also the best one in terams of scalability and platform reach. Such a solution can scale easily to thousands of people and reach all major browsers and mobile platforms.
I guess the third alternative is the one for you. You can read more about the whole capturing/publishing/transcoding/converting business in BlookGeek's greate blog post.
A webrtc based peer2peer connection is not the choice for one-to-n streaming. As there is no broadcast so far in webrtc you should consider another technique.
I'm working on a personal project where the iPhone connects to a server-type application running on a Mac. The iPhone send and receives textual/ASCII data via standard sockets. I now need to stream the microphone from the Mac to the iPhone. I've done some work with AudioServices before but wanted to check my thoughts here before getting too deep.
I'm thinking I can:
1. Create an Audio Queue in the standard Cocoa application on the Mac.
2. In my Audio Queue Callback function, rather than writing it to a file, write it to another socket I open for audio streaming.
3. On the iPhone, receive the raw sampled/encoded audio data from the TCP stream and dump it into an Audio Queue Player which outputs to headphone/speaker.
I know this is no small task and I've greatly simplified what I need to do but could it be as easy as that?
Thanks for any help you can provide,
Stateful
This looks broadly sensible, but you'll almost certainly need to do a few more things:
Buffering. On the "recording" end, you probably don't want to block the audio queue if the buffer is full. On the "playback" end, I don't think you can just pass buffers into the queue (IIRC you'll need to buffer it until you get a callback).
Concurrency. I'm pretty sure AQ callbacks happen on their own thread, so you'll need some sort of locking/barriers around your buffer accesses.
Buffer pools, if memory allocation ends up being a big overhead.
Compression. AQ might be able to give you "IMA4" frames (IMA ADPCM 4:1, or so); I'm not sure if it does hardware MP3 decompression on the iPhone.
Packetization, if e.g. you need to interleave voice chat with text chat.
EDIT: Playback sync (or whatever you're supposed to call it). You need to be able to handle different effective audio clock rates, whether it's due to a change in latency or something else. Skype does it by changing playback speed (with pitch-correction).
EDIT: Packet loss. You might be able to get away with using TCP over a short link, but that depends a lot on the quality of your wireless network. UDP is a minor pain to get right (especially if you have to detect an MTU hole).
Depending on your data rates, it might be worthwhile going for the lower-level (BSD) socket API and potentially even using readv()/writev().
If all you want is an "online radio" service and you don't care about the protocol used, it might be easier to use AVPlayer/MPMoviePlayer to play audio from a URL instead. This involves implementing a server which speaks Apple's HTTP streaming protocol; I believe Apple has some sample code that does this.
I'd like to get real-time video from the iPhone to another device (either desktop browser or another iPhone, e.g. point-to-point).
NOTE: It's not one-to-many, just one-to-one at the moment. Audio can be part of stream or via telephone call on iphone.
There are four ways I can think of...
Capture frames on iPhone, send
frames to mediaserver, have
mediaserver publish realtime video
using host webserver.
Capture frames on iPhone, convert to
images, send to httpserver, have
javascript/AJAX in browser reload
images from server as fast as
possible.
Run httpServer on iPhone, Capture 1 second duration movies on
iPhone, create M3U8 files on iPhone, have the other
user connect directly to httpServer on iPhone for
liveStreaming.
Capture 1 second duration movies on
iPhone, create M3U8 files on iPhone,
send to httpServer, have the other
user connected to the httpServer
for liveStreaming. This is a good answer, has anyone gotten it to work?
Is there a better, more efficient option?
What's the fastest way to get data off the iPhone? Is it ASIHTTPRequest?
Thanks, everyone.
Sending raw frames or individual images will never work well enough for you (because of the amount of data and number of frames). Nor can you reasonably serve anything from the phone (WWAN networks have all sorts of firewalls). You'll need to encode the video, and stream it to a server, most likely over a standard streaming format (RTSP, RTMP). There is an H.264 encoder chip on the iPhone >= 3GS. The problem is that it is not stream oriented. That is, it outputs the metadata required to parse the video last. This leaves you with a few options.
Get the raw data and use FFmpeg to encode on the phone (will use a ton of CPU and battery).
Write your own parser for the H.264/AAC output (very hard)
Record and process in chunks (will add latency equal to the length of the chunks, and drop around 1/4 second of video between each chunk as you start and stop the sessions).
"Record and process in chunks (will add latency equal to the length of the chunks, and drop around 1/4 second of video between each chunk as you start and stop the sessions)."
I have just wrote such a code, but it is quite possible to eliminate such a gap by overlapping two AVAssetWriters. Since it uses the hardware encoder, I strongly recommend this approach.
We have similar needs; to be more specific, we want to implement streaming video & audio between an iOS device and a web UI. The goal is to enable high-quality video discussions between participants using these platforms. We did some research on how to implement this:
We decided to use OpenTok and managed to pretty quickly implement a proof-of-concept style video chat between an iPad and a website using the OpenTok getting started guide. There's also a PhoneGap plugin for OpenTok, which is handy for us as we are not doing native iOS.
Liblinphone also seemed to be a potential solution, but we didn't investigate further.
iDoubs also came up, but again, we felt OpenTok was the most promising one for our needs and thus didn't look at iDoubs in more detail.
I'm looking to build a little toy app that is very similar to a voip application. One person would hold one iphone and talk to the other iphone. I don't want to use gamekit because it forces a p2p connection and does not work over 3g. I'm worrying about the server side of this later but just wanted to get started with the iPhone side of it. Which API to record audio in real time and which to play it back?
Look to Audio Queue Services for capture and recording. You'd need to come up with a wire protocol to transmit the audio, but the tools to capture and playback or save on either side of the connection can be built using the queue services.