I'm using WebRTC (win c++ native client) to broadcast real-time video to peers (hosted in Chrome).
Goal: send metadata along each video frame (metadata changes at frame level).
Would it be possible to send the metadata within a text track to be consumed by a javascript at the peer side?
If not, is there an alternative way of synchronizing WebRTC real-time video with metadata?e.g., using WebRTC DataChannel / WebSockets?
I guess you need this features
https://webrtc.googlesource.com/src/+/77c8e65b88c9d2d95442b66ada504e0f1c553d66
Update multiplex encoder to support having augmenting data attached to the video
No.
At the moment, WebRTC implementation (or specification) comes with no synchronization or the ability to synchronize data with video on a frame-by-frame basis. This is something being looked at for future WebRTC specification.
There are vendors who offer such a capability in their SDKs, but this is limited to their native SDKs and doesn't work in their browser based JS SDKs.
Related
I'm a bit lost looking through all the various Agora.io modules (and not sure what it means that only some of them have Unity-specific downloads).
I want to make a Unity app where two remote phones exchange data as follows:
Streaming voice in both directions
Streaming video in one direction (recorded from device camera)
Streaming a small amount of continuously-changing custom data in the other direction (specifically, a position + orientation in a virtual world; probably encoded as 7 floats)
The custom data needs to have low latency but does not need reliability (it's fine if some updates get lost; app only cares about the most recent update). Updates basically every frame.
Ideally I want to support both Android and iOS.
I started looking at Agora video (successfully built a test project) and it seems like it will cover the voice and video, but I'm struggling to find a good way to send the custom data (position + orientation). It's probably theoretically possible to encode it as a custom video feed but that sounds complex and inefficient. Is there some out-of-band signalling mechanism I could use to send some extra data alongside/instead of a video?
Agora real-time messaging sounds like it would probably work for this, but I can't seem to find any info about integrating it with Unity (either on Agora's web site or in a general web search). Can I roll this in somehow?
Agora interactive gaming could maybe also be relevant? The overview doesn't seem real clear about how it's different from regular Agora video. I suspect it's overkill but that might be fine if there isn't a large performance cost.
Could anyone point me in the right direction?
I would also consider alternatives to Agora if there's a better plugin for implementing this feature set in Unity.
Agora's Video SDK for Unity supports exporting projects to Android, iOS, MacOS, and Windows (non-UWP).
Regarding your data streaming needs, Agora's RTM SDK is in the process of being ported to work within Unity. At the moment the best way to send data using the Agora SDK is to use CreateDataStream to leverage Agora's ability to open a data stream that is sent along with the frames. Data stream messages are limited to 1kb per frame and 30kb/s so I would be cautious about running it on every frame if you are using a frame-rate above 30fps.
I am making a video chatting web application using C# socket programming to transfer data. I want to use the Web Audio API to capture audio and video in my view page, but I dont know how to transfer the audio using sockets (which are defined in controller class.) Can the API be used for socket programming if I can capture the raw bits from the API?
(I've also tried using WEB RTC, but I am unable to create multiple peer connections. As my application involves multiple peers, I prefer normal socket programming.)
If you mean, can you just get access to the raw audio/video bits from getUserMedia - yes, you can. (For audio, check out any of the input demos on webaudiodemos.appspot.com - particularly AudioRecorder shows how to get the bits from a ScriptProcessor node.) But I would caution that streaming audio and video over the net is not a trivial task. You can't really just push the bits over the wire with no thought to buffering (or adaptive capabilities, unless you can guarantee high-speed local network only).
I'm looking for a video streaming soluiton which has the ability to upload the video files to the server and deliver to multiple receivers on-demand across the hardware and software platforms (Desktop, Tablet, Mobile, Windows, Android, iOS, etc.). The solution should also support streaming live videos.
Can HTML-5 used as client for the above requirements? IF so, what should be the server side streaming solution? Any feedback and alternatives will be very helpful.
Appreciate it.
You may look at MediaMosa, it is a backend that handles video management. You may create your own application on the front-end.
I'm looking for a solution to provide streaming video to a variety of clients. I have iPhone clients as well as Flash-based clients. I'd like to not have to provide two separate mechanisms for delivering streaming content. Apple has decreed that HTTP Live Streaming is the way to provide streaming video to the iPhone (though does carve out an exception for small progressive downloads).
My question: Are there examples of Flash implementations consuming HTTP Live Streaming content? What challenges might be faced if I were to try and implement such a player? Are there other technologies I should consider?
Thanks!
Not yet. Maybe never. But...
What you could do is stream from a Wowza Media Server, which will allow you to publish one stream that can be consumed by various clients, including both Apple client devices and Flash browser clients.
Hello all i use windows encoder to stream video online and have a server that i use to broadcast this stream. i am trying to make an app that streams video to the iPhone/iPad using a unique link. i have seen apps out that stream their own DVR cameras so there must be a type of converter or encoder to use. any suggestions?
The short answer is no, not at this time. The iPhone/iPad/iPod Touch work natively with the Apple HTTP Adaptive segmented streaming protocols. MMS (Windows Media) streams are not compatible with "i" devices and will not play. You will need to look into encoding your video with this other format. Check out the Apple specs for a full description of the protocol. Future versions of Windows Media Services (4.0) are claiming that they will support the Apple protocols but this is only a preview/beta at this time and may not truly support the Apple specs.
If your trying to do on-demand iPhone video, you can utilize a service such as Encoding.com to pre-encode your files in the adaptive segmented format for your users to view. For live encoding, Telestream has a product called Wirecast which can encode in a h.264 Apple approved baseline format which can be sent to a service such as Akamai, Multicast Media, or Wowza Server for distribution to your clients.