Reference - Why sliced thread affect so much on realtime encoding using ffmpeg x264?
With Zerolatency / sliced-threads enabled, I am observing that the decoding time shoots up! I am encoding on my Windows 10 laptop and streaming to Samsung S4 phone where it is decoded and rendered. If usually, decoding takes 2-3 ms, it shoots up to around 25 ms if I use sliced-threads. It is a real time streaming application so I need low latency and that's why I enabled zerolatency. Can someone help please?
I am using the hardware decoder on the phone.
Related
I was reading about the thin clients and streaming videos. How is it different from downloading a file locally and then playing it on a browser. I mean internally how does streaming work? does streaming take less CPU and memory than playing from a file?
The concept behind streaming is very simple - essentially you can imagine the server sending the video either byte by byte, or in 'chunks' and the client receiving the bytes or chunks into a 'first in first out' queue and then playing them in the order they are received (and at the speed required to play the video properly).
More sophisticated streaming techniques will allow the client switch between different bit rate encodings while downloading the chunks of a file - this means that if the network conditions change during video playback the client can choose a lower or higher bit rate chunk as the next chunk to download appropriately. This is referred to as Adaptive Bit Rate streaming.
Advantages of streaming include fast video start up and seeking, better utilisation of bandwidth and no need to download the whole video if the user decides to seek or stop watching.
The following article gives a very good overview: http://www.jwplayer.com/blog/what-is-video-streaming/
I'm trying to minimize red5-screenshare's bandwith footprint by using jcodec's h264 encoder. The screenvideo codec takes up very little upload bandwidth, but only when used in 128 color mode. When used in full 24-bit RGB color mode it requires at least 5-10 Mbps on a lower resolution screen, which is unacceptable. I'm hoping that by using h264 I'd at least halve that upload bandwidth requirement.
To ask an actual question, would it be too hard to integrate jcodec into red5's screenshare, without having to rewrite the whole encoding and packaging process?
Keep in mind that I'd like to decode this video on the client side by using Adobe Flash Player.
Red5-screenshare: https://code.google.com/p/red5-screenshare/source/checkout
Jcodec: https://github.com/jcodec/jcodec
Also, could someone please give me some hints as to where I could find some info on how to approach this problem? I'm not very familiar with video codecs, encoding, decoding or packaging frames for streaming, so I'd appreciate some learning resources on that.
That would be all, thank you and have a good day!
I am the IT guy for a small chain of preschools. Each school current has about 30-40 Axis IP webcams each, ranging from the old Axis 205,206 models, up to the Axis M1011. These cams allow parents to log into a website to view their kid's classrooms throughout the day. The IP cams are on a local network, streaming on port 80. Each school has one public ip address, on which is a reverse-proxy HTTP server. We directly serve the MJPEG from the Axis webcams by basically doing a URL rewrite to allow the client viewing app direct access to the cam via the Axis HTTP API.
This has worked fine for years, and even worked on iPhone, up until recently. In fact, it still works on iPhone, but only when connected Wifi. When the iPhone is connected via 3G, the MJPEG stream no longer works. The carrier is AT&T.
I have done quite a bit of research the past few days, and know that I have to move towards HTTP Live Streaming for Apple devices, so I'm just trying to get my head around the various parts of the puzzle I have to solve to make it work.
My mix of Axis cams may be a problem. The Axis 205 and 206 are MJPEG only, but the M1011 can serve h.264, but apparently only via RTSP. I'm willing to swap out the older Axis cams for the M1011 if that is required or helpful to the overall solution.
From my understanding so far, the first thing I have to sort out is getting my 30 to 40 Axis M1011 h.264 streams (re)encoded / chunked as MPEG-2 (.m3u8 and .ts files).
Question #1
Is it possible and/or feasible to setup one computer, with a "batch" file of VLC command lines, to start and encode 30 to 40 input streams, coming from Axis webcams, into MPEG-2, ready to serve up to iPhone? I'm wondering about the stability, hardware requirements, etc. to handle that many webcams. This same computer would have a HTTP server (probably IIS) and a public IP address, so no need to get the MPEG-2 files any farther than the local computer.
Question #2
For the older Axis models (205, 206) that only serve MJPEG, is is possible to use VLC to encode those to MPEG-2 as well? Am I correct in assuming that (re)encoding or re-packaging .h264 into MPEG-2 is a lot less "work" than encoding MJPEG into MPEG-2, or is it about the same amount of CPU, etc? I have installed the latest version of Videolan and it was easy enough to connect to the Axis M1011 .h264 stream via it's RTSP URL, so I know that much works.
Question #3
Can anyone share any specific VLC command lines or configuration, for either side of my given configuration: Axis M1011 .h264 and/or Axis MJPEG as inputs, and for the output, MPEG-2 (.m3u8 and .ts files), "chunked" into the size as required by Apple iPhone, especially when connected 3G.
Question #4
Finally, if anyone has another suggested approach (different encoder, Media Server that would work better, etc. ) I would like to hear those suggestions as well.
If you have enough network capability to recieve the input stream and enough CPU power and RAM to encode it is possible. There is no inherent limitation otherwise on using a single machine for multiple instances of vlc or ffmpeg.
If you have a camera feeding in 512 kbps input for 30 streams you need 30x512 = 15 Mbps network capability which should be quite fine for a modern machine.
The difficult part is getting 30 transcodes for iPhone output. The output is a MPEG2 TS container but internally it has to have mpeg4 or h.264 encoded content. H.264 encoding is expensive but requires lesser outgoing bandwidth for same quality at MPEG4 [ballpark ~30% less]
Assuming it is ok for you to view a 320x240 which for decent quality with mpeg4 should be decent quality at 256 kbps output.
For h.264 you could go to 192 kbps [baseline for 3GS and below]
Trouble now is do you have the horsepower to do the transcoding. Here is how you check
Take one video file representative of your input. Check the input frame rate. Transcode it to the output you need. Check the fps you are getting from ffmpeg for encoding. Notice the multiple of input rate it is at. Remove 5-10% for handling multiple simultaneous streams. that is the maximum you are getting on your machine.
ffmpeg can take input from a http or rtsp source and create an output which is segmented. See this for understanding how to use ffmpeg to create segments. You still need to write a small program to generate the m3u8 files I guess.
It is non trivial here but you will enjoy it. It can be done.
EDIT: Another thing is you only need transcoding to happen when someone is viewing it. So at a given time if someone is not viewing 10 cameras, you dont have to do those transcodings. So perhaps you should first find out statistically how many cameras are viewed at a given time. Look at your history. Then you need a less powerful machine.
EDIT: If you can handle some programming instead of command lines look at: this
ffmpeg might be a possible alternative for you. It's command line based, cross platform and you'll be able to control quality, formats as well as stream. The encoding/re-encoding quality basically controls the processing speed of your application, taken the throughput neccessary to grab the data out of the loop.
I'm currently developing an app which plays an audio file (mp3, but can change to WAV to reduce decoding time), and records audio at the same time.
For synchronization purposes, I want to estimate the exact time when audio started playing.
Using AudioQueue to control each buffer, I can estimate the time when the first buffer was drained. My questions are:
What is the hardware delay between AudioQueue buffers being drained and them actually being played?
Is there a lower level API (specifically, AudioUnit), that has better performance (in hardware latency measures)?
Is it possible to place an upper limit on hardware latency using AudioQueue, w or w/o decoding the buffer? 5ms seems something that I can work with, more that that will require a different approach.
Thanks!
The Audio Queue API runs on top of Audio Units, so the RemoteIO Audio Unit using raw uncompressed audio will allow a lower and more deterministic latency. The minimum RemoteIO buffer duration that can be set on some iOS devices (using the Audio Session API) is about 6 to 24 milliseconds, depending on application state. That may set a lower limit on both play and record latency, depending on what events you are using for your latency measurement points.
Decoding compressed audio can add around an order of magnitude or two more latency from decode start.
Developed an internet radio streamer using jPlayer which utilizes the html5 audio tags with jQuery and has a flash fall back for unsupported browsers. Upon testing the player on the iPhone (iOS 5.0.1), we ran into a very peculiar issue.
When the iPhone is connected to WiFi, it streams perfectly using the HE-AAC V2 stream # 64kbps 44.1kHz (the preferred codec for apple products). However, when the iPhone is connected to the 3G mobile network, it "stutters" or stops streaming for 1-2 secs every 1-2 minutes (does not stop streaming completely). The troubling thing is when the iPhone is forced to use a separate MP3 stream at the same bit rate, it does not have this issue and works very well on 3G.
UPDATE 5
We recently acquired a 3G/4G Sprint mobile hotspot device and tested this issue with the device. When the iPhone is connected to the mobile hotspot, it shows as being connected to a wifi device and the issue does not render even tho the actual connection is via 3G/4G. This might point back to the issue being with the iPhone not handling HE-AAC via HTTP Live Streaming and when directly connected to the mobile network.
UPDATE 4
Updated the iPhone to iOS 5.1 yet the issue persists.
UPDATE 3
Read here on SO various issues of script not rendering correctly when connected to mobile networks. The finger seems to point to the mobile network carriers that may be inserting Proxy to serve webpages, e.g. for downsizing images. Also it might inject some JavaScript pages. The test page can be found HERE Note: this page is using HE-AAC so it will only work on iPhone...
UPDATE
According to Apple's HTTP Live Streaming doc for iOS devices, that "Audio-only content can be either MPEG-2 transport or MPEG elementary audio streams, either in AAC format with ADTS headers or in MP3 format." Our music server is using OddcastV3 encoder to send out three streams (MP3, HE-AAC V2, and Oggvorbis) to the icecastV2 server. Not sure if the encoder is inserting ADTS headers for the HE-AAC V2 stream. Is there a way to check for this?
Comming from a radio planning point of view - here are my two cents:
What you are describing sounds like bandwidth shaping - which is both a common and often neccesary design of radio networks (like 3G networks). In most 3G operators I worked at you would typically optimize your network to give high-speed burst (think downloading an image, sending one email or fetching one HTML page) - over "long-running" high bandwidth services.
This is due to the simple fact that this is what most users want/need.
This shaping can on a typical 3GPP (GSM 3G) network result in that you will first get a RAB (radio access bearer) supporting 384kbit and is then downgraded as long as your your device accepts it.
This means that typicall you will get switched from 384 -> 256 -> 128, then 64kbit where maybe your device starts recieving data to slowly, then the network upgrades it and again downgrade it after a while.
So why is not then the MP3 file stuttering? my guess is that the total kbit rate might differ - so you are fine in the 64kbit RAB. This is a common phenomena.
We have managed to get the exact same thing working. 64kbit AAC-v2 on mobile devices. We are streaming files and not a steady stream, I think Magnus is right when he explains how the network prioritized traffic to bursts, in our case that means we have large parts of the file right away and the player can continue to play until he next burst comes in. In your case that means the stream pauses until the next burst comes.
Either if you can switch to larger chunks in your streaming (larger buffer) or stream whole files instead?
We had a very strange phenomenon with iOS, we had to rename all files from .m4a to .aac to be able to get them streaming on iOS. If we didn't rename them iOS wouldnt play them.
Good luck.