I have a feature to implement in Flutter. I have to record user video and fetch audio from that for next level verification. Does anyone have any idea about this.
Thanks in advance.
I assume you are trying to extract audio only from Video, you can achive this by using the ffmpeg package.
Import this package in your project add the following lines in your android\build.gradle
ext.flutterFFmpegPackage = 'full'
flutter_ffmpeg provides eight packages that include different sets of external libraries. These packages are named according to the external libraries included in them. From documentation you can see which libraries are enabled in each package. And you can use according to your need, so if you need to change full to specific one you can do this but here i am using full to make your work done.
With below code snippet you can extract audio from video
final FlutterFFmpeg FFmpeg = FlutterFFmpeg();
FFmpeg.execute(
"-i /storage/emulated/0/download/Test_video.mp4 -map 0:a -acodec libmp3lame /storage/emulated/0/download/test3.mp3").then((rc) =>
{
print("FFmpeg process exited with rc $rc.")
}
Print statement should return 0 on success, Also i am using testvideo from my own storage you need to change this according to your storage location.
Related
I'm attempting to stream a H.264 video feed to a web browser. Media Foundation is used for encoding a fragmented MPEG4 stream (MFCreateFMPEG4MediaSink with MFTranscodeContainerType_FMPEG4, MF_LOW_LATENCY and MF_READWRITE_ENABLE_HARDWARE_TRANSFORMS enabled). The stream is then connected to a web server through IMFByteStream.
Streaming of the H.264 video works fine when it's being consumed by a <video src=".."/> tag. However, the resulting latency is ~2sec, which is too much for the application in question. My suspicion is that client-side buffering causes most of the latency. Therefore, I'm experimenting with Media Source Extensions (MSE) for programmatic control over the in-browser streaming. Chrome does, however, fail with the following error when consuming the same MPEG4 stream through MSE:
Failure parsing MP4: TFHD base-data-offset not allowed by MSE. See
https://www.w3.org/TR/mse-byte-stream-format-isobmff/#movie-fragment-relative-addressing
mp4dump of a moof/mdat fragment in the MPEG4 stream. This clearly shows that the TFHD contains an "illegal" base data offset parameter:
[moof] size=8+200
[mfhd] size=12+4
sequence number = 3
[traf] size=8+176
[tfhd] size=12+16, flags=1
track ID = 1
base data offset = 36690
[trun] size=12+136, version=1, flags=f01
sample count = 8
data offset = 0
[mdat] size=8+1624
I'm using Chrome 65.0.3325.181 (Official Build) (32-bit), running on Win10 version 1709 (16299.309).
Is there any way of generating a MSE-compatible H.264/MPEG4 video stream using Media Foundation?
Status Update:
Based on roman-r advise, I managed to fix the problem myself by intercepting the generated MPEG4 stream and perform the following modifications:
Modify Track Fragment Header Box (tfhd):
remove base_data_offset parameter (reduces stream size by 8bytes)
set default-base-is-moof flag
Add missing Track Fragment Decode Time (tfdt) (increases stream size by 20bytes)
set baseMediaDecodeTime parameter
Modify Track fragment Run box (trun):
adjust data_offset parameter
The field descriptions are documented in https://www.iso.org/standard/68960.html (free download).
Switching to MSE-based video streaming reduced the latency from ~2.0 to 0.7 sec. The latency was furthermore reduced to 0-1 frames by calling IMFSinkWriter::NotifyEndOfSegment after each IMFSinkWriter::WriteSample call.
There's a sample implementation available on https://github.com/forderud/AppWebStream
I was getting the same error (Failure parsing MP4: TFHD base-data-offset not allowed by MSE) when trying to play a fmp4 via MSE. The fmp4 had been created from a mp4 using the following ffmpeg comand:
ffmpeg -i myvideo.mp4 -g 52 -vcodec copy -f mp4 -movflags frag_keyframe+empty_moov myfmp4video.mp4
Based on this question I was able to find out that to have the fmp4 working in Chrome I had to add the "default_base_moof" flag. So, after creating the fmp4 with the following command:
ffmpeg -i myvideo.mp4 -g 52 -vcodec copy -f mp4 -movflags frag_keyframe+empty_moov+default_base_moof myfmp4video.mp4
I was able to play successfully the video using Media Source Extensions.
This Mozilla article helped to find out that missing flag:
https://developer.mozilla.org/en-US/docs/Web/API/Media_Source_Extensions_API/Transcoding_assets_for_MSE
The mentioned 0.7 sec latency (in your Status Update) is caused by the Media Foundation's MFTranscodeContainerType_FMPEG4 containterizer which gathers and outputs each roughly 1/3 seconds (from unknown reason) of frames in one MP4 moof/mdat box pair. This means that you need to wait 19 frames before getting any output from MFTranscodeContainerType_FMPEG4 at 60 FPS.
To output single MP4 moof/mdat per each frame, simply lie that MF_MT_FRAME_RATE is 1 FPS (or anything higher than 1/3 sec). To play the video at the correct speed, use Media Source Extensions' <video>.playbackRate or rather update timescale (i.e. multiply by real FPS) of mvhd and mdhd boxes in your MP4 stream interceptor to get the correctly timed MP4 stream.
Doing that, the latency can be squeezed to under 20 ms. This is barely recognizable when you see the output side by side on localhost in chains such as Unity (research) -> NvEnc -> MFTranscodeContainerType_FMPEG4 -> WebSocket -> Chrome Media Source Extensions display.
Note that MFTranscodeContainerType_FMPEG4 still introduces 1 frame delay (1st frame in, no output, 2nd frame in, 1st frame out, ...), hence the 20 ms latency at 60 FPS. The only solution to that seems to be writing own FMPEG4 containerizer. But that is order of magnitude more complex than intercepting of Media Foundation's MP4 streams.
The problem was solved by following roman-r's advise, and modifying the generated MPEG4 stream. See answer above.
Another way to do this is again using the same code #Fredrik mentioned but I write my own IMFByteStream and and I check the chunks written to the IMFByteStream.
FFMpeg writes the atoms almost once at a time. So you can check the atom name and do the mods. It is the same thing. I wish there was an MSE compliant windows sinker.
Is there one that can generate .ts files for HLS?
How to create an Mp4 file from H264 raw data that I am receiving from a live streamer (no predefined duration or moov atom), unfortunately can't use FFMPEG, I have to write my own code using live555. Can somebody help me with Mp4 container and how h264 data has to be pushed into it.? Thank you in advance : )
There are several operations to be made to store H.264 raw data into MP4, among them:
create box structures, in particular the moov box
store the NAL units in a mdatbox, possibly storing non-VCL NAL units in the moovbox
replace start codes with length fields
It also depends on your requirements. If you want to do the conversion on-the-fly, you have to use fragmented mp4. If you can store the H264 and then do the conversion, you may use non-fragmented mp4. In particular using MP4Box:
MP4Box -add file.264 file.mp4
I'm trying to convert this RTSP URL to something else (anything!) using this gst pipeline:
gst-launch rtspsrc location=rtsp://184.72.239.149/vod/mp4:BigBuckBunny_175k.mov ! rtpmp4vdepay ! filesink location=somebytes.bin
This gives the following error:
ERROR: from element /GstPipeline:pipeline0/GstRTSPSrc:rtspsrc0/GstUDPSrc:udpsrc0: Internal data flow error.
Additional debug info:
gstbasesrc.c(2791): gst_base_src_loop (): /GstPipeline:pipeline0/GstRTSPSrc:rtspsrc0/GstUDPSrc:udpsrc0:
streaming task paused, reason not-linked (-1)
So I guess it's something about connecting the rstp source to the depayloader. If I change the pipeline to use rtpmp4gdepay rather than vdepay, it works and produces something, but I'm not sure what the output format is.
Does anyone know what pipeline I should be using to get at the video from this URL? I'm assuming it's mp4/h264/aac, but maybe it's not.
Try this first:
gst-launch-0.10 -v playbin2 uri=rtsp://184.72.239.149/vod/mp4:BigBuckBunny_175k.mov
or
gst-launch-1.0 -v playbin uri=rtsp://184.72.239.149/vod/mp4:BigBuckBunny_175k.mov
Mov file is not directly streamable. So your RTSP source is probably sending you two elementary streams. (I am guessing this is darwin or some other similar server) So you may have to setup two outputs from rtspsrc one for audio and one for video.
rtpmp4vpay is for elementary mpeg4 streams. Is your source file having mpeg4 video codec? If it is h.264 replace it with rtph264depay. you can pass the output to decoder and play it if you want. Feed it to decodebin. To dump it in h.264 you will first have to parse and add nal headers to it ( h264parse parse perhaps? )
rtpmp4gpay is most probably accepted for the audio stream.
I am guessing your file is h.264/aac which is why rtpmp4vdepay wont work and rtpmp4gdepay will. But you are not doing anything about the video when you setup rtpmp4gdepay so you need to do that.
I'm trying to use GDCL MP4 Muxer with my RTSP Source Filter. They work fine together except after stopping the graph, muxer doesn't finilize the file and write the reqiured tables to the end of file via file writer (some parts are written starting from moov but not the time table values). When I try another RTSP source filter (which I don't have its source codes), table values are created with GDCL MP4 Muxer.
But when I try Elecard's MP4 Muxer, it works fine with my RTSP Source Filter. So, there is an incompatibility. I examined GDCL's source codes but couldn't find what it was expecting from me. I already calculate and set timestamp values to samples using SetTime method. But GDCL still doesn't finilaze file. Is it caused by missing information or missing signal when graph stops? What can be the problem, any ideas?
One thing you should be aware of regarding Geraint's MP4 Mux is that it is checking incoming media samples to have both start and stop time. You might be having only .tStart/AM_SAMPLE_TIMEVALID which still makes sense for video, but this would be a problem.
So the samples have to have stop time, or you need to fix this in multiplexer code.
A typical symptom for the problem is that generated files are empty or of zero duration.
I am building an application that needs to read H264 and AC3 streams from a MP4 container and mux them into a single ISMV file. The source MP4 file contains a number of video streams of different bitrates and a number of audio streams of different languages.
When I call IGraphBuilder::AddSourceFilter for my source file, I get a filter that has just two output pins: "Video" and "Audio". How do I choose which particular stream (e.g.: which bitrate of a video stream) to use for "Video" and "Audio"?
Do I have to instantiate multiple source filters to read that file and mux them into ISMV, or am I missing something?
That depends on the demux you are using for MP4. I don't think there is a stock MP4 demux, so you have probably got one as part of a decoder package, and that is acting as both source and demux.
You can try the free open-source MP4 demux at www.gdcl.co.uk/mpeg4. You will need to AddSourceFilter (getting a file source with a single output) and then explicitly connect the source output to the demux input. Then you will have output pins corresponding to all enabled streams that the demux understands, and you can select the ones you want.
G