What I'm doing :
I need to play audio and video files that are not supported by Apple on iPhone/iPad for example mkv/mka files which my contain several audio channels.
I'm using libffmpeg to find audio and video streams in media file.
Video is being decoded with avcodec_decode_video2 and audio with avcodec_decode_audio3
the return values are following for each function are following
avcodec_decode_video2 - returns AVFrame structure which encapsulates information about the video video frame from the pakcage, specifically is has data field which is a pointer to the picture/channel planes.
avcodec_decode_audio3 - returns samples of type int16_t * which I guess is the raw audio data
So basically I've done all this and successfully decoding the media content.
What I have to do :
I've to play the audio and video accordingly using Apples services. The playback I need to perform should support mixing of audio channels while playing video, i.e. let say mkv file contains two audio channel and a video channel. So I would like to know which service will be the appropriate choice for me ? My research showed that AudioQueue service might be useful audio playback, and probably AVFoundation for video.
Please help to find the right technology for my case i.e. video playeback + audio playback with possible audio channel mixing.
You are on the right path. If you are only playing audio (not recording at all) then I would use AudioQueues. It will do the mixing for you. If you are recording then you should use AudioUnits. Take a look at the MixerHost example project from Apple. For video I recommend using OpenGL. Assuming the image buffer is in YUV420 then you can render this with a simple two pass shader setup. I do believe there is an Apple example project showing how to do this. In any case you could render any pixel format using OpenGL and a shader to convert the pixel format to RGBA. Hope this help.
Related
I am trying to do an audio visualisation for a stream. The audio has to play in the background and currently I am playing it with an AVPlayer but I cannot get the metering from it. How can I get the metering and make the visualisation? Any suggestions?
Here you have an example with waveforms: A cocoa audio player component which displays the waveform of the audio file
Here you have a LED bar gauge and another example how can be used: ATTabandHoldAudioRecord
Apple, also have SpeakHere example - the code includes a LevelView .. but this Apple sample code is anything but simple to implement ...
I am working on media recording on IOS. I was able to record with both audio and video using UIImagePickerController. For my specification I want to record only audio. Is it possible with UIImagePickerController? or Do I need to think about other methods?
thanks.
UIImagePickerController cannot be used to record audio. Instead you should take a look at the SpeakHere sample code from Apple.
Also check Audio Queue - Recording to a compressed audio format on how to record as compressed audio format and conserve disk space.
I have to make an iPhone project that can process video data in
realtime. This app has to be able to reconize the color of the
object in the video frame. After I found information relating to
video processing in iOS, I found that I can use AVFoundation
Framework to achieve this task but I don't know which APIs or functions
of AVFoundation that's able to do this video processing task.
Can anyone suggest me which function to use to get image frames or
raw image data out of a video streaming in real-time?
I'd appreciate if you can give me some example code
Thank you very much for helping me...
You can first of all make use of AVAsset class by initiating it with your video file URL.
You can then use an AVAssetReader object for obtaining media data of that asset.
This will help you obtain video frames which you can read using AVAssetReaderVideoCompositionOutput class object. Accessing RGB channel data from these frames is pertinent to CGImage classes and it's methods.
Hope this helps you to get started
I have an iOS app that I want to record some of its visual output into a video. It looks like the way to create a video on iOS is to use AVMutableComposition and feed AVAssets to it via insertTimeRange.
All the documentation and examples that I can find only add video and audio assets to an AVMutableComposition. Is there a way to add image data to it (i.e. add an image for each frame of the video)? I can get this image data as straight RGB, PNG, JPG, UIImage, or whatever is easiest to feed to AV Foundation (if it's even possible).
If it's not possible to feed images into an AVMutableComposition for the video frames, is there another way to generate an .mp4 file from frames in iOS.
To generate movies from frame you can use AVAssetWriter, here is a question that sort of covers that here on SO, question
I am looking at Apple's 'aurioTouch' example for the iPhone and I would like to play an mp3 or wav instead of using the built in mic. I am very new to the audio portion of iPhone programming, but I think I need to modify the SetupRemoteIO(...) function and replace the AudioComponent named 'comp' with a custom AudioComponent that plays a file. Basically I want the app to function exactly the same as the original, but with an audio file as the input instead of the mic.
You just need to convert your audio file to pcm data and then feed that data to the RemoteIO interface during the playback callback.
To read your audio file in, you will want to use ExtAudioFileOpenURL and ExtAudioFileRead. Also make sure to set your audio format with ExtAudioFileSetProperty to convert to your target pcm format (which should be packed, signed integer PCM data).
Playback simply involves responding to the RemoteIO callback (which should be identical to aurioTouch's example) and feeding it the PCM data you loaded up.
The only other tricky part is that loading an entire mp3 file as PCM can take up a ton of memory. You might have to write a loading thread so that you can stay under your memory requirements by only loading your relevant chunk of the mp3.