First a little context about the application...
- There's a lot of heavy UI operation's involving video players (mostly scrolling)
- The videos are dynamic and change based on our current page .
- so the video's have to be dynamic and keep changing and also the UI needs to be responsive
I was initially using a MPMoviePlayerController but then due to certain requirements i had to fall back on the AVPlayer
I made my own wrapper for the AVPlayer .
To change the content in the videoPlayer this is what the method looks like in the AVPlayer-wrapper Class
/**We need to change the whole playerItem each time we wish to change a video url */
-(void)initializePlayerWithUrl:(NSURL *)url
{
AVPlayerItem *tempItem = [AVPlayerItem playerItemWithURL:url];
[tempItem addObserver:self forKeyPath:#"status"
options:NSKeyValueObservingOptionInitial | NSKeyValueObservingOptionNew
context:nil];
[tempItem addObserver:self forKeyPath:#"playbackBufferEmpty"
options:NSKeyValueObservingOptionInitial | NSKeyValueObservingOptionNew
context:nil];
//Not sure if this should be stopped or paused under the ideal circumstances
//These will be changed to custom enums later
[self setPlaybackState:MPMoviePlaybackStateStopped];
[self setLoadState:MPMovieLoadStateUnknown];
[self.videoPlayer replaceCurrentItemWithPlayerItem:tempItem];
//This is required only if we wish to pause the video immediately as we change the url
//[self.videoPlayer pause];
}
Now ofcourse everything was working fine ......except ..
[self.videoPlayer replaceCurrentItemWithPlayerItem:tempItem];
Seems to be blocking the UI for a fraction of a second and during scrolling these is making the UI really unresponsive and ugly also this operation cannot be performed in the background
Is there any fix or workaround for this .. ?
The solution I found was to ensure that the underlying AVAsset is ready to return basic info, such as its duration, before feeding it to the AVPlayer. AVAsset has a method loadValuesAsynchronouslyForKeys: which is handy for this:
AVAsset *asset = [AVAsset assetWithURL:self.mediaURL];
[asset loadValuesAsynchronouslyForKeys:#[#"duration"] completionHandler:^{
AVPlayerItem *newItem = [[AVPlayerItem alloc] initWithAsset:asset];
[self.avPlayer replaceCurrentItemWithPlayerItem:newItem];
}];
In my case the URL is a network resource, and replaceCurrentItemWithPlayerItem: will actually block for several seconds waiting for this information to download otherwise.
We had the same problem when building Ultravisual. I can't exactly remember how we solved it, but IIRC it involved doing as much of the item setup as possible on a background thread, and waiting until the new item reports that it is 'ready to play' before calling replaceCurrentItemWithPlayerItem.
Sadly this involves a voodoo dance with somewhat inconsistent asynchronous KVO, which is no fun.
Related
I am creating an application which displays a certain video based on an external event, which may require the playing video to change quickly - once per second or more. However, there must not be a gap or lag between the videos.
What is the best way to do this? There are only four videos, each about two megabytes.
I was considering creating four MPMoviePlayerControllers, and have their views added to the main view but hidden, and switching by pausing and hiding the current video, then unhiding and playing the next video. Is there a more elegant solution?
Edit
Here's some more information for my exact sitaution:
The different video frames share mostly common pixels- so it's OK for a frame to stick during switch, but NOT okay for black frames to appear.
Each video is only about ten seconds long, and there are only four videos. The general state transitions are 1<->2<->3<->4->1.
The video playback should compatible with simultaneous AVAudioRecorder recording. As far as I can tell, MPMoviePlayerController is not.
You'll need to set up and prebuffer all the video streams to avoid hiccups, so I don't think your multiple MPMoviePlayerController solution is too far off the mark.
However, that specific solution is potentially problematic because each movie player has its own UI. The UIs do not synchronize with each other, so one might be showing the control bar, another not; one might be in full screen mode, etc. Switching among them will cause visual discontinuities.
Since it sounds like your video switching is not necessarily user-initiated, I'm guessing you don't care too much about the standard video player UI.
I would suggest dropping down to the underlying layer, AV Foundation. In theory, you can just create an AVPlayerItem for each video. This is a stream-management object with no UI overhead, so it's perfect for what you're doing. You could then -- again, in theory -- create one AVPlayer and one AVPlayerLayer to handle the display. When you wanted to switch from one AVPlayerItem stream to another, you could call the AVPlayer's replaceCurrentItemWithPlayerItem: message to swap out the data stream.
I made a little test project (GitHub) to verify this, and unfortunately the straightforward solution isn't quite perfect. There is no video flow glitching, but in the transition from AVPlayer to AVPlayer, the presentation layer seems to briefly flash the previous movie's last-seen frame at the size appropriate to the next movie. It seems to help to allocate separate AVPlayer objects for each movie and switch among them to a constant player layer. There still seems to be an instantaneous flash of background, but at least it's a subtle defect. Here's the gist of the code:
#interface ViewController : UIViewController
{
NSMutableArray *players;
AVPlayerLayer *playerLayer;
}
#property (nonatomic) IBOutlet UIView *videoView;
- (IBAction) selectVideo:(id)sender;
#end
#implementation ViewController
#synthesize videoView;
- (void)viewDidLoad
{
[super viewDidLoad];
NSArray *videoTitles = [NSArray arrayWithObjects:#"Ultimate Dog Tease",
#"Backin Up", #"Herman Cain", nil];
players = [NSMutableArray array];
for (NSString *title in videoTitles) {
AVPlayerItem *player = [AVPlayer playerWithURL:[[NSBundle mainBundle]
URLForResource:title withExtension:#"mp4"]];
[player addObserver:self forKeyPath:#"status" options:0 context:nil];
[players addObject:player];
}
playerLayer = [AVPlayerLayer playerLayerWithPlayer:[players objectAtIndex:0]];
playerLayer.frame = self.videoView.layer.bounds;
playerLayer.videoGravity = AVLayerVideoGravityResizeAspect;
[self.videoView.layer addSublayer:playerLayer];
}
- (void) observeValueForKeyPath:(NSString *)keyPath ofObject:(id)object
change:(NSDictionary *)change context:(void *)context
{
[object removeObserver:self forKeyPath:#"status"];
for (AVPlayer *player in players) {
if (player.status != AVPlayerStatusReadyToPlay) {
return;
}
}
// All videos are ready to go
[self playItemAtIndex:0];
}
- (void) playItemAtIndex:(NSUInteger)idx
{
AVPlayer *newPlayer = [players objectAtIndex:idx];
if (newPlayer != playerLayer.player) {
[playerLayer.player pause];
playerLayer.player = newPlayer;
}
[newPlayer play];
}
- (IBAction) selectVideo:(id)sender
{
[self playItemAtIndex:((UILabel *)sender).tag];
}
#end
Half the code is there just to observe the state of the players and make sure that playback doesn't start until all videos have been buffered.
Allocating three separate AVPlayerLayers (in addition to three AVPlayers) prevents any sort of flash. Unfortunately, an AVPlayer connected to an undisplayed AVPlayerLayer seems to assume that it doesn't need to maintain a video buffer. Every switch among layers then produces a transient video stutter. So that's no good.
A couple of things to note when using AV Foundation are:
1) The AVPlayer object doesn't have built-in support for looped playback. You'll have to observe for the end of the current video and manually seek to time zero.
2) There's no UI at all other than the video frame, but again, I'm guessing that this might actually be an advantage for you.
The MPMoviePlayerController is a singleton. Four instances will share the same pointer, the same view, etc. With the native player, I think you have only two alternatives: one is to change the contentURL property when you want a transition. If the latency this way unacceptable, the other alternative is to produce a longer video with the shorter clips concatenated. You can create very quick jumps within the single, longer clip by setting currentPlaybackTime.
I'm running into a strange issue, I hope someone can help.
In my iOS app I create a video with a custom soundtrack using MutableComposition by combining a video from the user's photo library and an audio file from the app bundle. I then use an AVPlayer and AVPlayerItem to play the video back to the user using a custom video player I made.
Each time a new composition is created, the assets, the player and the composition are cleared, released and it basically starts from a clean, init state.
All works fine, until after exactly 4 successful videos created this way every other attempt to create the player fails with error Cannot Decode. It does not matter if its the same video I'm recreating, has no relation to the size/length of the video or the audio file it simply always fails exactly on the fifth attempt, like clockwork. Once it fails, it will then always fail!
This is weird, because it just decoded the same video four times with no problem, so all of a sudden it fails? So, if anyone has a clue, please let me know.
Ok everyone, I have the answer to this straight from Apple. I used one of my developer TSI lifelines to ask the question, and I'll summarize the response.
There is a limit on the number of concurrent video players that AVFoundation will allow. It is due to the limitations of iOS hardware. The limit for current devices is 4 players. If you create a 5th player, you will get the "cannot decode" error. It is not a limit on the number of instances of AVPlayer, or AVPlayerItem. Rather,it is the association of AVPlayerItem with an AVPlayer which creates a "render pipeline", and you are limited to 4 of these. For example, this causes a new render pipeline:
AVPlayer *player = [AVPlayer playerWithPlayerItem:somePlayerItem];
// assuming the AVPlayerItem is ready to go with an AVAsset that has been loaded
I was also warned that you cannot assume that you will have 4 pipelines available to you. Another App may be using one or more. Indeed, I have seen this happen on an iPad, but it was not clear which app was using a pipeline.
So, there you go, it was totally undocumented, but that is the story.
I ran into the same error message after creating 4 AVPlayer instances, the fix in my case wasn't exactly the same though. Perhaps this will help anyone else who comes across this problem.
What I eventually found is that the AVPlayers were not being released when I had thought they were. In my case I was pushing my AVPlayer View Controller onto a Navigation Controller. Even though I was only creating one AVPlayer instance at a time, when the View Controllers are popped off a nav controller they were not being released immediately. It was then very easy for me to reach 4 AVPlayer instances before the old View Controllers were cleaned up.
It wasn't until I made sure that the previous players were released that this problem went away. To be complete I released the AVPlayerItem, AVPlayer and set the player on the AVPlayerLayer to nil before releasing.
I have to wonder if there is some limit on AVPlayer instances, unintentional or not. A related bit of info from the docs:
https://developer.apple.com/library/ios/#documentation/AudioVideo/Conceptual/AVFoundationPG/Articles/02_Playback.html
"Multiple player layers: You can create arbitrarily many AVPlayerLayer objects from a single AVPlayer instance, but only the most-recently-created such layer will display any video content on-screen."
This one was absolutely killing me until I figured it out, picking up clues from this thread and a few others. The biggest single problem in my code was that I was instantiating my video player controller every time I wanted to play a video. Now, it gets instantiated once in the primary controller (in this case, my DetailViewContoller):
#interface DetailViewController () {
VideoPlayerViewController *videoPlayerViewController;
}
- (void) viewDidLoad
{
[super viewDidLoad];
videoPlayerViewController = [[VideoPlayerViewController alloc] initWithNibName: nil bundle: nil];
}
When I want to show a video, I call my DetailViewController's startVideoPlayback method:
- (void) startVideoPlayback: (NSString *)videoUID
{
videoPlayerViewController.videoUID = videoUID;
[self presentModalViewController: videoPlayerViewController animated: YES];
}
(NOTE: I'm passing it 'videoUID' -- a unique identified that was used to create the video in another part of the app.)
In the VideoPlayerViewController (which is largely cribbed from Apple's AVPlayerDemo sample), the one-time screen setup (initializing the AVPlayer, setting up the toolbar, etc.) is done in viewDidLoad -- which now only get's called once, and all per-video setup gets done within viewWillAppear, which then calls prepareToPlay:
- (void) prepareToPlay
{
[self initScrubberTimer];
[self syncPlayPauseButtons];
[self syncScrubber];
NSString *documentsDirectory = [NSSearchPathForDirectoriesInDomains(NSDocumentDirectory, NSUserDomainMask, YES) objectAtIndex:0];
//*** Retrieve and play video at associated with this videoUID
NSString *destinationPath = [documentsDirectory stringByAppendingFormat: #"/%#.mov", videoUID];
if ([self fileExists: destinationPath]) {
//*** Show the activity indicator spinny thing
[pleaseWait startAnimating];
[self setURL: [NSURL fileURLWithPath: destinationPath]];
//*** Get things going with the first video in this session
if (isFirst) {
isFirst = NO;
//*** Subseqeunt videos replace the first one
} else {
[self.mPlayer replaceCurrentItemWithPlayerItem: [AVPlayerItem playerItemWithURL: [NSURL fileURLWithPath: destinationPath]]];
}
}
}
OK, I figured out a solution, I hope this is helpful to anyone who may stumble on something similar to this problem.
The solution in my case was to initialize the asset for the AVPlayer and the AVPlayerItem on the main thread and make sure I don't create the actual AVPlayerLayer before the playerItem and the player objects return with status "ReadyToPlay".
This proved to be tricky to isolate and I still don't know why it worked the first 4 times and then failed consistently on the 5th time.
Till, I couldn't really include the code, it wasn't a matter of one line or even a few functions. It was a complex problem that I couldn't isolate to begin with. Thanks for the comments though.
It seems like that issue can be caused by any decoding tasks, not only actual players.
I randomly had this problem when I implemented a background task to extract frames from currently playing videos with generateCGImagesAsynchronously
I need to display 4 videos on screen and a race condition would sometime cause the frame extraction to start before the video started playing and I would wait for isReadyForDisplay forever.
Not sure what a good recover strategy is if you can't avoid the condition in the first place, I would probably try to replaceCurrentItem
I have a view controller that uses AVPlayer. this view controller can load a modal view controler where the user can record audio using AVAudioRecorder.
this is what happens:
if the user plays the composition in the fist controller with [AVPLayer play] the AVAudioRecorder will not record in the modal view controller. there are no errors but the current time returned by AVAudioRecorder is 0.0;
if the user dismisses the modal dialog and reloads it. AVAudioRecorder works fine.
this can be repeated over and over
AVAudioRecorder does not work the first time it is invoked after a [AVPlayer play] call
I have been fighting with this for days and just have reorganized my code related to both AVPlayer and AVAudioRecorder and it's still acting weird.
Any help or pointer is much appreciated
Thanks in advance
Jean-Pierre
I've been having the same problem too, and this is the solution I found.
When recording, write this line code after AVAudioRecord is initialized:
[[AVAudioSession sharedInstance] setCategory:AVAudioSessionCategoryRecord error:NULL];
Then invoke record method.
How are you setting the AVAudioSession's category, i.e., play or record? Try something like
[[AVAudioSession sharedInstance] setCategory:AVAudioSessionCategoryPlayAndRecord error:nil];
I've been having the same problem. I use AVPlayer to play compositions (previous recordings I've used AVAudioRecord for). However, I found that once I've used AVPlayer I could no longer use AVAudioRecorder. After some searching, I discovered that so long as AVPlayer is instantiated in memory and has been played at least once (which is usually what you do immediately after instantiating it) AVAudioRecorder will not record. However, once AVPlayer is dealloc'd, AVAudioRecorder is then free to record again. It appears that AVPlayer holds on to some kind of connection that AVAudioRecorder needs, and it's greedy...it won't let it go until you pry it from it's cold dead hands.
This is the solution I've found. Some people claim that instantiating AVPlayer takes too much time to keep breaking down and setting back up. However, this is not true. Instantiating AVPlayer is actually quite trivial. So also is instantiating AVPlayerItem. What isn't trivial is loading up AVAsset (or any of it's subclasses). You really only want to do that once. They key is to use this sequence:
Load up AVAsset (for example, if you're loading from a file, use AVURLAsset directly or add it to a AVMutableComposition and use that) and keep a reference to it. Don't let it go until you're done with it. Loading it is what takes all the time.
Once you're ready to play: instantiate AVPlayerItem with your asset, then AVPlayer with the AVPlayerItem and play it. Don't keep a reference to AVPlayerItem, AVPlayer will keep a reference to it and you can't reuse it with another player anyway.
Once it's done playing, immediately destroy AVPlayer...release it, set its var to nil, whatever you need to do. **
Now you can record. AVPlayer doesn't exist, so AVAudioRecorder is free to do its thing.
When you're ready to play again, re-instantiate AVPlayerItem with the asset you've already loaded & AVPlayer. Again, this is trivial. The asset has already been loaded so there shouldn't be a delay.
** Note that destroying AVPlayer may take more than just releasing it and setting its var to nil. Most likely, you've also added a periodic time observer to keep track of the play progress. When you do this, you receive back an opaque object you're supposed to hold on to. If you don't remove this item from the player AND release it/set it to nil, AVPlayer will not dealloc. It appears that Apple creates an intentional retain cycle you must break manually. So before you destroy AVPlayer you need to (example):
[_player removeTimeObserver:_playerObserver];
[_playerObserver release]; //Only if you're not using ARC
_playerObserver = nil;
As a side note, you may also have set up NSNotifications (I use one to determine when the player has completed playing)...don't forget to remove those as well.
If you need use AVPlayer and AVAudioRecorder at the same time do following:
set audio category to "play and record" (as described above or with C-based Audio Session Function)
"record" method must be invoked after invocation of "play" method with some delay. (I set 0.5 sec)
If you don't provide this, playback will start, but recording will not start.
shouldn't it be [AVAudioRecorder record]; instead of play?
I've had the same issue: AVAudioRecorder did not start recording.
My example is a bit different: I have a tabbar based app with several AVAudioPlayers in the different views. For the sake of the example, lets say that the first loaded view has 3 AVAudioPlayers, while the second view has one AVAudioPlayer and one AVAudioRecorder. They are all initiated in the viewDidLoad method.
The recorder was not working in second view. When I hit my record button, the view slightly shaked and the recording did not begin.
Strangely, the simulator worked fine and did not show the same symptoms... (anyone has an idea why?)
After some research and reading this thread I have realized that I have not set the players in the first view to nil, so I guess they were still in the memory preventing the recorder to start. Once I have set them to nil in the viewDidDisappear method, it was working alright again.
here is what helped me:
-(void)viewDidDisappear:(BOOL)animated {
firstPlayer = nil;
secondPlayer = nil;
thirdPlayer = nil;
}
ViewDidUnload did not help because the View, and therefore the variablbes in the class itself were not unloaded when switching views.
Hope this may be useful for others as well.
#Aaron Hayman explanation is accurate but the solution for me was very simple. Just implement the didFinishPlaying method of the player delegate and release the player there.
- (void)audioPlayerDidFinishPlaying:(AVAudioPlayer *)player successfully:(BOOL)flag{
self.playerWord = nil;
}
I'm working with the MPMoviePlayerController for the iOS platform and am not sure why the "Done" and "Loading Movie..." controls are not displaying automatically when loading a new video.
My original implementation of this was to use the UIWebView control to stream the videos, and when the videos are accessed, the "Done" and "Loading Movie..." controls are displayed until the video is loaded and ready to play. I would like to reproduce the same user experience with the MPMoviePlayerController.
Should I get the "Done" and "Loading Movie..." control overlays for free? If not, what do I have to set to get them to show up while the video is loading?
NSURL *videoUrl = [NSURL URLWithString:urlString];
self.moviePlayer = [[[MPMoviePlayerController alloc] init] autorelease];
self.moviePlayer.movieControlMode = MPMovieControlModeDefault;
[self.moviePlayer setContentURL:videoUrl];
[self.moviePlayer prepareToPlay];
[self.view addSubview:self.moviePlayer.view];
[[self.moviePlayer view] setFrame: [self.view bounds]];
[self.moviePlayer setFullscreen:YES animated:YES];
This is answer to your follow-up question, but anyways... You can speed-up things a little bit with this:
prepareToPlay
Prepares the current item for playback. (required)
- (void)prepareToPlay
This method is called automatically when you call the play method. Calling it before you call play gives the receiver a chance to prepare items sooner and may result in decreased latency when starting playback. However, calling this method may also interrupt any active audio sessions.
Available in iOS 3.2 and later.
Declared In MPMediaPlayback.h
I'm using the AVAudioPlayer to play sound FX on a separate thread in my app. Right now my app is fairly simple, I have a UIScrollView for a scrolling background, and a couple of CALayers that represent a player and an enemy in a simple side-scroller type game (think Mario or Kung Fu or whatnot)...I can move the player left and right, with the view scrolling appropriately with no problems. I have an "attack" button that's playing a sound effect...and many times when I play the sound effect, I get a little bit of a hitch, graphically. Is this just par for the course on the iPhone using Quartz? Is there some way to have this perform a little more smoothly without starting to investigate OpenGL?
Use NSData dataWithContentsOfMappedFile instead of dataWithContentsOfFile.
Also AVAudioPlayer handles it's own threading, you don't need to create one. [AVAudioPlayer play] returns after it started playing, not after it is done playing.
And don't forget to release your AVAudioPlayer when you no longer need it, for example if you only want to play a sound onces, implemement the audioPlayerDidFinishPlaying:successfully: delegate and release it in there.
How are you playing the sound? If I was to make a guess it would be that each time the sound is played the OS is actually loading it from disk as well as playing it.
Update:
Ok, you really don't need to be creating a new thread to play a sound. This alone could cause a stutter in a game and is also redundant - The AVAudioPlayer play method is asynchronous - it returns immediately and not when the sound ends.
I had the same problem with slowdown. To solve it, I first play and pause the sound when it first loads at volume 0 - this works better than prepareToPlay which still seems to result in slowdown:
player.volume = 0.0;
[player play];
[player pause];
player.volume = 1.0;
When it comes time to play a sound I just [player play];. Then every update cycle I loop through all my active sounds and catch ones that are about to end and rewind them instead of letting them expire - this avoids the overhead and slowdown of starting up the sound from scratch. The example here stops the sounds 1/15th of a second from the end of the sound:
for ({each player in active sounds})
{
if ((player.duration - player.currentTime) <= (1.0/15.0))
{
[player pause];
player.currentTime = 0.0;
}
}
Now it's ready to go for the next play. Seems to work fine!
Update: Turns out I was still getting some pauses the very first time the sound was played. I think playing it at zero volume wasn't actually doing anything, but I changed it to play at 0.0001 volume and then pause - it's inaudible but seems to do the job of prepping the sound.
Had the same problem at first. Per documentation you need to call "prepareToPlay" every time a sound has finished playing. Adopting the AVAudioPlayerDelegate protocol and adding the following method will do this.
-(void)audioPlayerDidFinishPlaying:(AVAudioPlayer *)player successfully:(BOOL)flag{
[player prepareToPlay];
}
(Using an answer rather than a comment for length and so I can show code):
My sound playing method is as follows:
- (void)playSoundThreaded:(id)soundPlayer
{
if(soundPlayer == nil)
return;
NSAutoreleasePool* pool = [[NSAutoreleasePool alloc] init];
[(AVAudioPlayer*)soundPlayer play];
[pool release];
}
- (void)playSound:(AVAudioPlayer*)player
{
if(player == nil)
return;
[NSThread detachNewThreadSelector:#selector(playSoundThreaded:)
toTarget:self withObject:(id)player];
}
And the sound is loaded up as follows:
fxPlayer = [[AVAudioPlayer alloc] initWithData:
[NSData dataWithContentsOfFile:
[[NSBundle mainBundle] pathForResource:#"voicehiya" ofType:#"caf"]] error:NULL];
We've had some inkling that the sound is being reloaded from disk every time...but have no idea as to why that is or how to prevent it.
I think the thread is over complicating your player. Try this for a go:
/* setup the sound player */
AVAudioPlayer *player1=[[AVAudioPlayer alloc] initWithContentsOfURL:[NSURL fileURLWithPath:audioFilePath] error:NULL];
player1.numberOfLoops = 0;
player1.volume=1.0f;
[player1 prepareToPlay];
and then playSound becomes
[player play];
And don't forget to have the AudioSession category set correctly too.
Why not just use AudioServicesPlaySystemSound? This is well suited to sound effects.
I use it in my game (vConqr) which is also based on a UIScrollView, with numerous other views (so all having their own CALayer), and it doesn't significantly impact performance.
I've been investigating this for a few days now thanks to a project I'm on at work. I added some home-rolled profiling to my code and found that the calls to play sounds were causing frame rate issues in my OpenGL game. I've got a few recommendations for you based on what I've found:
Use an AVAudioPlayer playing an mp3 for your background music.
Use OpenAL for your sound effects.
For sound effects, use the audio converter command line tool from apple to get your sounds to .caf format. This is an uncompressed format that doesn't require the CPU to decompress your sound to memory before sending it to the sound card. It can go directly, saving you time. Instructions on using the tool can be found HERE. (scroll to the very bottom of the page)
Use the lowest bit depths that still sound good. You don't really need 128 kbps - you can probably get away with 20 for a lot of things. Lower bit rate = less processing = faster sound.
I'll let you know if I find anything else that's useful! Good luck!