Concatenating AVAssets seamlessly - swift

Question

I've got some simple AVFoundation code to concatenate a bunch of four-second-long mp4 files together that looks like this:
func
compose(parts inParts: [Part], progress inProgress: (CMTime) -> ())
-> AVAsset?
{
guard
let composition = self.composition,
let videoTrack = composition.addMutableTrack(withMediaType: .video, preferredTrackID: kCMPersistentTrackID_Invalid),
let audioTrack = composition.addMutableTrack(withMediaType: .audio, preferredTrackID: kCMPersistentTrackID_Invalid)
else
{
debugLog("Unable to create tracks for composition")
return nil
}
do
{
var time = CMTime.zero
for p in inParts
{
let asset = AVURLAsset(url: p.path.url)
if let track = asset.tracks(withMediaType: .video).first
{
try videoTrack.insertTimeRange(CMTimeRange(start: .zero, duration: asset.duration), of: track, at: time)
}
if let track = asset.tracks(withMediaType: .audio).first
{
try audioTrack.insertTimeRange(CMTimeRange(start: .zero, duration: asset.duration), of: track, at: time)
}
time = CMTimeAdd(time, asset.duration)
inProgress(time)
}
}
catch (let e)
{
debugLog("Error adding clips: \(e)")
return nil
}
return composition
}
Unfortunately, every four seconds you can hear the audio cut out for a moment, indicating to me that this isn't an entirely seamless concatenation. Is there anything I can do to improve this?
Solution
Thanks to NoHalfBits’s excellent answer below, I’ve updated the above loop with the following, and it works very well:
for p in inParts
{
let asset = AVURLAsset(url: p.path.url)
// It’s possible (and turns out, it’s often the case with UniFi NVR recordings)
// for the audio and video tracks to be of slightly different start time
// and duration. Find the intersection of the two tracks’ time ranges and
// use that range when inserting both tracks into the composition…
// Calculate the common time range between the video and audio tracks…
let sourceVideo = asset.tracks(withMediaType: .video).first
let sourceAudio = asset.tracks(withMediaType: .audio).first
var commonTimeRange = CMTimeRange.zero
if sourceVideo != nil && sourceAudio != nil
{
commonTimeRange = CMTimeRangeGetIntersection(sourceVideo!.timeRange, otherRange: sourceAudio!.timeRange)
}
else if sourceVideo != nil
{
commonTimeRange = sourceVideo!.timeRange
}
else if sourceAudio != nil
{
commonTimeRange = sourceAudio!.timeRange
}
else
{
// There’s neither video nor audio tracks, bail…
continue
}
debugLog("Asset duration: \(asset.duration.seconds), common time range duration: \(commonTimeRange.duration.seconds)")
// Insert the video and audio tracks…
if sourceVideo != nil
{
try videoTrack.insertTimeRange(commonTimeRange, of: sourceVideo!, at: time)
}
if sourceAudio != nil
{
try audioTrack.insertTimeRange(commonTimeRange, of: sourceAudio!, at: time)
}
time = time + commonTimeRange.duration
inProgress(time)
}

Answer 1

In a mp4 container, every track can have its own start time and duration. Especially in recorded material it is not uncommon to have audio and video tracks with slightly different time ranges (insert some CMTimeRangeShow(track.timeRange) near the insertTimeRange to have a look at this).
To overcome this, instead of blindly inserting from CMTime.zero and the duration of the whole asset (the max endtime of all tracks):
get the timeRange of the sources audio and video track
calculate the common time range from these (CMTimeRangeGetIntersection does this for you)
use the common time range when inserting the segments from the source tracks to the destination tracks
increment your time by the duration of the common time range