How to send Microphone and InApp Audio CMSampleBuffer to webRTC in swift? - swift

Question

I am working on-screen broadcast application. I want to send my screen recording on WebRTC server.
override func processSampleBuffer(_ sampleBuffer: CMSampleBuffer, with sampleBufferType: RPSampleBufferType) {
//if source!.isSocketConnected {
switch sampleBufferType {
case RPSampleBufferType.video:
// Handle video sample buffer
source?.processVideoSampleBuffer(sampleBuffer)
break
case RPSampleBufferType.audioApp:
// Handle audio sample buffer for app audio
source?.processInAppAudioSampleBuffer(sampleBuffer)
break
case RPSampleBufferType.audioMic:
// Handle audio sample buffer for mic audio
source?.processAudioSampleBuffer(sampleBuffer)
break
#unknown default:
break
}
}
// VideoBuffer Sending Method
func startCaptureLocalVideo(sampleBuffer: CMSampleBuffer) {
let _pixelBuffer = CMSampleBufferGetImageBuffer(sampleBuffer)
if let pixelBuffer = _pixelBuffer {
let rtcPixelBuffer = RTCCVPixelBuffer(pixelBuffer: pixelBuffer)
let timeStampNs = CMTimeGetSeconds(CMSampleBufferGetPresentationTimeStamp(sampleBuffer)) * 1000000000
let rtcVideoFrame = RTCVideoFrame(buffer: rtcPixelBuffer, rotation: RTCVideoRotation._90, timeStampNs: Int64(timeStampNs))
localVideoSource!.capturer(videoCapturer!, didCapture: rtcVideoFrame)
}
}
I got success to send VIDEO Sample Buffer on WebRTC but I am getting stuck in AUDIO part.
I did not find any way how to send AUDIO buffer to WebRTC.
Thank you so much for your answer.

Answer 1

I found the solution for that, just come to this link and follow the guideline:
https://github.com/pixiv/webrtc/blob/branch-heads/pixiv-m78/README.pixiv.md
WebRTC team is no longer to support native framework, so we need to modify WebRTC source code and re-build it to use inside another app.
Luckily, I found the person who fork source code from WebRTC project and he update the function that pass CMSampleBuffer from Broadcast extension to RTCPeerConnection.