I am trying to send audio recorded from the iPhone microphone, to an ip camera.
I have an sdk(written in c) in order to communicate with the camera and this is the function i need to pass the data. We are talking of sending realtime audio.
/*
#Name: FosSdk_SendTalkData.
#Description: Send the data of talk.
#param handle: the handle of current connection information.
#param data: The data need to send.
#param len: Len of data.
#return: Please refer to the enum of FOSCMD_RESULT to get more information.
*/
FOSSDK FOSCMD_RESULT FOSAPI FosSdk_SendTalkData(FOSHANDLE handle, char *data, int len);
This is instead the Swift signature( i am currently using swift )
FosSdk_SendTalkData(handle: UInt32, data: UnsafeMutablePointer<Int8>!, len: Int32)
How do i record the audio from iPhone microphone and pass the audio buffer correctly to sendTalkData?
Thanks in advance for any clarification/help.
EDIT
I managed how to get the audio buffer using a AVFoundation and the AVCaptureAudioDataOutputSampleBufferDelegate function, from which i obtain a sample buffer. However, my implementation, does not work. One thing i noticed, is the app crashes when passing length parameter to sendTalkFunction. If for example, i pass Int32(1) as length, the app does not crash but however, it does not have any sense to pass Int32(1).
This snippet of code helped me a lot, but however it s a bit old so i needed to make some edit http://timestocomemobile.com/2014/10/swift-data-to-c-pointer-and-back-again.html
func captureOutput(_ captureOutput: AVCaptureOutput!, didOutputSampleBuffer sampleBuffer: CMSampleBuffer!, from connection: AVCaptureConnection!) {
guard let blockBufferRef = CMSampleBufferGetDataBuffer(sampleBuffer) else {return}
let lengthOfBlock = CMBlockBufferGetDataLength(blockBufferRef)
guard let data = NSMutableData(length: Int(lengthOfBlock)) else {return}
CMBlockBufferCopyDataBytes(blockBufferRef, 0, lengthOfBlock, data.mutableBytes)
// I need to pass the data as UnsafeMutablePointer<Int8> type
let dataUnsafeMutablePointer = data.mutableBytes.assumingMemoryBound(to: Int8.self)
FosSdk_SendTalkData(CameraConfigurationManager.mhandle, dataUnsafeMutablePointer, Int32(lengthOfBlock))
}
The AVCaptureSessions are starting correctly and i can see the sampleBuffers are collected every tot milliseconds. I just need to fill function parameters correctly and make these audio buffers be played on the ip camera
Related
I'd like to write a simple tool in Swift to read battery status of my mouse (Logitech G Pro Wireless). Since it's my first dive into both IOKit and managing HID devices, I am struggling with getting it done.
Here's my current approach:
I used this library to skip for now messing with Obj-C https://github.com/Arti3DPlayer/USBDeviceSwift
On launch of the app I create HID Device with hardcoded ids and start listening for its presence:
struct MyApp: App {
let rfDeviceMonitor = HIDDeviceMonitor([
HIDMonitorData(vendorId: 0x046d, productId: 0xC539)//0xc088)
], reportSize: 64)
var body: some Scene {
Window()
.onAppear {
let rfDeviceDaemon = Thread(target: self.rfDeviceMonitor, selector:#selector(self.rfDeviceMonitor.start), object: nil)
rfDeviceDaemon.start()
}
}
}
Another class is listening for connection and device's data.
func configure() {
NotificationCenter.default.addObserver(self, selector: #selector(self.usbConnected), name: .HIDDeviceConnected, object: nil)
NotificationCenter.default.addObserver(self, selector: #selector(self.hidReadData), name: .HIDDeviceDataReceived, object: nil)
}
#objc func usbConnected(notification: NSNotification) {
guard let nobj = notification.object as? NSDictionary else {
return
}
guard let deviceInfo: HIDDevice = nobj["device"] as? HIDDevice else {
return
}
self.deviceInfo = deviceInfo
write()
}
#objc func hidReadData(notification: Notification) {
let obj = notification.object as! NSDictionary
let data = obj["data"] as! Data
print([UInt8](data))
}
func write() {
let payload: [UInt8] = [
0x10, // short message
0xff, // receiver's index
0x06, // feature index - battery voltage for g pro wireless
0x00,
0x00,
0x00
]
var correctData = Data(payload)
var count = correctData.count
let ret: Int32 = IOHIDDeviceGetReport(
self.deviceInfo.device,
kIOHIDReportTypeFeature,
CFIndex(0x10),
&correctData,
&count
)
print(ret) // 0xe0005000
print([UInt8](correctData)) // [16, 255, 6, 0, 0, 0]
}
The issue is that IOKit always returns a value (0xe0005000) after calling IOHIDDeviceGetReport that is not a success. I have no idea what this means, since kIOReturn header doesn't mention this value at all.
Links that I found useful:
receiver's properties: https://github.com/pwr-Solaar/Solaar/blob/78341f87e969fcdb657d912953f919e7bdd7c491/docs/devices/Lightspeed%20Receiver%20C539.txt
Mouse's properties: https://github.com/pwr-Solaar/Solaar/blob/78341f87e969fcdb657d912953f919e7bdd7c491/docs/devices/G%20Pro%20Wireless%20Gaming%20Mouse%204079.txt
feature's and implementation of reading features via hidpp 2.0 https://github.com/pwr-Solaar/Solaar/blob/eac916b57c78b23a40bcded1a9c89e2cc30e06d4/lib/logitech_receiver/hidpp20.py
Logitech's specification's draft for hidpp 2.0 https://drive.google.com/drive/folders/0BxbRzx7vEV7eWmgwazJ3NUFfQ28?resourcekey=0-dQ-Lx1FORQl0KAdOHQaE1A
The most important informations I got from these sites:
My mouse has only a feature of passing current voltage. It's available on index 6. Also, there's mentioned hex value of 0x1001 next to that feature, but I am not sure what it could mean, maybe it's some Logitech's identifier for this feature.
buffer size has 7 or 20 bytes (this call should be 7B), where:
first means message size (short - 7B - 0x10, long - 20B - 0x11)
second is device index (0xFF is for receiver, yet to find which is meant for device connected via a wire)
Third means feature index (which is 6 in this case)
Fourth is divided on function and software identifier (in this order). The second one people say that is used to recognize responses meant for us from the rest of the output.
rest should be filled by report that I'd like to get.
https://github.com/pwr-Solaar/Solaar/blob/d41c60718876957158f2ef7ce51648cab78c72ad/lib/logitech_receiver/base.py#L437 - here's Python's implementation of sending such a request. This line particularly looks like it sends a request, then it waits for a message back? This is the point where I am struggling the most.
There's a line in documentation that describes bytes meaning, says:
"The device index, feature index, function identifier, and software identifier are always returned unchanged.", so I suppose I should use getReport instead of setReport, but again, my knowledge is very limited here, so I tried setReport as well, but with no luck. Same error has been thrown.
I am working on-screen broadcast application. I want to send my screen recording on WebRTC server.
override func processSampleBuffer(_ sampleBuffer: CMSampleBuffer, with sampleBufferType: RPSampleBufferType) {
//if source!.isSocketConnected {
switch sampleBufferType {
case RPSampleBufferType.video:
// Handle video sample buffer
source?.processVideoSampleBuffer(sampleBuffer)
break
case RPSampleBufferType.audioApp:
// Handle audio sample buffer for app audio
source?.processInAppAudioSampleBuffer(sampleBuffer)
break
case RPSampleBufferType.audioMic:
// Handle audio sample buffer for mic audio
source?.processAudioSampleBuffer(sampleBuffer)
break
#unknown default:
break
}
}
// VideoBuffer Sending Method
func startCaptureLocalVideo(sampleBuffer: CMSampleBuffer) {
let _pixelBuffer = CMSampleBufferGetImageBuffer(sampleBuffer)
if let pixelBuffer = _pixelBuffer {
let rtcPixelBuffer = RTCCVPixelBuffer(pixelBuffer: pixelBuffer)
let timeStampNs = CMTimeGetSeconds(CMSampleBufferGetPresentationTimeStamp(sampleBuffer)) * 1000000000
let rtcVideoFrame = RTCVideoFrame(buffer: rtcPixelBuffer, rotation: RTCVideoRotation._90, timeStampNs: Int64(timeStampNs))
localVideoSource!.capturer(videoCapturer!, didCapture: rtcVideoFrame)
}
}
I got success to send VIDEO Sample Buffer on WebRTC but I am getting stuck in AUDIO part.
I did not find any way how to send AUDIO buffer to WebRTC.
Thank you so much for your answer.
I found the solution for that, just come to this link and follow the guideline:
https://github.com/pixiv/webrtc/blob/branch-heads/pixiv-m78/README.pixiv.md
WebRTC team is no longer to support native framework, so we need to modify WebRTC source code and re-build it to use inside another app.
Luckily, I found the person who fork source code from WebRTC project and he update the function that pass CMSampleBuffer from Broadcast extension to RTCPeerConnection.
I am attempting to override Spotify's SPTCoreAudioController in order to get at the audio buffer so I can do some processing. I was able to subclass it and override the correct function. Just as a baseline, I was attempting to pass through the audioFrames and frameCount back to the audio pipeline by calling super.attempt...() The audio flows as expected however it got sped up after the first few seconds. I was expecting it just to pass through and playback normally. Can anyone explain why this is happening and/or point me to what I need to learn in order to work with the audio frames that are passed to me?
Here is the code:
class CoreAudioController: SPTCoreAudioController {
override func attempt(toDeliverAudioFrames audioFrames: UnsafeRawPointer!, ofCount frameCount: Int, streamDescription audioDescription: AudioStreamBasicDescription) -> Int {
print("attempt to deliver audio frames")
super.attempt(toDeliverAudioFrames: audioFrames, ofCount: frameCount, streamDescription: audioDescription)
return frameCount
}
}
Here is where I pass the custom controller above to the Spotify AudioStreamingController:
func initializaPlayer(authSession:SPTSession){
if self.player == nil {
self.audioController = CoreAudioController()
self.player = SPTAudioStreamingController.sharedInstance()
self.player!.playbackDelegate = self
self.player!.delegate = self
try! self.player?.start(withClientId: auth?.clientID, audioController: self.audioController, allowCaching: false)
//try! player?.start(withClientId: auth?.clientID)
self.player!.login(withAccessToken: authSession.accessToken)
}
}
I have successfully added https://github.com/chrisballinger/Opus-iOS to my project and I am able to call the functions declared in its header.
I want to convert from OPUS to AAC so my first step would be to decode my opus file. However, it keeps throwing an error code.
The file I am using is the 2-second file from https://people.xiph.org/~giles/2012/opus/.
This is my code
let url = Bundle.main.url(forResource: "detodos", withExtension: "opus")
print(url) //prints path correctly
var bytes = [UInt8]()
if let data = NSData(contentsOf: url!) {
var buffer = [UInt8](repeating: 0, count: data.length)
data.getBytes(&buffer, length: data.length)
bytes = buffer
let d = opus_decoder_create(48000, 1, nil)
var sampleBuffer = [opus_int16](repeating: 0, count: data.length)
print(opus_int32(bytes.count)) //6270
let w = opus_decode(d!, bytes, opus_int32(bytes.count), &sampleBuffer, opus_int32(5760), opus_int32(1))
print(sampleBuffer) // [0,0,...] every time
print(w) //-4 every time
//-4 = OPUS_INVALID_PACKET
}
I would've guessed that in this very minimal implementation nothing should go wrong but apparently it does. Printing my bytes object returns tons of different numbers so I know for a fact it doesn't stay at all-0's.
I realized that it might be due to the method expecting pure "audio data" but the file also contains a header etc. How can I strip this off?
The opus library decodes Opus packets. You give it an Opus packet and it decodes it; one packet at a time. You are trying to give it an entire Ogg Opus file, including all of the headers, framing, tags, and other metadata.
The easiest way to decode an entire Ogg Opus file is with the opusfile library. It can even stream the data from a URL if you want, so that you don't have to download it all before you start decoding as you are doing here. Alternatively, instead of opusfile you could use the ogg library to extract packets from the file and then pass each audio packet to the opus library for decoding.
I am using following to get video sample buffer:
- (void) writeSampleBufferStream:(CMSampleBufferRef)sampleBuffer ofType:(NSString *)mediaType
Now my question is that how can I get h.264 encoded NSData from above sampleBuffer. Please suggest.
Update for 2017:
You can do streaming Video and Audio now by using the VideoToolbox API.
Read the documentation here: VTCompressionSession
Original answer (from 2013):
Short: You can't, the sample buffer you receive is uncompressed.
Methods to get hardware accelerated h264 compression:
AVAssetWriter
AVCaptureMovieFileOutput
As you can see both write to a file, writing to a pipe does not work as the encoder updates header information after a frame or GOP has been fully written. So you better don't touch the file while the encoder writes to it as it does randomly rewrite header information. Without this header information the video file will not be playable (it updates the size field, so the first header written says the file is 0 bytes). Directly writing to a memory area is not supported currently. But you can open the encoded video-file and demux the stream to get to the h264 data (after the encoder has closed the file of course)
You can only get raw video images in either BGRA or YUV color formats from AVFoundation. However, when you write those frames to an mp4 via AVAssetWriter, they will be encoded using H264 encoding.
A good example with code on how to do that is RosyWriter
Note that after each AVAssetWriter write, you will know that one complete H264 NAL was written to a mp4. You could write code that reads a complete H264 NAL after each write by AVAssetWriter, which is going to give you access to an H264 encoded frame. It might take a bit to get it right with decent speed, but it is doable( I did it successfully).
By the way, in order to successfully decode these encoded video frames, you will need H264 SPS and PPS information which is located in a different place in the mp4 file. In my case, I actually create couple of test mp4 files, and then manually extracted those out. Since those don't change, unless you change the H264 encoded specs, you can use them in your code.
Check my post to SPS values for H 264 stream in iPhone to see some of the SPS/PPS I used in my code.
Just a final note, in my case I had to stream h264 encoded frames to another endpoint for decoding/viewing; so my code had to do this fast. In my case, it was relatively fast; but eventually I switched to VP8 for encoding/decoding just because it was way faster because everything was done in memory without file reading/writing.
Good luck, and hopefully this info helps.
Use VideoToolbox API. refer: https://developer.apple.com/videos/play/wwdc2014/513/
import Foundation
import AVFoundation
import VideoToolbox
public class LiveStreamSession {
let compressionSession: VTCompressionSession
var index = -1
var lastInputPTS = CMTime.zero
public init?(width: Int32, height: Int32){
var compressionSessionOrNil: VTCompressionSession? = nil
let status = VTCompressionSessionCreate(allocator: kCFAllocatorDefault,
width: width,
height: height,
codecType: kCMVideoCodecType_H264,
encoderSpecification: nil, // let the video toolbox choose a encoder
imageBufferAttributes: nil,
compressedDataAllocator: kCFAllocatorDefault,
outputCallback: nil,
refcon: nil,
compressionSessionOut: &compressionSessionOrNil)
guard status == noErr,
let compressionSession = compressionSessionOrNil else {
return nil
}
VTSessionSetProperty(compressionSession, key: kVTCompressionPropertyKey_RealTime, value: kCFBooleanTrue);
VTCompressionSessionPrepareToEncodeFrames(compressionSession)
self.compressionSession = compressionSession
}
public func pushVideoBuffer(buffer: CMSampleBuffer) {
// image buffer
guard let imageBuffer = CMSampleBufferGetImageBuffer(buffer) else {
assertionFailure()
return
}
// pts
let pts = CMSampleBufferGetPresentationTimeStamp(buffer)
guard CMTIME_IS_VALID(pts) else {
assertionFailure()
return
}
// duration
var duration = CMSampleBufferGetDuration(buffer);
if CMTIME_IS_INVALID(duration) && CMTIME_IS_VALID(self.lastInputPTS) {
duration = CMTimeSubtract(pts, self.lastInputPTS)
}
index += 1
self.lastInputPTS = pts
print("[\(Date())]: pushVideoBuffer \(index)")
let currentIndex = index
VTCompressionSessionEncodeFrame(compressionSession, imageBuffer: imageBuffer, presentationTimeStamp: pts, duration: duration, frameProperties: nil, infoFlagsOut: nil) {[weak self] status, encodeInfoFlags, sampleBuffer in
print("[\(Date())]: compressed \(currentIndex)")
if let sampleBuffer = sampleBuffer {
self?.didEncodeFrameBuffer(buffer: sampleBuffer, id: currentIndex)
}
}
}
deinit {
VTCompressionSessionInvalidate(compressionSession)
}
private func didEncodeFrameBuffer(buffer: CMSampleBuffer, id: Int) {
guard let attachments = CMSampleBufferGetSampleAttachmentsArray(buffer, createIfNecessary: true)
else {
return
}
let dic = Unmanaged<CFDictionary>.fromOpaque(CFArrayGetValueAtIndex(attachments, 0)).takeUnretainedValue()
let keyframe = !CFDictionaryContainsKey(dic, Unmanaged.passRetained(kCMSampleAttachmentKey_NotSync).toOpaque())
// print("[\(Date())]: didEncodeFrameBuffer \(id) is I frame: \(keyframe)")
if keyframe,
let formatDescription = CMSampleBufferGetFormatDescription(buffer) {
// https://www.slideshare.net/instinctools_EE_Labs/videostream-compression-in-ios
var number = 0
CMVideoFormatDescriptionGetH264ParameterSetAtIndex(formatDescription, parameterSetIndex: 0, parameterSetPointerOut: nil, parameterSetSizeOut: nil, parameterSetCountOut: &number, nalUnitHeaderLengthOut: nil)
// SPS and PPS and so on...
let parameterSets = NSMutableData()
for index in 0 ... number - 1 {
var parameterSetPointer: UnsafePointer<UInt8>?
var parameterSetLength = 0
CMVideoFormatDescriptionGetH264ParameterSetAtIndex(formatDescription, parameterSetIndex: index, parameterSetPointerOut: ¶meterSetPointer, parameterSetSizeOut: ¶meterSetLength, parameterSetCountOut: nil, nalUnitHeaderLengthOut: nil)
// parameterSets.append(startCode, length: startCodeLength)
if let parameterSetPointer = parameterSetPointer {
parameterSets.append(parameterSetPointer, length: parameterSetLength)
}
//
if index == 0 {
print("SPS is \(parameterSetPointer) with length \(parameterSetLength)")
} else if index == 1 {
print("PPS is \(parameterSetPointer) with length \(parameterSetLength)")
}
}
print("[\(Date())]: parameterSets \(parameterSets.length)")
}
}
}