I am building an app that uses microphone input to detect sounds and trigger events. I based my code on AKAmplitudeTap, but I when I ran it, I found that I was only obtaining sample data for intervals with missing sections.
The tap code looks like this (with the guts ripped out and simply keeping track of how many samples would have been processed):
open class MyTap {
// internal let bufferSize: UInt32 = 1_024 // 8-9 kSamples/sec
internal let bufferSize: UInt32 = 4096 // 39.6 kSamples/sec
// internal let bufferSize: UInt32 = 16536 // 43.3 kSamples/sec
public init(_ input: AKNode?) {
input?.avAudioNode.installTap(onBus: 0, bufferSize: bufferSize, format: nil ) { buffer, _ in
sampleCount += self.bufferSize
}
}
I initialize the tap with:
func afterLoad() {
assert(!loaded)
AKSettings.audioInputEnabled = true
do {
try AKSettings.setSession(category: .playAndRecord, with: .allowBluetoothA2DP)
} catch {
print("Could not set session category.")
}
mic = AKMicrophone()
myTap = MyTap(mic) // seriously, can it be that easy?
loaded = true
}
The original tap code was capturing samples to a buffer, but I saw that big chunks of time were missing with a buffer size of 1024. I suspected that the processing time for the sample buffer might be excessive, so...
I simplified the code to simply keep track of how many samples were being passed to the tap. In another part of the code, I simply print out sampleCount/elapsedTime and, as noted in the comments after 'bufferSize' I get different amounts of samples per second.
The sample rate converges on 43.1 KSamples/sec with a 16K buffer, and only collects about 20% of the samples with a 1K buffer. I would prefer to use the small buffer size to obtain near real-time response to detected sounds. As I've been writing this, the 4K buffer version has been running and has stabilized at 39678 samples/sec.
Am I missing something? Can a tap with a small buffer size actually capture 44.1 Khz sample data?
Problem resolved... the tap requires this line of code
buffer.frameLength = self.bufferSize
... and suddenly all the samples appear. I obviously stripped out a bit too much code from the code I obviously didn't understand.
Related
I am processing a PHLivePhoto using .frameProcessor to modify each frame. The frames appear to be processed in sequence, which is slow. Can I get PHLivePhotoEditingContext.frameProcessor to take advantage of more than one core?
func processLivePhoto(input: PHContentEditingInput) {
guard let context = PHLivePhotoEditingContext(livePhotoEditingInput: input)
else { fatalError("not a Live Photo editing input") }
context.frameProcessor = { frame, _ in
let renderedFrame = expensiveOperation(using: frame.image)
return renderedFrame
}
// ...logic for saving
}
I'm afraid there's no way to parallelize the frame processing in this case. You have to keep in mind:
Video frames need to be written in order.
The more frames you would process in parallel, the more memory you would need.
Core Image is processing the frames on the GPU, which can usually only process one frame at a time anyways.
Your expensiveOperation is not really happening in the frameProcessor block anyways, since the actual rendering is handled by the framework outside this scope.
I want to play the recorded audio using microphone.
After recording it as 32 bit arrays
let left = e.inputBuffer.getChannelData(0);
let tempLeftChannel = this.state.leftChannel;
tempLeftChannel.push(new Float32Array(left));
this.setState({ leftChannel: tempLeftChannel });
Now In the leftChannel array, I had chunk of audio data. Now, I want to play them in the browser. How can I do that?
You leave quite a bit out from your snippet, but perhaps the following will give you an idea of one way to play out the float array that you have. Let context be the AudioContext that you probably have.
let buffer = new AudioBuffer({length: leftChannel.length,
sampleRate: context.sampleRate});
buffer.copyToChannel(leftChannel, 0);
let source = new AudioBufferSourceNode(context, {buffer: buffer});
source.connect(context.destination);
source.start();
I'm building a 2d tile-based game for iOS with swift and Firebase. Because the world is large, I've designed it so that I only subscribe to the tiles that are on screen. That is, instead of adding listeners for all 10,000x10,000 tiles, I add them to just the tiles on screen. As the player moves, I de-register the old listeners and register the new ones. I've added a bit of a buffer zone around the edge of the screen, in the hopes that everything will be sufficiently loaded by the time it moves on screen. Unfortunately, there is often significant enough lag from Firebase that this strategy simply isn't working. On sub-optimal internet connections, it's possible to keep walking into the "unloaded world," taking several seconds at times to load the missing tiles.
Here's the thing, though: other MMO iOS games on the same connection and same device work fine. It's not a terrible connection. Which makes me suspect my implementation, or Firebase itself is at fault.
Fundamentally I'm waiting on the "load once" event for about 20 tiles each time I take a step. A step takes about 1/4 of a second, so each second I'm requesting about 100 items from Firebase. I can't think of a better way, though. Firebase documentation suggests that this should not be a problem, since it's all one socket connection. I could "bucket" the objects into, say, 10x10 blocks which would mean that I'd subscribe to fewer objects, but this would also be more wasteful in terms of total data transfer. If the socket connection is truly optimized, total data transfer should be the only bottleneck, implying this strategy would be wrong.
edit
Here's a video showing how it works. The buffer-size has been reduced to -1, so that you can easily see the edges of the screen and the tiles loading and unloading. Near the end of the video, lag strikes and I wander into the emptiness. I opened up another game and it loaded almost instantly. http://forevermaze.inzania.com/videos/FirebaseLag.mov (n.b., I ended the recording before the screen loaded again. It never fails to load, so it's not as if the code is failing to work. It's pure lag.)
Here is the code I'm using to load the tiles. It's called once for each tile. As I said, this means that this code is called about 20 times per step, in parallel. All other apps are running at a fine speed with no lag. I'm on a MiFi with LTE connectivity in Tokyo, so it's a solid connection.
/**
* Given a path to a firebase object, get the snapshot with a timeout.
*/
static func loadSnapshot(firebasePath: String!) -> Promise<FDataSnapshot?> {
let (promise, fulfill, _) = Promise<FDataSnapshot?>.pendingPromise()
let connection = Firebase(url: Config.firebaseUrl + firebasePath)
connection.observeSingleEventOfType(.Value, withBlock: { snapshot in
if !promise.resolved {
fulfill(snapshot)
}
})
after(Config.timeout).then { () -> Void in
if !promise.resolved {
DDLogWarn("[TIMEOUT] [FIREBASE-READ] \(firebasePath)")
fulfill(nil)
//reject(Errors.network)
}
}
return promise
}
The tiles reside at [ROOT]/tiles/[X]x[Y]. Most tiles contain very little data, but if there are objects on that tile (i.e., other players) those are stored. Here's a screenshot from Firebase:
edit2
Per request, I've recreated this issue very simply. Here is a 100-line XCTestCase class: http://forevermaze.com/code/LagTests.swift
Usage:
Drop the file into your Swift project (it should be stand-alone, requiring only Firebase)
Change the value of firebaseUrl to your root URL (i.e., https://MyProject.firebaseio.com)
Run the testSetupDatabase() function test once to setup the initial state of the database
Run the testWalking() function to test the lag. This is the main test. It will fail if any tile takes longer than 2 seconds to load.
I've tried this test on several different connections. A top-notch office connection passes with no problem, but even a high-end LTE or MiFi connection fails. 2 seconds is already a very long timeout, since it implies that I need to have a 10 tile buffer zone (0.2 seconds * 10 tiles = 2 seconds). Here's some output when I'm connected to a LTE connection, showing that it took nearly 10 seconds (!!) to load a tile:
error: -[ForeverMazeTests.LagTests testWalking] : XCTAssertTrue failed - Tile 2x20 took 9.50058007240295
I ran a few tests and the loading completes in 15-20 seconds when I test over a 3G connection. Over my regular connection it takes 1-2 seconds, so the difference is likely purely based on bandwidth.
I rewrote your test case into a JavaScript version, because I had a hard time figuring out what's going on. Find mine here: http://jsbin.com/dijiba/edit?js,console
var ref = new Firebase(URL);
var tilesPerStep = 20;
var stepsToTake = 100;
function testWalking() {
var startTime = Date.now();
var promises = [];
for (var x=0; x < stepsToTake; x++) {
promises.push(testStep(x));
}
Promise.all(promises).then(function() {
console.log('All '+promises.length+' steps done in '+(Date.now()-startTime)+'ms');
});
}
function testStep(x) {
var result = new Promise(function(resolve, reject){
var tiles = ref.child("/tiles_test");
var loading = 0;
var startTime = Date.now();
console.log('Start loading step '+x);
for (var y=0; y < tilesPerStep; y++) {
loading ++;
tiles.child(x+'x'+y).once('value', function(snapshot) {
var time = Date.now() - startTime;
loading--;
if (loading === 0) {
console.log('Step '+x+' took '+(Date.now()-startTime)+'ms');
resolve(Date.now() - startTime);
}
});
}
});
return result;
}
testWalking();
The biggest difference is that I don't delay starting any of the loading and I don't fail for a specific tile. I think that last bit is why your tests are failing.
All loading from Firebase happens asynchronously, but all requests are going through the same connection. When you start loading, you are queueing up a lot of requests. This timing is skewed by "preceding requests that haven't been fulfilled yet".
This is a sample of a test run with just 10 steps:
"Start loading step 0"
"Start loading step 1"
"Start loading step 2"
"Start loading step 3"
"Start loading step 4"
"Start loading step 5"
"Start loading step 6"
"Start loading step 7"
"Start loading step 8"
"Start loading step 9"
"Step 0 took 7930ms"
"Step 1 took 7929ms"
"Step 2 took 7948ms"
"Step 3 took 8594ms"
"Step 4 took 8669ms"
"Step 5 took 9141ms"
"Step 6 took 9851ms"
"Step 7 took 10365ms"
"Step 8 took 10425ms"
"Step 9 took 11520ms"
"All 10 steps done in 11579ms"
You'll probably note that the time taken for each step does not add up to the time taken for all steps combined. Essentially you are starting each request while there are still requests in the pipeline. This is the most efficient way of loading these items, but it does mean that you'll need to measure performance differently.
Essentially all steps start at almost the same time. Then you're waiting for the first response (which is in the above case includes establishing a WebSocket connection from the client to the correct Firebase server) and after that the responses come in reasonable intervals (given that there are 20 requests per step).
All of this is very interesting, but it doesn't solve your problem of course. I would recommend that you model your data into screen-sized buckets. So instead of having each tile separate, store every 10x10 tiles in a "bucket". You'll reduce the overhead of each separate request and only need to request at most one bucket for every 10 steps.
Update
I'm pretty sure we're just debugging multiple artifacts of your benchmark approach. If I update the code to this:
func testWalking() {
let expectation = expectationWithDescription("Load tiles")
let maxTime = self.timeLimit + self.stepTime * Double(stepsToTake)
let startTime = NSDate().timeIntervalSince1970
for (var x=0; x<stepsToTake; x++) {
let delay = Double(x) * stepTime
let data = ["x":x, "ex": expectation]
stepsRemaining++
NSTimer.scheduledTimerWithTimeInterval(0, target: self, selector: Selector("testStep:"), userInfo: data, repeats: false)
}
waitForExpectationsWithTimeout(maxTime) { error in
let time = NSDate().timeIntervalSince1970 - startTime
print("Completed loading after \(time)")
if error != nil {
print("Error: \(error!.localizedDescription)")
}
}
}
/**
* Helper function to test a single step (executes `tilesPerStep` number of tile loads)
*/
func testStep(timer : NSTimer) {
let tiles = Firebase(url: firebaseUrl).childByAppendingPath("/tiles_test")
let data = timer.userInfo as! Dictionary<String, AnyObject>
let x = data["x"] as! Int
let expectation = data["ex"] as! XCTestExpectation
var loading = 0
print("Start loading \(x)")
for (var y=0; y<tilesPerStep; y++) {
loading++
tiles.childByAppendingPath("\(x)x\(y)").observeSingleEventOfType(.Value, withBlock: { snapshot in
loading--
if loading == 0 {
print("Done loading \(x)")
self.stepsRemaining--
if self.stepsRemaining == 0 {
expectation.fulfill()
}
}
})
}
}
It completes the entire loading in less than 2 seconds over a high-speed network, over 3G it takes between 15 and 25 seconds.
But my recommendation of modeling at a level of more than each single tile remains.
I'm using the Audio Queue Services API to play audio streamed from a server over a TCP socket connection on an iPhone. I can play the buffers that were filled from the socket connection, I just cannot seem to make my AudioQueue call my AudioQueueOutputCallback function, and I'm out of ideas.
High level design
Data is passed to the player from the socket connection, and written
immediately into circular buffers in memory.
As AudioQueueBuffers become available, data is copied from the circular buffers into the
available AudioQueueBuffer, which is immediately re-queued. (Or would be, if my callback happened)
What happens
The buffers are all filled and enqueued successfully, and I hear the audio stream clearly. For testing, I use a large number of buffers (15) and all of them play through seamlessly, but the AudioQueueOutputCallback is never called, so I never re-queue any of those buffers, despite the fact that everything seems to be working perfectly. If I don't wait for my callback, assuming it will never be called, and instead drive the enqueueing of buffers based on the data as it is written, I can play the audio stream indefinitely, reusing and re-enqueueing buffers as if they had been explicitly returned to me by the callback. It is that fact: that I can play the stream perfectly while reusing buffers as needed, that confuses me the most. Why isn't the callback being called?
Possibly Relevant Code
The format of the stream is 16 bit linear PCM, 8 kHz, Mono:
_streamDescription.mSampleRate = 8000.0f;
_streamDescription.mFormatID = kAudioFormatLinearPCM;
_streamDescription.mBytesPerPacket = 2;
_streamDescription.mFramesPerPacket = 1;
_streamDescription.mBytesPerFrame = sizeof(AudioSampleType);
_streamDescription.mChannelsPerFrame = 1;
_streamDescription.mBitsPerChannel = 8 * sizeof(AudioSampleType)
_streamDescription.mReserved = 0;
_streamDescription.mFormatFlags = (kLinearPCMFormatFlagIsBigEndian |
kLinearPCMFormatFlagIsPacked);
My prototype and implementation of the callback are as follows. Nothing fancy, and pretty much identical to every example I've seen so far:
// Prototype, declared above the class's #implementation
void AQBufferCallback(void* inUserData, AudioQueueRef inAudioQueue, AudioQueueBufferRef inAudioQueueBuffer);
// Definition at the bottom of the file.
void AQBufferCallback(void* inUserData, AudioQueueRef inAudioQueue, AudioQueueBufferRef inAudioQueueBuffer) {
printf("callback\n");
[(MyAudioPlayer *)inUserData audioQueue:inAudioQueue didAquireBufferForReuse:inAudioQueueBuffer];
}
I create the AudioQueue like this:
OSStatus status = 0;
status = AudioQueueNewOutput(&_streamDescription,
AQBufferCallback, // <-- Doesn't work...
self,
CFRunLoopGetCurrent(),
kCFRunLoopCommonModes,
0,
&_audioQueue);
if (status) {
// This is not called...
NSLog(#"Error creating new audio output queue: %#", [MyAudioPlayer stringForOSStatus:status]);
return;
}
And I enqueue buffers like this. At this point, it is known that the local buffer contains the correct amount of data for copying:
memcpy(aqBuffer->mAudioData, localBuffer, kAQBufferSize);
aqBuffer->mAudioDataByteSize = kAQBufferSize;
OSStatus status = AudioQueueEnqueueBuffer(_audioQueue, aqBuffer, 0, NULL);
if (status) {
// This is also not called.
NSLog(#"Error enqueueing buffer %#", [MyAudioPlayer stringForOSStatus:status]);
}
Please save me.
Is this executed on the main thread or a background thread? probably not good if CFRunLoopGetCurrent() returns a run loop of a thread that could disappear (thread pool etc) or is a run loop that don't care about kCFRunLoopCommonModes.
Try to change CFRunLoopGetCurrent() to CFRunLoopGetMain() or make sure AudioQueueNewOutput() and CFRunLoopGetCurrent() is executed on the main thread or a thread that you have control over and has a proper run loop.
Try changing self for (void*)self. Like this:
status = AudioQueueNewOutput(&_streamDescription,
AQBufferCallback,
(void*)self,
CFRunLoopGetCurrent(),
kCFRunLoopCommonModes,
0,
&_audioQueue);
I am currently working on an audio DSP App development. The project requires direct access and modification of audio data. Right now I can successfully access and modify the raw audio data using AudioQueue but encounters error during playback. The output audio after any modification turns out be noise.
In short, the code is something like this:
(Modified from Speakhere sample code. The rest remains unchanged.)
void AQPlayer::AQBufferCallback(void * inUserData,
AudioQueueRef inAQ,
AudioQueueBufferRef inCompleteAQBuffer)
{
AQPlayer *THIS = (AQPlayer *)inUserData;
if (THIS->mIsDone) return;
UInt32 numBytes;
UInt32 nPackets = THIS->GetNumPacketsToRead();
OSStatus result = AudioFileReadPackets(THIS->GetAudioFileID(),
false,
&numBytes,
inCompleteAQBuffer->mPacketDescriptions,
THIS->GetCurrentPacket(),
&nPackets,
inCompleteAQBuffer->mAudioData);
if (result)
printf("AudioFileReadPackets failed: %d", (int)result);
if (nPackets > 0) {
inCompleteAQBuffer->mAudioDataByteSize = numBytes;
inCompleteAQBuffer->mPacketDescriptionCount = nPackets;
//My modification starts from here
//Modifying audio data
SInt16 *testBuffer = (SInt16*)inCompleteAQBuffer->mAudioData;
for (int i = 0; i < (inCompleteAQBuffer->mAudioDataByteSize)/sizeof(SInt16); i++)
{
//printf("before modification %d", (int)*testBuffer);
*testBuffer = (SInt16) *testBuffer/2; //Say some simple modification
//printf("after modification %d", (int)*testBuffer);
testBuffer++;
}
AudioQueueEnqueueBuffer(inAQ, inCompleteAQBuffer, 0, NULL);
}
During debugging, the data in buffer is displayed as expected, but the actual output is nothing but noise.
Here are some other strange behaviors of the code that makes both the whole team crazy:
If there is no change to the data (add/sub by 0, multiply by 1) or the whole buffer is assigned to a constant (say 0, then the audio will be muted), the playback behaves normally (Of course!) But if I perform anything more than it, it still turns out to be noise.
In the case I hardcode a single tone as test audio, the output noise spreads into another channel also.
So where is the bug in this code? Or if I am on the wrong track, what is the correct approach to modify the audio data and perform playback CORRECTLY? Any insight will be sincerely appreciated.
Thank you very much :-)
Cheers,
Manca
are you SURE the sample format is SInt16? And how many channels are there? You seem to treat the audio as a single channel short stream, but suppose the format is actually dual channel Float32 or so, and you do the modifications there, than the effect would be exactly as you describe, including the noise on other channels.