Quartz 2d Drawing: Perfect on simulator / bad on device. Distribution Vs Debug - iphone

I've just finished waveform drawing code for my app. I'm pretty happy with it and on the simulator it looks great.
The problem I have is when I run it on an ipad it doesnt draw properly. On the simulator the drawing looks like a nice regular waveform drawing whereas on the ipad the waveform just looks like one big rectangle.
I'm very unsure how I could even begin to start trouble shooting and resolving something like this.
Can you offer any suggestions as to why its working on the simulator & not the ipad?
If I can submit anymore information that might help please let me know.
calculation
-(void) plotwaveform:(AudioSourceOBJ )source
{
int count =source->framecount;
int blocksize= count/resolution;
currentmaxvalue=0;
int readindex=0;
CGRect *addrects= malloc(resolution * sizeof(CGRect));
float *heights=malloc(resolution * sizeof(float));
for (int i=0; i<resolution;i++) {
AudioUnitSampleType *blockofaudio;
blockofaudio =malloc(blocksize * sizeof(AudioUnitSampleType));
memcpy(blockofaudio, &source->leftoutput[readindex],(blocksize * sizeof(AudioUnitSampleType)));
float sample= [self getRMS:blockofaudio blocksize:blocksize];
heights[i]=sample;
readindex+=blocksize;
}
for (int scale=0; scale<resolution; scale++) {
float h= heights[scale];
h= (h/currentmaxvalue)* 45;
addrects[scale]=CGRectMake(scale, 0, 1, h);
}
if (waveform) {
[waveform release];
[waveform removeFromSuperview];
waveform=nil;
}
CGMutablePathRef halfpath=CGPathCreateMutable();
CGPathAddRects(halfpath, NULL, addrects, resolution);
CGMutablePathRef path= CGPathCreateMutable();
CGAffineTransform xf = CGAffineTransformIdentity;
xf= CGAffineTransformTranslate(xf, 0.0,45);
CGPathAddPath(path,&xf, halfpath);
xf= CGAffineTransformIdentity;
xf= CGAffineTransformTranslate(xf, 0.0, 45);
xf=CGAffineTransformScale(xf, 1.0, -1);
CGPathAddPath(path, &xf, halfpath);
CGPathRelease(halfpath);
free(addrects);
waveform = [[Waveform alloc] initWithFrameAndPlotdata:CGRectMake(0, 0, 400,90) thepoints:path];
[self.view addSubview:waveform];
}
-(float ) getRMS:(AudioUnitSampleType *)blockofaudio blocksize:(int)blocksize
{
float output;
float sqsummed;
float sqrootofsum;
float val;
for (int i=0;i<blocksize; i++) {
val= blockofaudio[i];
sqsummed+= val* val;
}
sqrootofsum=sqsummed / blocksize;
output = sqrt(sqrootofsum);
// find the max
if(output> currentmaxvalue)
{
currentmaxvalue=output;
}
return output;
}
Drawing
- (void)drawRect:(CGRect)rect
{
CGContextRef ctx= UIGraphicsGetCurrentContext();
CGContextSetRGBFillColor(ctx, 0, 0, 0, .5);
CGContextBeginPath(ctx);
CGContextAddPath(ctx, mutatablepath);
//CGContextStrokePath(ctx);
CGContextFillPath(ctx);
CFRelease(mutatablepath);
}
DESC EDIT
I pass a bunch of audio data to the plotwaveform function and divide it into chunks. For each chunk of audio I calculate the RMS for each chunk and keep a track of the maximum value. When all that is done I use the max value to scale my rms values to fit my view port.
I have noticed a strange thing. If I NSLog the values for the "output" variable in the getRMS function the waveform draws fine on the device. If I do not NSLog the values the waveform does not draw properly?!?
That to me is bizarre.

One major error I see is that you never initialize sqsummed inside the getRMS:blocksize: method, so its initial value is garbage. What the garbage happens to be depends on the details of the surrounding code, how the compiler allocates registers for variables, and so on. Adding an NSLog statement could well change what the garbage is next time around the loop.
If the garbage happens to always correspond to a very small float value you'll get expected behavior, while if it happens to always correspond to some extremely large float value (large enough to swamp the actual samples) you'll get one big rectangle, while if it happens to vary you'll get a noise-like output.

In any case, please remember that the simulator has your entire mac ram and cpu power to work with. Process capacity is sadly not emulated in the iphone/ipad simulator.

Related

In Unity, how to segment the user's voice from microphone based on loudness?

I need to collect voice pieces from a continuous audio stream. I need to process later the user's voice piece that has just been said (not for speech recognition). What I am focusing on is only the voice's segmentation based on its loudness.
If after at least 1 second of silence, his voice becomes loud enough for a while, and then silent again for at least 1 second, I say this is a sentence and the voice should be segmented here.
I just know I can get raw audio data from the AudioClip created by Microphone.Start(). I want to write some code like this:
void Start()
{
audio = Microphone.Start(deviceName, true, 10, 16000);
}
void Update()
{
audio.GetData(fdata, 0);
for(int i = 0; i < fdata.Length; i++) {
u16data[i] = Convert.ToUInt16(fdata[i] * 65535);
}
// ... Process u16data
}
But what I'm not sure is:
Every frame when I call audio.GetData(fdata, 0), what I get is the latest 10 seconds of sound data if fdata is big enough or shorter than 10 seconds if fdata is not big enough, is it right?
fdata is a float array, and what I need is a 16 kHz, 16 bit PCM buffer. Is it right to convert the data like: u16data[i] = fdata[i] * 65535?
What is the right way to detect loud moments and silent moments in fdata?
No. you have to read starting at the current position within the AudioClip using Microphone.GetPosition
Get the position in samples of the recording.
and pass the optained index to AudioClip.GetData
Use the offsetSamples parameter to start the read from a specific position in the clip
fdata = new float[clip.samples * clip.channels];
var currentIndex = Microphone.GetPosition(null);
audio.GetData(fdata, currentIndex);
I don't understand what exactly you convert this for. fdata will contain
floats ranging from -1.0f to 1.0f (AudioClip.GetData)
so if for some reason you need to get values between short.MinValue (= -32768) and short.MaxValue(= 32767) than yes you can do that using
u16data[i] = Convert.ToUInt16(fdata[i] * short.MaxValue);
note however that Convert.ToUInt16(float):
value, rounded to the nearest 16-bit unsigned integer. If value is halfway between two whole numbers, the even number is returned; that is, 4.5 is converted to 4, and 5.5 is converted to 6.
you might want to rather use Mathf.RoundToInt first to also round up if a value is e.g. 4.5.
u16data[i] = Convert.ToUInt16(Mathf.RoundToInt(fdata[i] * short.MaxValue));
Your naming however suggests that you are actually trying to get unsigned values ushort (or also UInt16). For this you can not have negative values! So you have to shift the float values up in order to map the range (-1.0f | 1.0f ) to the range (0.0f | 1.0f) before multiplaying it by ushort.MaxValue(= 65535)
u16data[i] = Convert.ToUInt16(Mathf.RoundToInt(fdata[i] + 1) / 2 * ushort.MaxValue);
What you receive from AudioClip.GetData are the gain values of the audio track between -1.0f and 1.0f.
so a "loud" moment would be where
Mathf.Abs(fdata[i]) >= aCertainLoudThreshold;
a "silent" moment would be where
Mathf.Abs(fdata[i]) <= aCertainSiltenThreshold;
where aCertainSiltenThreshold might e.g. be 0.2f and aCertainLoudThreshold might e.g. be 0.8f.

Repeated Scene items in iOS YUV video capturing output

I capture a video and handle the resulting YUV frames.
the output looks like the following:
Although it appears normally on my phone's screen. But my peer receives it like that img above.
Every item is repeated and shifted by some value horizontally and vertically
My captured video is 352x288 and my YPixelCount = 101376, UVPixelCount = YPIXELCOUNT/4
Any clue to solve this or a starting point to understand how to handle YUV video frames on iOS ?
NSNumber* recorderValue = [NSNumber numberWithUnsignedInt:kCVPixelFormatType_420YpCbCr8BiPlanarVideoRange];
[videoRecorderSession setSessionPreset:AVCaptureSessionPreset352x288];
And this is the captureOutput function
- (void)captureOutput:(AVCaptureOutput *)captureOutput didOutputSampleBuffer:(CMSampleBufferRef)sampleBuffer fromConnection:(AVCaptureConnection *)connection{
if(CMSampleBufferIsValid(sampleBuffer) && CMSampleBufferDataIsReady(sampleBuffer) && ([self isQueueStopped] == FALSE))
{
CVImageBufferRef imageBuffer = CMSampleBufferGetImageBuffer(sampleBuffer);
CVPixelBufferLockBaseAddress(imageBuffer,0);
UInt8 *baseAddress[3] = {NULL,NULL,NULL};
uint8_t *yPlaneAddress = (uint8_t *)CVPixelBufferGetBaseAddressOfPlane(imageBuffer,0);
UInt32 yPixelCount = CVPixelBufferGetWidthOfPlane(imageBuffer,0) * CVPixelBufferGetHeightOfPlane(imageBuffer,0);
uint8_t *uvPlaneAddress = (uint8_t *)CVPixelBufferGetBaseAddressOfPlane(imageBuffer,1);
UInt32 uvPixelCount = CVPixelBufferGetWidthOfPlane(imageBuffer,1) * CVPixelBufferGetHeightOfPlane(imageBuffer,1);
UInt32 p,q,r;
p=q=r=0;
memcpy(uPointer, uvPlaneAddress, uvPixelCount);
memcpy(vPointer, uvPlaneAddress+uvPixelCount, uvPixelCount);
memcpy(yPointer,yPlaneAddress,yPixelCount);
baseAddress[0] = (UInt8*)yPointer;
baseAddress[1] = (UInt8*)uPointer;
baseAddress[2] = (UInt8*)vPointer;
CVPixelBufferUnlockBaseAddress(imageBuffer,0);
}
}
Is there anything wrong with the above code ?
Your code doesn't look to0 bad. I can see two mistakes and one potential problem:
The uvPixelCount is incorrect. The YUV 420 format means that there is color information for each 2 by 2 pixel block. So the correct count is:
uvPixelCount = (width / 2) * (height / 2);
You write something about yPixelCount / 4, but I cannot see that in your code.
The UV information is interleaved, i.e. the second plane alternatingly contains a U and a V value. Or put differently: there's a U value on all even byte addresses and a V value on all odd byte addresses. If you really need to separate the U and V information, memcpy won't do.
There can be some extra bytes after each pixel row. You should use CVPixelBufferGetBytesPerRowOfPlane(imageBuffer, 0) to get the number of bytes between two rows. As a consequence, a single memcpy won't do. Instead you need to copy each pixel row separately to get rid of the extra bytes between the rows.
All these things only explain part of the resulting image. The remaining parts are probably due to differences between your code and what the receiving peer expect. You did't write anything about that? Does the peer really need separated U and V values? Does it you 4:2:0 compression as well? Does it you video range instead of full range as well?
If you provide more information, I can give your more hints.

Help with live-updating sound on the iPhone

My question is a little tricky, and I'm not exactly experienced (I might get some terms wrong), so here goes.
I'm declaring an instance of an object called "Singer". The instance is called "singer1". "singer1" produces an audio signal. Now, the following is the code where the specifics of the audio signal are determined:
OSStatus playbackCallback(void *inRefCon,
AudioUnitRenderActionFlags *ioActionFlags,
const AudioTimeStamp *inTimeStamp,
UInt32 inBusNumber,
UInt32 inNumberFrames,
AudioBufferList *ioData) {
//Singer *me = (Singer *)inRefCon;
static int phase = 0;
for(UInt32 i = 0; i < ioData->mNumberBuffers; i++) {
int samples = ioData->mBuffers[i].mDataByteSize / sizeof(SInt16);
SInt16 values[samples];
float waves;
float volume=.5;
for(int j = 0; j < samples; j++) {
waves = 0;
waves += sin(kWaveform * 600 * phase)*volume;
waves += sin(kWaveform * 400 * phase)*volume;
waves += sin(kWaveform * 200 * phase)*volume;
waves += sin(kWaveform * 100 * phase)*volume;
waves *= 32500 / 4; // <--------- make sure to divide by how many waves you're stacking
values[j] = (SInt16)waves;
values[j] += values[j]<<16;
phase++;
}
memcpy(ioData->mBuffers[i].mData, values, samples * sizeof(SInt16));
}
return noErr;
}
99% of this is borrowed code, so I only have a basic understanding of how it works (I don't know about the OSStatus class or method or whatever this is. However, you see those 4 lines with 600, 400, 200 and 100 in them? Those determine the frequency. Now, what I want to do (for now) is insert my own variable in there in place of a constant, which I can change on a whim. This variable is called "fr1". "fr1" is declared in the header file, but if I try to compile I get an error about "fr1" being undeclared. Currently, my technique to fix this is the following: right beneath where I #import stuff, I add the line
fr1=0.0;//any number will work properly
This sort of works, as the code will compile and singer1.fr1 will actually change values if I tell it to. The problems are now this:A)even though this compiles and the tone specified will play (0.0 is no tone), I get the warnings "Data definition has no type or storage class" and "Type defaults to 'int' in declaration of 'fr1'". I bet this is because for some reason it's not seeing my previous declaration in the header file (as a float). However, again, if I leave this line out the code won't compile because "fr1 is undeclared". B)Just because I change the value of fr1 doesn't mean that singer1 will update the value stored inside the "playbackcallback" variable or whatever is in charge of updating the output buffers. Perhaps this can be fixed by coding differently? C)even if this did work, there is still a noticeable "gap" when pausing/playing the audio, which I need to eliminate. This might mean a complete overhaul of the code so that I can "dynamically" insert new values without disrupting anything. However, the reason I'm going through all this effort to post is because this method does exactly what I want (I can compute a value mathematically and it goes straight to the DAC, which means I can use it in the future to make triangle, square, etc waves easily). I have uploaded Singer.h and .m to pastebin for your veiwing pleasure, perhaps they will help. Sorry, I can't post 2 HTML tags so here are the full links.
(http://pastebin.com/ewhKW2Tk)
(http://pastebin.com/CNAT4gFv)
So, TL;DR, all I really want to do is be able to define the current equation/value of the 4 waves and re-define them very often without a gap in the sound.
Thanks. (And sorry if the post was confusing or got off track, which I'm pretty sure it did.)
My understanding is that your callback function is called every time the buffer needs to be re-filled. So changing fr1..fr4 will alter the waveform, but only when the buffer updates. You shouldn't need to stop and re-start the sound to get a change, but you will notice an abrupt shift in the timbre if you change your fr values. In order to get a smooth transition in timbre, you'd have to implement something that smoothly changes the fr values over time. Tweaking the buffer size will give you some control over how responsive the sound is to your changing fr values.
Your issue with fr being undefined is due to your callback being a straight c function. Your fr variables are declared as objective-c instance variables as part of your Singer object. They are not accessible by default.
take a look at this project, and see how he implements access to his instance variables from within his callback. Basically he passes a reference to his instance to the callback function, and then accesses instance variables through that.
https://github.com/youpy/dowoscillator
notice:
Sinewave *sineObject = inRefCon;
float freq = sineObject.frequency * 2 * M_PI / samplingRate;
and:
AURenderCallbackStruct input;
input.inputProc = RenderCallback;
input.inputProcRefCon = self;
Also, you'll want to move your callback function outside of your #implementation block, because it's not actually part of your Singer object.
You can see this all in action here: https://github.com/coryalder/SineWaver

iPhone original render faster than iPhone 3GS?

I'm working on a game on iPhone, which uses C++ and OpenGL ES 1.x library.
It works fine on simulator. But when I install it on real iPhone, I found out that on iPhone original, it took about 20 milliseconds to render a frame. However, it took 35~40 milliseconds to render a frame on iPhone 3GS.
I've tried various OS, including 3GS + iOS 3.1.2, 3G + iOS 4.0, 3GS + iOS 4.1, iPad + iOS 3.2. All of them render much slower than iPhone original, which sounds really ridiculous to me. I tried google for anything I can think of, fixing every problem it might be related to, but nothing changed.
I have 2 machine which these pieces of code render faster: 1) iPhone original with iOS 3.1.3, 2) iPod Touch with iOS 3.1.3. Both took about 20 milliseconds to render a frame.
And 4 machine which render mysteriously slower: 1) iPhone 3G with iOS 4.0, 2) iPhone 3GS with iOS 3.1.2, 3) iPhone 3GS with iOS 4.1, 4) iPad with iOS 3.2. iPhone took about 35-40 milliseconds to render a frame and iPad took around 25.
I use PVRTC for texture, which is first cooked and make into a bundle. It uses total of ten 512x512 textures, three 1024x1024 textures.
The piece of code which binding texture is as follow:
GLenum internalFormat = 0;
GLenum pixelType = 0;
// resolve type
ResetFlags_();
assert(2==attr.Dimension && 1==attr.Depth);
switch (attr.Format)
{
case FORMAT_PVRTC2:
assert(attr.Width==attr.Height);
if (attr.AlphaBits>0)
internalFormat = GL_COMPRESSED_RGBA_PVRTC_2BPPV1_IMG;
else
internalFormat = GL_COMPRESSED_RGB_PVRTC_2BPPV1_IMG;
break;
case FORMAT_PVRTC4:
assert(attr.Width==attr.Height);
if (attr.AlphaBits>0)
internalFormat = GL_COMPRESSED_RGBA_PVRTC_4BPPV1_IMG;
else
internalFormat = GL_COMPRESSED_RGB_PVRTC_4BPPV1_IMG;
break;
... other formats ...
}
// prepare temp buffer to load
MemoryBuffer tmpBuffer(true);
uint8* buffer = tmpBuffer.GetWritePtr(attr.TextureSize);
// read data
stream.Read(buffer, attr.TextureSize);
if (stream.Fail())
return false;
// init
width_ = attr.Width;
height_ = attr.Height;
LODs_ = attr.LODs;
alphaBits_ = attr.AlphaBits;
// create and upload texture
glGenTextures(1, &glTexture_);
glBindTexture(GL_TEXTURE_2D, glTexture_);
uint32 offset = 0;
uint32 dim = width_; // = height
uint32 w, h;
switch (internalFormat)
{
case GL_COMPRESSED_RGBA_PVRTC_2BPPV1_IMG:
case GL_COMPRESSED_RGB_PVRTC_2BPPV1_IMG:
case GL_COMPRESSED_RGBA_PVRTC_4BPPV1_IMG:
case GL_COMPRESSED_RGB_PVRTC_4BPPV1_IMG:
for (uint32 i=0; i<LODs_; ++i) {
assert(offset<attr.TextureSize);
w = dim >> ((FORMAT_PVRTC2==attr.Format) ? 3:2);
h = dim >> 2;
// Clamp to minimum number of blocks
if (w<2) w = 2;
if (h<2) h = 2;
uint32 const image_size = w * h * 8; // 8 bytes for each block
glCompressedTexImage2D(GL_TEXTURE_2D, i, internalFormat, dim, dim, 0, image_size, buffer+offset);
dim >>= 1;
offset += image_size;
break;
... other formats ...
}
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_NEAREST_MIPMAP_NEAREST); // tri-linear?
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, GL_NEAREST);
SetContext_(&glTexture_);
return true;
Rendering part is huge because it uses an engine developed by others. As far as I can tell, it uses glDrawArrays and no shader was used.
Anyone had encounter the same problem before? I really can't see why iPhone original render much faster than iPhone 3GS.
p.s. I forgot to say. I draw only 2D rectangles with textures only. And it's around 20 rectangles in my game ( one background and one UI with 480x360 size. Others are commonly 64x64 units.)
The behaviour you are getting could be because of the possible emulation of Fixed Function Pipeline (FFP) via Programmable Pipeline (i.e. shaders).
Can you please execute a test that will load and display your textures in some way, completely without your engine.

EXC_BAD_ACCESS when calling avcodec_encode_video

I have an Objective-C class (although I don't believe this is anything Obj-C specific) that I am using to write a video out to disk from a series of CGImages. (The code I am using at the top to get the pixel data comes right from Apple: http://developer.apple.com/mac/library/qa/qa2007/qa1509.html). I successfully create the codec and context - everything is going fine until it gets to avcodec_encode_video, when I get EXC_BAD_ACCESS. I think this should be a simple fix, but I just can't figure out where I am going wrong.
I took out some error checking for succinctness. 'c' is an AVCodecContext*, which is created successfully.
-(void)addFrame:(CGImageRef)img
{
CFDataRef bitmapData = CGDataProviderCopyData(CGImageGetDataProvider(img));
long dataLength = CFDataGetLength(bitmapData);
uint8_t* picture_buff = (uint8_t*)malloc(dataLength);
CFDataGetBytes(bitmapData, CFRangeMake(0, dataLength), picture_buff);
AVFrame *picture = avcodec_alloc_frame();
avpicture_fill((AVPicture*)picture, picture_buff, c->pix_fmt, c->width, c->height);
int outbuf_size = avpicture_get_size(c->pix_fmt, c->width, c->height);
uint8_t *outbuf = (uint8_t*)av_malloc(outbuf_size);
out_size = avcodec_encode_video(c, outbuf, outbuf_size, picture); // ERROR occurs here
printf("encoding frame %3d (size=%5d)\n", i, out_size);
fwrite(outbuf, 1, out_size, f);
CFRelease(bitmapData);
free(picture_buff);
free(outbuf);
av_free(picture);
i++;
}
I have stepped through it dozens of times. Here are some numbers...
dataLength = 408960
picture_buff = 0x5c85000
picture->data[0] = 0x5c85000 -- which I take to mean that avpicture_fill worked...
outbuf_size = 408960
and then I get EXC_BAD_ACCESS at avcodec_encode_video. Not sure if it's relevant, but most of this code comes from api-example.c. I am using XCode, compiling for armv6/armv7 on Snow Leopard.
Thanks so much in advance for help!
I have not enough information here to point to the exact error, but I think that the problem is that the input picture contains less data than avcodec_encode_video() expects:
avpicture_fill() only sets some pointers and numeric values in the AVFrame structure. It does not copy anything, and does not check whether the buffer is large enough (and it cannot, since the buffer size is not passed to it). It does something like this (copied from ffmpeg source):
size = picture->linesize[0] * height;
picture->data[0] = ptr;
picture->data[1] = picture->data[0] + size;
picture->data[2] = picture->data[1] + size2;
picture->data[3] = picture->data[1] + size2 + size2;
Note that the width and height is passed from the variable "c" (the AVCodecContext, I assume), so it may be larger than the actual size of the input frame.
It is also possible that the width/height is good, but the pixel format of the input frame is different from what is passed to avpicture_fill(). (note that the pixel format also comes from the AVCodecContext, which may differ from the input). For example, if c->pix_fmt is RGBA and the input buffer is in YUV420 format (or, more likely for iPhone, a biplanar YCbCr), then the size of the input buffer is width*height*1.5, but avpicture_fill() expects the size of width*height*4.
So checking the input/output geometry and pixel formats should lead you to the cause of the error. If it does not help, I suggest that you should try to compile for i386 first. It is tricky to compile FFMPEG for the iPhone properly.
Does the codec you are encoding support the RGB color space? You may need to use libswscale to convert to I420 before encoding. What codec are you using? Can you post the code where you initialize your codec context?
The function RGBtoYUV420P may help you.
http://www.mail-archive.com/libav-user#mplayerhq.hu/msg03956.html