Hope there will be nothing confusing in what I'm going to talk about, because my mother tongue is not English and my grammar is poor:p
I'm working on a mipmap analyzing tool which need to calculate with pixels from the render texture. Here's a part of the C# code:
private IEnumerator CSGroupColor(RenderTexture rt, GroupColor[] groupColors)
{
var outputBuffer = new ComputeBuffer(groupColors.Length, 8);
csKernelID = cs.FindKernel("CSGroupColor");
cs.SetTexture(csKernelID, "rt", rt);
cs.SetBuffer(csKernelID, "groupColorOut", outputBuffer);
cs.Dispatch(csKernelID, rt.width / 8, rt.height / 8, 1);
var req = AsyncGPUReadback.Request(outputBuffer);
yield return new WaitUntil(() => req.done);
req.GetData<GroupColor>().CopyTo(groupColors);
foreach (var color in groupColors)
{
if (!m_staticsDatas.TryGetValue(color.groupindex, out var vl))
continue;
if (color.value > 0)
vl.allColors.Add(color.value);
}
}
And what I want to implement next, is to make every buffer smaller(e.g.with a length of 4096), like we usually do in other asynchronous communications. Maybe I can pass the first buffer to CPU right away when it's full, and then replace it with the second buffer, and so on.
As I see it, using SetBuffer() again after req.done must be permitted to make that viable. I have been finding on Internet all day for a sample usage, but still found nothing.
Is there anyone who would give some help? Thx very much.
Related
I have a BufferSource, which I create thusly:
const proxyUrl = location.origin == 'file://' ? 'https://cors-anywhere.herokuapp.com/' : '';
const request = new XMLHttpRequest();
request.open('GET', proxyUrl + 'http://heliosophiclabs.com/~mad/projects/mad-music/non.mp3', true);
// request.open('GET', 'non.mp3', true);
request.responseType = 'arraybuffer';
request.onload = () => {
audioCtx.decodeAudioData(request.response, buffer => {
buff = buffer;
}, err => {
console.error(err);
});
}
request.send();
Yes, the CORS workaround is pathetic, but this is the way I found to be able to work locally without needing to run a HTTP server. Anyway...
I would like to shift the pitch of this buffer. I've tried various different forms of this:
const source = audioCtx.createBufferSource();
source.buffer = buff;
const analyser = audioCtx.createAnalyser();
analyser.connect(audioCtx.destination);
analyser.minDecibels = -140;
analyser.maxDecibels = 0;
analyser.smoothingTimeConstant = 0.8;
analyser.fftSize = 2048;
const dataArray = new Float32Array(analyser.frequencyBinCount);
source.connect(analyser);
analyser.connect(audioCtx.destination);
source.start(0);
analyser.getFloatFrequencyData(dataArray);
console.log('dataArray', dataArray);
All to no avail. dataArray is always filled with -Infinity values, no matter what I try.
My idea is to get this frequency domain data and then to move all the frequencies up/down by some amount and create a new Oscillator node out of these, like this:
const wave = audioCtx.createPeriodicWave(real, waveCompnents);
oscillator.setPeriodicWave(wave);
Anyway. If anyone has a better idea of how to shift pitch, I'd love to hear it. Sadly, detune and playbackRate both seem to do basically the same thing (why are there two ways of doing the same thing?), namely just to speed up or slow down the playback, so that's not it.
First, there's a small issue with the code: you connect the analyser to the destination twice. You don't actually need to connect it at all.
Second, I think the reason you're getting all -infinity values is because you call getFloatFrequencyData right after you start the source. There's a good chance that no samples have been played so the analyser only has buffers of all zeros.
You need to call getFloatFrequencyData after a bit of time to see non-zero values.
Third, I don't think this will work at all, even for shifting the pitch of an oscillator. getFloatFrequencyData only returns the magnitude information. You will need the phase information for the harmonics to get everything shifted correctly. Currently there's no way to get the phase information.
Fourth, if you have an AudioBuffer with the data you need, consider using the playbackRate to change the pitch. Not sure if this will produce the shift you want.
Is there a callback function for p5.js’ createCapture function fails? (i.e. when user permission is denied or video camera stream is unsupported by user browser).
I notice in the src there is a success callback, but can’t seem to find one for failure.
In the browser console, p5 also reports ‘DOMException: Permission denied’, however, I would like to handle this in a more user-friendly fashion.
If there is no callback, what is the best practice for handling media failure with createCapture as it doesn’t seem to be discussed in the docs.
Ok so this answer is more than a year late but posting this as it may be useful for others stuck on the same issue. Rather than error testing the capture itself as suggested in comments below or perhaps reworking createCapture() (best done through opening an issue request if you feel it should be changed) I suggest testing the pixels array of the capture and only if it has been set then proceeding with doing whatever your script does. This could be done simply like so:
//load pixel data of webcam capture
cap.loadPixels();
//all values in the pixel array start as zero so just test if they are
//greater than zero
if (cap.pixels[1] > 0)
{
//put the rest of your script here
}
A full example of this in action is below:
var canvas;
var cap;
var xpos = 0;
function setup()
{
canvas = createCanvas(windowWidth, windowHeight);
canvas.style("display", "block");
background("#666666");
//create an instance of the webcam
cap = createCapture(VIDEO);
cap.size(640, 480);
cap.hide();
}
function draw()
{
//load pixel data of webcam capture
cap.loadPixels();
//if the first pixel's R value is set continue with the rest of the script
if (cap.pixels[1] > 0)
{
var w = cap.width;
var h = cap.height;
copy(cap, w/2, 0, 10, h, xpos, (height / 2) - (h / 2), 10, h);
xpos = xpos + 1;
if (xpos > width) xpos = 0;
}
}
I believe you can use a try and catch to detect when you get an error. Something like this:
try{
capture = createCapture(VIDEO);
}
catch(error){
// error handling here
}
More info on W3Schools and MDN.
I need to loop my source with some cross parametr (in sec). It will be great to listen looping without interrupting on the sample border.AudioBufferSourceNode is audioNode in my code.
I faced with the problem of inability to reuse the buffer, is it possible to get around this?
playNoteOn: function(indexNote){
var attack = this.get('attack'),
release = this.get('release'),
volume = 1 + this.get('volume') / 100,
reverb = _.clone(this.get('reverb')),
loop = this.get('loop'), cross;
//peace for Loop process
if (loop) {
//milli sec
attack = this.get('startLoop')*1000;
release = this.get('endLoop')*1000;
//sec
cross = this.get('crossLoop');
}
//peace for ADSR process
var t0 = this.get('audioNode').context.currentTime,
spread = attack/1000 + release/1000,
attackSpread = t0 + attack/1000;
[this.get('schema').leftGain, this.get('schema').rightGain].forEach(function(gain, index){
gain.gain.cancelScheduledValues(0);
gain.gain.setValueAtTime(0, t0);
gain.gain.linearRampToValueAtTime(volume, attackSpread);
// gain.gain.setValueAtTime(volume, decaySpread);
// gain.gain.linearRampToValueAtTime(0, releaseSpread);
});
this.get('audioNode').connect(this.get('schema').splitter, 0, 0);
this.get('audioNode').connect(this.get('schema').leftGain);
this.get('audioNode').connect(this.get('schema').rightGain);
this.get('audioNode').connect(this.get('schema').reverb);
this.get('audioNode').connect(APP.Models.Synth.get('schema').reverb);
APP.Models.Synth.get('effects').where({active: false}).forEach(function(effect){
effect.get('node').disconnect();
});
APP.Models.Synth.get('effects').where({active: true}).forEach(function(effect){
effect.get('node').disconnect();
effect.get('node').setParams(effect.toJSON()).getNode(this.get('audioNode'), [this.get('schema').leftGain, this.get('schema').rightGain]);
}, this);
if(loop){
this.get('audioNode').loop = true;
this.get('audioNode').loopEnd = this.get('audioNode').buffer.duration - cross;
}
this.get('audioNode').start(t0);
},
You cannot reuse a buffer. Once stopped the sourcebuffer is gone forever. Where is your audio node object anyway? But that's no problem. When you decoded from a sound file you can use the buffer from the decoding again. Just create more buffer sources. You can create as much as you like. What language are you using anyway? Say Some backgrounds to what you are doing and which frameworks you use.
Beware the difference between buffer from decoding and buffer source. Your audionode is a buffer source which can be feeded by a buffer. You can reuse the buffer but not the buffer source. So create the buffer source in your playnote code.
I'm working in Eclipse and I want to know if I can make an if statement that checks to see if the BufferedImage has been painted/drawn onto the frame. For some reason, it's not painting the correct image because clickable regions are appearing on that picture when they are not supposed to.
For example, when I click the region to go from 4>5 everything is good. When I click from 5 to go to 4 I end up at 6 because the 'regions' from 4 are appearing in 5 (The image should always be painted before the clickable regions are shown) before it's even being painted. I want to restrict this to check if the image has been painted onto the frame first.
I really don't want to use anything else besides what I have right now (so no new classes being implemented to do this task), I really just want a simple yet effective way to resolve this. Here is what I'm talking about:
...
MouseAdapter mouseHandler = new MouseAdapter()
{
public void mouseClicked(MouseEvent e)
{
repaint();
if(n==0)
{
if(e.getX()>=459 && e.getX()<491 && e.getY()>=111 && e.getY()<133
{
n = 4;
}
return;
}
if(n==5)
{
if(...)
{
n = 4;
}
return();
}
if(n==6)
{
if(...)
{
n = 5;
}
if(...)
{
n = 0;
}
if(...)
{
n = 6;
}
return();
}
}
...
I think you might need to give a little more information. The problem might lie in how you repaint, not whether it was painted.
If you are running another thread as your main program, you might instead want to send the mouse events synchronously to that so that the main thread can process the mouse click and then repaint.
Another solution might be to override the repaint method and paint the buffered images there, but you may have done that.
Also, a little off topic, I noticed that you used for loops to determine if the mouse was clicked in a specific area.
You could shorten the code:
for(int i=459; i<491; i++){
if(e.getX()==i){
for(int j=111; j<133; j++){
if(e.getY()==j){
//action taken
}
}
}
}
To:
if(e.getX()>=459 && e.getX()<491 && e.getY()>=111 && e.getY()<133{
//action taken
}
This would take up less space in your code and less time checking every pixel.
Back to your question.
I dont know of a function to tell if a buffered image has been painted. The ploblem that you are having though might of might not be in the code provided. Posting the rest of your code would be beneficial.
Okay I found the solution, I forgot to come back to this question and let you know. The problem was that the mouse was double clicking for some reason. You could almost say it was 'recursive'. I decided to move the mouseListener from inside the paintComponent to outside of it, and surely enough that fixed it.
I recently found that glDrawArrays allocating and releasing huge amounts of memory on every frame.
I suspect that it's related to "Shaders compiled outside of initialization" issue reported by openGL profiler. That occurs on every frame! Should it be only once, and after shaders are compiled, disappear?
EDIT: I also double checked that my vertex are properly aligned. So I'm really confused what memory driver needs to allocate on every frame.
EDIT #2: I'm using VBO's and degenerated triangle strips to render sprites and . I'm passing geometry on every frame (GL_STREAM_DRAW).
EDIT #3:
I think I'm close to issue but still unable to solve it. Problem disappears if I pass same texture id value to shader (see source code comment). Somehow this issue is relate to fragment shader I think.
In my sprite batch I have list of sprites and I render them by texture id and FIFO queue.
Here's source code of my sprite batch class:
void spriteBatch::renderInRange(shader& prog, int start, int count){
int curTexture = textures[start];
int startFrom = start;
//Looping through all vertexes and rendering them by texture id's
for(int i=start;i<start+count;++i){
if(textures[i] != curTexture || i == (start + count) -1){
//Problem occurs after decommenting this line
// prog.setUniform("texture", curTexture-1);
prog.setUniform("texture", 0); // if I pass same texture id everything is OK
int startVertex = startFrom * vertexesPerSprite;
int cnt = ((i - startFrom) * vertexesPerSprite);
//If last one has same texture we just adding it
//to last render call
if(i == (start + count) - 1 && textures[i] == curTexture)
cnt = ((i + 1) - startFrom) * vertexesPerSprite;
render(vbo, GL_TRIANGLE_STRIP, startVertex+1, cnt-1);
//if last element has different texture
//we need to render it separately
if(i == (start + count) - 1 && textures[i] != curTexture){
// prog.setUniform("texture", textures[i]-1);
render(vbo, GL_TRIANGLE_STRIP, (i * vertexesPerSprite) + 1, 5);
}
curTexture = textures[i];
startFrom = i;
}
}
}
inline GLint getUniformLocation(GLuint shaderID, const string& name) {
GLint iLocation = glGetUniformLocation(shaderID, name.data());
if(iLocation == -1){ // shader variable not found
stringstream errorText;
errorText << "Uniform \"" << name << " was not found!";
throw logic_error(errorText.str());
}
return iLocation;
}
void shader::setUniform(const string& name, const matrix& value) {
GLint location = getUniformLocation(this->programID, name.data());
glUniformMatrix4fv(location, 1, GL_FALSE, &(value[0]));
}
void shader::setUniform(const string& name, int value) {
GLint iLocation = getUniformLocation(this->programID, name.data());
//GLenum error = glGetError();
glUniform1i(iLocation, value);
// error = glGetError();
}
EDIT#4: I tried to profile app on IOS 6 and Iphone5 and allocations are much bigger. But methods are different in this case. I'm attaching new screenshot.
Issue is resolved by creating separate shader for each texture.
It looks like bug in driver implementation that does happen on all IOS devices (I tested on IOS 5/6). However on higher iPhone models it's not that noticeable.
On iPhone4 performance hit was very significant from 60 FPS to 38!
More code would help, but have you checked to see if the amount of memory involved is comparable to the amount of geometry you're updating? (although that would seem like a lot of geometry!) It looks like GL is holding your update until glDrawArrays, releasing it when it can be pulled into internal GL state.
If you can run the code in a MacOS app, the OpenGL Profiler tool may be able to further isolate the condition. (look in XCode documentation for more info, if you're not familiar with this tool). I'd also suggest looking at texture use, given the amount of memory involved.
The easiest thing to do might be to conditionally break on malloc() for a large allocation, note the address, and examine what's been loaded there.
try to query the texture uniform just once (in initialization) and cache it. calling "glGetUniformLocation" too much in one frame will hammer the performance (depending on the sprite count).