I'm trying to optimize some mesh generation using MeshData & the Job System, but for some reason when I try to use 2 params in meshData.SetVertexBufferParams, the resulting meshData.GetVertexData is half the length it should be (I set the vertex count to 5120, but the resulting VertexData NativeArray is only 2560 items long).
When I force it to be double the length (SetVertexBufferParams(numVerts * 2, ...)), it creates a mesh that appears to treat the norms and vert positions as all position data and also makes the screen go black so no screen shot.
Here's my code:
// generate 256 height values
int[] arr = new int[256];
for (int i = 0; i < arr.Length; i++)
{
arr[i] = (int) (Mathf.PerlinNoise(i / 16 / 16f, i % 16 / 16f) * 5);
}
// put it in a NativeArray
NativeArray<int> heights = new NativeArray<int>(arr, Allocator.TempJob);
// 4 verts per face * 5 faces = 20
int numVerts = heights.Length * 20; // this value is always 5120
// 2 tris per face * 5 daces * 3 indices = 30
int indices = heights.Length * 30;
// MeshData setup
Mesh.MeshDataArray meshDataArray = Mesh.AllocateWritableMeshData(1);
Mesh.MeshData meshData = meshDataArray[0];
meshData.SetVertexBufferParams(numVerts,
new VertexAttributeDescriptor(VertexAttribute.Position, VertexAttributeFormat.Float32, 3, stream:0),
new VertexAttributeDescriptor(VertexAttribute.Normal, VertexAttributeFormat.Float32, 3, stream:1)
);
meshData.SetIndexBufferParams(indices, IndexFormat.UInt16);
// Create job
Job job = new Job
{
Heights = heights,
MeshData = meshData
};
// run job
job.Schedule().Complete();
// struct I'm using for vertex data
[System.Runtime.InteropServices.StructLayout(System.Runtime.InteropServices.LayoutKind.Sequential)]
public struct VData
{
public float3 Vert;
public float3 Norm;
}
// Here's some parts of the job
public struct Job : IJob
{
[ReadOnly]
public NativeArray<int> Heights;
public Mesh.MeshData MeshData;
public void Execute()
{
NativeArray<VData> Verts = MeshData.GetVertexData<VData>();
NativeArray<ushort> Tris = MeshData.GetIndexData<ushort>();
// loops from 0 to 255
for (int i = 0; i < Heights.Length; i++)
{
ushort t1 = (ushort)(w1 + 16);
// This indicates that Verts.Length is 2560 when it should be 5120
Debug.Log(Verts.Length);
int t = i * 30; // tris
int height = Heights[i];
// x and y coordinate in chunk
int x = i / 16;
int y = i % 16;
float3 up = new float3(0, 1, 0);
// This throws and index out of bounds error because t1 becomes larger than Verts.Length
Verts[t1] = new VData { Vert = new float3(x + 1, height, y + 1), Norm = up};
// ...
}
}
}
meshData.SetVertexBufferParams(numVerts,
new VertexAttributeDescriptor(VertexAttribute.Position, VertexAttributeFormat.Float32, 3, stream:0),
new VertexAttributeDescriptor(VertexAttribute.Normal, VertexAttributeFormat.Float32, 3, stream:1)
);
Your SetVertexBufferParams here places VertexAttribute.Position and VertexAttribute.Normal on a separate streams thus halving the size of the buffer per stream and later the length of the buffers if buffer becomes reinterpreted with the wrong struct by mistake.
This is how documentation explains streams:
Vertex data is laid out in separate "streams" (each stream goes into a separate vertex buffer in the underlying graphics API). While Unity supports up to 4 vertex streams, most meshes use just one. Separate streams are most useful when some vertex attributes don't need to be processed, for example skinned meshes often use two vertex streams (one containing all the skinned data: positions, normals, tangents; while the other stream contains all the non-skinned data: colors and texture coordinates).
But why it might end up re-interpreted as half the length? Well, because of this line:
NativeArray<VData> Verts = MeshData.GetVertexData<VData>();
How? Because there is a implicit stream parameter value there (doc)
public NativeArray<T> GetVertexData(int stream = 0);
and it defaults to 0. So what happens here is this:
var Verts = Positions_Only.Reinterpret<Position_And_Normals>();
or in other words:
var Verts = NativeArray<float3>().Reinterpret<float3x2>();
case solved :T
TL;DR:
Change stream:1 to stream:0 so both vertex attributes end up on the same stream.
or var Positions = MeshData.GetVertexData<float3>(0); & var Normals = MeshData.GetVertexData<float3>(1);
or create a dedicated VData struct per stream var Stream0 = MeshData.GetVertexData<VStream0>(0); & var Stream1 = MeshData.GetVertexData<VStream1>(1);
I have a 4x4x4 3DTexture which I am initializing and showing correctly to color my 4x4x4 grid of vertices (see attached red grid with one white pixel - 0,0,0).
However when I render the 4 layers in a framebuffer (all four at one time using gl.COLOR_ATTACHMENT0 --> gl.COLOR_ATTACHMENT3, only four of the sixteen pixels on a layer are successfully rendered by my fragment shader (to be turned green).
When I only do one layer, with gl.COLOR_ATTACHMENT0, the same 4 pixels show up correctly altered for the 1 layer, and the other 3 layers stay with the original color unchanged. When I change the gl.viewport(0, 0, size, size) (size = 4 in this example), to something else like the whole screen, or different sizes than 4, then different pixels are written, but never more than 4. My goal is to individually specify all 16 pixels of each layer precisely. I'm using colors for now, as a learning experience, but the texture is really for position and velocity information for each vertex for a physics simulation. I'm assuming (faulty assumption?) with 64 points/vertices, that I'm running the vertex shader and the fragment shader 64 times each, coloring one pixel each invocation.
I've removed all but the vital code from the shaders. I've left the javascript unaltered. I suspect my problem is initializing and passing the array of vertex positions incorrectly.
//Set x,y position coordinates to be used to extract data from one plane of our data cube
//remember, z we handle as a 1 layer of our cube which is composed of a stack of x-y planes.
const oneLayerVertices = new Float32Array(size * size * 2);
count = 0;
for (var j = 0; j < (size); j++) {
for (var i = 0; i < (size); i++) {
oneLayerVertices[count] = i;
count++;
oneLayerVertices[count] = j;
count++;
//oneLayerVertices[count] = 0;
//count++;
//oneLayerVertices[count] = 0;
//count++;
}
}
const bufferInfo = twgl.createBufferInfoFromArrays(gl, {
position: {
numComponents: 2,
data: oneLayerVertices,
},
});
And then I'm using the bufferInfo as follows:
gl.useProgram(computeProgramInfo.program);
twgl.setBuffersAndAttributes(gl, computeProgramInfo, bufferInfo);
gl.viewport(0, 0, size, size); //remember size = 4
outFramebuffers.forEach((fb, ndx) => {
gl.bindFramebuffer(gl.FRAMEBUFFER, fb);
gl.drawBuffers([
gl.COLOR_ATTACHMENT0,
gl.COLOR_ATTACHMENT1,
gl.COLOR_ATTACHMENT2,
gl.COLOR_ATTACHMENT3
]);
const baseLayerTexCoord = (ndx * numLayersPerFramebuffer);
console.log("My baseLayerTexCoord is "+baseLayerTexCoord);
twgl.setUniforms(computeProgramInfo, {
baseLayerTexCoord,
u_kernel: [
0, 0, 0,
0, 0, 0,
0, 0, 0,
0, 0, 1,
0, 0, 0,
0, 0, 0,
0, 0, 0,
0, 0, 0,
0, 0, 0,
],
u_position: inPos,
u_velocity: inVel,
loopCounter: loopCounter,
numLayersPerFramebuffer: numLayersPerFramebuffer
});
gl.drawArrays(gl.POINTS, 0, (16));
});
VERTEX SHADER:
calc_vertex:
const compute_vs = `#version 300 es
precision highp float;
in vec4 position;
void main() {
gl_Position = position;
}
`;
FRAGMENT SHADER:
calc_fragment:
const compute_fs = `#version 300 es
precision highp float;
out vec4 ourOutput[4];
void main() {
ourOutput[0] = vec4(0,1,0,1);
ourOutput[1] = vec4(0,1,0,1);
ourOutput[2] = vec4(0,1,0,1);
ourOutput[3] = vec4(0,1,0,1);
}
`;
I’m not sure what you’re trying to do and what you think the positions will do.
You have 2 options for GPU simulation in WebGL2
use transform feedback.
In this case you pass in attributes and generate data in buffers. Effectively you have in attributes and out attributes and generally you only run the vertex shader. To put it another way your varyings, the output of your vertex shader, get written to a buffer. So you have at least 2 sets of buffers, currentState, and nextState and your vertex shader reads attributes from currentState and writes them to nextState
There is an example of writing to buffers via transform feedback here though that example only uses transform feedback at the start to fill buffers once.
use textures attached to framebuffers
in this case, similarly you have 2 textures, currentState, and nextState, You set nextState to be your render target and read from currentState to generate next state.
the difficulty is that you can only render to textures by outputting primitives in the vertex shader. If currentState and nextState are 2D textures that’s trival. Just output a -1.0 to +1.0 quad from the vertex shader and all pixels in nextState will be rendered to.
If you’re using a 3D texture then same thing except you can only render to 4 layers at a time (well, gl.getParameter(gl.MAX_DRAW_BUFFERS)). so you’d have to do something like
for(let layer = 0; layer < numLayers; layer += 4) {
// setup framebuffer to use these 4 layers
gl.drawXXX(...) // draw to 4 layers)
}
or better
// at init time
const fbs = [];
for(let layer = 0; layer < numLayers; layer += 4) {
fbs.push(createFramebufferForThese4Layers(layer);
}
// at draw time
fbs.forEach((fb, ndx) => {;
gl.bindFramebuffer(gl.FRAMEBUFFER, fb);
gl.drawXXX(...) // draw to 4 layers)
});
I’m guessing multiple draw calls is slower than one draw call so another solution is to instead treat a 2D texture as a 3D array and calculate texture coordinates appropriately.
I don’t know which is better. If you’re simulating particles and they only need to look at their own currentState then transform feedback is easier. If need each particle to be able to look at the state of other particles, in other words you need random access to all the data, then your only option is to store the data in textures.
As for positions I don't understand your code. Positions define a primitives, either POINTS, LINES, or TRIANGLES so how does passing integer X, Y values into our vertex shader help you define POINTS, LINES or TRIANGLES?
It looks like you're trying to use POINTS in which case you need to set gl_PointSize to the size of the point you want to draw (1.0) and you need to convert those positions into clip space
gl_Position = vec4((position.xy + 0.5) / resolution, 0, 1);
where resolution is the size of the texture.
But doing it this way will be slow. Much better to just draw a full size (-1 to +1) clip space quad. For every pixel in the destination the fragment shader will be called. gl_FragCoord.xy will be the location of the center of the pixel currently being rendered so first pixel in bottom left corner gl_FragCoord.xy will be (0.5, 0.5). The pixel to the right of that will be (1.5, 0.5). The pixel to the right of that will be (2.5, 0.5). You can use that value to calculate how to access currentState. Assuming 1x1 mapping the easiest way would be
int n = numberOfLayerThatsAttachedToCOLOR_ATTACHMENT0;
vec4 currentStateValueForLayerN = texelFetch(
currentStateTexture, ivec3(gl_FragCoord.xy, n + 0), 0);
vec4 currentStateValueForLayerNPlus1 = texelFetch(
currentStateTexture, ivec3(gl_FragCoord.xy, n + 1), 0);
vec4 currentStateValueForLayerNPlus2 = texelFetch(
currentStateTexture, ivec3(gl_FragCoord.xy, n + 2), 0);
...
vec4 nextStateForLayerN = computeNextStateFromCurrentState(currentStateValueForLayerN);
vec4 nextStateForLayerNPlus1 = computeNextStateFromCurrentState(currentStateValueForLayerNPlus1);
vec4 nextStateForLayerNPlus2 = computeNextStateFromCurrentState(currentStateValueForLayerNPlus2);
...
outColor[0] = nextStateForLayerN;
outColor[1] = nextStateForLayerNPlus1;
outColor[2] = nextStateForLayerNPlus1;
...
I don’t know if you needed this but just to test here’s a simple example that renders a different color to every pixel of a 4x4x4 texture and then displays them.
const pointVS = `
#version 300 es
uniform int size;
uniform highp sampler3D tex;
out vec4 v_color;
void main() {
int x = gl_VertexID % size;
int y = (gl_VertexID / size) % size;
int z = gl_VertexID / (size * size);
v_color = texelFetch(tex, ivec3(x, y, z), 0);
gl_PointSize = 8.0;
vec3 normPos = vec3(x, y, z) / float(size);
gl_Position = vec4(
mix(-0.9, 0.6, normPos.x) + mix(0.0, 0.3, normPos.y),
mix(-0.6, 0.9, normPos.z) + mix(0.0, -0.3, normPos.y),
0,
1);
}
`;
const pointFS = `
#version 300 es
precision highp float;
in vec4 v_color;
out vec4 outColor;
void main() {
outColor = v_color;
}
`;
const rtVS = `
#version 300 es
in vec4 position;
void main() {
gl_Position = position;
}
`;
const rtFS = `
#version 300 es
precision highp float;
uniform vec2 resolution;
out vec4 outColor[4];
void main() {
vec2 xy = gl_FragCoord.xy / resolution;
outColor[0] = vec4(1, 0, xy.x, 1);
outColor[1] = vec4(0.5, xy.yx, 1);
outColor[2] = vec4(xy, 0, 1);
outColor[3] = vec4(1, vec2(1) - xy, 1);
}
`;
function main() {
const gl = document.querySelector('canvas').getContext('webgl2');
if (!gl) {
return alert('need webgl2');
}
const pointProgramInfo = twgl.createProgramInfo(gl, [pointVS, pointFS]);
const rtProgramInfo = twgl.createProgramInfo(gl, [rtVS, rtFS]);
const size = 4;
const numPoints = size * size * size;
const tex = twgl.createTexture(gl, {
target: gl.TEXTURE_3D,
width: size,
height: size,
depth: size,
});
const clipspaceFullSizeQuadBufferInfo = twgl.createBufferInfoFromArrays(gl, {
position: {
data: [
-1, -1,
1, -1,
-1, 1,
-1, 1,
1, -1,
1, 1,
],
numComponents: 2,
},
});
const fb = gl.createFramebuffer();
gl.bindFramebuffer(gl.FRAMEBUFFER, fb);
for (let i = 0; i < 4; ++i) {
gl.framebufferTextureLayer(
gl.FRAMEBUFFER,
gl.COLOR_ATTACHMENT0 + i,
tex,
0, // mip level
i, // layer
);
}
gl.drawBuffers([
gl.COLOR_ATTACHMENT0,
gl.COLOR_ATTACHMENT1,
gl.COLOR_ATTACHMENT2,
gl.COLOR_ATTACHMENT3,
]);
gl.viewport(0, 0, size, size);
gl.useProgram(rtProgramInfo.program);
twgl.setBuffersAndAttributes(
gl,
rtProgramInfo,
clipspaceFullSizeQuadBufferInfo);
twgl.setUniforms(rtProgramInfo, {
resolution: [size, size],
});
twgl.drawBufferInfo(gl, clipspaceFullSizeQuadBufferInfo);
gl.bindFramebuffer(gl.FRAMEBUFFER, null);
gl.viewport(0, 0, gl.canvas.width, gl.canvas.height);
gl.drawBuffers([
gl.BACK,
]);
gl.useProgram(pointProgramInfo.program);
twgl.setUniforms(pointProgramInfo, {
tex,
size,
});
gl.drawArrays(gl.POINTS, 0, numPoints);
}
main();
<canvas></canvas>
<script src="https://twgljs.org/dist/4.x/twgl-full.min.js"></script>
I'm a new ILNumerics Visualization Engine user and I'm still coming up to speed on how to use it well. I've searched extensively for how to update the z-values of an ILSurface and read the posts, but I'm still not clear on how to do this.
I'm able to generate a surface and set up a camera to view it (Hamyo Kutschbach told me that's the best way to ensure that the aspect ratios of the surface don't change when rotating the surface, which is important in my application). Here's the code that displays a sin(x)/x function:
// Generate the data
ILArray<double> z = SincFunc(rows, cols, 10, 50);
ILArray<double> x = new double[cols];
ILArray<double> y = new double[rows];
for (int i = 0; i < cols; i++)
x[i] = (double)i;
for (int i = 0; i < rows; i++)
y[i] = (double)i;
// create the scene
scene = new ILScene();
pointCloudSurface = new ILSurface(z, x, y)
{
Colormap = Colormaps.Jet,
UseLighting = true,
Wireframe = { Visible = false },
Children = { new ILColorbar()
{
Height = 0.5f,
Location = new PointF(0.95f, 0.05f),
Children = { new ILLabel("microns") { Position = new Vector3(0.5,1,0), Anchor = new PointF(0.5f,0) } } }
},
Alpha = 1.0f
};
// Configure the surface and display it
scene.Camera.Add(pointCloudSurface);
scene.Camera.Position = new Vector3(50, 50, 700);
scene.Camera.LookAt = new Vector3(50, 50, 0);
scene.Camera.Top = new Vector3(0, 0, 700);
scene.Camera.Projection = Projection.Perspective;
scene.Camera.ZNear = 1.0f;
scene.Camera.ZFar = 0.0f;
scene.Camera.Top = new Vector3(1, 0, 0);
// Turn off the Powered by ILNumerics label
scene.Screen.First<ILLabel>().Visible = false;
ilPanel1.Scene = scene;
ilPanel1.Configure();
ilPanel1.Refresh();
And it works well. So now I want to change the z-values and update the plot without closing ilPanel1 because this plot is embedded in a Windows Form. Advice would be appreciated! Hopefully other newbies will find this post useful as well.
After further rummaging around, I came across a method, UpdateColormapped(), that does the trick. It's placed near the end of the code above like this:
scene.Camera.First<ILSurface>().UpdateColormapped(z);
ilPanel1.Scene = scene;
ilPanel1.Configure();
ilPanel1.Refresh();
It can be found in the API documentation here: UpdateColormapped()
It can also change the x and y data and perform other mods, but it requires that the z data be a float array, so if you're working double precision, you'll have to take the appropriate steps to get it into a float array.
I'm trying to get the Tango's camera stream in order to combine an homemade AR Kit to Tango.
I'm stuck at a point where everything works as intended in Tango's editor emulation, but not in the app pushed to the tablet.
The code I'm using is the following:
YUVTexture yuvTexture = m_tangoApplication.GetVideoOverlayTextureYUV();
Texture2D yTexture = yuvTexture.m_videoOverlayTextureY;
// m_videoOverlayTextureCr is not used by Tango yet for some reason
Texture2D uvTexture = yuvTexture.m_videoOverlayTextureCb;
// convert from YV12 to RGB
for (int i = 0; i < yTexture.height; ++i)
{
for (int j = 0; j < yTexture.width; ++j)
{
Color yPixel = yTexture.GetPixel(j, i);
Color uvPixel = uvTexture.GetPixel(j, i);
m_texture.SetPixel(4 * j + 0, yTexture.height - i - 1, YUV2Color(yPixel.r, uvPixel.r, uvPixel.g));
m_texture.SetPixel(4 * j + 1, yTexture.height - i - 1, YUV2Color(yPixel.g, uvPixel.r, uvPixel.g));
m_texture.SetPixel(4 * j + 2, yTexture.height - i - 1, YUV2Color(yPixel.b, uvPixel.b, uvPixel.a));
m_texture.SetPixel(4 * j + 3, yTexture.height - i - 1, YUV2Color(yPixel.a, uvPixel.b, uvPixel.a));
}
}
YUV2Color (extracted from Tango's YUV2RGB Shader):
public static Color YUV2Color(float y_value, float u_value, float v_value)
{
float r = y_value + 1.370705f * (v_value - 0.5f);
float g = y_value - 0.698001f * (v_value - 0.5f) - (0.337633f * (u_value - 0.5f));
float b = y_value + 1.732446f * (u_value - 0.5f);
return new Color(r, g, b, 1f);
}
Did someone already solved this problem? I've seen a lot of post related to it when the ITangoVideoOverlay was mostly used, but nothing with the current IExperimentalTangoVideoOverlay
I've experimented a lot of things, so far it has been the closest I got to what I expected ... Any help would be highly appreciated.
You are using the Texture ID method to get the YUV texture color, this is not very common to do. A easier path would be using the Raw Byte buffer method to get color camera image, to do that:
On TangoManager prefab, enable video overlay, and select Raw Byte method from the drop down box.
Register to ITangoVideoOverlay interface.
Convert the image buffer data from YUV to RGB, this part is exactly like the YUV2Color function, but use data from TangoImageData.data
I'm new in JavaCV and want to create a Histogram from a image.
I tried to translate some C++ code into Java but JavaCV don't have methods like cvCVtPixToPlane.
Can someone help me to create a histogram?
a translatet OpenCV-Code from the OpenCV-Wiki must be run.
I give you my code snipet to create a 1D-Diagram from the h-channel:
private CvHistogram getHueHistogram(IplImage image){
if(image==null || image.nChannels()<3) new Exception("Error!");
IplImage hsvImage= cvCreateImage(image.cvSize(), image.depth(), 3);
cvCvtColor(image, hsvImage, CV_BGR2HSV);
// Split the 3 channels into 3 images
IplImageArray hsvChannels = splitChannels(hsvImage);
//bins and value-range
numberOfBins=255;
float minRange= 0f;
float maxRange= 180f;
// Allocate histogram object
int dims = 1;
int[]sizes = new int[]{numberOfBins};
int histType = CV_HIST_ARRAY;
float[] minMax = new float[]{minRange, maxRange};
float[][] ranges = new float[][]{minMax};
int uniform = 1;
CvHistogram hist = cvCreateHist(dims, sizes, histType, ranges, uniform);
// Compute histogram
int accumulate = 1;
IplImage mask = null;
cvCalcHist(hsvChannels.position(0),hist, accumulate, null);
return hist;
}
And my splitChannels-Method, i used in this snipet:
private IplImageArray splitChannels(IplImage hsvImage) {
CvSize size = hsvImage.cvSize();
int depth=hsvImage.depth();
IplImage channel0 = cvCreateImage(size, depth, 1);
IplImage channel1 = cvCreateImage(size, depth, 1);
IplImage channel2 = cvCreateImage(size, depth, 1);
cvSplit(hsvImage, channel0, channel1, channel2, null);
return new IplImageArray(channel0, channel1, channel2);
}
If you want to draw a picture from the histogram you can iterate the bins. With cvQueryHistValue_1D() you can get the sum of pixels from bin_i