I am currently testing options for depth measurement with the smartphone and wanted to create a depth image initially for testing. I am using the Camera2Basic example as a basis for this. (https://github.com/android/camera-samples/tree/main/Camera2Basic) Using Depth16 I get a relatively sharp "depth image" back. But the millimetres are not correct. They are in a range around from 3600mm to 5000mm for an object like a wall that is about 500mm or 800mm away from the camera.
But what puzzles me the most is that the image does not transmit any information in the dark. If Android is really targeting the ToF sensor for DEPTH16, it shouldn't be affected in the dark, should it? Or do I have to use AR-Core or Huawei's HMS core to get a real ToF image?
I am using a Huawei P30 Pro and the code for extracting the depth information looks like this. And yes performance wise it is bullshit but it is only for testing purposes:)
private Map<String, PixelData> parseDepth16IntoDistanceMap(Image image) {
Map<String, PixelData> map = new HashMap();
Image.Plane plane = image.getPlanes()[0];
// using asShortBuffer() like in the documentation leads to a wrong format (for me) but does not help getting better values
ByteBuffer depthBuffer = plane.getBuffer().order(ByteOrder.nativeOrder());
int stride = plane.getRowStride();
int offset = 0;
int i = 0;
for (short y = 0; y < image.getHeight(); y++) {
for (short x = 0; x < image.getWidth(); x++) {
short depthSample = depthBuffer.getShort( (y / 2) * stride + x);
short depthSampleShort = (short) depthSample;
short depthRange = (short) (depthSampleShort & 0x1FFF);
short depthConfidence = (short) ((depthSampleShort >> 13) & 0x7);
float depthPercentage = depthConfidence == 0 ? 1.f : (depthConfidence - 1) / 7.f;
maxz = depthRange;
sum = sum + depthRange;
numPoints++;
listOfRanges.add((float) depthRange);
if (depthRange < minz && depthRange > 0) {
minz = depthRange;
}
map.put(x + "_" + y, new PixelData(x, y, depthRange, depthPercentage));
i++;
}
}
return map;
}
In any case, it would help a lot to know if you can get the data this way at all, so I know if I'm already doing something fundamentally wrong. Otherwise I will change to one of the ar systems. Either way, many thanks for your efforts
If you want to extract a depth map where you can see the distance to an object you might use ARCORE Depth API.
https://developers.google.com/ar/develop/java/depth/overview
Or you can follow the codelab where shows you how to get the data in millimeters.
https://codelabs.developers.google.com/codelabs/arcore-depth#0
Related
To be clear, I'm very new to STM32 and MBED programming, but is it possible to create valid VGA signal, using STM32 nucleo-F070RB board? I've picked up this standard, and my goal is to display "something" on screen. With this i mean I should have control which pixel i want to, well, turn on.
For demo, here is my (very crude) sketch:
#include <mbed.h>
DigitalOut led(LED1);
DigitalOut h_sync(PC_3);
DigitalOut v_sync(PC_2);
DigitalOut c_red(PC_0);
int main() {
h_sync = 1;
v_sync = 1;
int line_count = 0;
int color_red = 0;
while(1) {
wait_us(21);
h_sync = 0;
wait_us(3);
h_sync = 1;
wait_ns(2);
line_count++;
color_red++;
if (color_red == 16) { color_red = 0; c_red = !c_red; }
if (line_count == 601) v_sync = 0;
if (line_count == 605) v_sync = 1;
if (line_count == 628) { line_count = 0; c_red = 0; color_red = 0; }
}
}
I have connected V-Sync to V-Sync of my monitor, same with H-Sync and c_red (color red) through 560Ohm resistor to RED signal. And it (kind of) worked! It displayed red strips every 16 lines. Perfect, but I need to be able to control every pixel (if it's possible). I've seen some VGA libraries (maybe), but i really need to write it myself - something very crude. I just only want to have some sort of control over pixels, not the some super-super something (Just for me to learn :) ). And because I don't have much experience with STMs, after hours i was not able to "convince" my board to generate such "high-speed" signals, so, it is after all possible?
I was using MBED's Ticker function to generate the timing for each pixel, but it did not work - the fastest the Ticker went for me was something around few miliseconds, far too much. Can I use timer interrupts? Or something else?
I am implementing this neural network for some classification problem. I initially tried back propagation but it takes longer to converge. So I though of using RPROP. In my test setup RPROP works fine for AND gate simulation but never converges for OR and XOR gate simulation.
How and when should I update bias for RPROP?
Here my weight update logic:
for(int l_index = 1; l_index < _total_layers; l_index++){
Layer* curr_layer = get_layer_at(l_index);
//iterate through each neuron
for (unsigned int n_index = 0; n_index < curr_layer->get_number_of_neurons(); n_index++) {
Neuron* jth_neuron = curr_layer->get_neuron_at(n_index);
double change = jth_neuron->get_change();
double curr_gradient = jth_neuron->get_gradient();
double last_gradient = jth_neuron->get_last_gradient();
int grad_sign = sign(curr_gradient * last_gradient);
//iterate through each weight of the neuron
for(int w_index = 0; w_index < jth_neuron->get_number_of_weights(); w_index++){
double current_weight = jth_neuron->give_weight_at(w_index);
double last_update_value = jth_neuron->give_update_value_at(w_index);
double new_update_value = last_update_value;
if(grad_sign > 0){
new_update_value = min(last_update_value*1.2, 50.0);
change = sign(curr_gradient) * new_update_value;
}else if(grad_sign < 0){
new_update_value = max(last_update_value*0.5, 1e-6);
change = -change;
curr_gradient = 0.0;
}else if(grad_sign == 0){
change = sign(curr_gradient) * new_update_value;
}
//Update neuron values
jth_neuron->set_change(change);
jth_neuron->update_weight_at((current_weight + change), w_index);
jth_neuron->set_last_gradient(curr_gradient);
jth_neuron->update_update_value_at(new_update_value, w_index);
double current_bias = jth_neuron->get_bias();
jth_neuron->set_bias(current_bias + _learning_rate * jth_neuron->get_delta());
}
}
}
In principal you don't treat the bias differently than before when you did backpropagation. It's learning_rate * delta which you seem to be doing.
One source of error may be that the sign of the weight change depends on how you calculate your error. There's different conventions and (t_i-y_i) instead of (y_i - t_i) should result in returning (new_update_value * sgn(grad)) instead of -(new_update_value * sign(grad)) so try switching the sign. I'm also unsure about how you specifically implemented everything since a lot is not shown here. But here's a snippet of mine in a Java implementation that might be of help:
// gradient didn't change sign:
if(weight.previousErrorGradient * errorGradient > 0)
weight.lastUpdateValue = Math.min(weight.lastUpdateValue * step_pos, update_max);
// changed sign:
else if(weight.previousErrorGradient * errorGradient < 0)
{
weight.lastUpdateValue = Math.max(weight.lastUpdateValue * step_neg, update_min);
}
else
weight.lastUpdateValue = weight.lastUpdateValue; // no change
// Depending on language, you should check for NaN here.
// multiply this with -1 depending on your error signal's sign:
return ( weight.lastUpdateValue * Math.signum(errorGradient) );
Also, keep in mind that 50.0, 1e-6 and especially 0.5, 1.2 are empirically gathered values so they might need to be adjusted. You should definitely print out the gradients and weight changes to see if there's something weird going on (e.g. exploding gradients->NaN although you're only testing AND/XOR). Your last_gradient value should also be initialized to 0 at the first timestep.
Anyone know if it's possible to merge a set of cube geometries in a web worker and pass it back to the main thread? Was thinking this could reduce the lag when merging large amounts of cubes.
Does Three.JS work okay in a web worker, and if it does, would it be possible (and faster) to do this? Not sure if passing the geometry back would take just as long as merging it normally.
At the moment I'm using a timed for loop to reduce the lag:
// This array is populated by the server and contains the chunk position and data (which I do nothing with yet).
var sectionData = data.secData;
var section = 0;
var tick = function() {
var start = new Date().getTime();
for (; section < sectionData.length && (new Date().getTime()) - start < 1; section++) {
var sectionXPos = sectionData[section][0] * 10;
var sectionZPos = sectionData[section][1] * 10;
var combinedGeometry = new THREE.Geometry();
for (var layer = 0; layer < 1; layer++) { // Only 1 layer because of the lag...
for (var x = 0; x < 10; x++) {
for (var z = 0; z < 10; z++) {
blockMesh.position.set(x-4.5, layer-.5, z-4.5);
blockMesh.updateMatrix();
THREE.GeometryUtils.merge(combinedGeometry, blockMesh);
}
}
}
var sectionMesh = new THREE.Mesh(combinedGeometry, grassBlockMat);
sectionMesh.position.set(sectionXPos, 0, sectionZPos);
sectionMesh.matrixAutoUpdate = false;
sectionMesh.updateMatrix();
scene.add(sectionMesh);
}
if (section < sectionData.length) {
setTimeout(tick, 25);
}
};
setTimeout(tick, 25);
Using Three.JS rev59-dev.
Merged cubes make up the terrain in chunks, and at the moment (due to the lag) each chunk only has 1 layer.
Any tips would be appreciated! Thanks.
THREE.JS will not work in a web worker, however you can copy those parts of the library that you need to work both in the main thread and in your web worker.
Your first problem will be that you cannot send the geometry object itself back to the main thread.
Since the web worker onmessage variable passing works only by sending copies of JSON (not javascript objects) or references to ArrayBuffers, you would have to decode the geometry down to each float, pack it in an ArrayBuffer, and send a reference back to the main thread.
Note those are called transferable objects and once sent, they are cleared in the webworker / main thread from which they came.
See here for more details:
http://www.html5rocks.com/en/tutorials/workers/basics/
https://developer.mozilla.org/en-US/docs/Web/Guide/Performance/Using_web_workers
Here is an example of packing position vertices into an array for a physics type system:
//length * 3 axes * 4 bytes per vertex
var posBuffer = new Float32Array(new ArrayBuffer(len * 3 * 4));
//in a loop
//... do hard work
posBuffer[i * 3] = pos.x; //pos is a threejs vector
posBuffer[i * 3 + 1] = pos.y;
posBuffer[i * 3 + 2] = pos.z;
//after loop send buffer to main thread
self.postMessage({posBuffer:posBuffer}, [posBuffer.buffer]);
I copied the THREE.JS vector class inside my web worker and cut out all the methods I didn't need to keep it nice and lean.
FYI it's not slow and for something like n-body collisions it works well.
The main thread sends a command to the web worker telling it to run the update and then listens for the response. Kind of like a producer consumer model in regular threading.
I am currently in the process of building an application that reads in audio from my iPhone's microphone, and then does some processing and visuals. Of course I am starting with the audio stuff first, but am having one minor problem.
I am defining my sampling rate to be 44100 Hz and defining my buffer to hold 4096 samples. Which is does. However, when I print this data out, copy it into MATLAB to double check accuracy, the sample rate I have to use is half of my iPhone defined rate, or 22050 Hz, for it to be correct.
I think it has something to do with the following code and how it is putting 2 bytes per packet, and when I am looping through the buffer, the buffer is spitting out the whole packet, which my code assumes is a single number. So what I am wondering is how to split up those packets and read them as individual numbers.
- (void)setupAudioFormat {
memset(&dataFormat, 0, sizeof(dataFormat));
dataFormat.mSampleRate = kSampleRate;
dataFormat.mFormatID = kAudioFormatLinearPCM;
dataFormat.mFramesPerPacket = 1;
dataFormat.mChannelsPerFrame = 1;
// dataFormat.mBytesPerFrame = 2;
// dataFormat.mBytesPerPacket = 2;
dataFormat.mBitsPerChannel = 16;
dataFormat.mReserved = 0;
dataFormat.mBytesPerPacket = dataFormat.mBytesPerFrame = (dataFormat.mBitsPerChannel / 8) * dataFormat.mChannelsPerFrame;
dataFormat.mFormatFlags =
kLinearPCMFormatFlagIsSignedInteger |
kLinearPCMFormatFlagIsPacked;
}
If what I described is unclear, please let me know. Thanks!
EDIT
Adding the code that I used to print the data
float *audioFloat = (float *)malloc(numBytes * sizeof(float));
int *temp = (int*)inBuffer->mAudioData;
int i;
float power = pow(2, 31);
for (i = 0;i<numBytes;i++) {
audioFloat[i] = temp[i]/power;
printf("%f ",audioFloat[i]);
}
I found the problem with what I was doing. It was a c pointer issue, and since I have never really programmed in C before, I of course got them wrong.
You can not directly cast inBuffer->mAudioData to an int array. So what I simply did was the following
SInt16 *buffer = malloc(sizeof(SInt16)*kBufferByteSize);
buffer = inBuffer->mAudioData;
This worked out just fine and now my data is of correct length and the data is represented properly.
I saw your answer, there also is an underlying issue which gives wrong sample data bytes which is because of an endian issue of bytes being swapped.
-(void)feedSamplesToEngine:(UInt32)audioDataBytesCapacity audioData:(void *)audioData {
int sampleCount = audioDataBytesCapacity / sizeof(SAMPLE_TYPE);
SAMPLE_TYPE *samples = (SAMPLE_TYPE*)audioData;
//SAMPLE_TYPE *sample_le = (SAMPLE_TYPE *)malloc(sizeof(SAMPLE_TYPE)*sampleCount );//for swapping endians
std::string shorts;
double power = pow(2,10);
for(int i = 0; i < sampleCount; i++)
{
SAMPLE_TYPE sample_le = (0xff00 & (samples[i] << 8)) | (0x00ff & (samples[i] >> 8)) ; //Endianess issue
char dataInterim[30];
sprintf(dataInterim,"%f ", sample_le/power); // normalize it.
shorts.append(dataInterim);
}
I have built a tricopter from scratch based on a .NET Micro Framework board from TinyCLR.com. I used the FEZ Mini which runs at 72 MHz. Read more about my project at: http://bit.ly/TriRot.
So after a pre-flight check where I initialise and test each component, like calibrating the IMU and spinning each motor, checking that I get receiver data, etc., it enters a permanent loop which then calls the flight controller method on each loop.
I'm trying to tune my PID controller now using the Ziegler-Nichols method, but I am always getting a progressively larger overshoot. I was eventually able to get a [mostly] stable oscillation using proportional control only (setting Ki and Kd = 0); timing the period K with a stopwatch averaged out to 3.198 seconds.
I came across the answer (by Rex Logan) on a similar question by chris12892.
I was initially using the "Duration" variable in milliseconds which made my copter highly aggressive, obviously because I was multiplying the running integrator error by thousands on each loop. I then divided it by another thousand to bring it to seconds, but I'm still battling...
What I don't understand from Rex's answer is:
Why does he ignore the time variable in the integral and differential parts of the equations? Is that right or is it a typo?
What he means by the remark
In a normal sampled system the delta term would be one...
One what? Should this be one second under normal circumstances? What
if this value fluctuates?
My flight controller method is below:
private static Single[] FlightController(Single[] imuData, Single[] ReceiverData)
{
Int64 TicksPerMillisecond = TimeSpan.TicksPerMillisecond;
Int64 CurrentTicks = DateTime.Now.Ticks;
Int64 TickCount = CurrentTicks - PreviousTicks;
PreviousTicks = CurrentTicks;
Single Duration = (TickCount / TicksPerMillisecond) / 1000F;
const Single Kp = 0.117F; //Proportional Gain (Instantaneou offset)
const Single Ki = 0.073170732F; //Integral Gain (Permanent offset)
const Single Kd = 0.001070122F; //Differential Gain (Change in offset)
Single RollE = 0;
Single RollPout = 0;
Single RollIout = 0;
Single RollDout = 0;
Single RollOut = 0;
Single PitchE = 0;
Single PitchPout = 0;
Single PitchIout = 0;
Single PitchDout = 0;
Single PitchOut = 0;
Single rxThrottle = ReceiverData[(int)Channel.Throttle];
Single rxRoll = ReceiverData[(int)Channel.Roll];
Single rxPitch = ReceiverData[(int)Channel.Pitch];
Single rxYaw = ReceiverData[(int)Channel.Yaw];
Single[] TargetMotorSpeed = new Single[] { rxThrottle, rxThrottle, rxThrottle };
Single ServoAngle = 0;
if (!FirstRun)
{
Single imuRoll = imuData[1] + 7;
Single imuPitch = imuData[0];
//Roll ----- Start
RollE = rxRoll - imuRoll;
//Proportional
RollPout = Kp * RollE;
//Integral
Single InstanceRollIntegrator = RollE * Duration;
RollIntegrator += InstanceRollIntegrator;
RollIout = RollIntegrator * Ki;
//Differential
RollDout = ((RollE - PreviousRollE) / Duration) * Kd;
//Sum
RollOut = RollPout + RollIout + RollDout;
//Roll ----- End
//Pitch ---- Start
PitchE = rxPitch - imuPitch;
//Proportional
PitchPout = Kp * PitchE;
//Integral
Single InstancePitchIntegrator = PitchE * Duration;
PitchIntegrator += InstancePitchIntegrator;
PitchIout = PitchIntegrator * Ki;
//Differential
PitchDout = ((PitchE - PreviousPitchE) / Duration) * Kd;
//Sum
PitchOut = PitchPout + PitchIout + PitchDout;
//Pitch ---- End
TargetMotorSpeed[(int)Motors.Motor.Left] += RollOut;
TargetMotorSpeed[(int)Motors.Motor.Right] -= RollOut;
TargetMotorSpeed[(int)Motors.Motor.Left] += PitchOut;// / 2;
TargetMotorSpeed[(int)Motors.Motor.Right] += PitchOut;// / 2;
TargetMotorSpeed[(int)Motors.Motor.Rear] -= PitchOut;
ServoAngle = rxYaw + 15;
PreviousRollE = imuRoll;
PreviousPitchE = imuPitch;
}
FirstRun = false;
return new Single[] {
(Single)TargetMotorSpeed[(int)TriRot.LeftMotor],
(Single)TargetMotorSpeed[(int)TriRot.RightMotor],
(Single)TargetMotorSpeed[(int)TriRot.RearMotor],
(Single)ServoAngle
};
}
Edit: I found that I had two bugs in my code above (fixed now). I was integrating and differentiating with the last IMU values as opposed to the last error values. That got rid of the runaway sitation completely. The only problem now is that it seems to be a bit slow. When I perturb the system, it responds very quickly and stop it from continuing, but it takes a long time to get back to the setpoint (0), about 10 seconds or more. Is this now just down to tuning the PID? I'll give the suggestions below a go, and let you know if any of them make a difference.
One question I have is:
being a .NET board, I don't want to bank on any kind of accurate timing, so instead of trying to work out at what frequency I am executing that method, surely if I calculate the actual time and factor that into the equations, it should be better, or am I misunderstanding something?