I can't get a simple convolution to work in nd4j and documentation regarding this specific topic is scarse. What I'm trying to do:
INDArray values = Nd4j.create(new double[]{1, 2, 3, 4, 5, 6, 7, 8, 9, 10});
INDArray kernel = Nd4j.create(new double[]{0.5,0.5});
INDArray conv = Nd4j.getConvolution().convn(values, kernel, Convolution.Type.VALID);
No matter the values or the convolution type, I always get the same exception (see below). The error seems to occur when nd4j is trying to transform the array of values into a complex array to perform what I think is a Fourier transformation.
I've tried several versions of nd4j (0.9.1 - 0.8.0 - 0.7.0) but to no avail. Can anyone help?
Exception in thread "main" java.lang.UnsupportedOperationException
at org.nd4j.linalg.api.complex.BaseComplexNDArray.putScalar(BaseComplexNDArray.java:1947)
at org.nd4j.linalg.api.complex.BaseComplexNDArray.putScalar(BaseComplexNDArray.java:1804)
at org.nd4j.linalg.api.complex.BaseComplexNDArray.copyFromReal(BaseComplexNDArray.java:545)
at org.nd4j.linalg.api.complex.BaseComplexNDArray.<init>(BaseComplexNDArray.java:159)
at org.nd4j.linalg.api.complex.BaseComplexNDArray.<init>(BaseComplexNDArray.java:167)
at org.nd4j.linalg.cpu.nativecpu.complex.ComplexNDArray.<init>(ComplexNDArray.java:104)
at org.nd4j.linalg.cpu.nativecpu.CpuNDArrayFactory.createComplex(CpuNDArrayFactory.java:166)
at org.nd4j.linalg.factory.Nd4j.createComplex(Nd4j.java:3345)
at org.nd4j.linalg.convolution.DefaultConvolutionInstance.convn(DefaultConvolutionInstance.java:116)
at org.nd4j.linalg.convolution.BaseConvolution.convn(BaseConvolution.java:66)
at com.example.demo.Main.testing(Main.java:41)
at com.example.demo.Main.main(Main.java:34)
It's a bit trickier, as ND4j currently does not support mathematical convolution. You have to craft your own implementation.
double[] rawData = {12,10,15,12,10,11,15,12,12};
INDArray data = Nd4j.create(rawData);
double[] rawFilter = {1.0 / 2, 0, 1.0 / 2};
INDArray filter = Nd4j.create(rawFilter);
Nd4jConv1d convolution = new Nd4jConv1d(1, 1, (int) filter.shape()[1], 1, 0);
INDArray output = convolution.forward(data, filter);
As seen in: https://github.com/deeplearning4j/deeplearning4j/blob/af7155d61dc810d3e7139f15f98810e0255b2e17/arbiter/arbiter-deeplearning4j/src/test/java/org/deeplearning4j/arbiter/multilayernetwork/MNISTOptimizationTest.java
Note: you need an additional class Nd4jConv1d. Go to the repo to get it
Related
I am trying to compute a per-channel gradient image in PyTorch. To do this, I want to perform a standard 2D convolution with a Sobel filter on each channel of an image. I am using the torch.nn.functional.conv2d function for this
In my minimum working example code below, I get an error:
import torch
import torch.nn.functional as F
filters = torch.autograd.Variable(torch.randn(1,1,3,3))
inputs = torch.autograd.Variable(torch.randn(1,3,10,10))
out = F.conv2d(inputs, filters, padding=1)
RuntimeError: Given groups=1, weight[1, 1, 3, 3], so expected
input[1, 3, 10, 10] to have 1 channels, but got 3 channels instead
This suggests that groups need to be 3. However, when I make groups=3, I get a different error:
import torch
import torch.nn.functional as F
filters = torch.autograd.Variable(torch.randn(1,1,3,3))
inputs = torch.autograd.Variable(torch.randn(1,3,10,10))
out = F.conv2d(inputs, filters, padding=1, groups=3)
RuntimeError: invalid argument 4: out of range at
/usr/local/src/pytorch/torch/lib/TH/generic/THTensor.c:440
When I check that code snippet in the THTensor class, it refers to a bunch of dimension checks, but I don't know where I'm going wrong.
What does this error mean? How can I perform my intended convolution with this conv2d function? I believe I am misunderstanding the groups parameter.
If you want to apply a per-channel convolution then your out-channel should be the same as your in-channel. This is expected, considering each of your input channels creates a separate output channel that it corresponds to.
In short, this will work
import torch
import torch.nn.functional as F
filters = torch.autograd.Variable(torch.randn(3,1,3,3))
inputs = torch.autograd.Variable(torch.randn(1,3,10,10))
out = F.conv2d(inputs, filters, padding=1, groups=3)
whereas, filters of size (2, 1, 3, 3) or (1, 1, 3, 3) will not work.
Additionally, you can also make your out-channel a multiple of in-channel. This works for instances where you want to have multiple convolution filters for each input channel.
However, This only makes sense if it is a multiple. If not, then pytorch falls back to its closest multiple, a number less than what you specified. This is once again expected behavior. For example a filter of size (4, 1, 3, 3) or (5, 1, 3, 3), will result in an out-channel of size 3.
I just started learning accord.net, and while going through some examples I noticed that the Regress method on the SimpleLinearRegression is obsolete.
Apparently I should use the OrdinaryLeastSquares class, but I cannot find anything that will return the residual sum of squares, similar to the Regress method.
Do I need to create this method by myself?
Here is a full example on how to learn a SimpleLinearRegression and still be able to compute the residual sum of squares as you have been doing using the previous version of the framework:
// This is the same data from the example available at
// http://mathbits.com/MathBits/TISection/Statistics2/logarithmic.htm
// Declare your inputs and output data
double[] inputs = { 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11 };
double[] outputs = { 6, 9.5, 13, 15, 16.5, 17.5, 18.5, 19, 19.5, 19.7, 19.8 };
// Transform inputs to logarithms
double[] logx = Matrix.Log(inputs);
// Use Ordinary Least Squares to learn the regression
OrdinaryLeastSquares ols = new OrdinaryLeastSquares();
// Use OLS to learn the simple linear regression
SimpleLinearRegression lr = ols.Learn(logx, outputs);
// Compute predicted values for inputs
double[] predicted = lr.Transform(logx);
// Get an expression representing the learned regression model
// We just have to remember that 'x' will actually mean 'log(x)'
string result = lr.ToString("N4", CultureInfo.InvariantCulture);
// Result will be "y(x) = 6.1082x + 6.0993"
// The mean squared error between the expected and the predicted is
double error = new SquareLoss(outputs).Loss(predicted); // 0.261454
The last line in this example is the one that should be the most interesting to you. As you can see, the residual sum of squares that beforehand was being returned by the .Regress method can now be computed using the SquareLoss class. The advantages of this approach is that now you should be able to compute the most appropriate metric that matters the most to you, such as the ZeroOneLoss or the Euclidean loss or the Hamming loss.
In any case, I just wanted to reiterate that any methods marked as Obsolete in the framework are not going to stop working anytime soon. They are marked as obsolete meaning that new features will not be supported when using those methods, but your application will not stop working in case you have used any of those methods from within it.
In the Tensorflow docs, the tf.nn.conv2d-operation is described to:
Flatten the filter to a 2-D matrix with shape [filter_height * filter_width * in_channels, output_channels].
Extract image patches from the input tensor to form a virtual tensor of shape [batch, out_height, out_width, filter_height * filter_width * in_channels].
For each patch, right-multiply the filter matrix and the image patch vector.
Is there an operation to apply just step 2? I cannot find anything like that in the API docs. I might be searching with the wrong keywords.
This is now added to the tensorflow api: https://www.tensorflow.org/versions/r0.9/api_docs/python/array_ops.html#extract_image_patches
I guess a trick to do that would be to:
Take a filter of shape [filter_height, filter_width, in_channels, output_channels] with output_channels = filter_height * filter_width * in_channels
Fix the value of this filter in a way that when the filter is flattened to a 2-D matrix (cf. your step 2), it is the identity matrix. Check my example code below for a simple way to do that with np.eye().reshape()
Perform a normal tf.nn.conv2d(input, filter, strides=[1, 1, 1, 1], padding='SAME')
You now have an output of shape [batch, out_height, out_width, filter_height * filter_width * in_channels]
Here is a simple code for an input image of size 3*3 with 1 channel (and batch size 1).
import tensorflow as tf
import numpy as np
input_value = np.arange(1, 10).reshape((1, 3, 3, 1))
input = tf.constant(input_value)
input = tf.cast(input, tf.float32)
filter_value = np.eye(9).reshape((3, 3, 1, 9))
filter = tf.constant(filter_value)
filter = tf.cast(filter, tf.float32)
output = tf.nn.conv2d(input, filter, strides=[1, 1, 1, 1], padding='SAME')
I have a MATLAB program that loads two images and returns two camera matrices and a cameraParams object with distortion coefficients, etc. I would now like to use this exact configuration to undistort points and so on, in an OpenCV program that triangulates points given their 2D locations in two different videos.
function [cameraMatrix1, cameraMatrix2, cameraParams] = setupCameraCalibration(leftImageFile, rightImageFile, squareSize)
% Auto-generated by cameraCalibrator app on 20-Feb-2015
The thing is, the output of undistortPoints is different in MATLAB and OpenCV even though both use the same arguments.
As an example:
>> undistortPoints([485, 502], defaultCameraParams)
ans = 485 502
In Java, the following test mimics the above (it passes).
public void testUnDistortPoints() {
Mat srcMat = new Mat(2, 1, CvType.CV_32FC2);
Mat dstMat = new Mat(2, 1, CvType.CV_32FC2);
srcMat.put(0, 0, new float[] { 485, 502 } );
MatOfPoint2f src = new MatOfPoint2f(srcMat);
MatOfPoint2f dst = new MatOfPoint2f(dstMat);
Mat defaultCameraMatrix = Mat.eye(3, 3, CvType.CV_32F);
Mat defaultDistCoefficientMatrix = new Mat(1, 4, CvType.CV_32F);
Imgproc.undistortPoints(
src,
dst,
defaultCameraMatrix,
defaultDistCoefficientMatrix
);
System.out.println(dst.dump());
assertEquals(dst.get(0, 0)[0], 485d);
assertEquals(dst.get(0, 0)[1], 502d);
}
However, say I change the first distortion coefficient (k1). In MATLAB:
changedDist = cameraParameters('RadialDistortion', [2 0 0])
>> undistortPoints([485, 502], changedDist)
ans = 4.8756 5.0465
In Java:
public void testUnDistortPointsChangedDistortion() {
Mat srcMat = new Mat(2, 1, CvType.CV_32FC2);
Mat dstMat = new Mat(2, 1, CvType.CV_32FC2);
srcMat.put(0, 0, new float[] { 485, 502 } );
MatOfPoint2f src = new MatOfPoint2f(srcMat);
MatOfPoint2f dst = new MatOfPoint2f(dstMat);
Mat defaultCameraMatrix = Mat.eye(3, 3, CvType.CV_32F);
Mat distCoefficientMatrix = new Mat(1, 4, CvType.CV_32F);
distCoefficientMatrix.put(0, 0, 2f); // updated
Imgproc.undistortPoints(
src,
dst,
defaultCameraMatrix,
distCoefficientMatrix
);
System.out.println(dst.dump());
assertEquals(4.8756, dst.get(0, 0)[0]);
assertEquals(5.0465, dst.get(0, 0)[1]);
}
It fails with the following output:
[0.0004977131, 0.0005151587]
junit.framework.AssertionFailedError:
Expected :4.8756
Actual :4.977131029590964E-4
Why are the results different? I thought Java's distortion coefficient matrix includes both the radial and tangential distortion coefficients.
Also, is CV_64FC1 a good choice of type for the camera / distortion coefficient matrices?
I was trying to test the effect of changing the camera matrix itself (i.e. the value of f_x), but it's not possible to set the 'IntrinsicMatrix' parameter when using cameraparams, so I want to solve the distortion matrix problem first.
Any help would be greatly appreciated.
There is a couple of things you have to take into account when working with calibration models.
First, note there exist several camera calibration and distortion models: Tsai, ATAN, Pinhole, Ocam. I assume you want to use the Pinhole model, which is the used by OpenCV and the most common one. It models from 2 to 6 parameters for radial distortion (denoted as k1...k6) and 2 for tangential distortion (denoted as p1, p2), as you can read in the OpenCV doc. Bouget's calibration toolbox for Matlab uses this model too.
Second, there is not a standardized way to arrange distortion parameters in a vector. OpenCV expects items in this order: [k1 k2 p1 p2 k3...k6], being k3...k6 optional.
So, check the documentation of your Matlab calibration software and look for what model it uses and in which order the parameters are arranged. Then, make sure it meets the order in OpenCV.
The calibration parameters for OpenCV are ok as CV_32F and CV_64F as I recall.
Update
I don't know in Java, but in C++, when you create a Mat, its initial values are unspecified, so that this code may be creating a matrix with a 2f in the first item and garbage in the remaining ones:
Mat distCoefficientMatrix = new Mat(1, 4, CvType.CV_32F);
distCoefficientMatrix.put(0, 0, 2f);
Could you check if this is the problem?
A note for the future, to make things trickier, take into account that the intrinsic calibration matrix in OpenCV is the transpose of the one in Matlab.
Is there any direct opencv function for matlab function conv2? I tried using cvFilter2D(), but it seems to be giving me different results than conv2().
For example:
CvMat * Aa = cvCreateMat(2, 2, CV_32FC1);
CvMat * Bb = cvCreateMat(2, 2, CV_32FC1);
CvMat * Cc = cvCreateMat(2, 2, CV_32FC1);
cvSetReal2D(Aa, 0, 0, 1);
cvSetReal2D(Aa, 0, 1, 2);
cvSetReal2D(Aa, 1, 0, 3);
cvSetReal2D(Aa, 1, 1, 4);
cvSetReal2D(Bb, 0, 0, 5);
cvSetReal2D(Bb, 0, 1, 5);
cvSetReal2D(Bb, 1, 0, 5);
cvSetReal2D(Bb, 1, 1, 5);
cvFilter2D(Aa, Cc, Bb);
This produces the matrix [20 30; 40 50]
In MATLAB:
>> A=[1 2; 3 4]
A =
1 2
3 4
>> B=[5 5; 5 5]
B =
5 5
5 5
>> conv2(A,B,'shape')
ans =
50 30
35 20
Please Help me.its very much useful for me.Thank you.
Regards
Arangarajan.
The numerical computing environment Matlab (or e.g. its free alternative GNU Octave) provides a function called conv2 for the two-dimensional convolution of a given matrix with a convolution kernel. While writing some C++ code based upon the free image processing library OpenCV, I found that OpenCV currently offers no equivalent method.
Although there is a filter2D() method that implements two-dimensional correlation and that can be used to convolute an image with a given kernel (by flipping that kernel and moving the anchor point to the correct position, as explained on the corresponding OpenCV documentation page), it would be nice to have a method offering the same border handling options as Matlab (“full”, “valid” or “same” convolution), e.g. for comparing results of the same algorithm implemented in both Matlab and C++ using OpenCV.
Here is what I came up with:
enum ConvolutionType {
/* Return the full convolution, including border */
CONVOLUTION_FULL,
/* Return only the part that corresponds to the original image */
CONVOLUTION_SAME,
/* Return only the submatrix containing elements that were not influenced by the border
*/
CONVOLUTION_VALID
};
void conv2(const Mat &img, const Mat& kernel, ConvolutionType type, Mat& dest) {
Mat source = img;
if(CONVOLUTION_FULL == type) {
source = Mat();
const int additionalRows = kernel.rows-1, additionalCols = kernel.cols-1;
copyMakeBorder(img, source, (additionalRows+1)/2, additionalRows/2,
(additionalCols+1)/2, additionalCols/2, BORDER_CONSTANT, Scalar(0));
}
Point anchor(kernel.cols - kernel.cols/2 - 1, kernel.rows - kernel.rows/2 - 1);
int borderMode = BORDER_CONSTANT;
filter2D(source, dest, img.depth(), flip(kernel), anchor, 0, borderMode);
if(CONVOLUTION_VALID == type) {
dest = dest.colRange((kernel.cols-1)/2, dest.cols - kernel.cols/2)
.rowRange((kernel.rows-1)/2, dest.rows - kernel.rows/2);
}
}
In my unit tests, this implementation yielded results that were almost identical with the Matlab implementation. Note that both OpenCV and Matlab do the convolution in Fourier space if the kernel is large enough. The definition of ‘large’ varies in both implementations, but results should still be very similar, even for large kernels.
Also, the performance of this method might be an issue for the ‘full’ convolution case, since the entire source matrix needs to be copied to add a border around it. Finally, If you receive an exception in the filter2D() call and you are using a kernel with only one column, this might be caused by this bug. In that case, set the borderMode variable to e.g. BORDER_REPLICATE instead, or use the latest version of the library from the OpenCV trunk.
If you are using convolution, there is problem at the edge of the matrix. The convolution mask needs values which are outside of the matrix. The algorithms from OpenCV and matlab use different strategies to cope with this problem. OpenCV just replicates the pixels of the border whereas matlab just assumes that all this pixels are zero.
So if you want to emulate the behaviour of matlab in OpenCV you can add this zero padding manually. There even is a dedicated function for this. Let me give you an example of how your code could be modified:
CvMat * Ccb = cvCreateMat(3, 3, CV_32FC1);
CvMat * Aab = cvCreateMat(3, 3, CV_32FC1);
cvCopyMakeBorder(Aa,Aab, cvPoint(0,0),IPL_BORDER_CONSTANT, cvScalarAll(0));
cvFilter2D(Aab, Ccb, Bb);
The result this gives is:
20.000 30.000 20.000
40.000 50.000 30.000
30.000 35.000 20.000
To get your intended result you just need to delete the first column and row to get rid of the additional data introduced by the border we added.