Related
I have a csv file generated from another program which looks like this:
45, 133, 148, 213, 65, 26, 22, 73
84, 51, 41, 249, 25, 167, 102, 72
217, 198, 117, 123, 160, 9, 210, 211
230, 64, 37, 215, 91, 76, 240, 163
123, 169, 197, 16, 225, 160, 68, 65
89, 247, 170, 88, 173, 206, 158, 235
144, 138, 188, 164, 84, 38, 67, 29
98, 23, 106, 159, 96, 7, 77, 67
142, 140, 240, 56, 176, 0, 131, 160
241, 199, 96, 245, 213, 218, 51, 75
22, 226, 81, 106, 94, 252, 252, 110
0, 96, 132, 38, 189, 150, 162, 177
95, 252, 107, 181, 72, 7, 0, 247
228, 207, 203, 128, 91, 158, 164, 116
70, 124, 20, 37, 225, 169, 245, 103
103, 229, 186, 108, 151, 170, 18, 168
52, 86, 244, 244, 150, 181, 9, 146
115, 60, 50, 162, 70, 253, 43, 94
201, 72, 132, 207, 181, 106, 136, 70
92, 7, 97, 222, 149, 145, 155, 255
55, 188, 90, 58, 124, 230, 215, 229
231, 60, 48, 150, 179, 247, 104, 162
45, 241, 178, 122, 149, 243, 236, 92
186, 252, 165, 162, 176, 87, 238, 29
There is always a space following each 8x8 integer matrix.
I need to read each 8x8 matrix into a MATLAB program, generate an operation on it, and then write the result that has the same format. The result will be 8x8 matrix of floats, with space following each 8x8 matrix.
How do I do these 2 things in MATLAB R2017a? There are so many different functions with some depracated and some only available in later versions of MATLAB. I am not sure what do use to get the result I need.
you may use this approach
% read csv in older Matlab
m1 = csvread("test.csv");
% random operation to get some floats
m2 = m1 / 17 ;
%
newMatrix = ones(8,8);
newMatrix = string( newMatrix );
for i =1 : height(r)
for k = 1 : 8
newMatrix(i,k) = string(g(i,k));
end
end
cell2csv('testFile.csv' ,newMatrix , ", " )
% function below created by Sylvain Fiedler modified by Rob Kohr
%https://www.mathworks.com/matlabcentral/fileexchange/7601-cell2csv
function cell2csv(filename,cellArray,delimiter)
% Writes cell array content into a *.csv file.
%
% CELL2CSV(filename,cellArray,delimiter)
%
% filename = Name of the file to save. [ i.e. 'text.csv' ]
% cellarray = Name of the Cell Array where the data is in
% delimiter = seperating sign, normally:',' (it's default)
%
% by Sylvain Fiedler, KA, 2004
% modified by Rob Kohr, Rutgers, 2005 - changed to english and fixed delimiter
if nargin<3
delimiter = ',';
end
datei = fopen(filename,'w');
for z=1:size(cellArray,1)
for s=1:size(cellArray,2)
var = eval(['cellArray{z,s}']);
if size(var,1) == 0
var = '';
end
if isnumeric(var) == 1
var = num2str(var);
end
fprintf(datei,var);
if s ~= size(cellArray,2)
fprintf(datei,[delimiter]);
end
end
fprintf(datei,'\n');
end
fclose(datei);
end
For MATLAB release 2019a and above, you can use readmatrix and writematrix like this:
M = readmatrix('mat24x8.csv');
r = 1:8;
M1 = m(r,:);
M2 = m(r+8,:);
M3 = m(r+16,:);
% Apply whatever operations you want ..
M1 = sqrt(M1);
M2 = sin(M2);
M3 = cos(M3);
% Finally save again ..
M = [M1; M2; M3]
M = writematrix(M, 'new24x8.csv');
These functions are better than older ones like M = csvread(filename) and csvwrite(filename,M) because it preserves data accuracy. The file extension for older versions is .dat. Also note that white space has no meaning here, so, if you want each matrix stored separately, just save three files.
Update: To work in version R2017a of MATLAB, you should use the csvread(filename)andcsvwrite(filename,M)` I mentioned above in the same way as below.
M = csvread('myFile.csv');
r = 1:8;
M1 = M(r,:);
M2 = M(r+8,:);
M3 = M(r+16,:);
% Apply whatever operations you want ..
M1 = sqrt(M1);
M2 = sin(M2);
M3 = cos(M3);
% Finally save again ..
M = [M1; M2; M3];
csvwrite('myFile.csv',M);
No method of the above will work for arbitrary separators in the csv file other than blank lines.
This is my code for sobel filter:
def init_f(shape, dtype=None):
sobel_x = tf.constant([[-5, -4, 0, 4, 5], [-8, -10, 0, 10, 8], [-10, -20, 0, 20, 10], [-8, -10, 0, 10, 8], [-5, -4, 0, 4, 5]])
ker = np.zeros(shape, dtype)
ker_shape = tf.shape(ker)
kernel = tf.tile(sobel_x, ker_shape)//*Is this correct?*
return kernel
model.add(Conv2D(filters=30, kernel_size=(5,5), kernel_initializer=init_f, strides=(1,1), activation='relu'))
So far I have managed to do this.
But, this gives me error:
Shape must be rank 2 but is rank 4 for 'conv2d_17/Tile' (op: 'Tile') with input shapes: [5,5], [4].
Tensorflow Version: 2.1.0
You're close, but the args to tile don't appear to be correct. That is why you're getting the error "Shape must be rank 2 but is rank 4 for..." You're sobel_x must be a rank 4 tensor, so you need to add two more dimensions. I used reshape in this example.
from tensorflow import keras
import tensorflow as tf
import numpy
def kernelInitializer(shape, dtype=None):
print(shape)
sobel_x = tf.constant(
[
[-5, -4, 0, 4, 5],
[-8, -10, 0, 10, 8],
[-10, -20, 0, 20, 10],
[-8, -10, 0, 10, 8],
[-5, -4, 0, 4, 5]
], dtype=dtype )
#create the missing dims.
sobel_x = tf.reshape(sobel_x, (5, 5, 1, 1))
print(tf.shape(sobel_x))
#tile the last 2 axis to get the expected dims.
sobel_x = tf.tile(sobel_x, (1, 1, shape[-2],shape[-1]))
print(tf.shape(sobel_x))
return sobel_x
x1 = keras.layers.Input((128, 128, 3))
cvl = keras.layers.Conv2D(30, kernel_size=(5,5), kernel_initializer=kernelInitializer, strides=(2,2), activation='relu')
model = keras.Sequential();
model.add(x1)
model.add(cvl)
data = numpy.ones((1, 128, 128, 3))
data[:, 0:64, 0:64, :] = 0
pd = model.predict(data)
print(pd.shape)
d = pd[0, :, :, 0]
for row in d:
for col in row:
m = '0'
if col != 0:
m = 'X'
print(m, end="")
print("")
I looked at using expand_dims instead of reshape but there didn't appear any advantage. broadcast_to seems ideal, but you still have to add the dimensions, so I don't think it was better than tile.
Why 30 filters of the same filter though? Are they going to be changed afterwards?
class Model(nn.Module):
def __init__(self):
super(Model, self).__init__()
self.net = nn.Sequential(
nn.Conv2d(in_channels = 3, out_channels = 16),
nn.ReLU(),
nn.MaxPool2d(2),
nn.Conv2d(in_channels = 16, out_channels = 16),
nn.ReLU(),
Flatten(),
nn.Linear(4096, 64),
nn.ReLU(),
nn.Linear(64, 10))
def forward(self, x):
return self.net(x)
I have created this model without a firm knowledge in Neural Network and I just fixed parameters until it worked in the training. I am not sure how to get the output dimension for each layer (e.g. output dimension after the first layer).
Is there an easy way to do this in Pytorch?
You can use torchsummary, for instance, for ImageNet dimension(3x224x224):
from torchvision import models
from torchsummary import summary
vgg = models.vgg16()
summary(vgg, (3, 224, 224)
----------------------------------------------------------------
Layer (type) Output Shape Param #
================================================================
Conv2d-1 [-1, 64, 224, 224] 1,792
ReLU-2 [-1, 64, 224, 224] 0
Conv2d-3 [-1, 64, 224, 224] 36,928
ReLU-4 [-1, 64, 224, 224] 0
MaxPool2d-5 [-1, 64, 112, 112] 0
Conv2d-6 [-1, 128, 112, 112] 73,856
ReLU-7 [-1, 128, 112, 112] 0
Conv2d-8 [-1, 128, 112, 112] 147,584
ReLU-9 [-1, 128, 112, 112] 0
MaxPool2d-10 [-1, 128, 56, 56] 0
Conv2d-11 [-1, 256, 56, 56] 295,168
ReLU-12 [-1, 256, 56, 56] 0
Conv2d-13 [-1, 256, 56, 56] 590,080
ReLU-14 [-1, 256, 56, 56] 0
Conv2d-15 [-1, 256, 56, 56] 590,080
ReLU-16 [-1, 256, 56, 56] 0
MaxPool2d-17 [-1, 256, 28, 28] 0
Conv2d-18 [-1, 512, 28, 28] 1,180,160
ReLU-19 [-1, 512, 28, 28] 0
Conv2d-20 [-1, 512, 28, 28] 2,359,808
ReLU-21 [-1, 512, 28, 28] 0
Conv2d-22 [-1, 512, 28, 28] 2,359,808
ReLU-23 [-1, 512, 28, 28] 0
MaxPool2d-24 [-1, 512, 14, 14] 0
Conv2d-25 [-1, 512, 14, 14] 2,359,808
ReLU-26 [-1, 512, 14, 14] 0
Conv2d-27 [-1, 512, 14, 14] 2,359,808
ReLU-28 [-1, 512, 14, 14] 0
Conv2d-29 [-1, 512, 14, 14] 2,359,808
ReLU-30 [-1, 512, 14, 14] 0
MaxPool2d-31 [-1, 512, 7, 7] 0
Linear-32 [-1, 4096] 102,764,544
ReLU-33 [-1, 4096] 0
Dropout-34 [-1, 4096] 0
Linear-35 [-1, 4096] 16,781,312
ReLU-36 [-1, 4096] 0
Dropout-37 [-1, 4096] 0
Linear-38 [-1, 1000] 4,097,000
================================================================
Total params: 138,357,544
Trainable params: 138,357,544
Non-trainable params: 0
----------------------------------------------------------------
Input size (MB): 0.57
Forward/backward pass size (MB): 218.59
Params size (MB): 527.79
Estimated Total Size (MB): 746.96
----------------------------------------------------------------
Source: model-summary-in-pytorch
A simple way is:
Pass the input to the model.
Print the size of the output after passing every layer.
class Model(nn.Module):
def __init__(self):
super(Model, self).__init__()
self.net = nn.Sequential(
nn.Conv2d(in_channels = 3, out_channels = 16),
nn.ReLU(),
nn.MaxPool2d(2),
nn.Conv2d(in_channels = 16, out_channels = 16),
nn.ReLU(),
Flatten(),
nn.Linear(4096, 64),
nn.ReLU(),
nn.Linear(64, 10))
def forward(self, x):
for layer in self.net:
x = layer(x)
print(x.size())
return x
model = Model()
x = torch.randn(1, 3, 224, 224)
# Let's print it
model(x)
But be careful with the input size because you are using nn.Linear in your net. It would cause incompatible input size for nn.Linear if your input size is not 4096.
Like David Ng's answer but a tad shorter:
def get_output_shape(model, image_dim):
return model(torch.rand(*(image_dim))).data.shape
In this example I needed to figure out the input of the last Linear layer:
class Net(nn.Module):
def __init__(self):
super(Net, self).__init__()
self.expected_input_shape = (1, 1, 192, 168)
self.conv1 = nn.Conv2d(1, 32, 3, 1)
self.conv2 = nn.Conv2d(32, 64, 3, 1)
self.dropout1 = nn.Dropout2d(0.25)
self.dropout2 = nn.Dropout2d(0.5)
self.maxpool1 = nn.MaxPool2d(2)
self.maxpool2 = nn.MaxPool2d(3)
# Calculate the input of the Linear layer
conv1_out = get_output_shape(self.maxpool1, get_output_shape(conv1, self.expected_input_shape))
conv2_out = get_output_shape(self.maxpool2, get_output_shape(conv2, conv1_out))
fc1_in = np.prod(list(conv2_out)) # Flatten
self.fc1 = nn.Linear(fc1_in, 38)
def forward(self, x):
x = self.conv1(x)
x = F.relu(x)
x = self.maxpool1(x)
x = self.conv2(x)
x = F.relu(x)
x = self.maxpool2(x)
x = self.dropout1(x)
x = torch.flatten(x, 1) # flatten to a single dimension
x = self.fc1(x)
output = F.log_softmax(x, dim=1)
return output
This way, if I make changes to previous layers, I won't have to calculate all over again!
My answer is based on this answer
Another way to get the size after a certain layer in an nn.Sequential container is to add a custom Module that just prints out the size of the input.
class PrintSize(nn.Module):
def __init__(self):
super(PrintSize, self).__init__()
def forward(self, x):
print(x.shape)
return x
And now you can do:
model = nn.Sequential(
nn.Conv2d(3, 10, 5, 1),
// lots of convolutions, pooling, etc.
nn.Flatten(),
PrintSize(),
nn.Linear(1, 12), // the input dim of 1 is just a placeholder
)
Now, you can do model(x) and it will print out the shape of the output after the Conv2d layer ran. This is useful if you have a lot of convolutions and want to figure out what the final dimensions are for the first fully connected layer. You don't need to reformat your nn.Sequential as a Module and can just drop in this helper class with one-line.
for layer in model.children():
if hasattr(layer, 'out_features'):
print(layer.out_features)
Here's a solution in the form of a helper function:
def get_tensor_dimensions_impl(model, layer, image_size, for_input=False):
t_dims = None
def _local_hook(_, _input, _output):
nonlocal t_dims
t_dims = _input[0].size() if for_input else _output.size()
return _output
layer.register_forward_hook(_local_hook)
dummy_var = torch.zeros(1, 3, image_size, image_size)
model(dummy_var)
return t_dims
Example:
from torchvision import models, transforms
a_model = models.squeezenet1_0(pretrained=True)
get_tensor_dimensions_impl(a_model, a_model._modules['classifier'], 224)
Output is:
torch.Size([1, 1000, 1, 1])
Maybe you can try print(model.state_dict()['next_layer.weight'].shape).
This gives you a hint of the output shape from last layer.
I created the following three dimensional mockup matrix:
mockup(:,:,1) = ...
[100, 100, 100; ...
103, 95, 100; ...
101, 85, 100; ...
96, 90, 102; ...
91, 89, 99; ...
97, 91, 97; ...
105, 83, 100];
mockup(:,:,2) = ...
[50, NaN, NaN; ...
47, NaN, 40; ...
45, 60, 45; ...
47, 65, 45; ...
51, 70, 45; ...
54, 65, 50; ...
62, 80, 55];
I also defined percentTickerAvailable = 0.5.
As a result, The columns represent equity prices from three different assets. For futher processing I need to manipulate the NaN values in the following way.
If the percentage of NaNs in any given ROW is greater than 1 - percentTickerAvailable, replace all values in these particular rows with NaNs. That is, if not enough assets have prices in that particular row, ignore the row completely.
If the percentage of NaNs in any given ROW is less or equal to 1 - percentTickerAvailable, replace the respective NaNs with -inf.
To be clear, "percentage of NaNs in any given ROW" is calculated as follows:
Number of NaNs in any given ROW divided by number of columns.
The adjusted mockup matrix should look like this:
mockupAdj(:,:,1) = ...
[100, 100, 100; ...
103, 95, 100; ...
101, 85, 100; ...
96, 90, 102; ...
91, 89, 99; ...
97, 91, 97; ...
105, 83, 100];
mockupAdj(:,:,2) = ...
[NaN, NaN, NaN; ...
47, -inf, 40; ...
45, 60, 45; ...
47, 65, 45; ...
51, 70, 45; ...
54, 65, 50; ...
62, 80, 55];
So far, I did the following:
function vout = ranking(vin, percentTickerAvailable)
percentNonNaN = 1 - sum(isnan(vin), 2) / size(vin, 2);
NaNIdx = percentNonNaN < percentTickerAvailable;
infIdx = percentNonNaN > percentTickerAvailable & ...
percentNonNaN < 1;
[~, ~, numDimVin] = size(vin);
for i = 1 : numDimVin
vin(NaNIdx(:,:,i) == 1, :, i) = NaN;
end
about = vin;
end % EoF
By calling mockupAdj = ranking(mockup, 0.5) this already transforms the first row in mockup(1,:,2)correctly to {'NaN', 'NaN', 'NaN'}. However, I am struggling with the second point. With infIdx I already successfully identified the rows that corresponds to the second condition. But I don't know how to correctly use that information in order to replace the single NaN in mockup(2,2,2) with -inf.
Any hint is highly appreciated.
This is a good example of something that can be solved using vectorization. I am providing two versions of the code, one that uses the modern syntax (including implicit expansion) and one for older version of MATLAB.
Several things to note:
In the NaN substitution stage, I'm using a "trick" where 0/0 is evaluated to NaN.
In the Inf substitution stage, I'm using logical masking/indexing to access the correct elements in vin.
R2016b and newer:
function vin = ranking (vin, percentTickerAvailable)
% Find percentage of NaNs on each line:
pNaN = mean(isnan(vin), 2, 'double');
% Fills rows with NaNs:
vin = vin + 0 ./ (1 - ( pNaN >= percentTickerAvailable));
% Replace the rest with -Inf
vin(isnan(vin) & pNaN < percentTickerAvailable) = -Inf;
end
Prior to R2016b:
function vin = rankingOld (vin, percentTickerAvailable)
% Find percentage of NaNs on each line:
pNaN = mean(isnan(vin), 2, 'double');
% Fills rows with NaNs:
vin = bsxfun(#plus, vin, 0 ./ (1 - ( pNaN >= percentTickerAvailable)));
% Replace the rest with -Inf
vin(bsxfun(#and, isnan(vin), pNaN < percentTickerAvailable)) = -Inf;
end
1)
The percentage of NaN in any given row should be smaller than 1
... Are you talking about ratio? In which case this is a useless check, as it will always be the case. Or talking about percentages? In which case your code doesn't do what you describe. My guess is ratio.
2) Based on my guess, I have a follow up question: following your description, shouldn't mockup(2,2,2) stay NaN? There is 33% (<50%) of NaN in that row, so it does not fulfill your condition 2.
3) Based on the answers I deemed logical, I would have changed percentNaN = sum(isnan(vin), 2) / size(vin, 2); for readability, and NaNIdx = percentNaN > percentTickerAvailable; accordingly. Now just add one line in front of your loop:
vin(isnan(vin)) = -inf;
Why? Because like this you replace all the NaNs by -inf. Later on, the ones that respect condition 1 will be overwritten to NaN again, by the loop. You don't need the InfIdx.
4) Be aware that your function cannot return vout as of now. Just let it return vin, and you'll be fine.
You can also use logical indexing to achieve this task:
x(:,:,1) = ...
[100, 100, 100; ...
103, 95, 100; ...
101, 85, 100; ...
96, 90, 102; ...
91, 89, 99; ...
97, 91, 97; ...
105, 83, 100];
x(:,:,2) = ...
[50, NaN, NaN; ...
47, NaN, 40; ...
45, 60, 45; ...
47, 65, 45; ...
51, 70, 45; ...
54, 65, 50; ...
62, 80, 55];
% We fix the threshold
tres = 0.5; %fix the threshold.
% We check if a value = NaN or not.
in = isnan(x);
% Which line have more than 50% of NaN ?.
ind = (sum(in,2)./(size(x,2)))>0.5
% We generate an index
[x1,~,x3] = ind2sub(size(ind),ind);
% We set the NaN index to 0 if the line contains less than 50 % of NaN.
in(x1,:,x3) = 0;
% We calculate the new values.
x(in) = -inf;
x(x1,:,x3) = NaN;
I'm trying to vectorize the following code:
% code before
% code before
% a lot of code before we got to the current comment
%
% houghMatrix holds some values
for i=1:n
for j=1:m
for k = 1:maximalRadius
% get the maximal threshold
if houghMatrix(i,j,k) > getMaximalThreshold(k)
lhs = [j i k];
% verify that the new circle is not listed
isCircleExist = verifyCircleExists(circles,lhs,circleCounter);
% not listed - then we put it in the circles vector
if isCircleExist == 0
circles(circleCounter,:) = [j i k];
fprintf('Circle % d: % d, % d, % d \n', circleCounter, j, i, k);
circleCounter = circleCounter + 1;
end
end
end
end
end
Using tic tac I got the below outputs :
>> x = findCircles(ii);
Circle 1: 38, 38, 35
Circle 2: 89, 51, 34
Circle 3: 72, 66, 11
Circle 4: 33, 75, 30
Circle 5: 90, 81, 31
Circle 6: 54, 96, 26
Elapsed time is 3.111176 seconds.
>> x = findCircles(ii);
Circle 1: 38, 38, 35
Circle 2: 89, 51, 34
Circle 3: 72, 66, 11
Circle 4: 33, 75, 30
Circle 5: 90, 81, 31
Circle 6: 54, 96, 26
Elapsed time is 3.105642 seconds.
>> x = findCircles(ii);
Circle 1: 38, 38, 35
Circle 2: 89, 51, 34
Circle 3: 72, 66, 11
Circle 4: 33, 75, 30
Circle 5: 90, 81, 31
Circle 6: 54, 96, 26
Elapsed time is 3.135818 seconds.
Meaning - average of 3.1 seconds .
I tried to vectorize the code , but the problem is that I need to use
the index i,j,k in the body of the inner for (the 3rd for) .
Any suggestions how to vectorize the code would be greatly appreciated
Thanks
EDIT :
% -- function [circleExists] = verifyCircleExists(circles,lhs,total) --
%
%
function [circleExists] = verifyCircleExists(circles,lhs,total)
MINIMUM_ALLOWED_THRESHOLD = 2;
circleExists = 0;
for index = 1:total-1
rhs = circles(index,:);
absExpr = abs(lhs - rhs);
maxValue = max( absExpr );
if maxValue <= MINIMUM_ALLOWED_THRESHOLD + 1
circleExists = 1;
break
end
end
end
Heres what I think you want to do: For each valid coordinate triplet, you want to check whether there has been a nearby triplet already, otherwise, you add it to the list. This operation can be fully vectorized if there's no possibility of "chaining", i.e. if each cluster of possible candidate voxels can only accomodate one center. In this case, you simply use:
%# create a vector of thresholds
maximalThreshold = getMaximalThreshold(1:maximalRadius);
%# make it 1-by-1-by-3
maximalThreshold = reshape(maximalThreshold,1,1,[]);
%# create a binary array the size of houghMatrix with 1's
%# wherever we have a candidate circle center
validClusters = bsxfun(#gt, houghMatrix, maximalThreshold);
%# get the centroids of all valid clusters
stats = regionprops(validClusters,'Centroid');
%# collect centroids, round to get integer pixel values
circles = round(cat(1,stats.Centroid));
Alteratively, if you want to follow your scheme of selecting valid circles, you can get the ijk indices from validClusters as follows:
[potentialCircles(:,1),potentialCircles(:,2), potentialCircles(:,3)]= ...
sub2ind(size(houghMatrix),find(validClusters));
nPotentialCircles = size(potentialCircles,1);
for iTest = 2:nPotentialCircles
absDiff = abs(bsxfun(#minus,potentialCircles(1:iTest-1,:),potentialCircles(iTest,:)));
if any(absDiff(:) <= MINIMUM_ALLOWED_THRESHOLD + 1)
%# mask the potential circle
potentialCircles(iTest,:) = NaN;
end
end
circles = potentialCircles(isfinite(potentialCircles(:,1)),:);