I have a csv file generated from another program which looks like this:
45, 133, 148, 213, 65, 26, 22, 73
84, 51, 41, 249, 25, 167, 102, 72
217, 198, 117, 123, 160, 9, 210, 211
230, 64, 37, 215, 91, 76, 240, 163
123, 169, 197, 16, 225, 160, 68, 65
89, 247, 170, 88, 173, 206, 158, 235
144, 138, 188, 164, 84, 38, 67, 29
98, 23, 106, 159, 96, 7, 77, 67
142, 140, 240, 56, 176, 0, 131, 160
241, 199, 96, 245, 213, 218, 51, 75
22, 226, 81, 106, 94, 252, 252, 110
0, 96, 132, 38, 189, 150, 162, 177
95, 252, 107, 181, 72, 7, 0, 247
228, 207, 203, 128, 91, 158, 164, 116
70, 124, 20, 37, 225, 169, 245, 103
103, 229, 186, 108, 151, 170, 18, 168
52, 86, 244, 244, 150, 181, 9, 146
115, 60, 50, 162, 70, 253, 43, 94
201, 72, 132, 207, 181, 106, 136, 70
92, 7, 97, 222, 149, 145, 155, 255
55, 188, 90, 58, 124, 230, 215, 229
231, 60, 48, 150, 179, 247, 104, 162
45, 241, 178, 122, 149, 243, 236, 92
186, 252, 165, 162, 176, 87, 238, 29
There is always a space following each 8x8 integer matrix.
I need to read each 8x8 matrix into a MATLAB program, generate an operation on it, and then write the result that has the same format. The result will be 8x8 matrix of floats, with space following each 8x8 matrix.
How do I do these 2 things in MATLAB R2017a? There are so many different functions with some depracated and some only available in later versions of MATLAB. I am not sure what do use to get the result I need.
you may use this approach
% read csv in older Matlab
m1 = csvread("test.csv");
% random operation to get some floats
m2 = m1 / 17 ;
%
newMatrix = ones(8,8);
newMatrix = string( newMatrix );
for i =1 : height(r)
for k = 1 : 8
newMatrix(i,k) = string(g(i,k));
end
end
cell2csv('testFile.csv' ,newMatrix , ", " )
% function below created by Sylvain Fiedler modified by Rob Kohr
%https://www.mathworks.com/matlabcentral/fileexchange/7601-cell2csv
function cell2csv(filename,cellArray,delimiter)
% Writes cell array content into a *.csv file.
%
% CELL2CSV(filename,cellArray,delimiter)
%
% filename = Name of the file to save. [ i.e. 'text.csv' ]
% cellarray = Name of the Cell Array where the data is in
% delimiter = seperating sign, normally:',' (it's default)
%
% by Sylvain Fiedler, KA, 2004
% modified by Rob Kohr, Rutgers, 2005 - changed to english and fixed delimiter
if nargin<3
delimiter = ',';
end
datei = fopen(filename,'w');
for z=1:size(cellArray,1)
for s=1:size(cellArray,2)
var = eval(['cellArray{z,s}']);
if size(var,1) == 0
var = '';
end
if isnumeric(var) == 1
var = num2str(var);
end
fprintf(datei,var);
if s ~= size(cellArray,2)
fprintf(datei,[delimiter]);
end
end
fprintf(datei,'\n');
end
fclose(datei);
end
For MATLAB release 2019a and above, you can use readmatrix and writematrix like this:
M = readmatrix('mat24x8.csv');
r = 1:8;
M1 = m(r,:);
M2 = m(r+8,:);
M3 = m(r+16,:);
% Apply whatever operations you want ..
M1 = sqrt(M1);
M2 = sin(M2);
M3 = cos(M3);
% Finally save again ..
M = [M1; M2; M3]
M = writematrix(M, 'new24x8.csv');
These functions are better than older ones like M = csvread(filename) and csvwrite(filename,M) because it preserves data accuracy. The file extension for older versions is .dat. Also note that white space has no meaning here, so, if you want each matrix stored separately, just save three files.
Update: To work in version R2017a of MATLAB, you should use the csvread(filename)andcsvwrite(filename,M)` I mentioned above in the same way as below.
M = csvread('myFile.csv');
r = 1:8;
M1 = M(r,:);
M2 = M(r+8,:);
M3 = M(r+16,:);
% Apply whatever operations you want ..
M1 = sqrt(M1);
M2 = sin(M2);
M3 = cos(M3);
% Finally save again ..
M = [M1; M2; M3];
csvwrite('myFile.csv',M);
No method of the above will work for arbitrary separators in the csv file other than blank lines.
Related
Processing cannot locate my class in the main method.
I tried looking for extra or missing brackets, but didn't find any.
The error reads "Cannot find a class or type named "Avatar"", and i'm not sure what the problem is. I am fairly confident that the class is set up correctly
class Avatar {
int x1; //location variables of the main body parts
int y1;
int x2;
int y2;
int x3;
int y3;
Avatar() {
x1 = 300;
y1 = 200;
x2 = 300;
y2 = 350;
x3 = 300;
y3 = 550;
}//end of constructor
void display() {
//main body
fill(255); //white
noStroke(); //no line on the outside
ellipse(x1, y1, 150, 150); //head
ellipse(x2, y2, 200, 200); //middle portion
ellipse(x3, y3, 300, 300); //bottom portion
//additional parts
//sunglasses
rectMode(CENTER); //sets all rectangles to draw from the center
fill(0);
rect(260, 190, 40, 25);
rect(340, 190, 40, 25);
ellipse(260, 200.5, 40, 19.5);
ellipse(340, 200.5, 40, 19.5);
stroke(0);
line(240, 180, 350, 180);
line(200, 155, 250, 180);
line(400, 155, 359, 180);
//Smile
arc(float(300), float(211), float(65), float(75), float(0), PI);
//Leather Jacket
fill(0);
ellipse(x2, y2, 200, 200);
fill(175);
ellipse(300, 300, 15, 15);
ellipse(300, 350, 15, 15);
ellipse(300, 400, 15, 15);
//Arms
strokeWeight(25);
fill(0);
line(100, 550, 205, 300);
line(500, 550, 395, 300);
ellipse(100, 550, 25, 25);
ellipse(500, 550, 25, 25);
//belt
noStroke();
rectMode(CENTER);
fill(35);
rect(x2, y2+90, 200, 60);
}//end of display function
}//end of avatar class
the main function is just:
Avatar p1Avatar;
Avatar p2Avatar;
I'm currently trying to apply a 3D CNN to a set of images with the dimensions of 193 x 229 x 193 and would like to retain the same image dimensions through each convolutional layer (similar to tensorflow's padding=SAME). I know that the padding can be calculated as follow:
S=Stride
P=Padding
W=Width
K=Kernal size
P = ((S-1)*W-S+K)/2
Which yields a padding of 1 for the first layer:
P = ((1-1)*193-1+3)/2
P= 1.0
Although I also get a result of 1.0 for each of the subsequent layers. Anyone have any suggestions? Sorry, beginner here!
Reproducible example:
import torch
import torch.nn as nn
x = torch.randn(1, 1, 193, 229, 193)
padding = ((1-1)*96-1+3)/2
print(padding)
x = nn.Conv3d(in_channels=1, out_channels=8, kernel_size=3, padding=1)(x)
print("shape after conv1: " + str(x.shape))
x = nn.Conv3d(in_channels=8, out_channels=8, kernel_size=3,padding=1)(x)
x = nn.BatchNorm3d(8)(x)
print("shape after conv2 + batch norm: " + str(x.shape))
x = nn.ReLU()(x)
print("shape after reLU:" + str(x.shape))
x = nn.MaxPool3d(kernel_size=2, stride=2)(x)
print("shape after max pool" + str(x.shape))
x = nn.Conv3d(in_channels=8, out_channels=16, kernel_size=3,padding=1)(x)
print("shape after conv3: " + str(x.shape))
x = nn.Conv3d(in_channels=16, out_channels=16, kernel_size=3,padding=1)(x)
print("shape after conv4: " + str(x.shape))
Current output:
shape after conv1: torch.Size([1, 8, 193, 229, 193])
shape after conv2 + batch norm: torch.Size([1, 8, 193, 229, 193])
shape after reLU:torch.Size([1, 8, 193, 229, 193])
shape after max pooltorch.Size([1, 8, 96, 114, 96])
shape after conv3: torch.Size([1, 16, 96, 114, 96])
shape after conv4: torch.Size([1, 16, 96, 114, 96])
Desired output:
shape after conv1: torch.Size([1, 8, 193, 229, 193])
shape after conv2 + batch norm: torch.Size([1, 8, 193, 229, 193])
...
shape after conv3: torch.Size([1, 16, 193, 229, 193])
shape after conv4: torch.Size([1, 16, 193, 229, 193])
TLDR; your formula also applies to nn.MaxPool3d
You are using a max pool layer of kernel size 2 (implicitly (2,2,2)) with a stride of 2 (implicitly (2,2,2)). This means for every 2x2x2 block you're only getting a single value. In other words - as the name implies: only the maximum value from every 2x2x2 block is pooled to the output array.
That's why you're going from (1, 8, 193, 229, 193) to (1, 8, 96, 114, 96) (notice the division by 2).
Of course, if you set kernel_size=3 and stride=1 on nn.MaxPool3d, you will preserve the shape of your blocks.
Let #x be the input shape, and #w the kernel shape. If we want the output to have the same size, then #x = floor((#x + 2p - #w)/s + 1) needs to be true. That's 2p = s(#x - 1) - #x + #w = #x(s - 1) + #w - s (your formula)
Since s = 2 and #w = 2, then 2p = #x which is not possible.
class Model(nn.Module):
def __init__(self):
super(Model, self).__init__()
self.net = nn.Sequential(
nn.Conv2d(in_channels = 3, out_channels = 16),
nn.ReLU(),
nn.MaxPool2d(2),
nn.Conv2d(in_channels = 16, out_channels = 16),
nn.ReLU(),
Flatten(),
nn.Linear(4096, 64),
nn.ReLU(),
nn.Linear(64, 10))
def forward(self, x):
return self.net(x)
I have created this model without a firm knowledge in Neural Network and I just fixed parameters until it worked in the training. I am not sure how to get the output dimension for each layer (e.g. output dimension after the first layer).
Is there an easy way to do this in Pytorch?
You can use torchsummary, for instance, for ImageNet dimension(3x224x224):
from torchvision import models
from torchsummary import summary
vgg = models.vgg16()
summary(vgg, (3, 224, 224)
----------------------------------------------------------------
Layer (type) Output Shape Param #
================================================================
Conv2d-1 [-1, 64, 224, 224] 1,792
ReLU-2 [-1, 64, 224, 224] 0
Conv2d-3 [-1, 64, 224, 224] 36,928
ReLU-4 [-1, 64, 224, 224] 0
MaxPool2d-5 [-1, 64, 112, 112] 0
Conv2d-6 [-1, 128, 112, 112] 73,856
ReLU-7 [-1, 128, 112, 112] 0
Conv2d-8 [-1, 128, 112, 112] 147,584
ReLU-9 [-1, 128, 112, 112] 0
MaxPool2d-10 [-1, 128, 56, 56] 0
Conv2d-11 [-1, 256, 56, 56] 295,168
ReLU-12 [-1, 256, 56, 56] 0
Conv2d-13 [-1, 256, 56, 56] 590,080
ReLU-14 [-1, 256, 56, 56] 0
Conv2d-15 [-1, 256, 56, 56] 590,080
ReLU-16 [-1, 256, 56, 56] 0
MaxPool2d-17 [-1, 256, 28, 28] 0
Conv2d-18 [-1, 512, 28, 28] 1,180,160
ReLU-19 [-1, 512, 28, 28] 0
Conv2d-20 [-1, 512, 28, 28] 2,359,808
ReLU-21 [-1, 512, 28, 28] 0
Conv2d-22 [-1, 512, 28, 28] 2,359,808
ReLU-23 [-1, 512, 28, 28] 0
MaxPool2d-24 [-1, 512, 14, 14] 0
Conv2d-25 [-1, 512, 14, 14] 2,359,808
ReLU-26 [-1, 512, 14, 14] 0
Conv2d-27 [-1, 512, 14, 14] 2,359,808
ReLU-28 [-1, 512, 14, 14] 0
Conv2d-29 [-1, 512, 14, 14] 2,359,808
ReLU-30 [-1, 512, 14, 14] 0
MaxPool2d-31 [-1, 512, 7, 7] 0
Linear-32 [-1, 4096] 102,764,544
ReLU-33 [-1, 4096] 0
Dropout-34 [-1, 4096] 0
Linear-35 [-1, 4096] 16,781,312
ReLU-36 [-1, 4096] 0
Dropout-37 [-1, 4096] 0
Linear-38 [-1, 1000] 4,097,000
================================================================
Total params: 138,357,544
Trainable params: 138,357,544
Non-trainable params: 0
----------------------------------------------------------------
Input size (MB): 0.57
Forward/backward pass size (MB): 218.59
Params size (MB): 527.79
Estimated Total Size (MB): 746.96
----------------------------------------------------------------
Source: model-summary-in-pytorch
A simple way is:
Pass the input to the model.
Print the size of the output after passing every layer.
class Model(nn.Module):
def __init__(self):
super(Model, self).__init__()
self.net = nn.Sequential(
nn.Conv2d(in_channels = 3, out_channels = 16),
nn.ReLU(),
nn.MaxPool2d(2),
nn.Conv2d(in_channels = 16, out_channels = 16),
nn.ReLU(),
Flatten(),
nn.Linear(4096, 64),
nn.ReLU(),
nn.Linear(64, 10))
def forward(self, x):
for layer in self.net:
x = layer(x)
print(x.size())
return x
model = Model()
x = torch.randn(1, 3, 224, 224)
# Let's print it
model(x)
But be careful with the input size because you are using nn.Linear in your net. It would cause incompatible input size for nn.Linear if your input size is not 4096.
Like David Ng's answer but a tad shorter:
def get_output_shape(model, image_dim):
return model(torch.rand(*(image_dim))).data.shape
In this example I needed to figure out the input of the last Linear layer:
class Net(nn.Module):
def __init__(self):
super(Net, self).__init__()
self.expected_input_shape = (1, 1, 192, 168)
self.conv1 = nn.Conv2d(1, 32, 3, 1)
self.conv2 = nn.Conv2d(32, 64, 3, 1)
self.dropout1 = nn.Dropout2d(0.25)
self.dropout2 = nn.Dropout2d(0.5)
self.maxpool1 = nn.MaxPool2d(2)
self.maxpool2 = nn.MaxPool2d(3)
# Calculate the input of the Linear layer
conv1_out = get_output_shape(self.maxpool1, get_output_shape(conv1, self.expected_input_shape))
conv2_out = get_output_shape(self.maxpool2, get_output_shape(conv2, conv1_out))
fc1_in = np.prod(list(conv2_out)) # Flatten
self.fc1 = nn.Linear(fc1_in, 38)
def forward(self, x):
x = self.conv1(x)
x = F.relu(x)
x = self.maxpool1(x)
x = self.conv2(x)
x = F.relu(x)
x = self.maxpool2(x)
x = self.dropout1(x)
x = torch.flatten(x, 1) # flatten to a single dimension
x = self.fc1(x)
output = F.log_softmax(x, dim=1)
return output
This way, if I make changes to previous layers, I won't have to calculate all over again!
My answer is based on this answer
Another way to get the size after a certain layer in an nn.Sequential container is to add a custom Module that just prints out the size of the input.
class PrintSize(nn.Module):
def __init__(self):
super(PrintSize, self).__init__()
def forward(self, x):
print(x.shape)
return x
And now you can do:
model = nn.Sequential(
nn.Conv2d(3, 10, 5, 1),
// lots of convolutions, pooling, etc.
nn.Flatten(),
PrintSize(),
nn.Linear(1, 12), // the input dim of 1 is just a placeholder
)
Now, you can do model(x) and it will print out the shape of the output after the Conv2d layer ran. This is useful if you have a lot of convolutions and want to figure out what the final dimensions are for the first fully connected layer. You don't need to reformat your nn.Sequential as a Module and can just drop in this helper class with one-line.
for layer in model.children():
if hasattr(layer, 'out_features'):
print(layer.out_features)
Here's a solution in the form of a helper function:
def get_tensor_dimensions_impl(model, layer, image_size, for_input=False):
t_dims = None
def _local_hook(_, _input, _output):
nonlocal t_dims
t_dims = _input[0].size() if for_input else _output.size()
return _output
layer.register_forward_hook(_local_hook)
dummy_var = torch.zeros(1, 3, image_size, image_size)
model(dummy_var)
return t_dims
Example:
from torchvision import models, transforms
a_model = models.squeezenet1_0(pretrained=True)
get_tensor_dimensions_impl(a_model, a_model._modules['classifier'], 224)
Output is:
torch.Size([1, 1000, 1, 1])
Maybe you can try print(model.state_dict()['next_layer.weight'].shape).
This gives you a hint of the output shape from last layer.
I created the following three dimensional mockup matrix:
mockup(:,:,1) = ...
[100, 100, 100; ...
103, 95, 100; ...
101, 85, 100; ...
96, 90, 102; ...
91, 89, 99; ...
97, 91, 97; ...
105, 83, 100];
mockup(:,:,2) = ...
[50, NaN, NaN; ...
47, NaN, 40; ...
45, 60, 45; ...
47, 65, 45; ...
51, 70, 45; ...
54, 65, 50; ...
62, 80, 55];
I also defined percentTickerAvailable = 0.5.
As a result, The columns represent equity prices from three different assets. For futher processing I need to manipulate the NaN values in the following way.
If the percentage of NaNs in any given ROW is greater than 1 - percentTickerAvailable, replace all values in these particular rows with NaNs. That is, if not enough assets have prices in that particular row, ignore the row completely.
If the percentage of NaNs in any given ROW is less or equal to 1 - percentTickerAvailable, replace the respective NaNs with -inf.
To be clear, "percentage of NaNs in any given ROW" is calculated as follows:
Number of NaNs in any given ROW divided by number of columns.
The adjusted mockup matrix should look like this:
mockupAdj(:,:,1) = ...
[100, 100, 100; ...
103, 95, 100; ...
101, 85, 100; ...
96, 90, 102; ...
91, 89, 99; ...
97, 91, 97; ...
105, 83, 100];
mockupAdj(:,:,2) = ...
[NaN, NaN, NaN; ...
47, -inf, 40; ...
45, 60, 45; ...
47, 65, 45; ...
51, 70, 45; ...
54, 65, 50; ...
62, 80, 55];
So far, I did the following:
function vout = ranking(vin, percentTickerAvailable)
percentNonNaN = 1 - sum(isnan(vin), 2) / size(vin, 2);
NaNIdx = percentNonNaN < percentTickerAvailable;
infIdx = percentNonNaN > percentTickerAvailable & ...
percentNonNaN < 1;
[~, ~, numDimVin] = size(vin);
for i = 1 : numDimVin
vin(NaNIdx(:,:,i) == 1, :, i) = NaN;
end
about = vin;
end % EoF
By calling mockupAdj = ranking(mockup, 0.5) this already transforms the first row in mockup(1,:,2)correctly to {'NaN', 'NaN', 'NaN'}. However, I am struggling with the second point. With infIdx I already successfully identified the rows that corresponds to the second condition. But I don't know how to correctly use that information in order to replace the single NaN in mockup(2,2,2) with -inf.
Any hint is highly appreciated.
This is a good example of something that can be solved using vectorization. I am providing two versions of the code, one that uses the modern syntax (including implicit expansion) and one for older version of MATLAB.
Several things to note:
In the NaN substitution stage, I'm using a "trick" where 0/0 is evaluated to NaN.
In the Inf substitution stage, I'm using logical masking/indexing to access the correct elements in vin.
R2016b and newer:
function vin = ranking (vin, percentTickerAvailable)
% Find percentage of NaNs on each line:
pNaN = mean(isnan(vin), 2, 'double');
% Fills rows with NaNs:
vin = vin + 0 ./ (1 - ( pNaN >= percentTickerAvailable));
% Replace the rest with -Inf
vin(isnan(vin) & pNaN < percentTickerAvailable) = -Inf;
end
Prior to R2016b:
function vin = rankingOld (vin, percentTickerAvailable)
% Find percentage of NaNs on each line:
pNaN = mean(isnan(vin), 2, 'double');
% Fills rows with NaNs:
vin = bsxfun(#plus, vin, 0 ./ (1 - ( pNaN >= percentTickerAvailable)));
% Replace the rest with -Inf
vin(bsxfun(#and, isnan(vin), pNaN < percentTickerAvailable)) = -Inf;
end
1)
The percentage of NaN in any given row should be smaller than 1
... Are you talking about ratio? In which case this is a useless check, as it will always be the case. Or talking about percentages? In which case your code doesn't do what you describe. My guess is ratio.
2) Based on my guess, I have a follow up question: following your description, shouldn't mockup(2,2,2) stay NaN? There is 33% (<50%) of NaN in that row, so it does not fulfill your condition 2.
3) Based on the answers I deemed logical, I would have changed percentNaN = sum(isnan(vin), 2) / size(vin, 2); for readability, and NaNIdx = percentNaN > percentTickerAvailable; accordingly. Now just add one line in front of your loop:
vin(isnan(vin)) = -inf;
Why? Because like this you replace all the NaNs by -inf. Later on, the ones that respect condition 1 will be overwritten to NaN again, by the loop. You don't need the InfIdx.
4) Be aware that your function cannot return vout as of now. Just let it return vin, and you'll be fine.
You can also use logical indexing to achieve this task:
x(:,:,1) = ...
[100, 100, 100; ...
103, 95, 100; ...
101, 85, 100; ...
96, 90, 102; ...
91, 89, 99; ...
97, 91, 97; ...
105, 83, 100];
x(:,:,2) = ...
[50, NaN, NaN; ...
47, NaN, 40; ...
45, 60, 45; ...
47, 65, 45; ...
51, 70, 45; ...
54, 65, 50; ...
62, 80, 55];
% We fix the threshold
tres = 0.5; %fix the threshold.
% We check if a value = NaN or not.
in = isnan(x);
% Which line have more than 50% of NaN ?.
ind = (sum(in,2)./(size(x,2)))>0.5
% We generate an index
[x1,~,x3] = ind2sub(size(ind),ind);
% We set the NaN index to 0 if the line contains less than 50 % of NaN.
in(x1,:,x3) = 0;
% We calculate the new values.
x(in) = -inf;
x(x1,:,x3) = NaN;
I have a matrix of the form:
m = 1, 0, 10, 20, 30, 40, 50;
2, 1, 11, 20, 30, 40, 50;
3, 0, 12, 20, 30, 40, 50;
4, 1, 12, 21, 30, 40, 50;
For a given column index (say 3) and row index (say 1), I'd like to filter out all rows that have the same values to the right of that column in that row. Using the above example of m, columnIndex = 3, and rowIndex = 1 (noted with asterisks):
**
f(m, 3) = * 1, 0, 10, 20, 30, 40, 50; % [20, 30, 40, 50] matches itself, include
2, 1, 11, 20, 30, 40, 50; % [20, 30, 40, 50] matches the subvector in row 1, include
3, 0, 12, 20, 30, 40, 50; % [20, 30, 40, 50] matches the subvector in row 1, include
4, 1, 12, 21, 30, 40, 50; % [21, 30, 40, 50] does NOT match the subvector in row 1, filter this out
How can I achieve this behavior? I've tried this, but I'm getting an dimension mismatch error.
key = data( rowIndex, columnIndex:end );
filteredData = ( data( :, columnIndex:end ) == key );
Index those that you keep with == within bsxfun():
r = 3;
c = 2;
idx = all(bsxfun(#eq, m(:,c:end),m(r,c:end)),2);
m(idx,:)
I think you're looking to use the isequal operator, documented here.
isequal(m(1,columnIndex:end),key)
and here's kind of an inefficient one liner :-)
cellfun(#(x) isequal(key,x),mat2cell(m(:,columnIndex:end),ones(1,size(m,2)-columnIndex+1)))
Here's how it goes
Change the matrix to a cell array of the subvectors we're interested in: mat2cell(m(:,columnIndex:end),ones(1,size(m,2)-columnIndex+1))
Anonymous function to run on each cell element: #(x) isequal(key,x)
Function to cause each cell element to be passed to the anonymous function, cellfun
My answer using m above as an example:
cellfun(#(x) isequal(key,x),mat2cell(m(:,columnIndex:end),ones(1,size(m,2)-columnIndex+1)))
ans =
1
1
1
0
HTH!