I am trying to apply graph cut method for my segmentation task. I found some example codes at
Graph_Cut_Demo.
Part of the codes are showing below
img = im2double( imread([ImageDir 'cat.jpg']) );
[ny,nx,nc] = size(img);
d = reshape( img, ny*nx, nc );
k = 2; % number of clusters
[l0 c] = kmeans( d, k );
l0 = reshape( l0, ny, nx );
% For each class, the data term Dc measures the distance of
% each pixel value to the class prototype. For simplicity, standard
% Euclidean distance is used. Mahalanobis distance (weighted by class
% covariances) might improve the results in some cases. Note that the
% image intensity values are in the [0,1] interval, which provides
% normalization.
Dc = zeros( ny, nx, k );
for i = 1:k
dif = d - repmat( c(i,:), ny*nx,1 );
Dc(:,:,i) = reshape( sum(dif.^2,2), ny, nx );
end
It seems that the method used k-means clustering to initialise the graph and get the data term Dc. However, I don't understand how they calculate this data term. Why they use
dif = d - repmat( c(i,:), ny*nx,1 );
In the comments thy said the data term Dc measures the distance of each pixel value to the class prototype. What is the class prototype, and why it can be determined by k-means label?
In another implementation Graph_Cut_Demo2, it used
% calculate the data cost per cluster center
Dc = zeros([sz(1:2) k],'single');
for ci=1:k
% use covariance matrix per cluster
icv = inv(cov(d(l0==ci,:)));
dif = d- repmat(c(ci,:), [size(d,1) 1]);
% data cost is minus log likelihood of the pixel to belong to each
% cluster according to its RGB value
Dc(:,:,ci) = reshape(sum((dif*icv).*dif./2,2),sz(1:2));
end
This confused me a lot. Why they calculate the covariance matrix and how they formed the data term using minus log likelihood? Any papers or descriptions available for these implementation?
Thanks a lot.
Both graph-cut segmentation examples are strongly related. The authors of Image Processing, Analysis, and Machine Vision: A MATLAB Companion book (first example) used the graph cut wrapper code of Shai Bagon (with the author's permission naturally) - the second example.
So, what is the data term anyway?
The data term represent how each pixel independently is likely to belong to each label. This is why log-likelihood terms are used.
More concretely, in these examples you try to segment the image into k segments based on their colors.
You assume there are only k dominant colors in the image (not a very practical assumption, but sufficient for educational purposes).
Using k-means you try and find what are these two colors. The output of k-means is k centers in RGB space - that is k "representative" colors.
The likelihood of each pixel to belong to any of the k centers is inversely proportional to the distance (in color space) of the pixel from the representative k-th center: the larger the distance the less likely the pixel to belong to the k-th center, the higher the unary energy penalty one must "pay" to assign this pixel to the k-th cluster.
The second example takes this notion one step ahead and assumes that the k clusters may have different densities in color space, modeling this second order behavior using a covariance matrix for each cluster.
In practice, one uses a more sophisticated color model for each segment, usually a mixture of Gaussians. You can read about it in the seminal paper of GrabCut (section 3).
PS,
Next time you can email Shai Bagon directly and ask.
Related
Greetins,
How can I calculate how many distance calculations would need to be performed to classify the IRIS dataset using Nearest Mean Classifier.
I know that IRIS dataset has 4 features and every record is classified according to 3 different labels.
According to some textbooks, the calculation can be carried out as follow:
However, I am lost on these different notations and what does this equation mean. For example, what is s^2 is in the equation?
The notation is standard with most machine learning textbooks. s in this case is the sample standard deviation for the training set. It is quite common to assume that each class has the same standard deviation, which is why every class is assigned the same value.
However you shouldn't be paying attention to that. The most important point is when the priors are equal. This is a fair assumption which means that you expect that the distribution of each class in your dataset are roughly equal. By doing this, the classifier simply boils down to finding the smallest distance from a training sample x to each of the other classes represented by their mean vectors.
How you'd compute this is quite simple. In your training set, you have a set of training examples with each example belonging to a particular class. For the case of the iris dataset, you have three classes. You find the mean feature vector for each class, which would be stored as m1, m2 and m3 respectively. After, to classify a new feature vector, simply find the smallest distance from this vector to each of the mean vectors. Whichever one has the smallest distance is the class you'd assign.
Since you chose MATLAB as the language, allow me to demonstrate with the actual iris dataset.
load fisheriris; % Load iris dataset
[~,~,id] = unique(species); % Assign for each example a unique ID
means = zeros(3, 4); % Store the mean vectors for each class
for i = 1 : 3 % Find the mean vectors per class
means(i,:) = mean(meas(id == i, :), 1); % Find the mean vector for class 1
end
x = meas(10, :); % Choose a random row from the dataset
% Determine which class has the smallest distance and thus figure out the class
[~,c] = min(sum(bsxfun(#minus, x, means).^2, 2));
The code is fairly straight forward. Load in the dataset and since the labels are in a cell array, it's handy to create a new set of labels that are enumerated as 1, 2 and 3 so that it's easy to isolate out the training examples per class and compute their mean vectors. That's what's happening in the for loop. Once that's done, I choose a random data point from the training set then compute the distance from this point to each of the mean vectors. We choose the class that gives us the smallest distance.
If you wanted to do this for the entire dataset, you can but that will require some permutation of the dimensions to do so.
data = permute(meas, [1 3 2]);
means_p = permute(means, [3 1 2]);
P = sum(bsxfun(#minus, data, means_p).^2, 3);
[~,c] = min(P, [], 2);
data and means_p are the transformed features and mean vectors in a way that is a 3D matrix with a singleton dimension. The third line of code computes the distances vectorized so that it finally generates a 2D matrix with each row i calculating the distance from the training example i to each of the mean vectors. We finally find the class with the smallest distance for each example.
To get a sense of the accuracy, we can simply compute the fraction of the total number of times we classified correctly:
>> sum(c == id) / numel(id)
ans =
0.9267
With this simple nearest mean classifier, we have an accuracy of 92.67%... not bad, but you can do better. Finally, to answer your question, you would need K * d distance calculations, with K being the number of examples and d being the number of classes. You can clearly see that this is required by examining the logic and code above.
I've got an arbitrary probability density function discretized as a matrix in Matlab, that means that for every pair x,y the probability is stored in the matrix:
A(x,y) = probability
This is a 100x100 matrix, and I would like to be able to generate random samples of two dimensions (x,y) out of this matrix and also, if possible, to be able to calculate the mean and other moments of the PDF. I want to do this because after resampling, I want to fit the samples to an approximated Gaussian Mixture Model.
I've been looking everywhere but I haven't found anything as specific as this. I hope you may be able to help me.
Thank you.
If you really have a discrete probably density function defined by A (as opposed to a continuous probability density function that is merely described by A), you can "cheat" by turning your 2D problem into a 1D problem.
%define the possible values for the (x,y) pair
row_vals = [1:size(A,1)]'*ones(1,size(A,2)); %all x values
col_vals = ones(size(A,1),1)*[1:size(A,2)]; %all y values
%convert your 2D problem into a 1D problem
A = A(:);
row_vals = row_vals(:);
col_vals = col_vals(:);
%calculate your fake 1D CDF, assumes sum(A(:))==1
CDF = cumsum(A); %remember, first term out of of cumsum is not zero
%because of the operation we're doing below (interp1 followed by ceil)
%we need the CDF to start at zero
CDF = [0; CDF(:)];
%generate random values
N_vals = 1000; %give me 1000 values
rand_vals = rand(N_vals,1); %spans zero to one
%look into CDF to see which index the rand val corresponds to
out_val = interp1(CDF,[0:1/(length(CDF)-1):1],rand_vals); %spans zero to one
ind = ceil(out_val*length(A));
%using the inds, you can lookup each pair of values
xy_values = [row_vals(ind) col_vals(ind)];
I hope that this helps!
Chip
I don't believe matlab has built-in functionality for generating multivariate random variables with arbitrary distribution. As a matter of fact, the same is true for univariate random numbers. But while the latter can be easily generated based on the cumulative distribution function, the CDF does not exist for multivariate distributions, so generating such numbers is much more messy (the main problem is the fact that 2 or more variables have correlation). So this part of your question is far beyond the scope of this site.
Since half an answer is better than no answer, here's how you can compute the mean and higher moments numerically using matlab:
%generate some dummy input
xv=linspace(-50,50,101);
yv=linspace(-30,30,100);
[x y]=meshgrid(xv,yv);
%define a discretized two-hump Gaussian distribution
A=floor(15*exp(-((x-10).^2+y.^2)/100)+15*exp(-((x+25).^2+y.^2)/100));
A=A/sum(A(:)); %normalized to sum to 1
%plot it if you like
%figure;
%surf(x,y,A)
%actual half-answer starts here
%get normalized pdf
weight=trapz(xv,trapz(yv,A));
A=A/weight; %A normalized to 1 according to trapz^2
%mean
mean_x=trapz(xv,trapz(yv,A.*x));
mean_y=trapz(xv,trapz(yv,A.*y));
So, the point is that you can perform a double integral on a rectangular mesh using two consecutive calls to trapz. This allows you to compute the integral of any quantity that has the same shape as your mesh, but a drawback is that vector components have to be computed independently. If you only wish to compute things which can be parametrized with x and y (which are naturally the same size as you mesh), then you can get along without having to do any additional thinking.
You could also define a function for the integration:
function res=trapz2(xv,yv,A,arg)
if ~isscalar(arg) && any(size(arg)~=size(A))
error('Size of A and var must be the same!')
end
res=trapz(xv,trapz(yv,A.*arg));
end
This way you can compute stuff like
weight=trapz2(xv,yv,A,1);
mean_x=trapz2(xv,yv,A,x);
NOTE: the reason I used a 101x100 mesh in the example is that the double call to trapz should be performed in the proper order. If you interchange xv and yv in the calls, you get the wrong answer due to inconsistency with the definition of A, but this will not be evident if A is square. I suggest avoiding symmetric quantities during the development stage.
I was trying to code for feature extraction from the two images, which are actually similar. I tried to extract the intersection points from both of the image and calculated the distance from one intersection point to all other points. This procedure was iterated for all points and in both images.
Then I compared the distance between points in both images But I found that even for dissimilar images am getting same kind of distance and am not able to distinguish them.
Is there any way in this method which will improve the code or is there any other way to find the similarity.
I = bwmorph(I,'skel',Inf);
II = bwmorph(II,'skel',Inf);
[i,j] = ind2sub(size(I),find(bwmorph(bwmorph(I,'thin',Inf),'branchpoint') == 1));
[i1,j1] = ind2sub(size(II),find(bwmorph(bwmorph(II,'thin',Inf),'branchpoint') == 1));
figure,imshow(I); hold on; plot(j,i,'rx');
figure,imshow(II); hold on; plot(j1,i1,'rx')
m=size(i,1);
n=size(j,1);
m1=size(i1,1);
n1=size(j1,1);
for x=1:m
for y=1:n
d1(y,x)=round(sqrt((i(y,1)-i(x,1)).^2+(j(y,1)-j(x,1)).^2));
end
end
for x1=1:m1
for y1=1:n1
dd1(y1,x1)=round(sqrt((i1(y1,1)-i1(x1,1)).^2+(j1(y1,1)-j1(x1,1)).^2));
end
end
size(d1);
k1=reshape(d1,1,m*n);
k=sort(k1);
k=unique(k);
size(dd1);
k2=reshape(dd1,1,m1*n1);
k2=sort(k2);
k2=unique(k2);
z = intersect(k,k2)
length(z);
if length(z)>20
disp('similar images');
else
disp('dissimilar images');
end
This is a part of my code where I tried to extract features.
input1
input2
skel 1
skel2
I think your code is not the problem. Instead, it seems that either your feature descriptor is not powerful enough or your comparison method is not powerful enough, or a combination of the two. This gives us several options for how to explore solutions to the problem.
Feature Descriptor
You are constructing an image feature consisting of the distances between skeleton intersection points. This is an unusual approach and a very interesting one. It reminds me of peak constellations, a feature used by Shazam to audio-fingerprint songs. If you are interested in exploring, that more sophisticated technique, take a look at "An Industrial Strength Audio Search Algorithm" by Avery Li-Chun Wang. I believe you could adapt their feature descriptor to your application.
However, if you want a simpler solution there are some other options as well. Your current descriptor uses unique to find a set of unique distances between the skeleton intersection points. Take a look at the following images of a line and an equilateral triangle both with 5 unit line lengths. If we use the unique distances between vertices to make the feature, the two images have identical features, but we can also count the number of lines of each length in a histogram.
The histogram preserves more of the image structure as part of the feature. Using a histogram might help distinguish better between your similar and dissimilar cases.
Here's some demo code for histogram features using the Matlab demo images pears.png and peppers.png. I had difficulty extracting the skeleton from your provided images, but you should be able to adapt this code easily to your application.
I1 = = im2bw(imread('peppers.png'));
I2 = = im2bw(imread('pears.png'));
I1_skel = bwmorph(I1,'skel',Inf);
I2_skel = bwmorph(I2,'skel',Inf);
[i1,j1] = ind2sub(size(I1_skel),find(bwmorph(bwmorph(I1_skel,'thin',Inf),'branchpoint') == 1));
[i2,j2] = ind2sub(size(I2_skel),find(bwmorph(bwmorph(I2_skel,'thin',Inf),'branchpoint') == 1));
%You used a for loop to find the distance between each pair of
%intersections. There is a function for this.
d1 = round(pdist2([i1, j1], [i1, j1]));
d2 = round(pdist2([i2, j2], [i2, j2]));
%Choose a number of bins for the histogram.
%This will be the length of the feature.
%More bins will preserve more structure.
%Fewer bins will help generalize between similar but not identical images.
num_bins = 100;
%Instead of using `unique` to remove repetitions use `histcounts` in R2014b
%feature1 = histcounts(d1(:), num_bins);
%feature2 = histcounts(d2(:), num_bins);
%Use `hist` for pre R2014b Matlab versions
feature1 = hist(d1(:), num_bins);
feature2 = hist(d2(:), num_bins);
%Normalize the features
feature1 = feature1 ./ norm(feature1);
feature2 = feature2 ./ norm(feature2);
figure; bar([feature1; feature2]');
title('Features'); legend({'Feature 1', 'Feature 2'});
xlim([0, num_bins]);
Here are what the detected intersection points are in each image
Here are the resulting features. You can see the clear differences between images.
Feature Comparison
The second part to consider is how you compare your features. Currently, you are simply looking for >20 similar distances. With the 'peppers.png' and 'pears.png' test images distributed with Matlab, I find more than 2000 intersection points in one image and 260 in the other. With so many points, it is trivial to have an overlap of >20 similar distances. In your images, the number of intersection points is much smaller. You could carefully adjust the threshold of similar distances, but I think this metric is probably to simplistic.
In Machine Learning, a simple way to compare two feature vectors is vector similarity or distance. There are multiple distance metrics you could explore. Common ones include
Cosine Distance
score_cosine = feature1 * feature2'; %Cosine distance between vectors
%Set a threshold for cosine similarity [0, 1] where 1 is identical and 0 is perpendicular
cosine_threshold = .9;
disp('Cosine Compare')
disp(score_cosine)
if score_cosine > cosine_threshold
disp('similar images');
else
disp('dissimilar images');
end
Euclidean Distance
score_euclidean = pdist2(feature1, feature2);
%Set a threshold for euclidean similarity where smaller is more similar
euclidean_threshold = 0.1;
disp('Euclidean Compare')
disp(score_euclidean)
if score_euclidean < euclidean_threshold
disp('similar images');
else
disp('dissimilar images');
end
If these don't work, you may need to train a classifier to find a more complicated function to distinguish between similar and dissimilar images.
I have this matrix:
x = [2+2*i 2-2*i -2+2*i -2-2*i];
I want to simulate transmitting it and adding noise to it. I represented the components of the complex number as below:
A = randn(150, 2) + 2*ones(150, 2); C = randn(150, 2) - 2*ones(150, 2);
At the receiver, I received the below vector, where the components are ordered based on what I sent originally, i.e., the components of x).
X = [A A A C C A C C];
Now I want to apply the kmeans(X) to have four clusters, so kmeans(X, 4). I am experiencing the following problems:
I am not sure if I can represent the complex numbers as shown in X above.
I can't plot the result of the kmeans to show the clusters.
I could not understand the clusters centroid results.
How can I find the best error rate, if this example was to represent a communication system and at the receiver, k-means clustering was used in order to decide what the transmitted signal was?
If you don't "understand" the cluster centroid results, then you don't understand how k-means works. I'll present a small summary here.
How k-means works is that for some data that you have, you want to group them into k groups. You initially choose k random points in your data, and these will have labels from 1,2,...,k. These are what we call the centroids. Then, you determine how close the rest of the data are to each of these points. You then group those points so that whichever points are closest to any of these k points, you assign those points to belong to that particular group (1,2,...,k). After, for all of the points for each group, you update the centroids, which actually is defined as the representative point for each group. For each group, you compute the average of all of the points in each of the k groups. These become the new centroids for the next iteration. In the next iteration, you determine how close each point in your data is to each of the centroids. You keep iterating and repeating this behaviour until the centroids don't move anymore, or they move very little.
Now, let's answer your questions one-by-one.
1. Complex number representation
k-means in MATLAB doesn't define how complex data is handled. A common way for people to deal with complex numbered data is to split up the real and imaginary parts into separate dimensions as you have done. This is a perfectly valid way to use k-means for complex valued data.
See this post on the MathWorks MATLAB forum for more details: https://www.mathworks.com/matlabcentral/newsreader/view_thread/78306
2. Plot the results
You aren't constructing your matrix X properly. Note that A and C are both 150 x 2 matrices. You need to structure X such that each row is a point, and each column is a variable. Therefore, you need to concatenate your A and C row-wise. Therefore:
X = [A; A; A; C; C; A; C; C];
Note that you have duplicate points. This is actually no different than doing X = [A; C]; as far as kmeans is concerned. Perhaps you should generate X, then add the noise in rather than taking A and C, adding noise, then constructing your signal.
Now, if you want to plot the results as well as the centroids, what you need to do is use the two output version of kmeans like so:
[idx, centroids] = kmeans(X, 4);
idx will contain the cluster number that each point in X belongs to, and centroids will be a 4 x 2 matrix where each row tells you the mean of each cluster found in the data. If you want to plot the data, as well as the clusters, you simply need to do following. I'm going to loop over each cluster membership and plot the results on a figure. I'm also going to colour in where the mean of each cluster is located:
x = X(:,1);
y = X(:,2);
figure;
hold on;
colors = 'rgbk';
for num = 1 : 4
plot(x(idx == num), y(idx == num), [colors(num) '.']);
end
plot(centroids(:,1), centroids(:,2), 'c.', 'MarkerSize', 14);
grid;
The above code goes through each cluster, plots them in a different colour, then plots the centroids in cyan with a slightly larger thickness so you can see what the graph looks like.
This is what I get:
3. Understanding centroid results
This is probably because you didn't construct X properly. This is what I get for my centroids:
centroids =
-1.9176 -2.0759
1.5980 2.8071
2.7486 1.6147
0.8202 0.8025
This is pretty self-explanatory and I talked about how this is structured earlier.
4. Best representation of the signal
What you can do is repeat the clustering a number of times, then the algorithm will decide what the best clustering was out of these times. You would simply use the Replicates flag and denote how many times you want this run. Obviously, the more times you run this, the better your results may be. Therefore, do something like:
[idx, centroids] = kmeans(X, 4, 'Replicates', 5);
This will run kmeans 5 times and give you the best centroids of these 5 times.
Now, if you want to determine what the best sequence that was transmitted, you'd have to split up your X into 150 rows each (as your random sequence was 150 elements), then run a separate kmeans on each subset. You can try to find the best representation of each part of the sequence by using the Replicates flag each time.... so you can do something like:
for num = 1 : 8
%// Look at 150 points at a time
[idx, centroids] = kmeans(X((num-1)*150 + 1 : num*150, :), 4, 'Replicates', 5);
%// Do your analysis
%//...
%//...
end
idx and centroids would be the results for each portion of your transmitted signal. You probably want to look at centroids at each iteration to determine what symbol was transmitted at a particular time.
If you want to plot the decision regions, then you're probably looking for a Voronoi diagram. All you do is given a set of points that are defined within the domain of your problem, you just have to determine which cluster each point belongs to. Given that our data spans between -5 <= (x,y) <= 5, let's go through each point in the grid and determine which cluster each point belongs to. We'd then colour the appropriate point according to which cluster it belongs to.
Something like:
colors = 'rgbk';
[X,Y] = meshgrid(-5:0.05:5, -5:0.05:5);
X = X(:);
Y = Y(:);
figure;
hold on;
for idx = 1 : numel(X)
[~,ind] = min(sum(bsxfun(#minus, [X(idx) Y(idx)], centroids).^2, 2));
plot(X(idx), Y(idx), [colors(ind), '.']);
end
plot(centroids(:,1), centroids(:,2), 'c.', 'MarkerSize', 14);
The above code will plot the decision regions / Voronoi diagram of the particular configuration, as well as where the cluster centres are located. Note that the code is rather unoptimized and it'll take a while for the graph to generate, but I wanted to write something quick to illustrate my point.
Here's what the decision regions look like:
Hope this helps! Good luck!
Picture after segmentation with Euclidean distance (just absolute , not absolute squared)
Original texture picture
I'm getting the result above (picture 1) when I perform clustering using Kmeans algorithm and Laws Texture Energy filters (with cluster centroids / groups =6)
What are the possible ways of improving the result ? As can be seen from the result, there is no clear demarcation of the textures.
Could dilation /erosion somehow be implemented for the same ? If yes, please guide.
Analysing the texture using k-means cause you to disregard spatial relations between neighboring pixels: If i and j are next to each other, then it is highly likely that they share the same texutre.
One way of introducing such spatial information is using pair-wise energy that can be optimized using graph cuts or belief-propagation (among other things).
Suppose you have n pixels in the image and L centroids in your k-means, then
D is an L-by-n matrix with D(i,l) is the distance of pixel i to center l.
If you choose to use graph cuts, you can download my wrapper (don't forget to compile it) and then, in Matlab:
>> sz = size( img ); % n should be numel(img)
>> [ii jj] = sparse_adj_matrix( sz, 1, 1 ); % define 4-connect neighbor grid
>> grid = sparse( ii, jj, 1, n, n );
>> gch = GraphCut('open', D, ones( L ) - eye(L), grid );
>> [gch ll] = GraphCut('expand', gch );
>> gch = GraphCut('close', gch );
>> ll = reshape( double(ll)+1, sz );
>> figure; imagesc(ll);colormap (rand(L,3) ); title('resulting clusters'); axis image;
You can find sparse_adj_matrix here.
For a recent implementation of many optimization algorithms, take a look at opengm package.
With respect morphological filtering i suggest this reference: Texture Segmentation Using Area Morphology Local Granulometries. The paper basically describes a morphological area opening filter which removes grayscale components which are smaller than a given area parameter threshold. In binary images the local granulometric size distributions can be generated by placing a window at each image pixel position and, after each opening operation, counting the number of remaining pixels within. This results in a local size distribution, that can be normalised to give the local pdf . Differentiating the pattern spectra gives the density that yields the local pattern spectrum at the pixel, providing a probability density which contains textural information local to each pixel position.
Here is an example to use the granulometries of an image. They are basically non linear scale spaces which work on the area of the grayscale components. The basic intuition is each texture can be characterized based on their spectrum of areas of their grayscale components. A simple binary area opening filter is available in Matlab.