I have a question. I am conducting a study using cluster analysis. My number of observations is less than the number of variables which means my matrix in n<p. Is it true that I am violating cluster analysis assumptions? Do I have to reduce my variables?
Related
I performed a cluster analysis(using k means) in some marketing data and I found out that I should use 5 clusters. After that, I would like to see if the subjects in each cluster have a significant relationship with a disease. For each cluster I know if the subject had the disease or not. Is there any way to test if there is a significant relationship between cluster/disease? And if I can do that with a binomial test what should be the parameters? Thanks!
I know how to calculate the Recall, Precision and F_measure for clusters as explained in this course https://www.coursera.org/learn/cluster-analysis/lecture/BcYhV/6-4-external-measures-1-matching-based-measures
However, what if the number of clusters generated by my system is more than the number of clusters in the ground-truth, how can we calculate these measures?
It seems that there is no penalty for systems generating more clusters since we just matching each cluster in the ground-truth to the best cluster generated from my system. Am i missing something here?
Don't compute them as in classification!!!
Either you need to work with pairs of points - that is the most common approach, used by the very popular ARI measure.
Or you need to find the cluster with the maximum overlap, this then sometimes called "matching". I am not convinced of this approach.
Last but not least, you could use the Hungarian algorithm to find the best partial 1:1 correspondence, and consider unmatched clusters to be all false.
I have to cluster data which are power profiles of the solar panel output. I tried various algorithm including classical K-means to shape based clustering as well. I have to decide number of cluster possible in the pool of data. And I am always getting 2 cluster, so I think they are very dense.
Is there any way I can partition dense cluster?
I have test classification datasets from UCI Machine Learning repository which are labelled.
I am stripping of the labels and using the data to benchmark a few clustering algorithm and then I am planning to use external validation methods. I will run the algorithm with different initial configurations, for say, 50 times and then take the mean value. For 50 iterations the algorithm labels the data points of one single cluster with different numbers. Because in each run the cluster labels can change, also because each iteration might have slightly different cluster assignments, how to somehow remap each of the clusters to one uniform numbering.
Primary idea is to remap by checking how many of the points in the class labels intersect the maximum in the actual labels and then making a remap based on that, but this can get incorrect remappings because when the classes will have more or less equal number of points, this will not work.
Another idea is to keep the labels while clustering, but make the clustering algorithm ignore it. This way all the cluster data will have the label tags. This is doable but I have already have a benchmarked cluster assignment data to be processed therefore I am trying to avoid modifying and re-benchmarking my implementation (which will take quite some time and cpu) of the cluster analysis algorithms and include the label tag to the vectors and then ignore it.
Is there any way that I can compute average accuracy from the cluster assignments I have right now?
EDIT:
The domain in which I am studying (metaheuristic clustering algorithms) I could not find a paper comparing these indexes. The paper which compares seems to be incorrect in their values. Can anyone point me to a paper where clustering results are compared using any of these indexes?
What do you do when the number of clusters doesn't agree?
Do not try to map clusters.
Instead, use the proper external validation measures for clustering, which do not require a 1:1 correspondence of clusters. There are plenty, for details see Wikipedia.
When ever we want to cluster some data then It is required to give the number of cluster by user. Like K-Means algorithm we need to specify that how cluster are required.
My question is it possible that the algorithm decides itself that how cluster are feasible for particular data set.
There are several clustering algorithms that do not require a desired number of clusters as an input to the algorithm. An example of such an algorithm is the mean-shift clustering algorithm. However, you will need to specify a kernel as an input to the algorithm. This kernel selection (e.g., the size and shape of the kernel) will impact the number of clusters that you get as an output.
Some more information:
http://homepages.inf.ed.ac.uk/rbf/CVonline/LOCAL_COPIES/TUZEL1/MeanShift.pdf
http://scikit-learn.org/stable/auto_examples/cluster/plot_mean_shift.html
I'm not expert with that, but to answer to your question, yes there are methods to determine automatically the number of cluster for a kmeans for example.
It's quite complicated but given a dataset and a cluster method you can compute what is called gap statistic in order to estime the number of clusters.
If you are a R user, try to check clusGap and maxSE functions.