For teaching purposes, I need to generate random datasets of correlated random variables with different distributions. I have tried corr2data in Stata but it will not allow me to specify max and min values of the variables to be generated, just means, sd's and the covariance matrix. Therefore, I need to do messy adjustments after generation of the data. Various other details annoy me with corr2data. Is there a simpler way of doing this with MATLAB? I am not as familiar with this software as I am with Stata.
If you have access to Statistics Toolbox as well as MATLAB, you can use the copula functionality to do this fairly easily. Using a copula, you can specify the marginal distributions of each variable, and a correlation structure between the variables.
You can then generate random numbers from the copula, fit it to data etc. as well.
See in the MATLAB documentation:
Copulas: Generate Correlated Samples
Related
I have an array of values, with that values I plotted the histogram.I want to know the corresponding distribution from the histogram obtained. How is it possible.
Could you please explain the steps in obtaining appropriate probability distribution from histogram.
You'd better to ask this question in stats.stackexchange.com as it is more about the method than the programming. However, one thing that you can do is to fit a parametric distribution (using moment matching or maximum likelihood for example) then compare the fitted distribution to the normalized histogram using KL divergence or Bhattacharyya distance.
One option might be to use the "Distribution Fitting App" in the Statistics and Machine Learning Toolbox. That should help you evaluate if your data seems like it might have been drawn from some common distributions. You may never know for sure, since multiple distributions could account for the data, but if you have a lot of data it might help you narrow it down.
I think that in many cases an eye-ball comparison is enough. With a reasonable amount of data, it is quite difficult to not be able to distinguish between a gaussian or a weibull or...
I would use fitdist or fithist to eye-ball different distributions.
If you have no idea at all on the distribution and you want to know if two datasets are distributed differently it could be useful to compare their distributions by obtaining them with the option 'kernel'
I use normrnd and lognormrnd to sample numbers according to these two distribution functions. Nevertheless, since I am using quite large standard deviations, I sample several numbers that are above a threshold I put for my code. My question is: is there a way to sample numbers according to this distributions but within a certain threshold without using an if_bigger --> sample_again function?
The pdf for the multivariate normal distribution in MATLAB is mvnpdf(...). What about the case where multiple variables are uniformly distributed: Is there a function to describe their joint distribution analogous to the multivariate normal distribution? If there is no such function, is there a trick to handle this case?
The simplest way how several variables can be uniformly distributed is if they are mutually independent; in that case you simply have a uniform distribution over the hypercube in the space spanned by the variables. In order to get samples from this distribution, you just separately generate samples for each of the variables.
The point where a "trick" might be necessary is if you have dependencies between the variables even though the marginal distribution for each of them is still uniform. In this case you have to describe the dependency structure, and I'm not aware of any standard way to do this (the way dependencies between normally distributed variables are described by a correlation matrix).
Of course such distributions exist: For two dimensions, one possibility would be to have a joint distribution that looks like a solution to the "eight rooks" problem:
Another one actually derives from the introductory Matlab example, the magic square:
Both of these examples are discrete distributions, but can be produced at arbitrary granularity, or simply interpreted as piecewise constant continuous distributions.
As you can see there are many possibilities for a multivariate distribution each of whose marginal distributions are uniform. The question you have to answer for yourself is what kind of dependencies, if any, you are interested in?
If I'm understanding the question properly, we want to calculate the pdf of a multivariate uniform distribution. By definition, the pdf is constant for all values in the support the distribution. Thus to calculate the pdf all that is required is to calculate the norming constant, which is given by the inverse of the integral of the support. That is to say, the pdf is given by
f(x) = 1 / integral(A)
where A is the support set, and x is an element in A. If an analytic solution to integral(A) is not available, then a numerical integrator can be employed.
I have a set of data in a vector. If I were to plot a histogram of the data I could see (by clever inspection) that the data is distributed as the sum of three distributions;
One normal distribution centered around x_1 with variance s_1;
One normal distribution centered around x_2 with variance s_2;
Once lognormal distribution.
My data is obviously a subset of the 'real' data.
What I would like to do is to take a random subset of my data away from my data ensuring that the resulting subset is a reasonable representative sample of the original data.
I would like to do this as easily as possible in matlab but am new to both statistics and matlab and am unsure where to start.
Thank you for any help :)
If you can identify each of the 3 distributions (in the sense that you can estimate their parameters), one approach could be to select a random subset of your data and then try to estimate the parameters for each distribution and see whether they are close enough (according to your own definition of "close") to the parameters of the original distributions. You should repeat this process several time and look at the average difference given a random subset size.
I have a dataset of n data, where each data is represented by a set of extracted features. Generally, the clustering algorithms need that all input data have the same dimensions (the same number of features), that is, the input data X is a n*d matrix of n data points each of which has d features.
In my case, I've previously extracted some features from my data but the number of extracted features for each data is most likely to be different (I mean, I have a dataset X where data points have not the same number of features).
Is there any way to adapt them, in order to cluster them using some common clustering algorithms requiring data to be of the same dimensions.
Thanks
Sounds like the problem you have is that it's a 'sparse' data set. There are generally two options.
Reduce the dimensionality of the input data set using multi-dimensional scaling techniques. For example Sparse SVD (e.g. Lanczos algorithm) or sparse PCA. Then apply traditional clustering on the dense lower dimensional outputs.
Directly apply a sparse clustering algorithm, such as sparse k-mean. Note you can probably find a PDF of this paper if you look hard enough online (try scholar.google.com).
[Updated after problem clarification]
In the problem, a handwritten word is analyzed visually for connected components (lines). For each component, a fixed number of multi-dimensional features is extracted. We need to cluster the words, each of which may have one or more connected components.
Suggested solution:
Classify the connected components first, into 1000(*) unique component classifications. Then classify the words against the classified components they contain (a sparse problem described above).
*Note, the exact number of component classifications you choose doesn't really matter as long as it's high enough as the MDS analysis will reduce them to the essential 'orthogonal' classifications.
There are also clustering algorithms such as DBSCAN that in fact do not care about your data. All this algorithm needs is a distance function. So if you can specify a distance function for your features, then you can use DBSCAN (or OPTICS, which is an extension of DBSCAN, that doesn't need the epsilon parameter).
So the key question here is how you want to compare your features. This doesn't have much to do with clustering, and is highly domain dependant. If your features are e.g. word occurrences, Cosine distance is a good choice (using 0s for non-present features). But if you e.g. have a set of SIFT keypoints extracted from a picture, there is no obvious way to relate the different features with each other efficiently, as there is no order to the features (so one could compare the first keypoint with the first keypoint etc.) A possible approach here is to derive another - uniform - set of features. Typically, bag of words features are used for such a situation. For images, this is also known as visual words. Essentially, you first cluster the sub-features to obtain a limited vocabulary. Then you can assign each of the original objects a "text" composed of these "words" and use a distance function such as cosine distance on them.
I see two options here:
Restrict yourself to those features for which all your data-points have a value.
See if you can generate sensible default values for missing features.
However, if possible, you should probably resample all your data-points, so that they all have values for all features.