I have a set of data that I need to find a best fit polynomial for. Currently I am using python's polyfit, but there are obvious outliers (when the value drops to 0 on the parabola) that affect my polynomial. How would I go about filtering out the unwanted drops in the data so that I get a best fit parabola that does not include the noise?
There is an obvious parabolic path that I would like to get the polynomial to.
Is there a matlab function that I could use to filter out the noise so polyfit would be a better representation of the parabolic data path?
You can filter a 1D function by convolution.
I used this data set:
y=[0 10 50 80 100 0 100 0 110 0 130 130 0 130 0 110 0 100 0 70 0 50 20 10 0];
And when convoluting with
ones(1,10)/10
like this:
convolved=conv(y,ones(1,10)/10);
and then normalizing it to the same maximal value,
convolved2=convolved*max(y)/max(convolved);
this is the result:
And when convoluting with
convolved=conv(y,ones(1,50)/50);
This is the result (again multiplied by the max of y and divided by the max of convolved):
You just need to adjust the filter the way you need. Hopes that this helps.
Related
I am trying to find optimal parameters of my neural network model implemented on octave, this model is used for binary classification and 122 features (inputs) and 25 hidden units (1 hidden layer). For this I have 4 matrices/ Vectors:
size(X_Train): 125973 x 122
size(Y_Train): 125973 x 1
size(X_Test): 22543 x 122
size(Y_test): 22543 x 1
I have used 20% of the training set to generate a validation set (XVal and YVal)
size(X): 100778 x 122
size(Y): 100778 x 1
size(XVal): 25195 x 122
size(YVal): 25195 x 1
size(X_Test): 22543 x 122
size(Y_test): 22543 x 1
The goal is to generate the Learning curves of the NN. I have learned (the hard way xD) that this is very time consuming because I used the full size of Xval and X for this.
I don't know if there is an alternative solution for this. I am thinking to reduce the size of the training vector X (like 5000 samples for example), but I don't know if I can do that, or if the results will be biased since I'll only use a portion of the training set?
Bests,
The total number of parameters above is around 3k (122*25 + 25*1), which is not huge for one example. Since the number of examples is large, you might want to use stochastic gradient descent or mini-batches instead of gradient descent.
Note that Matlab and Octave are slow in general, specially with loops.
You need to write the code which uses matrix operations rather than loops for the speed to be manageable in Matlab/Octave.
I have a question that I don't know if there is a solution off the bat.
Here it goes,
I have two data sets, plotted on the same figure. I need to find their difference, simple so far...
the problem arises in the fact that say matrix A has 1000 data points while the second (matrix B) has 580 data points. How will I be able to find the difference between the two graphs since there is a dimensional miss match between the two figures.
One way that I thought of is artificially inflating matrix B to 1000 data points, but the trend of the plot will remain the same. Would this be possible? and if yes how?
for example:
A=[1 45 33 4 1009 ];
B=[1 22 33 44 55 66 77 88 99 1010];
Ya=A.*20+4;
Yb=B./10+3;
C=abs(B - A)
plot(A,Ya,'r',B,Yb)
xlim([-100 1000])
grid on
hold on
plot(length(B),C)
One way to do it is to resample the 580 element vector to 1000 samples. Use matlab resample (requires the Signal Processing Toolbox, I believe) for this:
x = randn(580,1);
y = randn(1000,1);
xr = resample(x, 50,29); # 50/29 = 1000/580 is the resampling ratio
You should then be able to compare the two data vectors.
There are two ways that I can think of:
1- Matching the size:
Generating more data for the matrix with lower number of elements (using interpolation, etc.)
Removing some data from the matrix with higher number of elements (i.e. outlier removal)
2- Comparing the matrices with their properties.
For instance, you can calculate the mean and the covariance of a matrix and compare it to the other matrix. The other options include, cov , mean , median , std, var , xcorr , xcov.
Сan anyone shine a light to my matlab program?
I have data from two sensors and i'm doing a kNN classification for each of them separately.
In both cases training set looks like a set of vectors of 42 rows total, like this:
[44 12 53 29 35 30 49;
54 36 58 30 38 24 37;..]
Then I get a sample, e.g. [40 30 50 25 40 25 30] and I want to classify the sample to its closest neighbor.
As a criteria of proximity I use Euclidean metrics, sqrt(sum(Y2)), where Y is a difference between each element and it gives me an array of distances between Sample and each Class of Training Set.
So, two questions:
Is it possible to convert distance into distribution of probabilities, something like: Class1: 60%, Class 2: 30%, Class 3: 5%, Class 5: 1%, etc.
added: Up to this moment I'm using formula: probability = distance/sum of distances, but I cannot plot a correct cdf or histogram.
This gives me a distribution in some way, but I see a problem there, because if distance is large, for example 700, then the closest class will get a biggest probability, but it'd be wrong because the distance is too big to be compared with any of classes.
If I would be able to get two probability density functions, I guess then I would do some product of them. Is it possible?
Any help or remark is highly appreciated.
I think there are multiple way of doing this:
as Adam suggested using 1/d / sum(1/d)
use the square, or even higher ordered of inverse of distance, e.g 1/d^2 / sum(1/d^2), This will make the class probability distribution more skewed. For example if 1/d generated 40%/60% prediction, the 1/d^2 may gave a 10%/90%.
use softmax (https://en.wikipedia.org/wiki/Softmax_function), the exponential of negative distance.
use exp(-d^2)/sigma^2 / sum[exp(-d^2)/sigma^2], this will imitate the Gaussian Distribution likelihoods. Sigma could be the average within-cluster distance, or simply set to 1 for all clusters.
You could try to inverse your distances to get a likelihood measure. I.e. the bigger the distance x, the smaller the inverse of it. Then, you can normalize as in probability = (1/distance) / (sum (1/distance) )
Hi: Have you ever tried with the formula probability = 1-distance assuming that you are using a standardized distance between 0 and 1?
`2.4 2.34 2.21 1.90 1.4 0.83 0
2.34 2.42 2.16 1.79 1.3 0.7 0
2.21 2.16 1.99 1.64 1.16 0.51 0
1.90 1.79 1.64 1.30 0.75 0 0
1.4 1.34 1.16 0.75 0 0 0
0.83 0.75 0.51 0 0 0 0
0 0 0 0 0 0 0
`
This is what my matrix looks like at one point in time and Id like to calculate the average slope or gradient of the surface( An indicator of steepness).
It seems like MatLab should have a built in function for this sort of thing but I can find one. I have also tried a code that I wrote but it is not accurate.
Best,
Abid
the gradient is vectorial, so you can treat the two dimensions separately.
Calculating the gradient consists of calculating the first derivative to every parameter of the function. In the discrete domain, this can be done by taking the finite differences: http://en.wikipedia.org/wiki/Finite_difference
If you want it exact and calculate the gradient at each point, you can't just take the forward or backward differences, because at the edge, you don't always have a forward (or backward) point.
But I think if you just want the mean of the gradient, you don't have to bother with the edges and you can just compute the finite differences for the central points between your original points by using diff:
dx = diff(data,1,1);
dy = diff(data,1,2);
mean_gradient = [mean(dx(:)) mean(dy(:))]
assuming equal spacing of the parameters of cours, else you'll have to divide dx and dy by the step sizes.
For the data you provided, this results in:
mean_gradient =
-0.26381 -0.26381
which matches the plot you've made that shows that the function is monotonically decreasing and symmetric in the parameters.
One way to do this is to perform multiple linear regression, for example by using regress.
A two-dimensional regression is a best-fit plane, from which you can calculate a slope value. The link has a nice example with code and a plot of the results.
For this type of data I want to do an interpolation like fit from one maxima to a minima and so on, so I can look for superimposed high frequencies:
I have a matrix of values such as:
a=[ 3 7 10 3 1 5 10 5 3 2 4 8 10 7 4 3 4 2 1 4 5 7 10 8 7 6 6 4 3 2];
Now I want to choose the relative and maximum and minimum values such that
a=[ 3 0 10 0 1 0 10 0 0 2 0 0 10 0 0 0 0 0 1 0 0 0 10 0 0 0 0 0 0 2];
I essentially want to fit a straight line from a(1) to a(3) and then from a(3) to a(5) and so on, and then subtract the fit from the data.
I know there is a function "detrend" that uses a breakpoint method it denotes as "bp", and that is the closest thing I found resembling my goal.
If you know of a way MATLAB can do this I will greatly appreciate it, otherwise it seems like I have to write an m-file to do it.
I think what your question is asking for is interpolation between the local minima and local maxima of a time series (what you call the "relative minimum and maximum values".)
See this similar question and add some code that does a linear interpolation between the local minima and maxima.
interp1() will do this handily. There is no need for the input points or the output points to be evenly spaced.
>> x = sort(rand(1,10));
>> y = rand(1,10);
>> plot (x,y,'r.');
>> xx = 0:0.01:1;
>> yy = interp1(x,y,xx);
>> hold on;
>> plot (xx,yy,'b-')
What I think you really want to do is decompose the signal into components based on local time scale. (that is, where the frequencies change with time). Use the empirical mode decomposition. Wavelet methods might be an alternative, but the output of the EMD is very easy to interpret visually.
Incidentally, the plain FFT won't work if applied to the entire length of a signal that is time varying - the FFT assumes a stationary (non-varying) signal.
You need to apply the Short-Time Fourier Transform, which is the FFT applied over sliding windows of the data to get a picture of frequency over time. See the spectrogram() function.
>> plot (x,y)
>> x=0:0.001:1;
>> y = chirp(x);
>> plot (x,y);
>> figure;
>> spectrogram(y);
There are issues with time vs. frequency resolution of the short-time Fourier transform that limit its application when your sampling rate is low compared to the frequency of the data. It would would be unlikely to work well for the example data you posted a picture of.