Related
I am trying to use the best practice techniques in the Computational fluid dynamics area to analyze and visualize a velocity field.
Given 6 arrays of moving particles' positions and velocities: x,y,z and vx,vy,vz respectively.
I want to visualize and calculate the induced velocity field and its properties such as: curl, divergence, isosurfaces etc.
Here is a modest script of the volume visualization functions I was able to use without calling meshgrid (to avoid interpolation and more noise).
Ultimately, one of the things that I am not sure about is how to wisely create a mesh grid from my 50 points in space, the second is how to use CFD approaches to visualize the velocity field regardless the small amount of data points.
close all
rng default
t=0.1:0.1:10;
x = sin(t)';
y = cos(t)';
z = t.^0.2';
vx=y;vy=x;vz=z;
figure
subplot(2,3,1);
quiver3(x,y,z,vx,vy,vz);
hold on
streamribbon({ [x y z] }, {vx},{vy},{vz});
subplot(2,3,2);
[curl_val, cav] = curl([x,y,z],[vx,vy,vz]);
surfc([x,y,z],cav);
subplot(2,3,3);
surfc([x,y,z],curl_val);
w = sqrt( vx.^2 + vy.^2 + vz.^2 );
subplot(2,3,4);
quiver3(x,y,z,vx,vy,vz);
streamtube({ [x y z] }, {w});
subplot(2,3,5);
quiver3(x,y,z,vx,vy,vz);
subplot(2,3,6);
surfc([x,y,z],[vx,vy,vz]);
When I run the above script (excluding the data generation) on a real data, I get the following plots which aren't very informative:
I strongly suspect that the problem here is with the data, not the visualization technique. But in general, the problem is one or more of the following:
1) You do not have enough data to capture the underlying dynamics (the dynamics in space operate at a higher spatial frequency than you sampled)
2) The data is too noisy for the number of datapoints your collected.
3) The flow is fundamentally turbulent, and hence hoping for a nice laminar-like plot is not going to happen.
When you have problems visualizing data, the first rule of thumb is always to throw away any visualization that attempts to approximate a derivative (or gradient) in any way. The reason is that when you try to approximate a derivative with real data, noise almost always makes that estimate nonsense. For example, let's suppose we have a cosine that gets corrupted by some noise, and we try to numerically estimate the derivative from the data
figure
% Create a signal
dt = .1;
t = 0:.1:10;
x = cos(t);
% Add some noise
y = x + .5 * randn(size(x));
% Compute the first order approximation of the derivatives of the signals
dx = diff(x)/dt;
dy = diff(y)/dt;
% Plot everything
subplot(2,1,1)
plot(t,x,t,y)
axis tight
subplot(2,1,2)
plot(t(2:end),dx,t(2:end),dy)
axis tight
In the first plot, which shows the raw data, the noise doesn't look to bad, but when we look at the derivative estimate! Ouch... The noise is really amplified. So forget about higher order properties of a flow, such as the curl and vorticity, which require gradients of the data.
So what can we do in cases like these? Well essentially, just look at the raw data. If there is a pattern, it will reveal itself. For instance, let's look at your raw velocity vectors from 3 different perspectives:
data = dlmread('data.csv','\s')
x = data(:,1);
y = data(:,2);
z = data(:,3);
vx = data(:,4);
vy = data(:,5);
vz = data(:,6);
close all
figure
subplot(1,3,1);
quiver3(x,y,z,vx,vy,vz);
view([1,0,0])
subplot(1,3,2);
quiver3(x,y,z,vx,vy,vz);
view([0,1,0])
subplot(1,3,3);
quiver3(x,y,z,vx,vy,vz);
view([0,0,1])
The only thing that looks even slightly structured is that last plot. However, that plot tells us that we probably also have turbulence (in addition to noise) to contend with.
Specifically, from view 3, it definitely seems like you are taking velocity measurements in a flow that is tightly hugging an object. In this case, your measurements are probably too tight though... and probably in the boundary layer. If that is the case (that the measurements are in the boundary layer), then you can get time-varying effects in the flow, meaning that it doesn't make sense to look at anything without also having a time component. The "nice" plots that you have in your answer are only really helpful when the flow is laminar, where we get to see these nice, consistent stream lines. If it is turbulent, then there is no discernable pattern in the flow, no matter how hard you look.
So in conclusion, I don't think you will be able to find a nice visualization for your data because either the sensors you used were too noisy, or the flow was too turbulent.
As an aside... consider what happens when we look at the raw velocity vectors from your "nice" dataset:
That, my friend, is a well-trained house pet. You have a wild mountain lion on your hands.
I have a set of curves varying over time, that are stored in a MATLAB matrix. Each row of the matrix is one of those curves, unfolding over time. Those are repetitions of a random experiment.
I need to plot the mean of these curves over time, along with the 95% confidence intervals.
My understanding of statistics this is rather poor, but I was suggested to use bootstrap confidence intervals using MATLAB's bootci function.
I implemented a minimal example in MATLAB, but I have some doubts. I hope you can help me gaining a better grasp on this and avoiding dumb mistakes.
Here's the example:
NVARIABLES = 200;
NOBSERVATIONS = 1000;
RESAMPLING = 10000;
DATA = rand(NOBSERVATIONS, NVARIABLES);
[CI, STAT] = bootci(RESAMPLING, #mean, DATA);
MEAN = mean(DATA); % <------- [1]
x = 1:NVARIABLES;
figure;
hold on;
plot(x, MEAN, 'LineWidth', 2);
plot(x, CI(1,:), '--', 'LineWidth', 2); % [2]
plot(x, CI(2,:), '--', 'LineWidth', 2);
% plot(x, MEAN-CI(1,:)); % ?
% plot(x, MEAN+CI(2,:)); % ?
hold off;
Here are my questions:
Am I using the function properly?
When reporting/plotting the mean, is it correct to plot mean(DATA) (see line 1) or I should plot a mean derived by the bootstrapping procedure? I saw that STAT contains the mean for each bootstrap example, but I don't know whether I should use this information, and how
Is it correct to plot the confidence intervals the way I am doing (see line [2]), or I should plot MEAN-CI(1,:) and MEAN+CI(2,:)?
Please find attached the plot generated by the code.
I can answer Q1 and Q3.
Q1. The first argument needs to be the number of bootstrap samples used in the computation, the second, a function that returns the statistic for which you wish to find the confidence intervals, and the third is the dataset itself that you want to give as input to the function. You will have to ensure if the first argument is correct for you, the rest seems correct.
Q3. What you've done is right - CI gives the range, and not the variation from the mean. There's also another way to plot it, which might be better in certain scenarios, or just based on personal preferences. plot_ci is a function that lets you plot confidence intervals and shows clean patches for these intervals on the same plot. The plots look like this (this is a sample plot, not based on the dataset in the question):
Here's the command:
plot_ci(x,[MEAN,CI(1,:),CI(2,:)],'PatchColor', 'k', 'PatchAlpha', 0.1, 'MainLineWidth', 2, 'MainLineStyle', '-', 'MainLineColor', 'b','LineWidth', 1.5, 'LineStyle','--', 'LineColor', 'k');
I have a discrete function y(n), n=1..8000 with two areas, which can be approximated by almost horizontal straight lines as shown in image.
I'd like to find coordinates x1, x2 of points where these areas meet the rapidly growing part of function. In Matlab, y(n) is a one-dimensional vector.
If data set is very smooth, you can use simple derivative magnitude detector.
If some noise peaks and small false fronts are possible, you'd better to use more robust method, for example
Choose some initial value for x1 (for example, 600)
Calculate linear regression parameters y = a*x + b for range 1..x1
If slope parameter a is small enough, increase x1 and repeat calculation
if slope is larger than some reasonable threshold, decrease x1 and repeat.
Binary search algorithm is rather quick way to reach needed x1 value.
Do the same for x2..N range.
As mentioned by #PatronBernard, you could look at the derivative of your trace, knowing that the derivative deviates from zero once you are no longer in the flat bit. This would involve manually picking a threshold which might have to be done for each new trace, and would be sensitive to noise in your trace. Here is an implementation, also showing the issues you might get with a noisy trace:
threshold = 0.2;
y1 = diff(y)/(x(2) - x(1));
y2 = diff(y1)/(x(2) - x(1));
figure;
hold on
plot(x,y, 'b')
xi = find(y1>threshold,1);
line([x(xi) x(xi)], ylim, 'Color', 'r')
xi = find(flipud(y1)>threshold,1);
line([x(length(x) - xi + 1) x(length(x) - xi + 1)], ylim, 'Color', 'r')
Alternatively, if you have genuinely noisy data I would recommend looking at IEC standard 60469:2013 "Transitions, Pulses and Related Waveforms - Terms, Definitions and Algorithms".
I have a list of data that I am trying to fit to a polynomial and I am trying to plot the 95% confidence bands for the parameters as well (in Matlab).
If my data are x and y
f=fit(x,y,'poly2')
plot(f,x,y)
ci=confint(f,0.95);
a_ci=ci(1,:);
b_ci=ci(2,:);
I do not know how to proceed after that to get the minimum and maximum band around my data. Does anyone know how to do that?
I can see that you have the curve fitting toolbox installed, which is good, because you need it for the following code to work.
Basic fit of example data
Let's define some example data and a possible fit function. (I could also have used poly2 here, but I wanted to keep it a bit more general.)
xdata = (0:0.1:1)'; % column vector!
noise = 0.1*randn(size(xdata));
ydata = xdata.^2 + noise;
f = fittype('a*x.^2 + b');
fit1 = fit(xdata, ydata, f, 'StartPoint', [1,1])
plot(fit1, xdata, ydata)
Side note: plot() is not our usual plot function, but a method of the cfit-object fit1.
Confidence intervals of the fitted parameters
Our fit uses the data to determine the coefficients a,b of the underlying model f(x)=ax2+b. You already did this, but for completeness here is how you can read out the uncertainty of the coefficients for any confidence interval. The coefficients are alphabetically ordered, which is why I can use ci(1,:) for a, and so on.
names = coeffnames(fit1) % check the coefficient order!
ci = confint(fit1, 0.95); % 2 sigma interval
a_ci = ci(1,:)
b_ci = ci(2,:)
By default, Matlab uses 2σ (0.95) confidence intervals. Some people (physicists) prefer to quote the 1σ (0.68) intervals.
Confidence and Prediction Bands
It's a good habit to plot confidence bands or prediction bands around the data – especially when the coefficients are correlated! But you should take a moment to think about which one of the two you want to plot:
Prediction band: If I take a new measurement value, where would I expect it to lie? In Matlab terms, this is called the “observation band”.
Confidence band: Where do I expect the true value to lie? In Matlab terms, this is called the “functional band”.
As with the coefficient’s confidence intervals, Matlab uses 2σ bands by default, and the physicists among us switch this to 1σ intervals. By its nature, the prediction band is bigger, because it is the combination of the error of the model (the confidence band!) and the error of the measurement.
There is a another destinction to make, one that I don’t fully understand. Both Matlab and Wikipedia make that distinction.
Pointwise: How big is the prediction/confidence band for a single measurement/true value? In virtually all cases I can think of, this is what you would want to ask as a physicist.
Simultaneous: How big do you have to make the prediction/confidence band if you want a set of all new measurements/all prediction points to lie within the band with a given confidence?
In my personal opinion, the “simultaneous band” is not a band! For a measurement with n points, it should be n individual error bars!
The prediction/confidence distinction and the pointwise/simultaneous distinction give you a total of four options for “the” band around the plot. Matlab makes the 2σ pointwise prediction band easily accessible, but what you seem to be interested in is the 2σ pointwise confidence band. It is a bit more cumbersome to plot, because you have to specify dummy data over which to evaluate the prediction band:
x_dummy = linspace(min(xdata), max(xdata), 100);
figure(1); clf(1);
hold all
plot(xdata,ydata,'.')
plot(fit1) % by default, evaluates the fit over the currnet XLim
% use "functional" (confidence!) band; use "simultaneous"=off
conf1 = predint(fit1,x_dummy,0.95,'functional','off');
plot(x_dummy, conf1, 'r--')
hold off
Note that the confidence band at x=0 equals the confidence interval of the fit-coefficient b!
Extrapolation
If you want to extrapolate to x-values that are not covered by the range of your data, you can evaluate the fit and the prediction/confidence band for a bigger range:
x_range = [0, 2];
x_dummy = linspace(x_range(1), x_range(2), 100);
figure(1); clf(1);
hold all
plot(xdata,ydata,'.')
xlim(x_range)
plot(fit1)
conf1 = predint(fit1,x_dummy,0.68,'functional','off');
plot(x_dummy, conf1, 'r--')
hold off
I have an image and my aim is to binarize the image. I have filtered the image with a low pass Gaussian filter and have computed the intensity histogram of the image.
I now want to perform smoothing of the histogram so that I can obtain the threshold for binarization. I used a low pass filter but it did not work. This is the filter I used.
h = fspecial('gaussian', [8 8],2);
Can anyone help me with this? What is the process with respect to smoothing of a histogram?
imhist(Ig);
Thanks a lot for all your help.
I've been working on a very similar problem recently, trying to compute a threshold in order to exclude noisy background pixels from MRI data prior to performing other computations on the images. What I did was fit a spline to the histogram to smooth it while maintaining an accurate fit of the shape. I used the splinefit package from the file exchange to perform the fitting. I computed a histogram for a stack of images treated together, but it should work similarly for an individual image. I also happened to use a logarithmic transformation of my histogram data, but that may or may not be a useful step for your application.
[my_histogram, xvals] = hist(reshape(image_volume), 1, []), number_of_bins);
my_log_hist = log(my_histogram);
my_log_hist(~isfinite(my_log_hist)) = 0; % Get rid of NaN values that arise from empty bins (log of zero = NaN)
figure(1), plot(xvals, my_log_hist, 'b');
hold on
breaks = linspace(0, max_pixel_intensity, numberofbreaks);
xx = linspace(0, max_pixel_intensity, max_pixel_intensity+1);
pp = splinefit(xvals, my_log_hist, breaks, 'r');
plot(xx, ppval(pp, xx), 'r');
Note that the spline is differentiable and you can use ppdiff to get the derivative, which is useful for finding maxima and minima to help pick an appropriate threshold. The numberofbreaks is set to a relatively low number so that the spline will smooth the histogram. I used linspace in the example to pick the breaks, but if you know that some portion of the histogram exhibits much greater curvature than elsewhere, you'd want to have more breaks in that region and less elsewhere in order to accurately capture the shape of the histogram.
To smooth the histogram you need to use a 1-D filter. This is easily done using the filter function. Here is an example:
I = imread('pout.tif');
h = imhist(I);
smooth_h = filter(normpdf(-4:4, 0,1),1,h);
Of course you can use any smoothing function you choose. The mean would simply be ones(1,8).
Since your goal here is just to find the threshold to binarize an image you could just use the graythresh function which uses Otsu's method.