In a multiple linear regression model with two regressors, how would a test for significance change with a known variance?
The null hypothesis would be H0 = Beta_1 = Beta_2 = 0
With the variance known, we wouldn't have to estimate sigma squared hat, but can we still use the F test? F0 = MSr/MSres
I know that the estimate sigma squared hat = MSres but is that the same when sigma squared is known?
Related
Data x is input to an autoregreesive model (AR) model. The output of the AR model is corrupted with Additive White Gaussian Noise at SNR = 30 dB. The observations are denoted by noisy_y.
Let there be close estimates h_hat of the AR model (these are obtained from Least Squares estimation). I want to see how close the input obtained from deconvolution with h_hat and the measurements is to the known x.
My confusion is which variable to use for deconvolution -- clean y or noisy y?
Upon deconvolution, I should get x_hat. I am not sure if the correct way to perform deconvolution is using the noisy_y or using the y before adding noise. I have used the following code.
Can somebody please help in what is the correct method to plot x and x_hat.
Below is the plot of x vs x_hat. As can be seen, that these do not match. Where is my understand wrong? Please help.
The code is:
clear all
N = 200; %number of data points
a1=0.1650;
b1=-0.850;
h = [1 a1 b1]; %true coefficients
x = rand(1,N);
%%AR model
y = filter(1,h,x); %transmitted signal through AR channel
noisy_y = awgn(y,30,'measured');
hat_h= [1 0.133 0.653];
x_hat = filter(hat_h,1,noisy_y); %deconvolution
plot(1:50,x(1:50),'b');
hold on;
plot(1:50,x_hat(1:50),'-.rd');
A first issue is that the coefficients h of your AR model correspond to an unstable system since one of its poles is located outside the unit circle:
>> abs(roots(h))
ans =
1.00814
0.84314
Parameter estimation techniques are then quite likely to fail to converge given a diverging input sequence. Indeed, looking at the stated hat_h = [1 0.133 0.653] it is pretty clear that the parameter estimation did not converge anywhere near the actual coefficients. In your specific case you did not provide the code illustrating how you obtained hat_h (other than specifying that it was "obtained from Least Squares estimation"), so it isn't possible to further comment on what went wrong with your estimation.
That said, the standard formulation of Least Mean Squares (LMS) filters is given for an MA model. A common method for AR parameter estimation is to solve the Yule-Walker equations:
hat_h = aryule(noisy_y - mean(noisy_y), length(h)-1);
If we were to use this estimation method with the stable system defined by:
h = [1 -a1 -b1];
x = rand(1,N);
%%AR model
y = filter(1,h,x); %transmitted signal through AR channel
noisy_y = awgn(y,30,'measured');
hat_h = aryule(noisy_y - mean(noisy_y), length(h)-1);
x_hat = filter(hat_h,1,noisy_y); %deconvolution
The plot of x and x_hat would look like:
I can't get my mind around the concept of how to calculate bias and variance from a random set.
I have created the code to generate a random normal set of numbers.
% Generate random w, x, and noise from standard Gaussian
w = randn(10,1);
x = randn(600,10);
noise = randn(600,1);
and then extract the y values
y = x*w + noise;
After that I split my data into a training (100) and test (500) set
% Split data set into a training (100) and a test set (500)
x_train = x([ 1:100],:);
x_test = x([101:600],:);
y_train = y([ 1:100],:);
y_test = y([101:600],:);
train_l = length(y_train);
test_l = length(y_test);
Then I calculated the w for a specific value of lambda (1.2)
lambda = 1.2;
% Calculate the optimal w
A = x_train'*x_train+lambda*train_l*eye(10,10);
B = x_train'*y_train;
w_train = A\B;
Finally, I am computing the square error:
% Compute the mean squared error on both the training and the
% test set
sum_train = sum((x_train*w_train - y_train).^2);
MSE_train = sum_train/train_l;
sum_test = sum((x_test*w_train - y_test).^2);
MSE_test = sum_test/test_l;
I know that if I create a vector of lambda (I have already done that) over some iterations I can plot the average MSE_train and MSE_test as a function of lambda, where then I will be able to verify that large differences between MSE_test and MSE_train indicate high variance, thus overfit.
But, what I want to do extra, is to calculate the variance and the bias^2.
Taken from Ridge Regression Notes at page 7, it guides us how to calculate the bias and the variance.
My questions is, should I follow its steps on the whole random dataset (600) or on the training set? I think the bias^2 and the variance should be calculated on the training set. Also, in Theorem 2 (page 7 again) the bias is calculated by the negative product of lambda, W, and beta, the beta is my original w (w = randn(10,1)) am I right?
Sorry for the long post, but I really want to understand how the concept works in practice.
UPDATE 1:
Ok, so following the previous paper didn't generate any good results. So, I took the standard form of Ridge Regression Bias-Variance which is:
Based on that, I created (I used the test set):
% Bias and Variance
sum_bias=sum((y_test - mean(x_test*w_train)).^2);
Bias = sum_bias/test_l;
sum_var=sum((mean(x_test*w_train)- x_test*w_train).^2);
Variance = sum_var/test_l;
But, after 200 iterations and for 10 different lambdas this is what I get, which is not what I expected.
Where in fact, I was hoping for something like this:
sum_bias=sum((y_test - mean(x_test*w_train)).^2); Bias = sum_bias/test_l
Why have you squared the difference between y_test and y_predicted = x_test*w_train?
I don't believe your formula for bias is correct. In your question, the 'bias term' above in blue is the bias^2 however surely your formula is neither the bias nor the bias^2 since you have only squared the residuals, not the entire bias?
I am trying to design a lowpass filter in Matlab:
fc = 100; % Cutoff frequency
fs = 4020; % Sampling frequency
w_norm = 2*fc/fs;
filter_order = 1;
[num,denom] = butter(filter_order,w_norm)
sys = tf(num, denom)
[z,p,k] = zpkdata(sys)
Matlab gives me a pole at:
s = 0.8541
My questions are:
Where does this number come from? Shouldn't the pole be at w = 2*pi*fc = 628 rad/s (normalised to 1 if divided by wc)?
Shouldn't it be negative since butterworth LP filter poles are in the left half plane?
Why does Matlab also give me a zero at -1?
Many thanks.
By default, butter produces a discrete-time filter design. Therefore the transfer function is defined in terms of z (z-transform), not s (Laplace transform).
A discrete-time Butterworth filter of order n has an n-order zero at z=-1 and n poles within the unit circle. This is in accordance with your results. (In contrast, a continuous-time Butterworth filter would have an n-order zero at infinity and n poles in the left-hand unit semicircle).
I'm trying to get wavelet decomposition of arcsin(x) using, say, Haar wavelets
When using both Matlab's dwt or wavedec functions, I get strange values for approximating coefficients. Since applying low-pass Haar wavelets's filter equals to performing half-sum and the maximum of arcsin is pi/2, I assume that approximating coefficients can't surpass pi/2, yet this code:
x = linspace(0,1,128);
y = asin(x);
[cA, cD] = dwt(y, 'haar'); %//cA for approximating coefficients
returns values more than pi/2 in cA. Why is that?
I believe what makes you confused here is thinking that Haar's filter just averages two adjacent numbers when computing 1-level approximation coefficients. Due to the energy preservation feature of the scaling function, each pair of numbers gets divided by sqrt(2) instead of 2. In fact, you could see what a particular wavelet filter does by typing in the following command (for the Haar filter in this case):
[F1,F2] = wfilters('haar','d')
F1 =
0.7071 0.7071
F2 =
-0.7071 0.7071
You can then check the validity of what you have gotten above by constructing a simple loop:
CA_compare = zeros(1,64);
for k = 1 : 64
CA_compare(k) = dot( y(2*k-1 : 2*k), F1 );
end
You will then see that "CA_compare" contains exactly the same values as your "cA" does.
Hope this helps.
I have data and I need to do a linear regression on the data to obtain
y=Alpha*x+Beta
Alpha and Beta are estimators given by the regression, polyfit can give me those with no problem but this is a physical science report and I need to give error estimators on those values
I know from statistics that standard deviation exists for simple linear regression coefficients.
How can I calculate then in Matlab
Thank you
The second output of the regress command will give you 95% confidence intervals for the regression coefficients. Here's an example:
>> x = [ones(100,1), (1:100)'];
>> alpha = 3; beta = 2;
>> y = x*[alpha; beta]+randn(100,1);
>> [coeffs, coeffints] = regress(y,x);
>> coeffs
coeffs =
2.9712
1.9998
>> coeffints
coeffints =
2.5851 3.3573
1.9932 2.0064
Here alpha has been estimated 2.9712, with 95% confidence interval of between 2.5851 and 3.3573, and beta has been estimated as 1.9998, with 95% confidence interval between 1.9932 and 2.0064.
The easiest solution is to use LSCOV:
%# create some data
x = 1:10;
y = 3*x+randn(size(x))*0.1;
%# create the design matrix
A = [x(:),ones(length(x),1)];
[u,std_u] = lscov(A,y(:));
u =
3.0241
-0.070209
std_u =
0.018827
0.11682
Or just use the relationship that the length of the 95% Confidence Interval is 2*1.9654 standard errors so the standard errors in the example with regress above are given by:
st errors = (coeffints(:,2)-coeffints(:,1))/(2*1.9654).
The number 1.9654 appears because of the normality assumption in the regress function