When parsing an S/MIME certificate, there are a lot of variations and possible combinations, especially of the Key Usage and Extended Key Usage fields. It's not easy to understand what exactly they mean, how they mutually influence their meanings, and in what order to interpret them best.
How to interpret these properly in order to determine if the certificate can be used for encrypting or signing emails or both?
After much research, I've finally determined the following decision matrix of how to properly interpret an S/MIME certificate's properties:
Case
a
b
c
d
e
f
g
Extension Extended Key Usage exists?
Y
-
-
-
-
-
-
Extension Extended Key Usage marked critical?
Y
-
-
-
-
-
-
Extended Key Usage has value Email (1.3.6.1.5.5.7.3.4)?
N
-
-
-
-
-
-
Extension Key Usage exists?
-
Y
Y
Y
Y
Y
N
Extension Key Usage marked critical?
-
Y
Y
Y
Y
N
-
Key Usage has value keyEncipherment?
-
Y
Y
N
N
-
-
Key Usage has value digitalSignature?
-
Y
N
Y
N
-
-
→ Certificate can be used to encrypt
x
x
x
x
→ Certificate can be used to sign
x
x
x
x
Extension Extended Key Usage is OID 2.5.29.37
Extension Key Usage is OID 2.5.29.15
How to read the decision table
Just in case your are not familiar with decision tables, this is how to read it:
start with case a
go down to the first Y or N value (dashes mean not relevant)
check the condition (question and answer)
if it is true, continue down vertically
if it is false, continue to the right horizontally
repeat until all conditions are true
do down in the same column to see the results
Example
Case a in pseudo code
if exists(ExtensionExtendedKeyUsage) and isExtensionExtendedKeyUsageCritical == true and ExtendedKeyUsageValue == "1.3.6.1.5.5.7.3.4"
useForEncryption = false
useForSigning = false
end-if
Related
I have a 3x3 matrix in which some elements depend on a parameter called x which can attain values between 0 and 1.
I multiply the inverse of this matrix by a 3x1 vector, and I therefore obtain a 3x1 vector called optimal taxesin which each element depends on x.
I then create a variable called reinf which is equal to the first element of optimal taxes minus the other two elements.
I would like to plot the values of reinf depending on the parameter x in a range between 0 and 1.
I have tried used the following code, where H_V1x is the 3x3 matrix.
syms x positive real
optimal_taxes1x = inv(H_V1x)*inc_cons
reinf1x = optimal_taxes1x(1,1) - optimal_taxes1x(2,1) - optimal_taxes1x(3,1)
However when I run the last line it says I get the error "Index in position 1 exceeds array bounds (must not exceed 1)."
which is surprising since optimal_taxes1x is a 3x1 syms vector.
How could I solve this problem and create the plot?
Thank you
As requested, here are the matrix and the vector:
H_V1x = [ -5.9280e-07, 3.7066e-07,2.2215e-07; 3.7066e-07,(1537*x)/258614388 - 6736739427183417/2361183241434822606848,-(1537*x)/258614388;2.2215e-07,-(1537*x)/258614388,(1537*x)/258614388 - 6645950193052613/4722366482869645213696];
inc_cons = [0.823603356515493,
-0.219495869657492,
-0.0752767166225879]
I can run the code with no problem in 2016b. Here is my code:
clear variables; clc; close all
syms x real
assume(x > 0)
H_V1x = [ -5.9280e-07, 3.7066e-07,2.2215e-07; 3.7066e-07,(1537*x)/258614388 - 6736739427183417/2361183241434822606848,-(1537*x)/258614388;2.2215e-07,-(1537*x)/258614388,(1537*x)/258614388 - 6645950193052613/4722366482869645213696];
inc_cons = [0.823603356515493;
-0.219495869657492;
-0.0752767166225879]
optimal_taxes1x = inv(H_V1x)*inc_cons;
reinf1x = optimal_taxes1x(1,1) - optimal_taxes1x(2,1) - optimal_taxes1x(3,1)
x=0:0.02:1;
plot(x, subs(reinf1x))
And this is the plot result:
reinf1x =
(1040053994187210661864407040*(495592707753015189241062869767647395840x - 89156821447333517620228626174465662937))/(747(2815381160608336475815926275042762431495254208116923826176x - 445811259036595286456502337342890549874079624653932849201)) - (460318575875822254261534720(11210459727128231750210203484711869743104x - 2497571854516097684109652531721363759511))/(30627(2815381160608336475815926275042762431495254208116923826176x - 445811259036595286456502337342890549874079624653932849201)) - (42633524607347156989247488(162551666043359360378047950528322111275008x - 3799129388554029548248573922328032752851))/(30627(2815381160608336475815926275042762431495254208116923826176x - 445811259036595286456502337342890549874079624653932849201)) - (7258277284170644693450752(81275833021679680189023975264161055637504x - 35848130569758668835608493944994001436187))/(10209(2815381160608336475815926275042762431495254208116923826176x - 445811259036595286456502337342890549874079624653932849201)) + (402307845408630815045189632(495592707753015189241062869767647395840x - 73377230813429148255176338884456817971))/(249(2815381160608336475815926275042762431495254208116923826176x - 445811259036595286456502337342890549874079624653932849201)) - (7623637334217303404343984128(18254049353170950597767874357190897696768x - 2894673088146242304784596506648828329737))/(30627(2815381160608336475815926275042762431495254208116923826176*x - 445811259036595286456502337342890549874079624653932849201))
I'm in the process of implementing Three Fish block cipher using MATLAB. At first, I implemented the algorithm on uint8 numbers to validate my code. Every thing was OK and the decryption was successful. But when I replaced the numbers to uint64 the plain text did not retrieved correctly.
I traced the rounds results again and over again to find the reason, but I couldn't find it so far. There is difference in the first four digits between encryption and decryption, that is, along the rounds x encrypted as 9824265115183455531, but it decrypts as 9824265115183455488.
I think the reason behind this difference is in the functions AddMod64 and SubMod64 to find arithmetic modulo 2 to the power 64. but really I could not fix it so far.
I know that
double(2^64) = 18446744073709552000
and
uint64(2^64) = 18446744073709551615 % z = ( x + y ) % 2^64
function z = AddMod64(x , y)
m = uint64(2^64);
z = double(mod(mod(double(x),m)+mod(double(y),m),m));
end
% z = (x - y ) % 2^64
function z = SubMod64(x , y)
m = uint64(2^64);
z = double(mod(mod(double(x),m) - mod(double(y),m),m));
end
double(2^64) is already the wrong result, the double type can hold only up to 2^52-1 as an integer without rounding.
Also, when you do uint64(2^64), the power is computed using double, giving the wrong result, which you then cast to uint64. And because the maximum value that a uint64 van hold is 2^64-1, that whole operation is wrong.
Use maxint instead:
m = maxint('uint64');
To do modulo addition in MATLAB is rather tricky, because MATLAB does saturated arithmetic with integers. You need to test for overflow before doing the computation.
if x > m - y
x = y - (m - x + 1);
else
x = x + y
end
I have a 30x30 matrix as a base matrix (OD_b1), I also have two base vectors (bg and Ag). My aim is to optimize a matrix (X) who's dimensions are 30X30 such that:
1) the squared difference between vector (bg) and vector of sum of all the columns is minimized.
2)the squared difference between vector (Ag) and vector of sum of all rows is minimized.
3)the squared difference between the elements of matrix (X) and matrix (OD_b1) is minimized.
The mathematical form of the equation is as follows:
I have tried this:
fun=#(X)transpose(bg-sum(X,2))*(bg-sum(X,2))+ (Ag-sum(X,1))*transpose(Ag-sum(X,1))+sumsqr(X_b-X);
[val,X]=fmincon(fun,OD_b1,AA,BB,Aeq,beq,LB,UB)
I don't get errors but it seems like it's stuck.
Is it because I have too many variables or is there another reason?
Thanks in advance
This is a simple, unconstrained least squares problem and hence has a simple solution that can be expressed as the solution to a linear system.
I will show you (1) the precise and efficient way to solve this and (2) how to solve with fmincon.
The precise, efficient solution:
Problem setup
Just so we're on the same page, I initialize the variables as follows:
n = 30;
Ag = randn(n, 1); % observe the dimensions
X_b = randn(n, n);
bg = randn(n, 1);
The code:
A1 = kron(ones(1,n), eye(n));
A2 = kron(eye(n), ones(1,n));
A = (A1'*A1 + A2'*A2 + eye(n^2));
b = A1'*bg + A2'*Ag + X_b(:);
x = A \ b; % solves A*x = b
Xstar = reshape(x, n, n);
Why it works:
I first reformulated your problem so the objective is a vector x, not a matrix X. Observe that z = bg - sum(X,2) is equivalent to:
x = X(:) % vectorize X
A1 = kron(ones(1,n), eye(n)); % creates a special matrix that sums up
% stuff appropriately
z = A1*x;
Similarly, A2 is setup so that A2*x is equivalent to Ag'-sum(X,1). Your problem is then equivalent to:
minimize (over x) (bg - A1*x)'*(bg - A1*x) + (Ag - A2*x)'*(Ag - A2*x) + (y - x)'*(y-x) where y = Xb(:). That is, y is a vectorized version of Xb.
This problem is convex and the first order condition is a necessary and sufficient condition for the optimum. Take the derivative with respect to x and that equation will define your solution! Sample example math for almost equivalent (but slightly simpler problem is below):
minimize(over x) (b - A*x)'*(b - A*x) + (y - x)' * (y - x)
rewriting the objective:
b'b- b'Ax - x'A'b + x'A'Ax +y'y - 2y'x+x'x
Is equivalent to:
minimize(over x) (-2 b'A - 2y'*I) x + x' ( A'A + I) * x
the first order condition is:
(A'A+I+(A'A+I)')x -2A'b-2I'y = 0
(A'A+I) x = A'b+I'y
Your problem is essentially the same. It has the first order condition:
(A1'*A1 + A2'*A2 + I)*x = A1'*bg + A2'*Ag + y
How to solve with fmincon
You can do the following:
f = #(X) transpose(bg-sum(X,2))*(bg-sum(X,2)) + (Ag'-sum(X,1))*transpose(Ag'-sum(X,1))+sum(sum((X_b-X).^2));
o = optimoptions('fmincon');%MaxFunEvals',30000);
o.MaxFunEvals = 30000;
Xstar2 = fmincon(f,zeros(n,n),[],[],[],[],[],[],[],o);
You can then check the answers are about the same with:
normdif = norm(Xstar - Xstar2)
And you can see that gap is small, but that the linear algebra based solution is somewhat more precise:
gap = f(Xstar2) - f(Xstar)
If the fmincon approach hangs, try it with a smaller n just to gain confidence that my linear algebra based solution is more precise, way way faster etc... n = 30 is solving a 30^2 = 900 variable optimization problem: not easy. With the linear algebra approach, you can go up to n = 100 (i.e. 10000 variable problem) or even larger.
I would probably solve this as a QP using quadprog using the following reformulation (keeping the objective as simple as possible to make the problem "less nonlinear"):
min sum(i,v(i)^2)+sum(i,w(i)^2)+sum((i,j),z(i,j)^2)
v = bg - sum(c,x)
w = ag - sum(r,x)
Z = xbase-x
The QP solver is more precise (no gradients using finite differences). This approach also allows you to add additional bounds and linear equality and inequality constraints.
The other suggestion to form the first order conditions explicitly is also a good one: it also has no issue with imprecise gradients (the first order conditions are linear). I usually prefer a quadratic model because of its flexibility.
For a given set of q and r, I want to find the maximum of Tp=x * log(1 + (q* r (1 - 1/y)* (2/x - y))/(1 + r* (1 - 1/y) + q* (2/x - y))) for x in (0,1) and y in (1,2).
I can calculate them using two for loops, but when I use really small steps sizes for x and y, e.g., 0.00001, this takes a long time. But I know that if I get Tp as a matrix for all x and y, i.e., Tp is matrix of size length(x) x length(y), it may easier and faster. As I read, bsxfun(#times,..) may be helped, but I don't know how I can apply it in my problem.
Here is what I have tried, but it doesn't give correct output. Here I used larger step size for understanding. Can someone fix this issue in my code?
function maxTp
hvar=0.1:0.2:1;
hl=length(hvar);
q=hvar; r=hvar;
stepx=0.2;stepy=0.1;
y0=1.1; x0=0.1;
x=x0:stepx:1; y=y0:stepy:2;
ox = zeros(hl,1); oy = zeros(hl,1);
MaxTp = zeros(hl,1);
for k=1:hl
Tp = bsxfun(#times,log(1 + (q(k)*r(k)*(1 - 1./y).*(2./x - y))./(1 + r(k)*(1 - 1./y) + q(k)*(2./x - y))).',x);
MaxTp(k,1)=max(max(Tp));
[p, q] = ind2sub(size(Tp),find(Tp==MaxTp(k,1)));
ox(k,1)=x0+(p-1)*stepx;
oy(k,1)=y0+(q-1)*stepy;
end
Try this inside your for loop:
Tp = bsxfun(#(x,y) log(1+(q(k)*r(k)*(1 - 1./y).*(2./x - y))./(1 + r(k)*(1 - 1./y) + q(k)*(2./x - y)))*x,x,y.'); %\\'
MaxTp(k,1)=max(max(Tp));
[p2, q2] = ind2sub(size(Tp),find(Tp==MaxTp(k,1)));
ox(k,1)=x0+(p2-1)*stepx;
oy(k,1)=y0+(q2-1)*stepy;
I changed the bsxfun to do the calculation in the function part rather than the vector inputs, and you were also overwriting p and q as the results of ind2sub.
You can also use fmincon (be aware the maximisation means we need to minimise the negative of the function). The following code goes inside the for loop:
f=#(x,y) log(1+(q(k)*r(k)*(1 - 1./y).*(2./x - y))./...
(1 + r(k)*(1 - 1./y) + q(k)*(2./x - y)))*x;
o(:,k)=fmincon(#(x) -f(x(1),x(2)),[0.5;0.5],[],[],[],[],[0;1],[1;2]);
o(:,k) gives the x and y coordinates of the maximum, I think it's different to your ox and oy variables though.
I am fitting data with weights using scipy.odr but I don't know how to obtain a measure of goodness-of-fit or an R squared. Does anyone have suggestions for how to obtain this measure using the output stored by the function?
The res_var attribute of the Output is the so-called reduced Chi-square value for the fit, a popular choice of goodness-of-fit statistic. It is somewhat problematic for non-linear fitting, though. You can look at the residuals directly (out.delta for the X residuals and out.eps for the Y residuals). Implementing a cross-validation or bootstrap method for determining goodness-of-fit, as suggested in the linked paper, is left as an exercise for the reader.
The output of ODR gives both the estimated parameters beta as well as the standard deviation of those parameters sd_beta. Following p. 76 of the ODRPACK documentation, you can convert these values into a t-statistic with (beta - beta_0) / sd_beta, where beta_0 is the number that you're testing significance with respect to (often zero). From there, you can use the t-distribution to get the p-value.
Here's a working example:
import numpy as np
from scipy import stats, odr
def linear_func(B, x):
"""
From https://docs.scipy.org/doc/scipy/reference/odr.html
Linear function y = m*x + b
"""
# B is a vector of the parameters.
# x is an array of the current x values.
# x is in the same format as the x passed to Data or RealData.
#
# Return an array in the same format as y passed to Data or RealData.
return B[0] * x + B[1]
np.random.seed(0)
sigma_x = .1
sigma_y = .15
N = 100
x_star = np.linspace(0, 10, N)
x = np.random.normal(x_star, sigma_x, N)
# the true underlying function is y = 2*x_star + 1
y = np.random.normal(2*x_star + 1, sigma_y, N)
linear = odr.Model(linear_func)
dat = odr.Data(x, y, wd=1./sigma_x**2, we=1./sigma_y**2)
this_odr = odr.ODR(dat, linear, beta0=[1., 0.])
odr_out = this_odr.run()
# degrees of freedom are n_samples - n_parameters
df = N - 2 # equivalently, df = odr_out.iwork[10]
beta_0 = 0 # test if slope is significantly different from zero
t_stat = (odr_out.beta[0] - beta_0) / odr_out.sd_beta[0] # t statistic for the slope parameter
p_val = stats.t.sf(np.abs(t_stat), df) * 2
print('Recovered equation: y={:3.2f}x + {:3.2f}, t={:3.2f}, p={:.2e}'.format(odr_out.beta[0], odr_out.beta[1], t_stat, p_val))
Recovered equation: y=2.00x + 1.01, t=239.63, p=1.76e-137
One note of caution in using this approach on nonlinear problems, from the same ODRPACK docs:
"Note that for nonlinear ordinary least squares, the linearized confidence regions and intervals are asymptotically correct as n → ∞ [Jennrich, 1969]. For the orthogonal distance regression problem, they have been shown to be asymptotically correct as σ∗ → 0 [Fuller, 1987]. The difference between the conditions of asymptotic correctness can be explained by the fact that, as the number of observations increases in the orthogonal distance regression problem one does not obtain additional information for ∆. Note also that Vˆ is dependent upon the weight matrix Ω, which must be assumed to be correct, and cannot be confirmed from the orthogonal distance regression results. Errors in the values of wǫi and wδi that form Ω will have an adverse affect on the accuracy of Vˆ and its component parts. The results of a Monte Carlo experiment examining the accuracy
of the linearized confidence intervals for four different measurement error models is presented in [Boggs and Rogers, 1990b]. Those results indicate that the confidence regions and intervals for ∆ are not as accurate as those for β.
Despite its potential inaccuracy, the covariance matrix is frequently used to construct confidence regions and intervals for both nonlinear ordinary least squares and measurement error models because the resulting regions and intervals are inexpensive to compute, often adequate, and familiar to practitioners. Caution must be exercised when using such regions and intervals, however, since the validity of the approximation will depend on the nonlinearity of the model, the variance and distribution of the errors, and the data itself. When more reliable intervals and regions are required, other more accurate methods should be used. (See, e.g., [Bates and Watts, 1988], [Donaldson and Schnabel, 1987], and [Efron, 1985].)"
As mentioned by R. Ken, chi-square or variance of the residuals is one of the more
commonly used tests of goodness of fit. ODR stores the sum of squared
residuals in out.sum_square and you can verify yourself
that out.res_var = out.sum_square/degrees_freedom corresponds to what is commonly called reduced chi-square: i.e. the chi-square test result divided by its expected value.
As for the other very popular estimator of goodness of fit in linear regression, R squared and its adjusted version, we can define the functions
import numpy as np
def R_squared(observed, predicted, uncertainty=1):
""" Returns R square measure of goodness of fit for predicted model. """
weight = 1./uncertainty
return 1. - (np.var((observed - predicted)*weight) / np.var(observed*weight))
def adjusted_R(x, y, model, popt, unc=1):
"""
Returns adjusted R squared test for optimal parameters popt calculated
according to W-MN formula, other forms have different coefficients:
Wherry/McNemar : (n - 1)/(n - p - 1)
Wherry : (n - 1)/(n - p)
Lord : (n + p - 1)/(n - p - 1)
Stein : (n - 1)/(n - p - 1) * (n - 2)/(n - p - 2) * (n + 1)/n
"""
# Assuming you have a model with ODR argument order f(beta, x)
# otherwise if model is of the form f(x, a, b, c..) you could use
# R = R_squared(y, model(x, *popt), uncertainty=unc)
R = R_squared(y, model(popt, x), uncertainty=unc)
n, p = len(y), len(popt)
coefficient = (n - 1)/(n - p - 1)
adj = 1 - (1 - R) * coefficient
return adj, R
From the output of your ODR run you can find the optimal values for your model's parameters in out.beta and at this point we have everything we need for computing R squared.
from scipy import odr
def lin_model(beta, x):
"""
Linear function y = m*x + q
slope m, constant term/y-intercept q
"""
return beta[0] * x + beta[1]
linear = odr.Model(lin_model)
data = odr.RealData(x, y, sx=sigma_x, sy=sigma_y)
init = odr.ODR(data, linear, beta0=[1, 1])
out = init.run()
adjusted_Rsq, Rsq = adjusted_R(x, y, lin_model, popt=out.beta)