Matlab SHA-1 custom implementation doesn't give the right result - matlab

I trying to have a sha1 algorithm in Matlab.
I know I can use System.Security.Cryptography.HashAlgorithm.Create('SHA1');, but that relies on .NET, which I'd like to avoid.
I also found this thread, which suggest to use MessageDigest.getInstance("SHA-1"), but the output is not good.
I haven't found any other portable way.
Here's my code based on Wikipedia pseudo code, the result is also off. The only solution that gives result similar to online SHA1 is the .Net function.
Can somebody see the error in my function?
function [hh] = sha1(bytes_in)
% Note 1: All variables are unsigned 32-bit quantities and wrap modulo 232 when calculating, except for
% ml, the message length, which is a 64-bit quantity, and
% hh, the message digest, which is a 160-bit quantity.
% Note 2: All constants in this pseudo code are in big endian.
% Within each word, the most significant byte is stored in the leftmost byte position
%
% Initialize variables:
bytes_in = squeeze(uint8(bytes_in));
bytes_in = reshape(bytes_in, length(bytes_in), 1);
h0 = uint32(0x67452301);
h1 = uint32(0xEFCDAB89);
h2 = uint32(0x98BADCFE);
h3 = uint32(0x10325476);
h4 = uint32(0xC3D2E1F0);
% Pre-processing:
% append the bit '1' to the message e.g. by adding 0x80 if message length is a multiple of 8 bits.
% append 0 ? k < 512 bits '0', such that the resulting message length in bits
% is congruent to ?64 ? 448 (mod 512)
% append ml, the original message length in bits, as a 64-bit big-endian integer.
% Thus, the total length is a multiple of 512 bits.
%
message_len64 = uint64(length(bytes_in));
messages_len_bytes = zeros(8, 1);
for i=1:8
messages_len_bytes(i) = uint8(bitshift(message_len64, -64+i*8));
end
bytes_in = [bytes_in; 0x80];
padlen = 64-8 - mod(length(bytes_in), 64);
bytes_in = [bytes_in; zeros(padlen,1);messages_len_bytes];
assert(mod(length(bytes_in), 64) == 0);
chunk_count = length(bytes_in)/64;
% Process the message in successive 512-bit chunks:
% break message into 512-bit chunks
for i=1:chunk_count
chunk = bytes_in( ((i-1)*64+1):(i*64));
assert(length(chunk) == 64);
% Break chunk into sixteen 32-bit big-endian words w[i], 0 ? i ? 15
w = uint32(zeros(80,1));
for j=0:15
p1 = bitshift(uint32(chunk(j*4+1)),24);
p2 = bitshift(uint32(chunk(j*4+2)),16);
p3 = bitshift(uint32(chunk(j*4+3)),8);
p4 = bitshift(uint32(chunk(j*4+4)),0);
w(j+1) = p1 + p2 + p3 + p4;
end
% Message schedule: extend the sixteen 32-bit words into eighty 32-bit words:
for j=17:80
temp = bitxor(bitxor(bitxor(w(j-3),w(j-8)), w(j-14)), w(j-16));
w(j) = leftrotate32(temp, 1);
end
% Initialize hash value for this chunk:
a = h0;
b = h1;
c = h2;
d = h3;
e = h4;
for j=1:80
if j >= 1 && j <= 20
f = bitor(bitand(b,c), bitand(bitcmp(b), d));
k = 0x5A827999;
elseif j >= 21 && j <= 40
f = bitxor(bitxor(b,c), d);
k = 0x6ED9EBA1;
elseif j >= 41 && j <= 60
f = bitor(bitor(bitand(b, c),bitand(b, d)), bitand(c, d)) ;
k = 0x8F1BBCDC;
elseif j >= 61 && j <= 80
f = bitxor(bitxor(b,c), d);
k = 0xCA62C1D6;
end
temp = uint64(leftrotate32(a,5)) + uint64(f) + uint64(e) + uint64(k) + uint64(w(j));
temp = uint32(bitand(temp, uint64(0xFFFFFFFF)));
e = d;
d = c;
c = leftrotate32(b, 30);
b = a;
a = temp;
end
% Add this chunk's hash to result so far:
h0 = uint32(bitand(uint64(h0) + uint64(a), uint64(0xFFFFFFFF)));
h1 = uint32(bitand(uint64(h1) + uint64(b), uint64(0xFFFFFFFF)));
h2 = uint32(bitand(uint64(h2) + uint64(c), uint64(0xFFFFFFFF)));
h3 = uint32(bitand(uint64(h3) + uint64(d), uint64(0xFFFFFFFF)));
h4 = uint32(bitand(uint64(h4) + uint64(e), uint64(0xFFFFFFFF)));
% Produce the final hash value (big-endian) as a 160-bit number:
hh = [dec2hex(h0, 8), dec2hex(h1, 8), dec2hex(h2, 8), dec2hex(h3, 8), dec2hex(h4, 8)];
assert(length(hh) == 160/8*2)
end
end
function vout = leftrotate32(v32, v)
vout = uint32(bin2dec(circshift(dec2bin(v32, 32), -v)));
end

Related

Berlekamp Massey Algorithm for BCH simplified binary version

I am trying to follow Lin, Costello's explanation of the simplified BM algorithm for the binary case in page 210 of chapter 6 with no success on finding the error locator polynomial.
I'm trying to implement it in MATLAB like this:
function [locator_polynom] = compute_error_locator(syndrome, t, m, field, alpha_powers)
%
% Initial conditions for the BM algorithm
polynom_length = 2*t;
syndrome = [syndrome; zeros(3, 1)];
% Delta matrix storing the powers of alpha in the corresponding place
delta_rho = uint32(zeros(polynom_length, 1)); delta_rho(1)=1;
delta_next = uint32(zeros(polynom_length, 1));
% Premilimnary values
n_max = uint32(2^m - 1);
% Initialize step mu = 1
delta_next(1) = 1; delta_next(2) = syndrome(1); % 1 + S1*X
% The discrepancy is stored in polynomial representation as uint32 numbers
value = gf_mul_elements(delta_next(2), syndrome(2), field, alpha_powers, n_max);
discrepancy_next = bitxor(syndrome(3), value);
% The degree of the locator polynomial
locator_degree_rho = 0;
locator_degree_next = 1;
% Update all values
locator_polynom = delta_next;
delta_current = delta_next;
discrepancy_rho = syndrome(1);
discrepancy_current = discrepancy_next;
locator_degree_current = locator_degree_next;
rho = 0; % The row with the maximum value of 2mu - l starts at 1
for i = 1:t % Only the even steps are needed (so make t out of 2*t)
if discrepancy_current ~= 0
% Compute the correction factor
correction_factor = uint32(zeros(polynom_length, 1));
x_exponent = 2*(i - rho);
if (discrepancy_current == 1 || discrepancy_rho == 1)
d_mu_times_rho = discrepancy_current * discrepancy_rho;
else
alpha_discrepancy_mu = alpha_powers(discrepancy_current);
alpha_discrepancy_rho = alpha_powers(discrepancy_rho);
alpha_inver_discrepancy_rho = n_max - alpha_discrepancy_rho;
% The alpha power for dmu * drho^-1 is
alpha_d_mu_times_rho = alpha_discrepancy_mu + alpha_inver_discrepancy_rho;
% Equivalent to aux mod(2^m - 1)
alpha_d_mu_times_rho = alpha_d_mu_times_rho - ...
n_max * uint32(alpha_d_mu_times_rho > n_max);
d_mu_times_rho = field(alpha_d_mu_times_rho);
end
correction_factor(x_exponent+1) = d_mu_times_rho;
correction_factor = gf_mul_polynoms(correction_factor,...
delta_rho,...
field, alpha_powers, n_max);
% Finally we add the correction factor to get the new delta
delta_next = bitxor(delta_current, correction_factor(1:polynom_length));
% Update used data
l = polynom_length;
while delta_next(l) == 0 && l>0
l = l - 1;
end
locator_degree_next = l-1;
% Update previous maximum if the degree is higher than recorded
if (2*i - locator_degree_current) > (2*rho - locator_degree_rho)
locator_degree_rho = locator_degree_current;
delta_rho = delta_current;
discrepancy_rho = discrepancy_current;
rho = i;
end
else
% If the discrepancy is 0, the locator polynomial for this step
% is passed to the next one. It satifies all newtons' equations
% until now.
delta_next = delta_current;
end
% Compute the discrepancy for the next step
syndrome_start_index = 2 * i + 3;
discrepancy_next = syndrome(syndrome_start_index); % First value
for k = 1:locator_degree_next
value = gf_mul_elements(delta_next(k + 1), ...
syndrome(syndrome_start_index - k), ...
field, alpha_powers, n_max);
discrepancy_next = bitxor(discrepancy_next, value);
end
% Update all values
locator_polynom = delta_next;
delta_current = delta_next;
discrepancy_current = discrepancy_next;
locator_degree_current = locator_degree_next;
end
end
I'm trying to see what's wrong but I can't. It works for the examples in the book, but not always. As an aside, to compute the discrepancy S_2mu+3 is needed, but when I have only 24 syndrome coefficients how is it computed on step 11 where 2*11 + 3 is 25?
Thanks in advance!
It turns out the code is ok. I made a different implementation from Error Correction and Coding. Mathematical Methods and gives the same result. My problem is at the Chien Search.
Code for the interested:
function [c] = compute_error_locator_v2(syndrome, m, field, alpha_powers)
%
% Initial conditions for the BM algorithm
% Premilimnary values
N = length(syndrome);
n_max = uint32(2^m - 1);
polynom_length = N/2 + 1;
L = 0; % The curent length of the LFSR
% The current connection polynomial
c = uint32(zeros(polynom_length, 1)); c(1) = 1;
% The connection polynomial before last length change
p = uint32(zeros(polynom_length, 1)); p(1) = 1;
l = 1; % l is k - m, the amount of shift in update
dm = 1; % The previous discrepancy
for k = 1:2:N % For k = 1 to N in steps of 2
% ========= Compute discrepancy ==========
d = syndrome(k);
for i = 1:L
aux = gf_mul_elements(c(i+1), syndrome(k-i), field, alpha_powers, n_max);
d = bitxor(d, aux);
end
if d == 0 % No change in polynomial
l = l + 1;
else
% ======== Update c ========
t = c;
% Compute the correction factor
correction_factor = uint32(zeros(polynom_length, 1));
% This is d * dm^-1
dd_sum = modulo(alpha_powers(d) + n_max - alpha_powers(dm), n_max);
for i = 0:polynom_length - 1
if p(i+1) ~= 0
% Here we compute d*d^-1*p(x_i)
ddp_sum = modulo(dd_sum + alpha_powers(p(i+1)), n_max);
if ddp_sum == 0
correction_factor(i + l + 1) = 1;
else
correction_factor(i + l + 1) = field(ddp_sum);
end
end
end
% Finally we add the correction factor to get the new locator
c = bitxor(c, correction_factor);
if (2*L >= k) % No length change in update
l = l + 1;
else
p = t;
L = k - L;
dm = d;
l = 1;
end
end
l = l + 1;
end
end
The code comes from this implementation of the Massey algorithm

huffmandict error The symbol and probability vector must have the same length

I'm implementing this code for JPEG compression, but I get the error
??? Error using ==> huffmandict at 97
The symbol and probability vector must have the same length
this is the code, please help:
function y = mat2huff(x)
y.size = uint32(size(x));
x = round(double(x));
x = unique(x)
xmin = min(x(:));
xmax = max(x(:));
pmin = double(int16(xmin));
pmin = uint16(pmin+32768);
y.min = pmin;
x = x(:)';
h = histc(x, xmin:xmax);
if max(h) > 65535
h = 65535 * h / max(h);
end
[map , w] = huffmandict(x,h);
hx = map(x(:) - xmin + 1); % Map image
hx = char(hx)'; % Convert to char array
hx = hx(:)';
hx(hx == ' ') = [ ]; % Remove blanks
ysize = ceil(length(hx) / 16); % Compute encoded size
hx16 = repmat('0', 1, ysize * 16); % Pre-allocate modulo-16 vector
hx16(1:length(hx)) = hx; % Make hx modulo-16 in length
hx16 = reshape(hx16, 16, ysize); % Reshape to 16-character words
hx16 = hx16' - '0'; % Convert binary string to decimal
twos = pow2(15 : - 1 : 0);
y.code = uint16(sum(hx16 .* twos(ones(ysize ,1), :), 2))';
To ensure that your histc call does count the amount of x values per unique x value call it as
h=histc(x,linspace(xmin,xmax,numel(unique(x(:))));
Else, if you rimage is binary and your only values are 0 and 255, histc will return a size(h)=256 size array with lots of zeroes, because xmin:xmax is 0:255=[0 1 2 3 ... 254 255]

Have to convert Integer to binary

I'm writing a user-defined function to convert integers to binary. The largest number that could be converted with the function should be a binary number with
16 1 s. If a larger number is entered as d, the function should display an error
message. With my code, I'm trying to add the numbers 0 or 1 to my vector x based on the remainder, then I want to reverse my final vector to display a number in binary. Here's what I have:
function [b] = bina(d)
% Bina is a function that converts integers to binary
x = [];
y = 2;
in = d/2;
if d >=(2^16 -1)
fprintf('This number is too big')
else
while in > 1
if in >= 1
r = rem(in,y);
x = [x r]
end
end
end
end
As you insist on a loop:
x = [];
y = 2;
in = d;
if d >=(2^16 -1)
fprintf('This number is too big')
else
ii = 1;
while in > 0
r = logical(rem(in,y^ii));
x = [r x];
in = in - r*2^(ii-1);
ii = ii+1;
end
end
b = x;
You had the right ideas, but you need to update the variables in your while-loop with every iteration. This is mainly in, where you need to subtract the remainder. And just store the binary remainders in your variable x.
You can check your result with
x = double( dec2bin(d, 16) ) - 48
You could also use a for loop, by pre-calculating the number of iterations with
find( d < 2.^(1:16),1)
and then
if d >=(2^16 -1)
fprintf('This number is too big')
else
for ii = 1:find( d < 2.^(1:16),1)
r = logical(rem(in,y^ii));
x = [r x];
in = in - r*2^(ii-1)
end
end

Matlab FFT and home brewed FFT

I'm trying to verify an FFT algorithm I should use for a project VS the same thing on Matlab.
The point is that with my own C FFT function I always get the right (the second one) part of the double sided FFT spectrum evaluated in Matlab and not the first one as "expected".
For instance if my third bin is in the form a+i*b the third bin of Matlab's FFT is a-i*b. A and b values are the same but i always get the complex conjugate of Matlab's.
I know that in terms of amplitudes and power there's no trouble (cause abs value) but I wonder if in terms of phases I'm going to read always wrong angles.
Im not so skilled in Matlab to know (and I have not found useful infos on the web) if Matlab FFT maybe returns the FFT spectre with negative frequencies first and then positive... or if I have to fix my FFT algorithm... or if it is all ok because phases are the unchanged regardless wich part of FFT we choose as single side spectrum (but i doubt about this last option).
Example:
If S is the sample array with N=512 samples, Y = fft(S) in Matlab return the FFT as (the sign of the imaginary part in the first half of the array are random, just to show the complex conjugate difference for the second part):
1 A1 + i*B1 (DC, B1 is always zero)
2 A2 + i*B2
3 A3 - i*B3
4 A4 + i*B4
5 A5 + i*B5
...
253 A253 - i*B253
254 A254 + i*B254
255 A255 + i*B255
256 A256 + i*B256
257 A257 - i*B257 (Nyquyst, B257 is always zero)
258 A256 - i*B256
259 A255 - i*B255
260 A254 - i*B254
261 A253 + i*B253
...
509 A5 - i*B5
510 A4 - i*B4
511 A3 + i*B3
512 A2 - i*B2
My FFT implementation returns only 256 values (and that's ok) in the the Y array as:
1 1 A1 + i*B1 (A1 is the DC, B1 is Nyquist, both are pure Real numbers)
2 512 A2 - i*B2
3 511 A3 + i*B3
4 510 A4 - i*B4
5 509 A5 + i*B5
...
253 261 A253 + i*B253
254 260 A254 - i*B254
255 259 A255 - i*B255
256 258 A256 - i*B256
Where the first column is the proper index of my Y array and the second is just the reference of the relative row in the Matlab FFT implementation.
As you can see my FFT implementation (DC apart) returns the FFT like the second half of the Matlab's FFT (in reverse order).
To summarize: even if I use fftshift as suggested, it seems that my implementation always return what in the Matlab FFT should be considered the negative part of the spectrum.
Where is the error???
This is the code I use:
Note 1: the FFT array is not declared here and it is changed inside the function. Initially it holds the N samples (real values) and at the end it contains the N/2 +1 bins of the single sided FFT spectrum.
Note 2: the N/2+1 bins are stored in N/2 elements only because the DC component is always real (and it is stored in FFT[0]) and also the Nyquyst (and it is stored in FFT[1]), this exception apart all the other even elements K holds a real number and the oven elements K+1 holds the imaginary part.
void Fft::FastFourierTransform( bool inverseFft ) {
double twr, twi, twpr, twpi, twtemp, ttheta;
int i, i1, i2, i3, i4, c1, c2;
double h1r, h1i, h2r, h2i, wrs, wis;
int nn, ii, jj, n, mmax, m, j, istep, isign;
double wtemp, wr, wpr, wpi, wi;
double theta, tempr, tempi;
// NS is the number of samples and it must be a power of two
if( NS == 1 )
return;
if( !inverseFft ) {
ttheta = 2.0 * PI / NS;
c1 = 0.5;
c2 = -0.5;
}
else {
ttheta = 2.0 * PI / NS;
c1 = 0.5;
c2 = 0.5;
ttheta = -ttheta;
twpr = -2.0 * Pow( Sin( 0.5 * ttheta ), 2 );
twpi = Sin(ttheta);
twr = 1.0+twpr;
twi = twpi;
for( i = 2; i <= NS/4+1; i++ ) {
i1 = i+i-2;
i2 = i1+1;
i3 = NS+1-i2;
i4 = i3+1;
wrs = twr;
wis = twi;
h1r = c1*(FFT[i1]+FFT[i3]);
h1i = c1*(FFT[i2]-FFT[i4]);
h2r = -c2*(FFT[i2]+FFT[i4]);
h2i = c2*(FFT[i1]-FFT[i3]);
FFT[i1] = h1r+wrs*h2r-wis*h2i;
FFT[i2] = h1i+wrs*h2i+wis*h2r;
FFT[i3] = h1r-wrs*h2r+wis*h2i;
FFT[i4] = -h1i+wrs*h2i+wis*h2r;
twtemp = twr;
twr = twr*twpr-twi*twpi+twr;
twi = twi*twpr+twtemp*twpi+twi;
}
h1r = FFT[0];
FFT[0] = c1*(h1r+FFT[1]);
FFT[1] = c1*(h1r-FFT[1]);
}
if( inverseFft )
isign = -1;
else
isign = 1;
n = NS;
nn = NS/2;
j = 1;
for(ii = 1; ii <= nn; ii++) {
i = 2*ii-1;
if( j>i ) {
tempr = FFT[j-1];
tempi = FFT[j];
FFT[j-1] = FFT[i-1];
FFT[j] = FFT[i];
FFT[i-1] = tempr;
FFT[i] = tempi;
}
m = n/2;
while( m>=2 && j>m ) {
j = j-m;
m = m/2;
}
j = j+m;
}
mmax = 2;
while(n>mmax) {
istep = 2*mmax;
theta = 2.0 * PI /(isign*mmax);
wpr = -2.0 * Pow( Sin( 0.5 * theta ), 2 );
wpi = Sin(theta);
wr = 1.0;
wi = 0.0;
for(ii = 1; ii <= mmax/2; ii++) {
m = 2*ii-1;
for(jj = 0; jj <= (n-m)/istep; jj++) {
i = m+jj*istep;
j = i+mmax;
tempr = wr*FFT[j-1]-wi*FFT[j];
tempi = wr*FFT[j]+wi*FFT[j-1];
FFT[j-1] = FFT[i-1]-tempr;
FFT[j] = FFT[i]-tempi;
FFT[i-1] = FFT[i-1]+tempr;
FFT[i] = FFT[i]+tempi;
}
wtemp = wr;
wr = wr*wpr-wi*wpi+wr;
wi = wi*wpr+wtemp*wpi+wi;
}
mmax = istep;
}
if( inverseFft )
for(i = 1; i <= 2*nn; i++)
FFT[i-1] = FFT[i-1]/nn;
if( !inverseFft ) {
twpr = -2.0 * Pow( Sin( 0.5 * ttheta ), 2 );
twpi = Sin(ttheta);
twr = 1.0+twpr;
twi = twpi;
for(i = 2; i <= NS/4+1; i++) {
i1 = i+i-2;
i2 = i1+1;
i3 = NS+1-i2;
i4 = i3+1;
wrs = twr;
wis = twi;
h1r = c1*(FFT[i1]+FFT[i3]);
h1i = c1*(FFT[i2]-FFT[i4]);
h2r = -c2*(FFT[i2]+FFT[i4]);
h2i = c2*(FFT[i1]-FFT[i3]);
FFT[i1] = h1r+wrs*h2r-wis*h2i;
FFT[i2] = h1i+wrs*h2i+wis*h2r;
FFT[i3] = h1r-wrs*h2r+wis*h2i;
FFT[i4] = -h1i+wrs*h2i+wis*h2r;
twtemp = twr;
twr = twr*twpr-twi*twpi+twr;
twi = twi*twpr+twtemp*twpi+twi;
}
h1r = FFT[0];
FFT[0] = h1r+FFT[1]; // DC
FFT[1] = h1r-FFT[1]; // FS/2 (NYQUIST)
}
return;
}
In matlab try using fftshift(fft(...)). Matlab doesn't automatically shift the spectrum after the FFT is called which is why they implemented the fftshift() function.
It is simply a matlab formatting thing. Basically, matlab arrange Fourier transform in following order
DC, (DC-1), .... (Nyquist-1), -Nyquist, -Nyquist+1, ..., DC-1
Let's say you have a 8 point sequence: [1 2 3 1 4 5 1 3]
In your signal processing class, your professor probably draws the Fourier spectrum based on a Cartesian system ( negative -> positive for x axis); So your DC should be located at 0 (the 4th position in your fft sequence, assuming position index here is 0-based) on your x axis.
In matlab, the DC is the very first element in the fft sequence, so you need to to fftshit() to swap the first half and second half of the fft sequence such that DC will be located at 4th position (position is 0-based indexed)
I am attaching a graph here so you may have a visual:
where a is the original 8-point sequence; FT(a) is the Fourier transform of a.
The matlab code is here:
a = [1 2 3 1 4 5 1 3];
A = fft(a);
N = length(a);
x = -N/2:N/2-1;
figure
subplot(3,1,1), stem(x, a,'o'); title('a'); xlabel('time')
subplot(3,1,2), stem(x, fftshift(abs(A),2),'o'); title('FT(a) in signal processing'); xlabel('frequency')
subplot(3,1,3), stem(x, abs(A),'o'); title('FT(a) in matlab'); xlabel('frequency')

matlab: convert a string of hex values to a decimal value?

I wrote functions to convert 100,000 hex strings to values, but it takes 10 seconds to perform on the whole array. Does Matlab have a function to do this, so that it is faster, ... ie: less than 1 second for the array?
function x = hexstring2dec(s)
[m n] = size(s);
x = zeros(1, m);
for i = 1 : m
for j = n : -1 : 1
x(i) = x(i) + hexchar2dec(s(i, j)) * 16 ^ (n - j);
end
end
function x = hexchar2dec(c)
if c >= 48 && c <= 57
x = c - 48;
elseif c >= 65 && c <= 70
x = c - 55;
elseif c >= 97 && c <= 102
x = c - 87;
end
Try using hex2dec. It should be faster much faster than looping over each character.
shoelzer's answer is obviously the best.
However, if you want to do the conversion by yourself, then you might find this useful:
Assuming s is a char matrix: all hex numbers are of the same length (zero padded if necessary) and each row has a single number. Then
ds = double( upper(s) ); % convert to double
sel = ds >= double('A'); % select A-F
ds( sel ) = ds( sel ) - double('A') + 10; % convert to 10 - 15
ds(~sel) = ds(~sel) - double('0'); % convert 0-9
% do the sum through vector product
v = 16.^( (size(s,2)-1):-1:0 );
x = s * v(:);