I am trying to read the values from an energy meter, and convert them to REAL (32bit float).
In this case I am reading phase 1 voltage.
Each value is read across two registers.
So I have received to WORDS of values 17268 (MSW) and 2456 (LSW) converted them into a DWORD, and then to a REAL value after multiplying by 0.1, but I am not getting the answer I'm expecting.
I should be getting 245.0375 volts.
However I am getting 1.13E+08
Please see snip of structured text with live values.
snip
The problem is that DWORD_TO_REAL is trying to do a type conversion; that is, make the value of a DWORD match the format of a REAL. In your case, the contents of MSW and LSW are simply a IEEE754 value split in half and just need to be forced into the corresponding bits of a REAL variable. With TwinCAT (Beckhoff) I would do a direct memory copy:
MEMCPY(ADR(realValue)+2, ADR(MSW), 2);
MEMCPY(ADR(realValue), ADR(LSW), 2);
I would assume Schneider has a similar command.
Related
I am converting a program from MATLAB 2012 to 2016. I've been getting some strange errors, which I believe some of are due to a lack of precision in MATLAB functions.
For instance, I have a timeseries oldTs as such:
Time Data
-----------------------------
1.00000000000000001 1.277032377439511
1.00000000000000002 1.277032378456123
1.00000000000000003 1.277032380112478
I have another timeseries newTs with similar data, but many more rows. oldTs may have half a million rows, whereas newTs could have a million. I want to interpolate the data from the old timeseries with the new timeseries, for example:
interpolatedTs = interp(oldTs.time, oldTs.data, newTs.time)
This is giving me an error: x values must be distinct
The thing is, my x values are distinct. I think that MATLAB may be truncating some of the data, and therefore believing that some of the data is not unique. I found that other MATLAB functions do this:
test = [1.00000000000000001, 1.00000000000000002, 1.0000000000000000003]
unique(test)
ans =
1
test2 = [10000000000000000001, 10000000000000000002, 10000000000000000003]
unique(test2)
ans =
1.000000000000000e+19
MATLAB thinks that this vector only has one unique value in it instead of three! This is a huge issue for me, as I need to maintain the highest level of accuracy and precision with my data, and I cannot sacrifice any of that precision. Speed/Storage is not a factor.
Do certain MATLAB functions, by default, truncate data at a certain nth decimal? Has this changed from MATLAB 2012 to MATLAB 2016? Is there a way to force MATLAB to use a certain precision for a program? Why does MATLAB do this to begin with?
Any light shed on this topic is much appreciated. Thanks.
No, this has not changed since 2012, nor since the very first version of MATLAB. MATLAB uses, and has always used, double precision floating point values by default (8 bytes). The first value larger than 1 that can be represented is 1 + eps(1), with eps(1) = 2.2204e-16. Basically you have less than 16 decimal digits to play with. Your value 1.00000000000000001 is identical to 1 in double precision floating point representation.
Note that this is not something specific to MATLAB, it is a standard that your hardware conforms to. MATLAB simply uses your hardware's capabilities.
Use the variable precision arithmetic from the Symbolic Math Toolbox to work with higher precision numbers:
data = [vpa(1) + 0.00000000000000001
vpa(1) + 0.00000000000000002
vpa(1) + 0.00000000000000003]
data =
1.00000000000000001
1.00000000000000002
1.00000000000000003
Note that vpa(1.00000000000000001) will not work, as the number is first interpreted as a double-precision float value, and only after converted to VPA, but the damage has already been done at that point.
Note also that arithmetic with VPA is a lot slower, and some operations might not be possible at all.
My knowledge of matlab is very limited, so I'll use more general terms to explain my problem:
We have a recording system that samples variables in an embedded system in realtime, and delivers the recorded data as matlab files for analysis.
My problem is that if a recorded variable is a "double" (more specifically a 64-bit, IEEE 754-1985, floating point value), the result is delivered as two unsigned 32-bit integers, and I have no idea how to turn it back into a floating-point value in matlab.
For example, if I record the variable SomeFloat, which is a double, I will get the recorded data as two sets of data, SomeFloat1 and SomeFloat2. Both are unsigned, 32-bit integers. SomeFloat1 contains the 32 most significant bits of SomeFloat, and SomeFloat2 contains the 32 least significant bits.
I was hoping to find an existing function for converting it back do a double, I mean something like:
MyDouble = MyDreamFunction(SomeFloat1, SomeFloat2)
I have not been able to find MyDreamFunction, but being new to matlab, I'm not really sure where to look...
So, does anyone know of a simple way to do this?
I think you want typecast (convert datatypes without changing underlying data):
>> x1 = uint32(7346427); %// example uint32 value
>> x2 = uint32(1789401); %// example uint32 value
>> typecast([x2 x1],'double')
ans =
1.4327e-306
>> typecast([x1 x2],'double')
ans =
3.7971e-308
I have a list of real data in a file. The real data looks like this..
25.935
25.550
24.274
29.936
23.122
27.360
28.154
24.320
28.613
27.601
29.948
29.367
I write fortran90 code to read this data into an array as below:
PROGRAM autocorr
implicit none
INTEGER, PARAMETER :: TRUN=4000,TCOR=1800
real,dimension(TRUN) :: angle
real :: temp, temp2, average1, average2
integer :: i, j, p, q, k, count1, t, count2
REAL, DIMENSION(0:TCOR) :: ACF
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
open(100, file="fort.64",status="old")
do k = 1,TRUN
read(100,*) angle(k)
end do
Then, when I print again to see the values, I get
25.934999
25.549999
24.274000
29.936001
23.122000
27.360001
28.153999
24.320000
28.613001
27.601000
29.948000
29.367001
32.122002
33.818001
21.837000
29.283001
26.489000
24.010000
27.698000
30.799999
36.157001
29.034000
34.700001
26.058001
29.114000
24.177000
25.209000
25.820999
26.620001
29.761000
May I know why the values are now 6 decimal points?
How to avoid this effect so that it doesn't affect the calculation results?
Appreciate any help.
Thanks
You don't show the statement you use to write the values out again. I suspect, therefore, that you've used Fortran's list-directed output, something like this
write(output_unit,*) angle(k)
If you have done this you have surrendered the control of how many digits the program displays to the compiler. That's what the use of * in place of an explicit format means, the standard says that the compiler can use any reasonable representation of the number.
What you are seeing, therefore, is your numbers displayed with 8 sf which is about what single-precision floating-point numbers provide. If you wanted to display the numbers with only 3 digits after the decimal point you could write
write(output_unit,'(f8.3)') angle(k)
or some variation thereof.
You've declared angle to be of type real; unless you've overwritten the default with a compiler flag, this means that you are using single-precision IEEE754 floating-point numbers (on anything other than an exotic computer). Bear in mind too that most real (in the mathematical sense) numbers do not have an exact representation in floating-point and that the single-precision decimal approximation to the exact number 25.935 is likely to be 25.934999; the other numbers you print seem to be the floating-point approximations to the numbers your program reads.
If you really want to compute your results with a lower precision, then you are going to have to employ some clever programming techniques.
I'm generating ~1million text files containing arrays of doubles, tab delimited (these are simulations for research). Example output below. Each million text files I expect to be ~5 TB, which is unacceptable. So I need to compress.
However, all my data analysis will be done in matlab. And every matlab script will need to access all million of these text files. I can't decompress the whole million using C++, then run the matlab scripts, because I lack the HD space. So my question is, are there some very simple, easy to implement algorithms or other ways of reducing my text file sizes so that I can write the compression in C++ and read it in matlab?
example text file
0.0220874 0.00297818 0.000285954 1.70E-05 1.52E-07
0.0542912 0.00880725 0.000892849 6.94E-05 4.51E-06
0.0848582 0.0159799 0.00185915 0.000136578 7.16E-06
0.100415 0.0220033 0.00288016 0.000250445 1.38E-05
0.101889 0.0250725 0.00353148 0.000297856 2.34E-05
0.0942061 0.0256 0.00393893 0.000387219 3.01E-05
0.0812377 0.0238492 0.00392418 0.000418365 4.09E-05
0.0645259 0.0206528 0.00372185 0.000419891 3.23E-05
0.0487525 0.017065 0.00313825 0.00037539 3.68E-05
If it matters.. the complete text files represent joint probability mass functions, so they sum to 1. And I need lossless compression.
UPDATE Here is an IDIOTS guide to writing binary in C++ and reading it Matlab, with some very basic explanation along the way.
C++ code to write a small array to a binary file.
#include <iostream>
using namespace std;
int main()
{
float writefloat;
const int rows=2;
const int cols=3;
float JPDF[rows][cols];
JPDF[0][0]=.19493;
JPDF[0][1]=.111593;
JPDF[0][2]=.78135;
JPDF[1][0]=.33333;
JPDF[1][1]=.151535;
JPDF[1][2]=.591355;
JPDF is an array of type float that I save 6 values to. It's a 2x3 array.
FILE * out_file;
out_file = fopen ( "test.bin" , "wb" );
To be honest, I don't quite get what the first line is doing. It seems to be making a pointer of type FILE named out_file. The second line fopen says make a new file for writing (the 'w' of the second parameter), and make it a binary file (the 'b' of the wb).
fwrite(&rows,sizeof(int),1,out_file);
fwrite(&cols,sizeof(int),1,out_file);
Here I encode the size of my array (# rows, # cols). Note that we fwrite the reference to the variables rows and cols, not the variables themselves (& is by ref). The second parameter tells it how many bytes to write. Since rows and cols are both ints, I use sizeof(int). The '1' says do this 1 time. I think. And out_file is our pointer to the file we're writing to.
for (int i=0; i<3; i++)
{
for (int j=0; j<2; j++)
{
writefloat=JPDF[j][i];
fwrite (&writefloat , sizeof(float), 1, out_file);
}
}
fclose (out_file);
return 0;
}
Now I'll iterate through my array and write each value in bytes to my file. The indexing is a little backwards looking in that I'm iterating down each column rather than across a column in the inner loop. We'll see why in a sec. Again, I'm writing the reference to writefloat, which takes on the value of the current array element in each iteration. Since each array element is a float, I'm using sizeof(float) here instead of sizeof(int).
Just to be incredibly, stupidly clear, here's a diagram of how I think of the file we've just created.
[4 bytes: rows][4 bytes: cols][4 bytes: JPDF[0][0]][4 bytes: JPDF[1][0]] ...
[4 bytes: JPDF[1][2]]
..where each chunk of bytes is written in binary (0s and 1s).
To interpret in MATLAB:
FID=fopen('test.bin');
sizes=fread(FID,2,'int')
FID sort of works like a pointer here. Secretly, it probably is a pointer. Then we use fread which operates very similarly to C++ fread. FID is our 'pointer' to our file. The 'int' tells the function how many bytes each chunk contains. So sizes=fread(FID,2,'int') says 'open FID in binary, and read 2 chunks of size INT bytes, and return the 2 elements in vector form. Now, sizes(1)=rows, and sizes(2)=cols.
s=fread(FID,[sizes(1) sizes(2)],'float')
The next part wasn't completely clear to me originally, I thought I'd have to tell fread to skip the 'header' of my binary that contains row/col info. However, it secretly maintains a pointer to where you left off. So now I empty the rest of the binary file, using the fact that I know the dimensions of the array. Note, while the second parameter [M,N] is [rows,cols], fread reads in "column order", which is why we wrote the array data in column order.
The one * is that I think I can only use matlab code 'int' and 'float' if the architecture of the C++ program is concordant with matlab (e.g., both are 64-bit, or both are 32-bit). But I'm not sure about this.
The output is:
sizes =
2
3
s =
0.194930002093315 0.111593000590801 0.781350016593933
0.333330005407333 0.151535004377365 0.59135502576828
To do better than four bytes per number, you need to determine to what precision you need these numbers. Since they are probabilities, they are all in [0,1]. You should be able to specify a precision as a power of two, e.g. that you need to know each probability to within 2-n of the actual. Then you can simply multiply each probability by 2n, round to the nearest integer, and store just the n bits in that integer.
In the worst case, I can see that you are never showing more than six digits for each probability. You can therefore code them in 20 bits, assuming a constant fixed precision past the decimal point. Multiply each probability by 220 (1048576), round, and write out 20 bits to the file. Each probability will take 2.5 bytes. That is smaller than the four bytes for a float value.
And either way is way smaller than the average of 11.3 bytes per value in your example file.
You can get better compression even than that if you can exploit known patterns in your data. Assuming that there are any. I see that in your example, on each line the values go down by some factor at each step. If that is real and not just an artifact of the generation of the example, then you can successively use fewer bits for each sample. Also if the first sample is really always less than 1/8, then you can drop the top three bits off that one, since those bits would always be zero. If the second column is always less than 1/32, you can drop the first five bits off all of those. And so on. Assuming that the magnitudes in the example are maximums across all of the data sets (obviously not true, but just using that as an illustrative case), and assuming you need six decimal digits after the decimal point, I could code each row of six values in 50 bits, for an average of a little over one byte per probability.
And for one last smidgen of compression, since the values add to one, you don't have store the last value.
Matlab can read binary files. Why not save your files as binary instead of text?
Saving each number as a float would only require 4 bytes (if you're running 32 bit linux), you could use doubles but it appears that you aren't using the full double resolution. Under your current scheme each digit every number consumes a byte of space. All of your numbers are easily 4+ char longs, some as long as 10 chars. Implementing this change should cut down your file sizes by more than 50%.
Additionally you might consider using a more elegant data format like HDF5 (more here) that both supports compression and is supported by matlab
Update:
There are lots of examples of how to write binary files in C++, just google it.
Additionally to read in a binary file in Matlab simply use fread
The difference between representing a number as ascii vs binary is really simple. All files are written using binary, the difference is in how that information gets interpreted. Text files are generally read using ASCII, which provides a nice mapping between an 8bit word and characters. When you see a string like "255" what you have is a array of bytes where each byte encodes on character in the array. However when you are storing numbers its really wasteful to store each digit of using a different byte. A single byte can store values between 0-255. So why use three bytes to store the string "255" when I can use a single byte to store the value 255.
You can always go ahead and zip everything using a standard library like zlib. Afterwards you could use a custom dll written in C++ that unzips your data in chunks you can manage. So basically:
Data --> Zip --> Dll (Loaded by Matlab via LoadLibrary) --> Matlab
I'm trying to convert very long binary strings, often greater than 52 bits into numbers. I cannot have a fixed lookahead window because I am doing this to calculate a version of Lempel-Ziv complexity for neural data.
When I try to convert any long string, bin2dec throws and error that the binary string must be 52 bits or less.
Is there a way to get around this size limitation?
dec2bin throws that error because a single is not capable of storing that much precision. Your very question asks an impossibility. You have two choices: store the value in something other than a floating point value, or throw away some precision before you convert.
Or describe more completely what you're trying to accomplish.
EDITING:
Based on your additional information, I am even more certain that converting to floating point is not what you want to do. If you want to reduce the storage size to something more efficient, convert to a vector of bytes (uint8), which is as dense as you can get. Just split the binary string into N rows of 8 digits each, using reshape. This seems to be an accepted approach for biological data.
str = char((rand(1, 100)>0.5) + '0'); % test data
data = uint8(bin2dec(reshape(str(1:end-mod(end,8)), [], 8)));
In this code, I toss any bits that don't divide evenly into 8. Or, skip the uint8 step and just perform your processing on the resulting vector, where each double-precision float represents one 8-bit word from your sequence.
You could roll your own implementation:
len = 60;
string = [];
for i = 1:len
string = [string sprintf('%d', randi([0 1]))];
end
% error
% bin2dec(string);
% roll your own...
value = 0;
for i = length(string):-1:1
value = value + str2num(string(i))*2^(length(string)-i);
end
I'm just looping through the string and adding to some value. At the end, value will contain the decimal value of the string. Does this work for you?
Note: This solution is slow. You can speed it up a bit by preallocating the string, which I did on my own machine. Also, it's going to have issues if your number gets up to 1e6 digits. At that point, you need variable precision arithmetic to keep track of it. And adding that to the calculation really slowed things down. If I were you, I'd strongly consider compiling this from a .mex file if you need the functionality in MATLAB.
credits due to #aardvarkk, but here's a sped up version of his algorithm (+- 100x faster):
N=100;
strbin = char(randi(2,1,N)+'0'-1);
pows2 = 2.^(N-1:-1:0);
value=pows2*(strbin-'0')';
double's range goes only up to 1.79769e+308 which is 2^1024 give or take. From there on, value will be Inf or NaN. So you still need to find another way storing the resulting number.
A final pro on this algorithm: you can cache pows2 for a large number and then use a piece of it for any new strbin of length N:
Nmax = 1e8; % already 700MB for pows2, watch out!
pows2 = 2.^(Nmax-1:-1:0);
and then use
value = pows2(Nmax-N+1:end)*(strbin-'0')';
Solution to matlab's numeric upper bound
There's a tool on the File Exchange called vpi: http://www.mathworks.com/matlabcentral/fileexchange/22725
It allows you to use really big integers (2^5000? no prob). It's only slower (a lot) in calculating everything, I don't suggest using my method above with this. But hey, you can't have everything!
Download the package, addpath it and the following might work:
N=3000;
strbin = char(randi(2,1,N)+'0'-1);
binvals=strbin-'0';
val=0;
twopow=vpi(1);
for ii=1:N
val=val+twopow*binvals(N-ii+1);
twopow=twopow*2;
end