How to read DAF (double precision array file) "transfer" files? - encoding

I downloaded some data in DAF "transfer" format, which NASA completely
fails to explain here:
http://naif.jpl.nasa.gov/pub/naif/toolkit_docs/C/req/daf.html#Conversion%20and%20Transfer%20of%20DAF%27s
How do I read this file. Here are the first few lines I'm trying to comprehend:
DAFETF NAIF DAF ENCODED TRANSFER FILE
'DAF/SPK '
'2'
'6'
'NIO2SPK '
BEGIN_ARRAY 1 3895604
'URA111 '
'-BC186A96D0E76^8'
'BC0DDF032F041^8'
'2BD'
'7'
'1'
'3'
1024
'-BC18166^8'
'FD2^4'
'-DA4A19AC2BCD18^4'
'-4D5E7E1A67739^4'
'1D46248537C30E^5'
'EBA587DFA5E3B^3'
'-26885CE73CB0D^4'
'-BF0DC6EDB5B2C8^2'
'129C1CFEABE48^3'
'5594FC676368^1'
'-472EBF2225A^1'
'-2198AE1963D^0'
'79CC4CA0C^-1'
'FDD9792D82^-2'
'2001D81A^-2'
'333BCEE2BDD724^4'
'-D78AA10831D9C8^4'
'-6D712677574DF8^4'
'283A14783CDC^4'
'90AC22194ABF6^3'
'-1DEF6219F664FE^3'
'-47318F604096^2'
'9B805F405B1C^1'
'1275B947E2AC^1'
'-16A664664D^0'
'-2F614B9F5^-1'
'-B7C3E41D^-3'
'2F3D71F8^-3'
According to NASA, this is/was a popular format for Fortran programs,
but google was not at all helpful (wikipedia doesn't have an entry
either).

OK, I think I finally figured it out at least part of this. For
reference, the original file (a whopping 162M in size) is the
ura111.bsp file in:
http://naif.jpl.nasa.gov/pub/naif/generic_kernels/spk/satellites/
and converted to ura111.xsp using the toxfr program in:
http://naif.jpl.nasa.gov/pub/naif/utilities/SunIntel_32bit/
The small files:
http://naif.jpl.nasa.gov/pub/naif/generic_kernels/spk/satellites/ura111.cmt
http://naif.jpl.nasa.gov/pub/naif/generic_kernels/spk/satellites/ura111.inp
explain more about the main file.
Things like "-BC18166^8" really are double precision numbers, written
in modified hexadecimal IEEE-754 format. Wikipedia sort of explains
this format here:
http://en.wikipedia.org/wiki/IEEE-754
and there are IEEE-754-to-decimal convertors like this:
http://www.h-schmidt.net/FloatConverter/ (and many others)
However, these don't explain/convert the exact format NASA uses, which
was one reason for my confusion.
For reference "-BC18166^8" is converted as follows:
The decimal value of "BC18166" is 197230950
We now divide by 16 repeatedly until the result is less than 1 (in
other words, we divide by 16^(length of "BC18166")), yielding
0.734742544591427
The '^8' means we multiply by 16**8 to get 3155695200
the leading "-" just means we add a minus sign to get -3155695200
Of course, we could've combined steps 2 and 3 and just multiplied
197230950 by 16.
#klugerama, to answer your question, yes, I am trying to write a file
parser, this time in Perl, as part of a program that accurately
identifies the positions of various objects in our solar system.
I've already parsed the NASA files relating to planets (and Earth's
own moon) here:
ftp://ssd.jpl.nasa.gov/pub/eph/planets/ascii/
but these are in a much different and far easier-to-parse format.

This document (hosted at ucla.edu) has a complete description of the file format.
Addtionally, check out this python project on Github. It appears to provide the DAFTB function you're looking for.
Edit For the record (cough), it doesn't look like this format was ever intended to be read, per se, by humans. It's a transfer format intended to be converted back to usable binary in whatever executable code is appropriate.
You didn't explain why you want to do this. So unless you are writing a file parser (which has been done already in at least two languages), I'm not sure what the benefit is of being able to read the raw values.
Strictly speaking, the answer to your question is that you use software (see link above) to read it.

Related

Post-processing the output result from Dymola when there are only 2 elements in the result

I am trying to use the MATLAB scripts shipped with Dymola to post-process the output result of Dymola. But in some cases, the output data in the .mat file only have 2 elements, how could I get the data between 10s and 100s in this kind of cases?
It's a parameter or variable that is not time depending so it's stored in a compact way. I understand the mechanism, but it is not user-friendly when post-processing the data in MATLAB, I have to find the "wrong" dimensional data. How could I fix this issue?
I recommend creating some simple logic that looks at the size of the variable and then automatically puts it into some dictionary, list, etc. From there you can manipulate the variable. I know you are asking for Matlab but here is a Python solution that I have used which may help you get started:
varNames_param_base=[]
varNames_var_base=[]
for i, val in enumerate(r.varNames()):
if np.size(r.values(val)) == 4:
varNames_param_base.append(val)
else:
varNames_var_base.append(val)
I used those lines in this file.
In the example r.varNames() is a list of all the variable names (i.e., strings) which are read from the resulting Dymola .mat file. r.values gets the value of the variable name currently being used in the for loop (i.e., val).
You may also consider converting your result file to SDF (a simple HDF5 representation), because that format does not use any clever storage options (if I remember correctly).

Can I force visdiff to display more than the first 2000 bytes?

I have two binary files that I'm trying to compare using Matlab's built-in function visdiff, but it only displays the first 2000 bytes as a default. Is there any way to force the comparison tool to display the entire contents of both files side by side?
Edit the file matlabroot\toolbox\shared\comparisons\private\bindiff.m, where matlabroot is your MATLAB installation directory. On line 149, you'll see it sets the variable MAXLEN to 2000. Change this to something bigger (even Inf seems to work).
You may need to type rehash toolboxcache after making this change, in order to get MATLAB to notice.
Please note:
As you're making a change to the MATLAB source, this is at your own risk (it seems fine to me though). Keep a backup of the file you've edited.
That truncation at 2000 bytes is there for a reason - comparing the whole of larger binary files does seem to take quite a while, so be patient. Maybe try gradually increasing MAXLEN, rather than going straight to Inf.
I only have R2011b available to me right now, so if you're on a newer version the file path and line number I mentioned above may have changed. It was very easy to trace through the code from visdiff to comparisons_private to bindiff though, so unless they've changed the deeper structure of the Comparisons Tool between 11b and now, it will probably be very similar.

How determine date format in MATLAB

I want to know in MATLAB which is the date pattern used by Excel. This is because I read an Excel file from MATLAB, but depending of the user machine locate the date is represented as dd-mm-yyyy or mm-dd-yyyy.
CLARIFICATION: Sorry for my bad explanation. This is my scenario. I have an Excel file with dates (and other collumns, no relevant for this problem). I have two computers, which need to run my matlab application. In the first one when I use xlsread (in MATLAB) the dates appears in dd-mm-yyyy format due to the regional configuration of my computer. In the second one, I read the same file, in the same MatLab version, but the readed dates are in mm-dd-yyyy format (again, due to the regional configuration of computer 2, which is different from computer 1).
Now, when I try to use datenum, to date transformation, I cant use formatIn parameter in a right way, because if I specify the formatIn equals to mm-dd-yyyy this will Works correctly in computer 1, by not in computer 2, and vice versa.
So, I think that I need to identify in MATLAB which is the date pattern used by Excel in the computer, in order to find the right input parameter for formatIn.
It is impossible to do unless you know your data really well. For instance if you have yearly readings for 01/07/20XX, it is impossible to know if it is 7th Jan or 1st July.
However, you can try the following:
MyString='01-23-2012';
FirstTwo=str2num(MyString(1:2));
if(FirstTwo>12)
display('DD/MM');
else
display('MM/DD');
end
If the first two digits of the date are greater than 12, then you can probably conclude that you have DD/MM/YYYY. You can loop this over all your dates.
If you're talking about an actual .xls file, I don't know enough to say if there's a some flag for this kind of thing, but one heuristic approach (and possibly the only approach with a CSV format) would be to look for numbers greater than 12. That will immediately tell you which format you have, because such a number can't be correspond to a month. Of course, with a small data set, this isn't reliable (strictly, it's never perfectly reliable, but with non-trivial data, it's highly likely to work).
You may be able to do something with Java to tell you the date format.
>> import java.text.DateFormat;
>> import java.text.SimpleDateFormat;
>> df = DateFormat.getDateInstance(DateFormat.SHORT);
>> dateFormat = char(df.toPattern())
dateFormat =
dd/MM/yy
I think xlsread uses this format, although you'll need to test it on both of your machines.
Note there is also a Locale input to getDateInstance that may be useful.
I am kind of confused by your question, both MATLAB and Excel are able to easily support mm-dd and dd-mm. In excel, the default will depend on where you live. In America, it will be mm-dd, and in Europe (and probably most of the rest of the world), it will be dd-mm.
In MATLAB, I am not sure if it is location dependent like Excel is, as an American, the standard is of course mm-dd, but you can fully customize how matlab parses date strings!
Check out http://www.mathworks.com/help/matlab/ref/datenum.html and then go to input arguments, then "formatIn", it will provide you a list of ways to read in dates and convert it to a serial date number. (or vector if you want)
EDIT:
Nevermind, I misunderstood your question
I run into the same issue with computers from Australia and USA.
This is a way around but it is a clean solution.
In excel convert date to text for example
in International format 'yyyymmdd'
% B1=TEXT(A1,"yyyymmdd") % This is in excel
% in matlab read excel file 'dates.xlsx'
[data, dates_header] =xlsread('dates.xlsx');
% use datevec to read-in data
t = datevec(dates_header(:,2),'yyyymmdd');

Displaying EUnit code coverage in Emacs

I am using Rebar to build my erlang project and want to integrate it more tightly with Emacs. I found that if I add {cover_print_enabled, true}. to my rebar config file I get code coverage in the build output.
However there is also an option cover_export_enabled which outputs a binary file of some form. Is there an emacs plugin to parse that file and color code my code to show what code is covered by tests?
I really don't like having to switch to a browser to see code coverage.
As far as I know, there is no such plugin.
The exported cover data file can be read as follows:
Read one byte, giving the length of the next term; let's call it N.
Read N bytes in Erlang binary term format. This can be decoded with binary_to_term/1.
If the term from step 2 is of the form {'$size',X}, then read X bytes and decode as a term. (This happens when the binary representation of the term is longer than 255 bytes.)
Continue from step 1, until end of file.
Distel has an Emacs Lisp implementation of binary_to_term called erlext-read-obj in erlext.el.
I haven't looked into what to do with the terms in the file, once decoded, but hopefully this is enough to get someone started. Read lib/tools/src/cover.erl if in doubt.
Just added this feature to rebar.el in commit https://github.com/leoliu/rebar.el/commit/9ba8699ff6310721226b93341e62491ebfd0ee99
Leo

Create Numbers file and open it with Numbers on iPad

I would like to do a task that is quite simple on other OS, but it is not so trivial on iOS. Namely, I want to create file and open it in Numbers.
I can preview the file with UIDocumentInteractionController and then offer it to user that he/she opens it.
THis seems to me quite a reasonable solution. However, I need to offer proper file format. I suppose CSV and XLS would be reasonable to implement and it would most probably work, but I would still like to do it in native Numbers format if possible. However, I can't find any info about this file format.
Basically, this task is about exporting data to another app and then working further with them.
I don't know of a library that can create native Numbers files. There are hoewever some libraries that allow creating XLS files. Since Numbers fully supports XLS, this is probably the way to go.
There is a comercial library available that might work on the iPhone (costs $200): http://www.libxl.com/
As for free XLS libraries, I only know xlwt, a Python module. You could set up a webservice that creates an XLS file for your app, using xlwt on the server side.
If you want to pass information to Numbers, you can probably also use CSV files. If you use CSV files, you must be aware of some things. There are two kinds of CSV files: the comma separated version (used in english speaking countries) and the semicolon separated (used in continental europe).
The comma separated CSV files look for example like this:
"ID","First Name","Last Name","Salary"
1,"John","Malkovich",3400.20
2,"Fred","Astaire",2000.60
The second kind of CSV files are semicolon separated and use a comma as decimal mark. They look like this:
"ID";"First Name";"Last Name";"Salary"
1;"John";"Malkovich";3400,20
2;"Fred";"Astaire";2000,60
On the Macintosh, Numbers expects a different format depending on the Region setting. If you have your Region set to the US, it will expect the first kind. If you choose Germany, it will expect the second kind.
I don't know what kind of files Numbers on the iPad expects.
Another alternative would be using copy and paste. Try to copy tab separated text into the clipboard.
I hope this may help you. I've contacted libxl team and they responded with the link to the demo version of their iPhone library: http://www.libxl.com/download/libxl-iphone.zip