I am looking for the hash algorithm that symstore uses to create the directory name. I found this link Microsoft Symbol Server / Local Cache Hash Algorithm that describes the data elements that are used to generate the hash, but it does not go into any detail on how the hash value is calculated. I am interested to see how symstore generates the hash directory and if anyone has any sample code that they can show, that would be great!
symstore.exe calculates hash directory names as follows:
For PDB files, the GUID + Age, are used. Here is a python example:
pdb = pdbparse.parse("some.pdb")
pdb.STREAM_PDB.load()
guid = pdb.STREAM_PDB.GUID
guid_str = "%.8X%.4X%.4X%s" % (guid.Data1, guid.Data2, guid.Data3,
guid.Data4.encode("hex").upper())
symstore_hash = "%s%s" % (guid_str, pdb.STREAM_PDB.Age)
For PE (exe/dll) files, the TimeDateStamp (from IMAGE_FILE_HEADER) and SizeOfImage (from IMAGE_OPTIONAL_HEADER) are used. Here is a python example:
pe = pefile.PE("some.exe")
symstore_hash = "%X%X" % (pe.FILE_HEADER.TimeDateStamp,
pe.OPTIONAL_HEADER.SizeOfImage)
Here is an example python script that prints symstore hash values for PDB and PE files:
https://gist.github.com/lennartblanco/9a70961a5aa66fe49df6
Not sure if you have already reviewed this but it is the U.S. Patent describing the symbol store process. Its pretty dense as you can imagine but it does describe in quite a bit of detail how the symbol store directories are expanded and deleted (specifically in sections 6, 7, 8). Hope this helps or points you in the right direction.
Related
I am trying to copy the C and H files to another directory, which are generated with the command:
rtwbuild('path/to/subsystem')
But I am having trouble determining the exact output directory used for the code generation. Until now I am using the following snippet:
% Assuming 'sys' is a subsystem string like from gcb
sysName = get_param(sys, 'Name');
bldInfo = RTW.GetBuildDir(bdroot(sys));
codeDir = bldInfo.CodeGenFolder;
bldSuffix = bldInfo.BuildDirSuffix;
if strcmp(mdl, sysName)
sysName = [sysName '0'];
end
bldDir = fullfile(codeDir, [sysName bldSuffix]);
This seems to work for most cases, but I now have a subsystem which uses the suffix '1' (i.e. [sysName '1']) instead of '0'.
How is this numeric suffix generated?
Is there an API command which returns the correct build directory for a given subsystem? (As the above code also does not handle special characters and probably other corner cases)
Note to overambitious downvoters: I already did search in the documentation, on Stackoverflow and Google.
I am trying to read all the images in the folder in MATLAB using this code
flst=dir(str_Expfold);
But it shows me output like this. which is not the sequence as i want.
Can anyone please tell me how can i read all of them in sequence?
for giving downmark, please explain the reason for that too.
By alphabetical order depth10 comes before depth2. If at all possible, when creating string + num type filenames, use a fixed width numerical part (e.g. depth01, depth02) - this tends to avoid sorting problems.
If you are stuck with the filenames you have, and know the filename pattern, though, you can not bother using dir at all and create your filename list in the correct order in the first place:
for n = 1:50
fname = sprintf('depth%d.png',n);
% code to read and process images goes here
end
From the Matlab forums, the dir command output sorting is not specified, but it seems to be purely alphabetical order (with purely I mean that it does not take into account sorter filenames first). Therefore, you would have to manually sort the names. The following code is taken from this link (you probably want to change the file extension):
list = dir(fullfile(cd, '*.mat'));
name = {list.name};
str = sprintf('%s#', name{:});
num = sscanf(str, 'r_%d.mat#');
[dummy, index] = sort(num);
name = name(index);
I am looking for a way to list the differences between two .mat files, something that can be usefull for many people.
Though I searched everywhere I could think of, I have not found anything that meets my requirements:
Pick 2 mat files
Find the differences
Save them properly
The closest I have come is visdiff. As long as I stay within matlab, it will allow me to browse the differences, but when I save the result it only shows me the top level.
Here is a simplified example of what my files typically look like:
a = 6;
b.c.d = 7;
b.c.e = 'x';
save f1
f = a;
clear a
b.c.e = 'y';
save f2
visdiff('f1.mat','f2.mat')
If I click here on b, I can find the difference. However if I run this and use 'file>save', I am not able to click on b. Thus I still don't know what has been changed.
Note: I don't have Simulink
Hence my question is:
How can I show all differences between 2 mat files to someone without Matlab
Here are the answers that I personally consider to be most suitable for different situations:
Answer for users with Simulink
General answer
Answer displaying all value differences
Find all differences between mat files without MATLAB?
You can find the differences between HDF5 based .mat files with the HDF5 Tools.
Example
Let me shorten your MATLAB example and assume you create two mat files with
clear ; a = 6 ; b.c = 'hello' ; save -v7.3 f1
clear ; a = 7 ; b.e = 'world' ; save -v7.3 f2
Outside MATLAB use
h5ls -v -r f1.mat
to get a listing about the kind of data included f1.mat:
Opened "f1.mat" with sec2 driver.
/ Group
Location: 1:96
Links: 1
/a Dataset {1/1, 1/1}
Attribute: MATLAB_class scalar
Type: 6-byte null-terminated ASCII string
Data: "double"
Location: 1:2576
Links: 1
Storage: 8 logical bytes, 8 allocated bytes, 100.00% utilization
Type: native double
/b Group
Attribute: MATLAB_class scalar
Type: 6-byte null-terminated ASCII string
Data: "struct"
Location: 1:800
Links: 1
/b/c Dataset {5/5, 1/1}
Attribute: H5PATH scalar
Type: 2-byte null-terminated ASCII string
Data: "/b"
Attribute: MATLAB_class scalar
Type: 4-byte null-terminated ASCII string
Data: "char"
Attribute: MATLAB_int_decode scalar
Type: native int
Data: 2
Location: 1:1832
Links: 1
Storage: 10 logical bytes, 10 allocated bytes, 100.00% utilization
Type: native unsigned short
Use of
h5ls -d -r f1.mat
returns the values of the stored data:
/ Group
/a Dataset {1, 1}
Data:
(0,0) 6
/b Group
/b/c Dataset {5, 1}
Data:
(0,0) 104, 101, 108, 108, 111
The data 104, 101, 108, 108, 111 represents the word hello, which can be seen with
h5ls -d -r f1.mat | tail -1 | awk '{FS=",";printf("%c%c%c%c%c \n",$2,$3,$4,$5,$6)}'
You can get the same listing for f2.mat and compare the two outputs with the tool of your choice.
Comparison also works directly with HDF5 Tools. To compare the two numbers a from both files use
h5diff -r f1.mat f2.mat /a
which will show you the values and their difference
dataset: </a> and </a>
size: [1x1] [1x1]
position a a difference
------------------------------------------------------------
[ 0 0 ] 6 7 1
1 differences found
attribute: <MATLAB_class of </a>> and <MATLAB_class of </a>>
0 differences found
Remarks
There are a few more commands and options in the HDF5 Tools, which may help to get your real problem solved.
Binary distributions are available for Linux and Windows from The HDF Group. For OS X you can get them installed via MacPorts. If needed there is also a GUI: HDFView.
If you have simulink you can use Simulink.saveVars to generate an m-file that upon execution creates the same variables in work space:
a = 6;
b.c.d = 7;
b.c.e = 'x';
Simulink.saveVars('f1');
f = a;
clear a
b.c.e = 'y';
Simulink.saveVars('f2');
visdiff('f1.m','f2.m')
as illustrated in this sctreenshot
Note that by default it limits the number of elements in arrays to 1000 and you can increase it to 10000. Arrays larger than that limit will be saved in a separate mat-file.
UPDATE: From R2014a a new function similar to Simulink.saveVars has been added to MATLAB. see matlab.io.saveVariablesToScript
This is only part of the answer, but maybe it helps.
You could use gencode, a Matlab function that generates Matlab code from a variable such that running the code reproduces the variable. You do this for all of the variables in each mat-file (takes some programming, but should be doable) and put the results in different .m-files.
Then you use a standard text comparison tool (maybe even visdiff) to compare the .m-files.
There are several good tools to compare XML-Files, this I would proceed this way:
Download struct2xml.m
Load both matfiles
Export each with struct2xml
compare, using XMLSpy or similar
Simple general answer, without displaying value differences
Due to the insight I gained from the answers of #BHF, #Daniel R and #Dennis Jaheruddin, I have managed to find a simple scalable solution:
[fs1, fs2, er] = comp_struct(load('f1.mat'),load('f2.mat'))
Note that it works for .mat containing an arbritrary number of variables.
This uses the Compare Structures - File Exchange submission.
Answer for small files, displaying all value differences
Based on the suggestion by #A. Donda I have tried to use gencode to create a variable for everything.
Though it works for my toy example, it is quite slow and tells me that I exceed the allowed amount of variables for my real .mat files.
Anyway, for those who are looking for something that works with small files, I will post this option:
wList=who;
for iLoop = 1:numel(wList)
eval(['generated_' wList{iLoop} '= gencode(' wList{iLoop} ');'])
for jLoop = 1:numel(eval(['generated_' wList{iLoop}]))
eval(['generated_' wList{iLoop} '_' num2str(jLoop) '= generated_' wList{iLoop} '(' num2str(jLoop) ');' ])
end
end
Though it may work, I don't feel like this is the best way to go.
General answer, without displaying value differences
Due to the insight I gained from the answers of #BHF and #Daniel R I have managed to find a reasonably scalable solution.
Step 1: Save all variables from each files as a single struct
This uses the Save workspace to struct - File Exchange submission.
Here are the steps to take assuming you want to compare f1.mat and f2.mat:
clear
load f1
myStruct1 = ws2struct;
save myStruct1 myStruct1
clear
load f2
myStruct2 = ws2struct;
save myStruct2 myStruct2
clear
load myStruct1
load myStruct2
Step 2: Compare the structs
This uses the Compare Structures - File Exchange submission
Given that you want to compare myStruct1 and myStruct2 you can simply call:
[fs1, fs2, er] = comp_struct(myStruct1,myStruct2)
I was positively surprised at how readable the list of differences in er is, here is the output for the example that was used in the question:
er =
's2 is missing field a'
's1(1).b(1).c(1).e and s2(1).b(1).c(1).e do not match'
Note that it will not show values, from a technical point of view it is probably not too hard to change the m file if value difference displays are desirable. However, especially if there are some big matrices I suppose this could result in problematic output.
I have a folder (new_images) which contains 52 sub-folders (person1 to person52) and each sub-folders contain 50 images which are not sequential (like: person1 1, person1 3, person1 10). I want to these read images from each sub-folders and do a processing, How can I do this?
I would really appreciate your answers
You can load images using the Matlab imread function.
I'm thinking of a quick way to do that:
new_images_rep = pwd;
for i=1:52
eval(srcat('pics = dir(new_images_rep','/person',num2str(i),')');
for k=1:50
a(i,k) = imread(pics(k+2).name,'fmt'); %there is k+2 because the dir function also stores repositories like '.' or '..'.
end
end
You have to be careful with the dir function. You should test it in your new_images repository in order to see what are the files/repositories stored in pics.
eval is a Matlab function that allows you to execute any Matlab expression. The string created by strcatis here (if i=3):
'pics = dir(new_images_rep/person3)'
Be sure that the new_imagesrepository is in your Matlab path, or replace new_images_rep = pwd; by new_images_rep = 'the_actual_full_path'.
'fmt' is the actual format of the images you want to store (like .tif or .jpg or anything else).
Using the code I gave you (and a few modifications), all the files from the first subfolder will be stored in a(1,:). Do not hesitate to read Matlab help on every function used here.
I'm currently attempting to load a number of MATLAB files which all contain the same variable in order to make a matrix of all the values.
These files all start with a number (i.e. 40_analysed.mat), which was previously extracted from different raw data files using regular expressions, meaning I have a vector composed of all the individual numbers (id).
When I try to load the values and display the data for all individuals in a single matrix using the code below, the files aren't loaded alphanumerically (i.e. according to id), instead appearing to be loaded randomly.
file = dir('*_analysed.mat');
for i=1:length(id);
load(file(i).name,'means');
overallThresholds{i} = means;
end
overallMeans = cell2mat(overallThresholds)
How could I do this so the resulting matrix would be in the correct order? Apologies if this question doesn't make much sense, the problem is a little hard to articulate!
If your filenames don't have a fixed-precision number (as #FakeDIY points out, that would mean they would already be sorted), you could do something like this:
file = dir('*_analysed.mat');
overalThresholds = cell(1, length(id));
IDs = zeros(1, length(id));
for i = 1:length(id)
fileName = file(i).name;
IDs(i) = str2double( strrep( fileName, '_analysed.mat', '' ) );
data = load(fileName, 'means');
overallThresholds{i} = data.means;
end
[~, reordering] = sort(IDs);
overallThresholds = overallThresholds(reordering);
In other words, store the file ID in a separate array as you're going along, and then reorder overallThresholds to be in sorted order of IDs using the second output of SORT.
(I also pre-allocated the arrays, and use the functional form of LOAD, but you don't really need to do that).
When one uses dir command, it is not promised that the results will be in alphabetical order. In fact, the manual says explicitly that:
dir lists the files and folders in the MATLAB current folder. Results
appear in the order returned by the operating system.
Even if you did get this in alphabetical order, nothing assures you that you will get it next time. Thus, you must order the results from dir using sort command.
[~,order] = sort( {file.name} );
file = file(order);