While deploying packages of Matlab code using Matlab 2015 I encountered the problem of gathering all required files from my repository to run a certain file or set of files. Matlab has a method for simplifying this process, matlab.codetools.requiredFilesAndProducts.
However, sometimes some files are missing in the repository (either because I got a package from someone else who was not that careful or because it was not checked in in the repository).
When running the code one would get of course an error:
Undefined function or variable 'XXX'.
However, this may take long to fix (running takes long, you would have to repeat for every missing file). Therefore I thought to use the function above. Unfortunately, it only lists existing files in the output (I tested this). Functions that are called from your code, but that are not present in the current path are omitted by matlab.codetools.requiredFilesAndProducts.
My problem: I would like to get a list of all files that are required by running a certain file but are not present in the current path so that I can find them and add them to my collection.
I know that this must be an iterative process because the missing files could themselves call other missing files and I know that there would be false positives, some of these items could be unknown variables instead, and I know that the missing files would only have a name, no path (of course).
My question: What is the easiest way to find a list of potentially missing files of my code in one go?
Please note that function depfun has been removed in recent versions of Matlab.
Related
I would like to use two versions of the same Matlab software package at the same time (I would like to compare their outputs for testing). The package modifies the path so that functions in subdirectories can be found. This seems problematic since the package assumes that it is the only copy running on the machine. The path is essentially a global variable which is unintentionally shared between the two copies of the code.
Example simplified code structure:
/main_code.m
/compare_results.m
/code_a/somefn.m
/code_a/submethods/
/code_b/somefn.m
/code_b/submethods/
Note that somefn.m adds the submethods directory to the path, and calls code from the submethods folder by relying on the path.
Example of code that I would like to run:
for i = 1:1000000
% Run version A:
result_a = code_a.somefn(i);
% Run version B:
result_b = code_b.somefn(i);
% Compare the output from the two versions:
compare_results(a,b);
end
One solution that I can think of is to manually update the Matlab path every time that I want to switch to a different version of the package. This seems like unnecessary coding overhead, and potentially a performance problem (due to switching the path so often).
Another solution might be to rewrite the code to be object oriented, so that the functions are attached to objects, and I can create objects of different versions. The problem with this is that in reality the code package contains hundreds of files, and I was not the original author so rewriting would be a huge task .
(Yet another option would be to change directory all the time, so that the code to run is always in the current directory. This would be so much of a headache due to the number of subfolders that I do not think this is a serious solution. It also has potential performance overhead drawbacks similar to always changing the path.)
Is there a cleaner way to handle this? Can I somehow specify the folder of the code that I want to run? What is the best way to design such a code package so that this problem does not come up?
Just create packages, which can contain functions with the same names: Packages Create Namespaces
Basically create two folders named, say, +package1 and +package2. The "+" in the folder name is important. Then place your functions under both of them, say, foo.m. Then, you can call each separately without messing with MATLAB path as:
>> package1.foo
>> package2.foo
You can use private functions. Change the directory structure as:
/main_code.m
/compare_results.m
/+code_a/somefn.m
/+code_a/private/
/+code_b/somefn.m
/+code_b/private/
Each somefn has access to the functions contained in its private sub-folder. So there is no need to create global variable and to add private sub-folders to the path.
I am trying to run Kevin Murphy's Bayes Net Toolbox in Octave and encountering some problems. It doesn't help that I'm a novice at Bayesian networks, Matlab and Octave.
This toolbox was originally written for Matlab. There is a large test file called test_BNT.m which runs through all the functionality in the toolbox. Most of the error messages relate to the difference between & and && in Matlab and Octave. This is easy to fix. However, I've now come across a new problem and I don't know what to do about it.
For instance, the qmr1.m script creates an instance of the pearl_inf_engine class, sets some of the member member variables and passes the instance of the class to another function. Later on, the member variables are accessed again in a different script (parallel_protocol.m). But when this happens, the following message appears:
error: invalid index for class
error: evaluating argument list element number 1
It seems that from one script to another, it has forgotten that the class has any member variables and gives the invalid index message when you try to access them.
Is this a common error with an easy solution? Is something wrong with the path or working directory? Maybe someone else has already converted the BNT to octave and knows what to do?
Edit
I was able to get past this error message. The trick was to read the installation instructions (haha) and run addpath(genpathKPM(<BNT base directory)). genpathKPM.m is a script includes in BNT which adds all the required directories to the path.
After doing this, run test_BNT.m and change & to && and | to || at each line where it gives a warning. This will clear up most of the errors.
However, I'm still unable to run mpe1.m, mp2.m, mildew1.m and some others. The new error message I'm stuck on is:
error: invalid empty index list
error: called from:
error: C:\FullBNT-1.0.7\bnt\BNT\inference\static\#var_elim_inf_engine\find_mpe
.m at line 63, column 5
on this line of code:
eval(['sCPT.T(', sargs, num2str(jj), ')=0;']);
If I can get all the scripts to work, I'll post an answer here with the steps I took to do it.
Edit 2
I was able to get past the problem in the previous edit. Replace
eval(['sCPT.T(', sargs, num2str(jj), ')=0;']);
with
eval(['sCPT.T(', sargs, sprintf('%d',jj), ')=0;']);
The next problem is identical. Just replace num2str in the same way.
This file was apparently contributed by a user of BNT, and not written by the original author. Using eval kind of a hack, I think. A better fix would be to just rewrite the code so it doesn't use eval at all.
There is one more error in draw_graph.m, which was apparently also an outside contribution to the project. I just commented out the call to that function since I'm not interested in drawing graphs right now. After doing this, and continuing to fix shortcircuit operators, all of the tests in test_BNT.m will run.
Still, I won't create an answer for this until I can get draw_graph.m to run, too.
As a significant amount of time has passed, and the answer to the core problem was provided in the question, I will post it here so it will not stay listed as unanswered:
tl;dr: Change a few operators, solve the remaining bugs specified below, and everything works except the drawing of graphs.
Edit
I was able to get past this error message. The trick was to read the
installation instructions (haha) and run addpath(genpathKPM(<BNT base
directory)). genpathKPM.m is a script includes in BNT which adds
all the required directories to the path.
After doing this, run test_BNT.m and change & to && and | to
|| at each line where it gives a warning. This will clear up most of
the errors.
However, I'm still unable to run mpe1.m, mp2.m, mildew1.m and
some others. The new error message I'm stuck on is:
error: invalid empty index list
error: called from:
error: C:\FullBNT-1.0.7\bnt\BNT\inference\static\#var_elim_inf_engine\find_mpe
.m at line 63, column 5
on this line of code:
eval(['sCPT.T(', sargs, num2str(jj), ')=0;']);
If I can get all the scripts to work, I'll post an answer here with
the steps I took to do it.
Edit 2
I was able to get past the problem in the previous edit. Replace
eval(['sCPT.T(', sargs, num2str(jj), ')=0;']); with
eval(['sCPT.T(', sargs, sprintf('%d',jj), ')=0;']);
The next problem is identical. Just replace num2str in the same way.
This file was apparently contributed by a user of BNT, and not written
by the original author. Using eval kind of a hack, I think. A better
fix would be to just rewrite the code so it doesn't use eval at all.
There is one more error in draw_graph.m, which was apparently also
an outside contribution to the project. I just commented out the call
to that function since I'm not interested in drawing graphs right now.
After doing this, and continuing to fix shortcircuit operators, all of
the tests in test_BNT.m will run.
i'm looking for a way to import an existing xpo-export via command-line into ax2009 aot and afterwards compile just this imported xpo. google tells me how to compile the whole aot by commandline, which takes quite long.
so is there a way to import an xpo ( shared project ) and compile just these objects?
what possibilities are available, if the objects which should be imported are version-controlled by ax and are checked-in?
hoping for an easy way to automate optionally check-out, import, avoid overwrite?-questions, compile and run ;)
thanks in advance!
You can make you own startup command:
Make a new class and extend SysStartupCmd
Change the construct method of SysStartupCmd to call you class.
Do whatever you need, this includes parsing the parm variable.
Also you will have to deal with version control by calling checkin/checkout in your code, handling compile errors etc.
There are no easy way, this is complicated stuff.
Over the last two years I have introduced and refined a command line process for deploying XPOs to AX 4.0 with great success. The class SysAutoRun is key as mentioned above. The following is a brief explanation of the resulting process:
Developers export AX objects from the AOT to a corresponding folder(layer) i.e. CUS, VAR, etc... for the most part the file name is the default file name set by AX.
Developers commit using SVN in this scenario. This would have to be evaluted to meet your needs.
Console application for the build process reads all file names from each directory(layer) and creates corresponding AX project definition files.
Console application reads all file names from each directory (again) and creates an import definition file for each corresponding layer(folder). The project definition created above is also instructed to be imported after all other objects are loaded and finally compiled. The import definition contains some specialized elements that are recognized by the SysAutoRun.execCommand(XmlNode _command) method.
A call is made to ax32.exe "config.axc" -StartupCmd=AUTORUN_ImportDefinitionMentionedAbove.xml -lazyclassloading -lazytableloading -nocompileonimport -internal=noModalBoxes
AX parses this import definition file invoking customizations as instructed. Logging is added to the process for outputting compilation results to an XML log file. Finally step 3's project definition file is compiled.
Console application validates the outputted XML log and handles appropriately.
Step 5-7 is repeated for each (folder)layer.
I understand this is very vague. The intent of this post is to get feedback on interest before I invest more time on describing the process. The import definition file is probably of most interest as it is responsible for loading the objects in the right order, synchronizing the ORM, compiling, repeating, etc...
Thanks M#
We have a lot of MATLAB code in my lab. The problem is there's really no way to organize it. Since all the functions have to be in the same folder to be called (or you have to add a bunch of folders to MATLAB's path environment variable), it seems that we're doomed have loads of files in the same folder, all in the global namespace. Is there a better way to organize our files and functions? I really wish there were some sort of module system...
MATLAB has a notion of packages which can be nested and include both classes and functions.
Just make a directory somewhere on your path with a + as the first character, like +mypkg. Then, if there is a class or function in that directory, it may be referred to as mypkg.mything. You can also import from a package using import mypkg.mysubpkg.*.
The one main gotcha about moving a bunch of functions into a package is that functions and classes do not automatically import the package they live in. This means that if you have a bunch of functions in different m-files that call each other, you may have to spend a while dropping imports in or qualifying function calls. Don't forget to put imports into subfunctions that call out as well. More info:
http://www.mathworks.com/help/matlab/matlab_oop/scoping-classes-with-packages.html
I don't see the problem with having to add some folder to Matlab's search path. I have modified startup.m so that it recursively looks for directories in my Matlab startup directory, and adds them to the path (it also runs svn update on everything). This way, if I change the directory structure, Matlab is still going to see all the functions the next time I start it.
Otherwise, you can look into object-oriented code, where you store all the methods in a #objectName folder. However, this may lead to a lot of re-writing code that can be avoided by updating the path (there is even a button add with subfolders if you add the folder to the path from the File menu) and doing a bit of moving code.
EDIT
If you would like to organize your code so that some functions are only visible to the functions that call them directly (and if you don't want to re-write in OOP), you put the calling functions in a directory, and within this directory, you create a subdirectory called private. The functions in there will only be visible to the functions in the parent directory. This is very useful if you have to overload some built-in Matlab functions for a subset of your code.
Another way to organize & reuse code is using matlab's object-oriented features. Each Object is customarily in a folder that begins with an "#" and has the file(s) for that class inside. (though the newer syntax does not require this for a class defined in a single file.) Using private folders inside class folders, matlab even supports private class members. Matlab's new class notation is relatively fully-featured, but even the old syntax is useful.
BTW, my startup.m that examines a well-known location that I do my SVN checkouts into, and adds all of the subfolders onto my path automatically.
The package system is probably the best. I use the class system (#ClassName folder), but I actually write objects. If you're not doing that, it's silly just to write a bunch of static methods. One thing that can be helpful is to put all your matlab code into a folder that isn't on the matlab path. Then you can selectively add just the code you need to the path.
So say you have two projects, stored in "c:\matlabcode\foo" and "c"\matlabcode\bar", that both use common code stored in "c:\matlabcode\common," you might have a function "setupPaths.m" like this:
function setupPaths(projectName)
basedir = fullfile('c:', 'matlabcode');
addpath(genpath(fullfile(basedir, projectName)));
switch (projectName)
case {'foo', 'bar'}
addpath(genpath(fullfile(basedir, 'common')));
end
Of course you could extend this. An obvious extension would be to include a text file in each directory saying what other directories should be added to the path to use the functions in that directory.
Another useful thing if you share code is to set up a "user specific/LabMember" directory structure, where you have different lab members save code they are working on. That way you have access to their code if you need it, but don't get clobbered when they write a function with the same name as one of yours.
I have a workspace built using MS-Visual Studio 2005 with all C code.In that i see many functions which are not called but they are still compiled(they are not under any compile time macro to disable them from compiling).
I set following optimization settings for the MS-VS2005 project to remove that unused code:-
Optimization level - /Ox
Enable whole program optimization - /GL
I tried both Favor speed /Ot and Favor Size /Os
Inspite of all these options, when i see the linker generated map file, I see the symbols(unsed functions) names present in the map file.
Am I missing something? I want to completely remove the unused code.
How do I do this?
The compiler compiles C files one-at-a-time. Therefore, while compiling a C-file that does contains an unused function, the compiler cannot be sure that it will not be called from another file and hence it will compile that function too. However, if that function were declared as static (file-scope), then the compiler would know it is not used and hence remove it.
Even with whole program optimization, I think it would still not be done since the compilation could be for a library.
Linkers do something similar to what you are looking for. If your code links against a library containing multiple objects, then any objects that do not contain functions used by your code (directly or indirectly) would not be included in the final executable.
One option would be to separate your code into individual libraries and object files.
PS - This is just my guess. The behavior of the compiler (with whole program optimization) or linker essentially depends on the design choices of that particular compiler or linker
On our projects we have a flag set under the project properties\Linker\Refrences. We set it to Eliminate Unreferenced Data (/OPT:REF), according to the description this is supposed to remove function calls or data that are never used. I am just going by the description, I have never tested this or worked with it. But I just happened to see it within the last hour and figured it might be something you could try.