I am writing a class-based test suite in MATLAB for a timeseries handling package. The first test in my suite needs to check whether a connection exists to a Haver database on a network drive. If the connection does not exist, then the first test should abort the rest of the suite using one of the fatalAssert methods.
One complicating factor, which I have excluded from the exposition below, but I will mention now is that I need to use an anonymous function to check the connection to Haver (unless someone has a better idea). My package handles data from multiple sources, Haver only being one of them.
I have a parent-class test suite that performs general tests for all of the sources. I then inherit this parent-class into specific child-class test suites and set specific parameters in their respective TestMethodSetup method. One of these parameters is an anonymous function, connfun, and a location, connloc, which I use in the parent-class to test the connection. The reason I do this is because the parent tests are executed first, so I would have to wait for all of those to end if I wanted to test the connection in the child class.
This also complicates the order of execution. If I want to assign the connfun in the child class, then I have to use either the TestMethodSetup or TestClassSetup of the child class (open to recommendations on which is best here) and put this connection test in the Test method of the parent class. I noticed the if I put checkConn in the TestMethodSetup and TestClassSetup of the parent class was running before that of the child class, I was unable to pass the anonymous function and the test would be incomplete.
Putting the previous point aside for a moment, this was my first attempt at writing the test in the parent-class (note that I used a fatalAssertEqual instead of a fatalAssertTrue because isconnection() does not return a logical):
methods (Test)
function checkConn(testCase)
connloc = 'pathToHaverDatabase';
connfun = #(x) isconnection(haver(x));
testCase.fatalAssertEqual(connfun(connloc), 1);
end
end
The above works when there is a connection, but the problem that I bumped into with this is that when I cannot access connloc, an error ocurrs during the call to haver(). So instead of returning a 1 or 0 from the isconnection() call that I can fatalAssertEqual on, all of checkConn errors out due to haver(). This then leads to the rest of the tests running (and failing, which is exactly what I want to avoid).
My next idea works for both cases, but it feels like bad code, and does not have the anonymous function specification described above.
methods (Test)
function checkConn(testCase)
connloc = 'pathToHaverDatabase';
connfun = #(x) isconnection(haver(x));
try
isconn = connfun(connloc);
catch
isconn = 0;
end
testCase.fatalAssertEqual(isconn, 1)
end
end
When I wrote this, I did not necessarily want to distinguish between not having access to the network drive, not being able to call the haver() function, and getting an isconnection equal to 0 because the last case covers all three. But I realized that if I did differentiate them, then it would be a bit more robust, but it's still missing the anonymous function taht I could pass from child to parent.
properties
connloc = 'pathToHaverDatabase';
end
methods (Test)
function checkDrive(testCase)
isfound = fillattrib(testCase.connloc);
testCase.fatalAssertTrue(isfound);
end
function checkHaver(testCase)
try
hav = haver(testCase.connloc);
ishaver = ~isempty(hav);
catch
ishaver = false;
end
testCase.fatalAssertTrue(ishaver);
end
function checkConn(testCase)
connfun = #(x) isconnection(haver(x));
testCase.fatalAssertEqual(connfun(testCase.connloc), 1);
end
end
Ideally, what I would want is a fatalAssert method (or something similar) that ends the test suite when its input is an error. Something that would perhaps be called fatalAssertNotError, but I don't think that exists. If it did, the last line of my first function would simply be testCase.fatalAssertNotError(connfun(connloc)) and I would not have to worry about all the cases.
I'm very open to dynamic rewrite of this whole test setup, so any specific comments or general advice are welcome!
First of all, I think the fatalAssert case is a strong use case to provide something like fatalAssertNotError. One reason why it is not part of the package is because many/most times people don't want to check whether something doesn't error, they just want to call the code, and if it errors, it fails for the test author automatically and it is much simpler. However, other qualification types like fatal assertions and assumptions perhaps point to the need to provide this so you can choose the outcome of the test in the presence of an error, in cases where you don't want it to fail (like with assumptions) or you want it to fail "more strongly" like with fatal assertions.
That being said, I am still not convinced that you can't achieve what you are ultimately trying to do without it. The question I have centers around why you can't use TestClassSetup. It is not clear to me exactly why you weren't able to customize in the derived test class to get the behavior you want. For example, does something like this work?
classdef BaseTest < matlab.unittest.TestCase
properties(Abstract)
connloc
connfun
end
methods(TestClassSetup)
function validateConnection(testCase)
% If this errors it behaves like an assertion (not fatal assertion)
% and fails all tests in the test class. If it doesn't error but
% doesn't return 1 then the assertion failure will occur.
testCase.assertEqual(connfun(connloc), 1,
'Could not establish a connection to the database');
end
end
end
classdef DerivedTest < BaseTest
properties
connloc = 'pathToHaverDatabase';
connfun = #(x) isconnection(haver(x));
end
methods(Test)
function testSomething(testCase)
% Have at least one test method to test it out
end
end
end
Hope that helps!
If you really want to use a function you can define a nested one like this:
methods (Test)
function checkConn(testCase)
connloc = 'pathToHaverDatabase';
function res = connfun(x)
try
res = isconnection(haver(x));
catch
res = false
end
end
testCase.fatalAssertEqual(connfun(connloc), 1);
end
end
Nested functions can be a bit confusing to me because of the way they share data with the parent function. There really is no difference between an anonymous function and a nested function.
The alternative is to put the function at the end of the file, outside the classdef block:
classdef ...
%...
methods (Test)
function checkConn(testCase)
connloc = 'pathToHaverDatabase';
function res = connfun(x)
try
res = isconnection(haver(x));
catch
res = false
end
end
testCase.fatalAssertEqual(connfun(connloc), 1);
end
end
%...
end
function res = connfun(x)
try
res = isconnection(haver(x));
catch
res = false
end
end
But I honestly don't understand why you need to have a function call within fatalAssertEqual. The code you have seems perfectly fine to me.
Related
I would like to test datasets with a variable number of values. Each value should be tested and I would like to have a standardized output that I can read in afterward again. My used framework is Matlab.
Example:
The use case would be a dataset which includes, e.g., 14 values that need to be testet. The comparison is already completely handled by my implementation. So I have 14 values, which I would like to compare against some tolerance or similar and get an output like
1..14
ok value1
ok value2
not ok value3
...
ok value14
Current solution:
I try to use the unit-testing framework and the according TAPPlugin that would produce exactly such an output (tap), one for every unit-test. My main problem is that the unit-testing framework does not take any input parameters. I already read about parametrization, but I do not know how this helps me. I could put the values as a list into the parameter, but how to I pass them there? Afaik the unit-test class does not allow additional parameters during initialization, so I cannot include this in the program the way I want.
I would like to avoid to need to format the TAP output on my own, because it is already there, but only for unit-test objects. Unfortunately, I cannot see how to implement this wisely.
How can I implement the output of a test anything protocol where I have a variable amount of comparisons (values) in Matlab?
If you are using class based unit tests you could access its properties from outside the test.
So let's say you have following unit test:
classdef MyTestCase < matlab.unittest.TestCase
properties
property1 = false;
end
methods(Test)
function test1(testCase)
verifyTrue(testCase,testCase.property1)
end
end
You could access and change properties from outside:
test=MyTestCase;
MyTestCase.property1 = true;
MyTestCase.run;
This should no succeed since you changed from false to true. If you want to have a more flexible way, you could have a list with the variables and a list with the requirements and then cycle through both in one of the test functions.
properties
variables = [];
requirements = [];
end
methods(Test)
function test1(testCase)
for i = 1:length(variables):
verifyEqual(testCase,testCase.variables[i],testCase.requirements [i])
end
end
end
Now you would set variables and requirements:
test=MyTestCase;
MyTestCase.variables = [1,2,3,4,5,6];
MyTestCase.requirements = [1,3,4,5,5,6];
MyTestCase.run;
Please note, that in theory you should not have multiple assert statements in one test.
Suppose we have
Check.m:
classdef Check < handle
methods (Static)
function doStuff()
if isCalledFromAssertSubclass
% do this stuff only if called as Assert.doStuff(), not if called as Check.doStuff()
end
% do other stuff
end
end
end
and Assert.m:
classdef Assert < Check
% nop
end
As written in the comment, I would like to have that Check.doStuff() executes only the "do other stuff" line and that Assert.doStuff() executes the if isCalledFromAssertSubclass block as well.
I want to use static methods, so that I don't neet to create an Assert object whenever I need an assert. Using a global assert object is also very ugly and needs a global assert line in each function I want to use an assert. Likewise for checks.
So there are two problems:
Since these are static classes, there is no chance of using class(obj) or any other non-static class property or function.
dbstack is not inheritance aware and always returns Check as calling class, also for Assert.doStuff.
I did find a working solution, which uses a combination of dbstack and dbtype to read the line where the call came from, i.e. the line where it says Assert.doStuff(). However it involves two debug functions which probably should not be used in productive code and more importent, dbtype is very slow (in my case, 30 our of 70 seconds!).
I could use a package instead (directory +Check with function files in there) and create a symlink +Assert -> +Check. Then I could check the file name, but that's a) not portable, b) quite ugly and c) also somewhat slow (I suppose).
Is there any faster method for this?
Why not overload the static method for Assert, and have it call the parent's method when it's done? This is the normal way of using inheritance: you don't want the parent, Check, to know anything about its child, Assert.
This is what it would look like:
classdef Assert < Check
methods (Static)
function doStuff()
% do some stuff
Check.doStuff()
end
end
end
As #Wolfie suggests in a comment, the above works as long as Check.doStuff is not sealed. A sealed method cannot be overloaded. See the documentation.
I have a code which consists of a single file containing multiple functions, some of which use persistent variables. In order for it to work correctly, persistent variables must be empty.
There are documented ways to clear persistent variables in a multi-function file, such as:
clear functionName % least destructive
clear functions % more destructive
clear all % most destructive
Unfortunately, I cannot guarantee that the user remembers to clear the persistent variables before calling the function, so I'm exploring ways to perform the clearing operation at the beginning of the code. To illustrate the problem, consider the following example:
function clearPersistent(methodId)
if ~nargin, methodId = 0; end
switch methodId
case 0
% do nothing
case 1
clear(mfilename);
case 2
eval(sprintf('clear %s', mfilename));
case 3
clear functions;
case 4
clear all;
end
subfunction();
subfunction();
end
function [] = subfunction()
persistent val
if isempty(val)
disp("val is empty");
val = 123;
else
disp("val is not empty");
end
end
When first running this, we get:
>> clearPersistent
val is empty
val is not empty
I would expect that running the function again at this point, with any of the non-0 inputs would result in the val variable being cleared, but alas - this is not the case. After val is set, unless we use one of the alternatives shown in the top snippet externally, or modify the .m file, it remains set.
My question: Is it possible to clear persistent variable in subfunctions from within the body of the main function, and if yes - how?
In other words, I'm looking for some code that I can put in clearPersistent before calling the subfunctions, such that the output is consistently:
val is empty
val is not empty
P.S.
Here's a related past question (which doesn't deal with this specific use case): List/view/clear persistent variables in Matlab.
I'm aware of the possibility of rewriting the code to not use persistent variables at all (e.g. by passing data around, using appdata, adding a 'clear' flag to all subfunctions, etc.).
Please note that editing the source code of the function and saving implicitly clears it (along with all persistent variables).
I'm aware that the documentation states that "The clear function does not clear persistent variables in local or nested functions.
Additional background on the problem:
The structure of the actual code is as follows:
Main function (called once)
└ Global optimization solver (called once)
└ Objective function (called an unknown N≫1 times)
└ 1st function that uses persistents
└ 2nd function that uses persistents
As mentioned in the comments, there are several reasons why some variables were made persistent:
Loose coupling / SoC: The objective function does not need to be aware of how the subfunctions work.
Encapsulation: It is an implementation detail. The persistent variables do not need to exist outside the scope of the function that uses them (i.e. nobody else ever needs them).
Performance: The persistent variables contain matrices that are fairly expensive to compute, but this operation needs to happen only once per invocation of the main function.
One (side?) effect of using persistent variables is making the entire code stateful (with two states: before and after the expensive computations). The original issue stems from the fact that the state was not being correctly reset between invocations of the main function, causing runs to rely on a state created with previous (and thus invalid) configurations.
It is possible to avoid being stateful by computing the one-time values in the main function (which currently only parses user-supplied configurations, calls the solver, and finally stores/displays outputs), then passing them alongside the user configurations into the objective function, which would then pass them on to the subfunctions. This approach solves the state-clearing problem, but hurts encapsulation and increases coupling, which in turn might hurt maintainability.
Unfortunately, the objective function has no flag that says 'init' etc., so we don't know if it's called for the 1st or the nth time, without keeping track of this ourselves (AKA state).
The ideal solution would have several properties:
Compute expensive quantities once.
Be stateless.
Not pass irrelevant data around (i.e. "need to know basis"; individual function workspaces only contain the data they need).
clear fname and clear functions removes the M-file from memory. The next time you run the function, it is read from disk again, parsed and compiled into bytecode. Thus, you slow down the next execution of the function.
Clearing a function or sub-function from within a function thus does not work. You're running the function, you cannot clear its file from memory.
My solution would be to add an option to subfunction to clear its persistent variable, like so:
function clearPersistent()
subfunction('clear');
subfunction();
subfunction();
end
function [] = subfunction(option)
persistent val
if nargin>0 && ischar(option) && strcmp(option,'clear')
val = [];
return
end
if isempty(val)
disp("val is empty");
val = 123;
else
disp("val is not empty");
end
end
Of course you could initialize your value when called as subfunction('init') instead.
A different solution that might work for your usecase is to separate the computation of val and its use. I would find this easier to read than any of the other solutions, and would be more performant too.
function main()
val = computeval();
subfunction(val);
subfunction(val);
end
Given your edit, you could put the objective function in a separate file (in the private subdirectory). You will be able to clear it.
An alternative to persistent variables would be to create a user class with a constructor that computed the expensive state, and another method to compute the objective function. This could also be a classdef file in the private subdirectory. I think this is nicer because you won’t need to remember to call clear.
In both these cases you don’t have a single file containing all the code any more. I think you need to give up on one of those two ideals: either break data encapsulation or split the code across two files (code encapsulation?).
Why not using global variables?
You can create a global struct that contains your variables and it can be managed using a variable_manager:
function main
variable_manager('init')
subfunction1()
subfunction2()
end
function variable_manager(action)
global globals
switch action
case 'init'
globals = struct('val',[],'foo',[]);
case 'clear'
globals = structfun(#(x)[],globals,'UniformOutput', false);
% case ....
% ...
end
end
function subfunction1
global globals
if isempty(globals.val)
disp("val is empty");
globals.val = 123;
else
disp("val is not empty");
end
end
function subfunction2
global globals
if isempty(globals.foo)
disp("foo is empty");
globals.foo = 321;
else
disp("foo is not empty");
end
end
As mentioned in the question, one of the possibilities is using appdata, which is not too different from global (at least when associating them with "object 0" - which is the MATLAB instance itself). To avoid "collisions" with other scripts/functions/etc. we introduce a random string (if we generate a string in every function that uses this storage technique, it would almost certainly guarantee no collisions). The main downside of this approach is that the string has to be hard-coded in multiple places, or the structure of the code should be changed such that the functions that use this appdata are nested within the function that defines it.
The two ways to write this are:
function clearPersistent()
% Initialization - clear the first time:
clearAppData();
% "Business logic"
subfunction();
subfunction();
% Clear again, just to be sure:
clearAppData();
end % clearPersistent
function [] = subfunction()
APPDATA_NAME = "pZGKmHt6HzkkakvdfLV8"; % Some random string, to avoid "global collisions"
val = getappdata(0, APPDATA_NAME);
if isempty(val)
disp("val is empty");
val = 123;
setappdata(0, APPDATA_NAME, val);
else
disp("val is not empty");
end
end % subfunction
function [] = clearAppData()
APPDATA_NAME = "pZGKmHt6HzkkakvdfLV8";
if isappdata(0, APPDATA_NAME)
rmappdata(0, APPDATA_NAME);
end
end % clearAppData
and:
function clearPersistent()
APPDATA_NAME = "pZGKmHt6HzkkakvdfLV8";
% Initialization - clear the first time:
clearAppData();
% "Business logic"
subfunction();
subfunction();
% Clear again, just to be sure:
clearAppData();
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
function [] = subfunction()
val = getappdata(0, APPDATA_NAME);
if isempty(val)
disp("val is empty");
val = 123;
setappdata(0, APPDATA_NAME, val);
else
disp("val is not empty");
end
end % subfunction
function [] = clearAppData()
if isappdata(0, APPDATA_NAME)
rmappdata(0, APPDATA_NAME);
end
end % clearAppData
end % clearPersistent
Background -- I was reading up on accessing shadowed functions, and started playing with builtin . I wrote a little function:
function klear(x)
% go to parent environment...
evalin('base', builtin('clear','x')) ;
end
This throws the error:
Error using clear
Too many output arguments.
I think this happens because evalin demands an output from whatever it's being fed, but clear is one of the functions which has no return value.
So two questions: am I interpreting this correctly, and if so, is there an alternative function that allows me to execute a function in the parent environment (that doesn't require an output)?
Note: I'm fully aware of the arguments against trying to access shadowed funcs (or rather, to avoid naming functions in a way that overload base funcs, etc). This is primarily a question to help me learn what can and can't be done in MATLAB.
Note 2
My original goal was to write an overload function that would require an input argument, to avoid the malware-ish behavior of clear, which defaults to deleting everything. In Q&D pseudocode,
function clear(x)
if ~exist('x','var') return
execute_in_base_env(builtin(clear(x)))
end
There's a couple issues with your clear override:
It will always clear in the base workspace regardless of where it's called from.
It doesn't support multiple inputs, which is a common use case for clear.
Instead I'd have it check for whether it was called from the base workspace, and special-case that for your check for whether it's clearing everything. If some function is calling plain clear to clear all its variables, that's bad practice, but it's still how that function's logic works, and you don't want to break that. Otherwise it could error, or worse, return incorrect results.
So, something like this:
function clear(varargin)
stk = dbstack;
if numel(stk) == 1 && (nargin == 0 || ismember('all', varargin))
fprintf('clear: balking at clearing all vars in base workspace. Nothing cleared.\n');
return;
end
% Check for quoting problems
for i = 1:numel(varargin)
if any(varargin{i} == '''')
error('You have a quote in one of your args. That''s not valid.');
end
end
% Construct a clear() call that works with evalin()
arg_strs = strcat('''', varargin, '''');
arg_strs = [{'''clear'''} arg_strs];
expr = ['builtin(' strjoin(arg_strs, ', '), ')'];
% Do it
evalin('caller', expr);
end
I hope it goes without saying that this is an atrocious hack that I wouldn't recommend in practice. :)
What happens in your code:
evalin('base', builtin('clear','x'));
is that builtin is evaluated in the current context, and because it is used as an argument to evalin, it is expected to produce an output. It is exactly the same as:
ans = builtin('clear','x');
evalin('base',ans);
The error message you see occurs in the first of those two lines of code, not in the second. It is not because of evalin, which does support calling statements that don't produce an output argument.
evalin requires a string to evaluate. You need to build this string:
str = 'builtin(''clear'',''x'')';
evalin('base',ans);
(In MATLAB, the quote character is escaped by doubling it.)
You function thus would look like this:
function clear(var)
try
evalin('base',['builtin(''clear'',''',var,''')'])
catch
% ignore error
end
end
(Inserting a string into another string this way is rather awkward, one of the many reasons I don't like eval and friends).
It might be better to use evalin('caller',...) in this case, so that when you call the new clear from within a function, it deletes something in the function's workspace, not the base one. I think 'base' should only be used from within a GUI that is expected to control variables in the user's workspace, not from a function that could be called anywhere and is expected (by its name in this case) to do something local.
There are reasons why this might be genuinely useful, but in general you should try to avoid the use of clear just as much as the use of eval and friends. clear slows down program execution. It is much easier (both on the user and on the MATLAB JIT) to assign an empty array to a variable to remove its contents from memory (as suggested by rahnema1 in a comment. Your base workspace would not be cluttered with variables if you used function more: write functions, not scripts!
I have this chunk of code which display a question box with three choices:
Yes
Save as
No
I just want to create a unit test for each of those choices, and I need to interact with the box for it.
When the box is created, every matlab process is frozen until the user react. Of course my tests are automated, and there's no user present.
Is there a solution to send a "Yes" event to Matlab ?
I could use autoit but I would rather avoid that
Thanks
Currently, you're best bet is to structure your code to not depend on the questdlg function directly but to wrap an interface around the function so that you can inject a test specific dependency, or "mock". This might look something like the following:
Source Code Under Test
function codeWhichUsesQuestDlg(dlgProvider)
validateattributes(dlgProvider, {'DialogProvider'},{});
if (needToAskQuestion)
result = dlgProvider.provideQuestionDialog(...)
else
...
end
Source Code Interface/Infrastructure
classdef DialogProvider < handle
methods(Abstract)
result = provideQuestionDialog(varargin);
% perhaps others? errordlg, etc?
end
end
Production Implementation
classdef ProductionProvider < DialogProvider
methods
function result = provideQuestionDialog(varargin)
result = questdlg(varargin{:});
end
end
end
Production Usage
>> codeWhichUsesQuestDlg(ProductionProvider)
Test Usage
classdef TestCode < matlab.mock.TestCase
methods(Test)
function testCode(testCase)
% Create the test specific "mock" implementation
[mockProvider, behavior] = testCase.createMock(?DialogProvider);
% Define mock behavior needed for the test
when(withAnyInputs(behavior.provideQuestionDialog), ...
AssignOutputs('Yes'));
% Call code under test with the mock
codeWhichUsesQuestDlg(mockProvider);
% Verify correct result/behavior
...
end
end
end
Take a look at the mocking framework to see more info. Also you might be interested in this post covering the mocking framework as well as this post on dependency injection.