I am currently using LibFuzzer (or more particular, cargo fuzz, but it's just a wrapper around LibFuzzer) in order to fuzz a function that only accepts input of even length. So you can imagine the code being something like this:
function toFuzz(String input){
if((input.length() % 2)==1){
return 0; // Not equal length
}
// logic goes here
}
I do not want to fuzz this particular check - is there any way I can tell libFuzzer to only generate inputs of even length? Or is filtering out even inputs in the test harness the best way to go?
You can filter out inputs in the test harness or just call the function and let it do the early return. Either way LibFuzzer will notice that it isn't an "interesting" case and not spend much time on it.
Related
I would like to test datasets with a variable number of values. Each value should be tested and I would like to have a standardized output that I can read in afterward again. My used framework is Matlab.
Example:
The use case would be a dataset which includes, e.g., 14 values that need to be testet. The comparison is already completely handled by my implementation. So I have 14 values, which I would like to compare against some tolerance or similar and get an output like
1..14
ok value1
ok value2
not ok value3
...
ok value14
Current solution:
I try to use the unit-testing framework and the according TAPPlugin that would produce exactly such an output (tap), one for every unit-test. My main problem is that the unit-testing framework does not take any input parameters. I already read about parametrization, but I do not know how this helps me. I could put the values as a list into the parameter, but how to I pass them there? Afaik the unit-test class does not allow additional parameters during initialization, so I cannot include this in the program the way I want.
I would like to avoid to need to format the TAP output on my own, because it is already there, but only for unit-test objects. Unfortunately, I cannot see how to implement this wisely.
How can I implement the output of a test anything protocol where I have a variable amount of comparisons (values) in Matlab?
If you are using class based unit tests you could access its properties from outside the test.
So let's say you have following unit test:
classdef MyTestCase < matlab.unittest.TestCase
properties
property1 = false;
end
methods(Test)
function test1(testCase)
verifyTrue(testCase,testCase.property1)
end
end
You could access and change properties from outside:
test=MyTestCase;
MyTestCase.property1 = true;
MyTestCase.run;
This should no succeed since you changed from false to true. If you want to have a more flexible way, you could have a list with the variables and a list with the requirements and then cycle through both in one of the test functions.
properties
variables = [];
requirements = [];
end
methods(Test)
function test1(testCase)
for i = 1:length(variables):
verifyEqual(testCase,testCase.variables[i],testCase.requirements [i])
end
end
end
Now you would set variables and requirements:
test=MyTestCase;
MyTestCase.variables = [1,2,3,4,5,6];
MyTestCase.requirements = [1,3,4,5,5,6];
MyTestCase.run;
Please note, that in theory you should not have multiple assert statements in one test.
I am new to Specman and trying to learn it by reading existing code.
I came across the following function and can't find an explanation in specman documentation...
VerifyNode(end_point:string, derived_value:uint) is also {
if ('(end_point)' === ~derived_value) {
message("Error1");
}
else if ('(end_point)' === derived_value) {
message("Error2");
}
else {
message("Error3");
}
}
I assume logically the '(end_point)' is getting actual value of end_point signal in run-time. Is that true?
Error1 will be the message if value of end_point at run time is negate the derived_value unsigned integer.
Error2 will be the message if value of end_point at run time is the same as the derived_value unsigned integer.
How can i explain the "Error3" condition?
Indeed this is the very old school of signal access based on string. I recommend you search for "simple_port".
The end_point is a string, that should be the full HDL path of the signal you want to read. Example of calling this method:
// check if signal has the value 3:
VerifyNode("top.module_a.sig_val", 3);
The official name of '' is called the tick notation in Specman documentation. Search for the term "tick notation" in cdnshelp and you will find it.
'(end_point)' is two things. It is old-school, non-port access to the device under test (DUT) and the (end_point) part of that snippet is the way to do string substitution directly in the '' access to the DUT. Using the old style of access to the DUT can drastically slow down simulation speed, if your design is big enough for that to be a concern for you. Also, this methodology lacks more static compile time checks as those strings can be anything and you'll only know the path is wrong when you simulate it and the DUT path cannot be found, creating an error.
Im working with scalatest and scalacheck, alsso working with FeatureSpec.
I have a generator class that generate object for me that looks something like this:
object InvoiceGen {
def myObj = for {
country <- Gen.oneOf(Seq("France", "Germany", "United Kingdom", "Austria"))
type <- Gen.oneOf(Seq("Communication", "Restaurants", "Parking"))
amount <- Gen.choose(100, 4999)
number <- Gen.choose(1, 10000)
valid <- Arbitrary.arbitrary[Boolean]
} yield SomeObject(country, type, "1/1/2014", amount,number.toString, 35, "something", documentTypeValid, valid, "")
Now, I have the testing class which works with FeatureSpec and everything that I need to run the tests.
In this class I have scenarios, and in each scenario I want to generate a different object.
The thing is from what I understand is that to generate object is better to use forAll func, but for all will not sure to bring you an object so you can add minSuccessful(1) to make sure you get at list 1 obj....
I did it like this and it works:
scenario("some scenario") {
forAll(MyGen.myObj, minSuccessful(1)) { someObject =>
Given("A connection to the system")
loginActions shouldBe 'Connected
When("something")
//blabla
Then("something should happened")
//blabla
}
}
but im not sure exactly what it means.
What I want is to generate an invoice each scenario and do some actions on it...
im not sure why i care if the generation work or didnt work...i just want a generated object to work with.
TL;DR: To get one object, and only one, use myObj.sample.get. Unless your generator is doing something fancy that's perfectly safe and won't blow up.
I presume that your intention is to run some kind of integration/acceptance test with some randomly generated domain object—in other words (ab-)use scalacheck as a simple data generator—and you hope that minSuccessful(1) would ensure that the test only runs once.
Be aware that this is not the case!. scalacheck will run your test multiple times if it fails, to try and shrink the input data to a minimal counterexample.
If you'd like to ensure that your test runs only once you must use sample.
However, if running the test multiple times is fine, prefer minSuccessful(1) to "succeed fast" but still profit from minimized counterexamples in case the test fails.
Gen.sample returns an option because generators can fail:
ScalaCheck generators can fail, for instance if you're adding a filter (listingGen.suchThat(...)), and that failure is modeled with the Option type.
But:
[…] if you're sure that your generator never will fail, you can simply call Option.get like you do in your example above. Or you can use Option.getOrElse to replace None with a default value.
Generally if your generator is simple, i.e. does not use generators that could fail and does not use any filters on its own, it's perfectly safe to just call .get on the option returned by .sample. I've been doing that in the past and never had problems with it. If your generators frequently return None from .sample they'd likely make scalacheck fail to successfully generate values as well.
If all that you want is a single object use Gen.sample.get.
minSuccessful has a very different meaning: It's the minimal number of successful tests that scalacheck runs—which by no means implies
that scalacheck takes only a single value out of the generator, or
that the test runs only once.
With minSuccessful(1) scalacheck wants one successful test. It'll take samples out of the generator until the test runs at least once—i.e. if you filter the generated values with whenever in your test body scalacheck will take samples as long as whenever discards them.
If the test passes scalacheck is happy and won't run the test a second time.
However if the test fails scalacheck will try and produce a minimal example to fail the test. It'll shrink the input data and run the test as long as it fails and then provides you with the minimized counter example rather than the actual input that triggered the initial failure.
That's an important property of property testing as it helps you to discover bugs: The original data is frequently too large to lend itself for debugging. Minimizing it helps you discover the piece of input data that actually triggers the failure, i.e. corner cases like empty strings that you didn't think of.
I think the way you want to use Scalacheck (generate only one object and execute the test for it) defeats the purpose of property-based testing. Let me explain a bit in detail:
In classical unit-testing, you would generate your system under test, be it an object or a system of dependent objects, with some fixed data. This could e.g. be strings like "foo" and "bar" or, if you needed a name, you would use something like "John Doe". For integers and other data, you can also randomly choose some values.
The main advantage is that these are "plain" values—you can directly see them in the code and correlate them with the output of a failed test. The big disadvantage is that the tests will only ever run with the values you specified, which in turn means that your code is also only tested with these values.
In contrast, property-based testing allows you to just describe how the data should look like (e.g. "a positive integer", "a string of maximum 20 characters"). The testing framework will then—with the help of generators—generate a number of matching objects and execute the test for all of them. This way, you can be more sure that your code will actually be correct for different inputs, which after all is the purpose of testing: to check if your code does what it should for the possible inputs.
I never really worked with Scalacheck, but a colleague explained it to me that it also tries to cover edge-cases, e.g. putting in a 0 and MAX_INT for a positive integer, or an empty string for the aforementioned string with max. 20 characters.
So, to sum it up: Running a property-based test only once for one generic object is the wrong thing to do. Instead, once you have the generator infrastructure in place, embrace the advantage you then have and let your code be checked a lot more times!
I'm trying to check the input of a Matlab function to see whether the user has forgotten about it or not (which is easy to do in this case).
If the user has not supplied number_obs then I want to pause the program and wait for the user to input this information.
Some other StackOverflow posts seem to suggest using ~exist however this doesn't seem to work. Can anybody suggest what I'm doing wrong here?
function output=test(number_obs)
if ~exist('number_obs'),
number_obs=input('How many observations do you have in your experiments?')
end
The Python equivalent would be something like:
def test(number_obs):
if nummber_obs != None:
output=raw_input('How many observations do you have in your experiments? :')
return output
You can do this with nargin
function output=test(number_obs)
if nargin<1
number_obs=input('How many observations do you have in your experiments?')
end
(Edited to correct the truth) It may not matter here, but to be on the safe side, you should always specify the type of object you're checking. In your case, its a 'var', so
if ~exist('number_obs','var'),
Thanks to dasdingonesin for pointing this out.
Is there an idiomatic way in Matlab to bind the value of an expression to the nth return value of another expression?
For example, say I want an array of indices corresponding to the maximum value of a number of vectors stored in a cell array. I can do that by
function I = max_index(varargin)
[~,I]=max(varargin{:});
cellfun(#max_index, my_data);
But this requires one to define a function (max_index) specific for each case one wants to select a particular return value in an expression. I can of course define a generic function that does what I want:
function y = nth_return(n,fun,varargin)
[vals{1:n}] = fun(varargin{:});
y = vals{n};
And call it like:
cellfun(#(x) nth_return(2,#max,x), my_data)
Adding such functions, however, makes code snippets less portable and harder to understand. Is there an idiomatic to achieve the same result without having to rely on the custom nth_return function?
This is as far as I know not possible in another way as with the solutions you mention. So just use the syntax:
[~,I]=max(var);
Or indeed create an extra function. But I would also suggest against this. Just write the extra line of code, in case you want to use the output in another function. I found two earlier questions on stackoverflow, which adress the same topic, and seem to confirm that this is not possible.
Skipping outputs with anonymous function in MATLAB
How to elegantly ignore some return values of a MATLAB function?
The reason why the ~ operator was added to MATLAB some versions ago was to prevent you from saving variables you do not need. If there would be a syntax like the one you are searching for, this would not have been necessary.