How to capture Failed Assert exception message or state when using NUnit ir MeTest Assert method.
I was trying to capture "AssertionException" when my assert get failed, how can i capture that using either NUnit or MsTest. since Assert method doesn't return any type. My requirements are Test should continue even when assert get failed to finish remaining assertions, should capture the error and should fail this assert. I was using below code statement. When i use Nunit framework its failing and continuous to next assert buy not able to capture even when i use try.. catch block where as in MsTest its failing, capturing using try..catch block and not continuing to next assertion.
Much appreciated for any help!
public static void ResponseValueAssert(dynamic actualValue, dynamic expectedValue, string nameOfAssert)
{
//var ex = Assert.Throws<AssertionException>(() =>
//Assert.AreEqual(expectedValue, actualValue, "Actual value doesn't match with Expected value {0}", nameOfAssert));
if (ResponseValueAssertImplicit(actualValue, expectedValue, nameOfAssert))
{
Console.WriteLine("\r\nResponse Assert:- {0}: <PASS>", nameOfAssert);
}
else
{
Console.WriteLine("\r\nResponse Assert:- {0}: <<FAIL>>", nameOfAssert);
Console.Error.WriteLine("\r\nResponse Assert:- {0}: <<FAIL>>", nameOfAssert);
// Assert.Fail();
}
Console.WriteLine("Expected Value: {0}.\r\nActual Value: {1}.", actualValue, expectedValue);
}
public static bool ResponseValueAssertImplicit(dynamic actualValue, dynamic expectedValue, string nameOfAssert)
{
try
{
Assert.AreEqual(expectedValue, actualValue, "Actual value doesn't match with Expected value {0}", nameOfAssert);
return true;
}
catch (AssertionException ex)
{
return false;
}
}
It's not useful to ask a question about two completely different pieces of software as one thing. The answer would obviously be different for NUnit and MSTest, which are implemented quite differently these days.
So I'll answer only with respect to NUnit, since I have no idea what you would do with MSTest anyway.
In NUnit, if you want a test to continue so that more than one assertion can be reported on in the same test, you use multiple assertions. That is...
Assert.Multiple(() =>
{
// Put your various asserts here
};
NUnit will report all the failed asserts. At the end of the block, the test will be terminated if any of the asserts failed.
Note that many people will say that more than one assert in a test is a bad idea. I believe it is most of the time but that there are situations, like checking multiple properties of the same object, where it can be useful.
Also, for the record, you should never be catching exceptions that are used internally by the test framework. They are basically hidden implementation details and all your work can be lost in the next release of the software... as has happened already to some people in this case.
Related
I have a GoLang API with an SPA to consume it. What I do to errors in my API is return them until the handler where I test if an error from previous functions exist. If there is an error, I put it inside the response body, set status code to either 400 or 500 then return the response
in the handler function, to be able to create a clear message to the client side, I need to know what kind of error was returned, how do I do it?
I know about error types but I read about Dave Cheney's recommendation to just return an error along with a message (or wrap them in other words).
But if there are so many kinds of errors which might occur in the API call, then does it mean before returning the response, I need to check them all just to know what message I should return?
The first thing to say about errors is that just because there's an error interface
type error interface {
Error() string
}
Does not mean that the error returned from any given method can only have that method / information attached to it.
One common method is to define your own error interface:
type myError interface {
error // embeds the standard error interface
OtherMethod() string // can define own methods here
}
When writing methods and functions it's really important to return an error and not myError, else you couple that method to your error implementation and cause dependency nightmares for yourself later.
Now that we've decided we can return extra information from error, using our own error interfaces you've got 3 main choices.
Sentinel errors
Error Failure types
Errors with Behaviour
Sentinel errors
Sentinel errors are error values that are defined as package level variables, are exported and allow comparison to check for errors.
package myPackage
var ErrConnectionFailed = errors.New("connection failed")
func Connect() error {
// trimmed ...
return ErrConnectionFailed
}
A consumer of this example could use the connect function:
if err := myPackage.Connect(); err == myPackage.ErrConnectionFailed {
// handle connection failed state
}
You can do a comparison to check if the error returned is equal to the sentinel error of the package. The drawback is that any error created with errors.New("connection failed") will be equal, and not just the error from myPackage.
Error failure types
Slightly better than sentinel errors are error failure types.
We've already seen that you can define your own error interface, and if we say ours is now:
type MyError interface {
error
Failure() string
}
type Err struct {
failure string
}
func (e *Err) Error() string {
// implement standard error
}
func (e *Err) Failure() string {
return e.failure
}
const ConnFailed = "connection failed"
err := &Err{failure: ConnFailed}
In the consumer code you can get an error, check if it implements MyError and then do things with it.
err := myPackage.Connect()
if myErr, ok := err.(myPackage.MyError); ok {
// here you know err is a MyError
if myErr.Failure() == myPackage.ConnFailed {
// handle connection failed, could also use a switch instead of if
}
}
Now you have an idea of what caused the error which is nice. But do you really care what the cause was? Or do you only really care what you want to do to handle that error.
This is where errors with behaviour are nice.
Errors with behaviour
This is similar to defining your own error type, but instead you define methods that report information about that error. Given the example above, do you really care that the connection failed, or do you really only care if you can retry or if you need to error up the call stack again?
package myPackage
// this interface could report if the error
// is temporary and if you could retry it
type tempErr interface {
Temporary() bool
}
func (e *Err) Temporary() bool {
// return if the error is temporary or not
}
Now in the consumer (note you don't need to use the myPackage.tempErr), you can test using type assertions if the error is temporary and handle the retry case:
err := myPackage.Connect()
if tmp, ok := err.(interface { Temporary() bool }); ok && tmp.Temporary() {
// the error is temporary and you can retry the connection
}
To answer the question, it's very hard to say without the specifics of the service that you are trying to implement. But as broad advice, I would try and use the last of the 3 examples as much as possible.
If the consumer of your service sends you some input that's not valid:
err := doThing(...)
if inv, ok := err.(interface { Invalid() bool }); ok && inv.Invalid() {
// input is invalid, return 400 bad request status code etc.
}
If you want to return a specific message to a consumer, you could make that a method of your error type. Warning: this would give your packages knowledge that they are being used in a web service, etc.
err := doThing(...)
if msg, ok := err.(interface { ResponseMsg() string }); ok {
// write the message to the http response
io.WriteString(response, msg.ResponseMsg())
}
TL;DR you would need to handle all the cases, but you can create error types that make the code much easier to work with!
I'm trying to catch my custom Error, but for some reason my catch statements where I name the error that I know is being thrown, it skips those, goes to the default catch, and then gives me a EXC_BAD_INSTRUCTION (code=EXC_I386_INVOP, subcode=0x0) when I try to do print("Unexpected error \(error)")
Here's some abbreviated code:
This is the error that I have declared in my file that houses the class that I'm calling the method on (the class is called CC8DB):
public enum CC8RSVPError: Error {
case noEventOnDate
case invalidRSVPValue
}
I have a method declared as:
public func rsvpForEvent(_ inEventDate:Date?, forUserID inUserID:String, withValue inRSVPValue:String) throws -> CC8RSVPStatus
In another class were I'm calling this method, I have this:
do {
let rsvpResponse = try self.cc8DB.rsvpForEvent(inRSVPDate, forUserID: String(inMessage.author.id.rawValue), withValue: inRSVPValue);
...(other code to do when this doesn't fail)...
} catch CC8RSVPError.invalidRSVPValue {
...(Report specific error to user)...
} catch CC8RSVPError.noEventOnDate {
...(Report specific error to user)...
} catch {
...(Report general error to user)...
print("Error doing RSVP: \(error)");
}
And finally, in the CC8DB.rsvpForEvent() method, I'm triggering an error that does this:
throw CC8RSVPError.invalidRSVPValue;
The germane part of this method is:
public func rsvpForEvent(_ inEventDate:Date?, forUserID inUserID:String, withValue inRSVPValue:String) throws -> CC8RSVPStatus
{
var retStatus = CC8RSVPStatus(eventDate: nil, previousRSVP: "", newRSVP: "");
var upperRSVPValue:String = inRSVPValue.uppercased();
if (["YES", "MAYBE", "NO"].contains(upperRSVPValue)) {
//...(Code to do things when the info is correct)...
} else {
throw CC8RSVPError.invalidRSVPValue;
}
return retStatus;
}
For my test case where I'm seeing this, the inRSVPValue is "bla", to test what happens when a user doesn't enter a valid status value.
What I'm seeing is that rather than going into the catch that's specific for the CC8RSVPError.invalidRSVPValue case, it's going down to the general catch. In addition, I'm getting the EXC_BAD_INSTRUCTION on the line where I try and print the error value.
I've stepped through it to verify that I am indeed hitting the throw line that I think I am, and I can see in the debugger that the value of error is CC8DB.CC8RSVPError.invalidRSVPValue, but even if I try to do po error from the lldb command, I get the same exception error.
Has anyone seen this or know what I could have done to make do-try-catch not work right?
You could assign a constant named error inside your catch statement and inside the catch block read the constant and figure out what to do with it.
do something like:
} catch let error {
switch error {
case CC8RSVPError.noEventOnDate:
// code
case CC8RSVPError.invalidRSVPValue:
// code
}
}
Ok, I figured it out. I realized that somewhere along the way, some build setting got set so that I was statically linking into the binary (this is a command-line tool, a bot for Discord to be specific).
I saw some warnings about some of the Swift runtime libs being found in both the binary and the XCode developer runtime area, and realized that it might be that the error object was being used both in my CC8DB module in the binary and in the built-modules folder (or something to that effect).
I need to statically link for when I actually deploy the bot to where it's going to run, so I must have turned something on that won't turn off (I deleted the extra flags that I thought turned that on, but that wasn't fixing it).
Basically, I recreated my .xcodeproj file with swift package generate-xcodeproj to clear out whatever I broke, and now it works as expected.
Thanks to everyone who looked at this and offered help (especially #gmogames for his time and help). I'm sure it helped lead me down the path of figuring this out.
I like the way Catch has nested hierarchies of tests, and it works through the combinations. It feels more natural than the setup/teardown of xUnit frameworks.
I now have a set of tests. What I want to do, about halfway down is insert a load/save serialization test, and then repeat all the tests below that point, first without the load/save, then again using the data it loaded from the serialization process. I.e. to prove that the load/save was correct.
I cannot get my head around if Catch has anything that can help with this? If it was phpUnit, I would be thinking about a string of #depends tests, and use a #dataProvider with a boolean input. A bit ugly.
(If that does not make sense, let me know, and I'll try to work out a minimal example)
The issue here is that Catch is designed to descend a tree-like organisation of tests and it automatically discovers all of the leaf-nodes of the structure and calls back into the test cases with untested code paths until they're all tested. The leaf nodes (tests, sections) are meant to be independent.
It sounds like you want to test a repository - something that can persist some data and then load it back in.
To repeat the exact same tests in two different scenarios (before serialisation, after serialisation) you'd need to put the same tests into some common place and call into that place. You can still use the same Catch macros in a non-test-case function, as long as you call it from a test case.
One possible way to do this is:
struct TestFixture {
Data data;
Repository repository;
TestFixture() : data(), instance() { }
};
void fillUpData(Data& data) {
// ...
}
void isDataAsExpected(Data& data) {
// Verify that 'data' is what we expect it to be, whether we
// loaded it or filled it up manually
SECTION("Data has ...) {
REQUIRE(data...);
}
}
TEST_CASE_METHOD(TestFixture, "Test with raw data") {
fillUpData(data);
isDataAsExpected(data);
REQUIRE(repository.save(data));
}
TEST_CASE_METHOD(TestFixture, "Operate on serialised data") {
REQUIRE(repository.load(data));
isDataAsExpected(_data);
}
One possible alternative is to supply your own main and then use command-line arguments to control whether/not the data is first serialised.
There's a third way I can think of that uses a non-quite-ready-yet feature of Catch - Generators:
TEST_CASE("...") {
using Catch::Generators;
int iteration(GENERATE(values(0, 1)));
const bool doSave(iteration == 0);
const bool doLoad(iteration == 1);
Repository repository;
Data data;
if (doLoad) {
REQUIRE(repository.load(data));
} else {
// fill up data
}
REQUIRE(..data..test..);
if (doSave) {
REQUIRE(repository.save(data));
}
}
The advantage of this method is you can see the flow and the test runs twice (for the two values) but the major disadvantage is that Generators are incompatible with SECTIONs and BDD-style features.
I am using Xamarin Studio 5.2 on Mac OS X 10.9.4 with NUnit 2.6.3 and FakeItEasy 1.23.0.
When I run tests for this code:
using System;
using ValueSet = System.Collections.Generic.HashSet<uint>;
using NUnit.Framework;
using FakeItEasy;
namespace SetTest
{
[TestFixture]
class TestFixture
{
[Test]
public void CallsUsersWithSetAndReducedSet()
{
var values = new ValueSet { 1, 2, 3 };
var setUser = A.Fake<SetUser>();
ClassUnderTest testInstance = new ClassUnderTest();
using (var scope = Fake.CreateScope())
{
testInstance.RunWith(setUser);
using (scope.OrderedAssertions())
{
A.CallTo(() => setUser.Use(A<ValueSet>.That.IsEqualTo(values))).MustHaveHappened(Repeated.Exactly.Once);
A.CallTo(() => setUser.Use(A<ValueSet>.That.Matches(set =>
set.Count == 2 && set.Contains(1)))).MustHaveHappened(Repeated.Exactly.Once);
}
}
}
}
public class SetUser
{
public virtual void Use(ValueSet set)
{
}
}
class ClassUnderTest
{
public static void Main(string[] arguments)
{
}
public void RunWith(SetUser setUser)
{
var values = new ValueSet { 1, 2, 3 };
setUser.Use(values);
values.Remove(3);
setUser.Use(values);
}
}
}
I get the following error output:
FakeItEasy.ExpectationException: Assertion failed for the following call: SetTest.SetUser.Use(1[System.UInt32]>) Expected to find it exactly once but found it #0 times among the calls:
1. SetTest.SetUser.Use(set: System.Collection.Generic.HashSet1[System.UInt32]) repeated 2 times
I don't understand what is causing this failure and how to fix it.
What is needed to get this type of test to pass?
#Tim Long is on the right track in his comment.
Here's a little more detail, as well as updates to respond to your comments of 2014-08-11 03:25:56:
The first reason the first MustHaveHappened fails:
According to the FakeItEasy argument constraints documentation, That.IsEqualTo tests for "object equality using object.Equals". That's what's causing the unexpected behaviour.
Not passing values into the method isn't necessarily a problem, or wouldn't be if ValueSet.Equals performed a value comparison, but ValueSet is a HashSet<uint>, so you can see from that class's method documentation that it doesn't—it uses object.Equals, which tests for reference equality. Thus, your IsEqualTo assertion fails. If you use a more sophisticated matcher that performed a value-type comparison for HashSet, perhaps something closer to what you use in your second A.CallTo, or maybe something using That.Contains, I think you'll have better success.
You may think to use That.IsSameSequenceAs, but be careful if doing so: the HashSet doesn't guarantee the order of the elements in the enumeration, so even if the set has the same elements, you may get a failure.
The second reason the first MustHaveHappened fails:
RunWith changes the contents of the values set between calls to setUser.Use. So the same set is used in two calls, first with 3 elements, then when it has only 2 elements. This means that by the time the first MustHaveHappened call is made, the set has only 2 elements, so the comparison fails. You could see this more clearly by writing an argument formatter for the ValueSet. That would provide more information.
The cause of the mismatch is that when a call is made to a faked method, FakeItEasy captures the arguments. However, for reference types, such as ValueSet (HashSet), only the reference to the argument is kept. Thus, if the object is modified later, in particular between the execution and the verification stages of the test, the object will look different than it did at the time of the faked call. See #jimmy_keen's answer to MustHaveHappened fails when called twice on the same object. There's a little more discussion over at FakeItEasy Issue 306 - Verifying multiple method calls with reference parameters.
In this case, the usual approach is to do as he suggests—provide code to capture the important state of the incoming argument at call time, and then query that saved state later.
You might be able to use something like this:
[Test]
public void CallsUsersWithSetAndReducedSet()
{
var capturedValueSets = new List<List<uint>>();
var setUser = A.Fake<SetUser>();
A.CallTo(() => setUser.Use(A<ValueSet>._)) // matches any call to setUser.Use
.Invokes((ValueSet theSet) => capturedValueSets.Add(theSet.ToList()));
ClassUnderTest testInstance = new ClassUnderTest();
testInstance.RunWith(setUser);
Assert.That(capturedValueSets, Has.Count.EqualTo(2),
"not enough calls to setUser.Use");
Assert.That(capturedValueSets[0], Is.EquivalentTo(new uint[] {1, 2, 3}),
"bad set passed to first call to setUser.Use");
Assert.That(capturedValueSets[1], Has.Count.EqualTo(2) & Has.Member(1),
"bad set passed to second call to setUser.Use");
}
You can see that each time Use is called, we add the contents of the ValueSet argument to capturedValueSets. Then at the end we
make sure 2 calls were made, by checking the length of capturedValueSets
make sure that the first time Use was called, the set had the elements 1, 2, and 3. Is.EquivalentTo checks the two lists but ignores order
make sure that the second time Use was called, the set had 2 elements, one of which was 1
By checking the two captured value sets in turn, all the bits about the scopes and ordered assertions became unnecessary.
Is there a hook in NUnit to execute code only when assertion fails without catching the exception itself. Basically, it should accept action delegate to be executed when assertion fails and then re-throw exception. Why do I need this?
I need to compare two objects and dump the result on the screen, for easier debugging, when assertion fails.
Something like this works but is a bad hack, The problem is that it eagerly evaluates ProcessCompareError so I have unnecessary overhead, plus it does it no matter if there is an error or not. So, is there overload that will accept the delegate that would be executed when assertion fails?
Assert.That(benefitLimitComparer.Compare(copyBenefitLimit, origBenefitLimit), Is.EqualTo(0),limitError, ProcessCompareError(origBenefitLimit, copyBenefitLimit));
}
}
}
private string ProcessCompareError(BenefitLimit origBenefitLimit, BenefitLimit copyBenefitLimit)
{
Console.WriteLine("Original: ");
ObjectDumper.Write(origBenefitLimit);
Console.WriteLine("Copy");
ObjectDumper.Write(copyBenefitLimit);
return "";
}
I'm not sure how it might be done through a delegate. One alternative is to store the result of the Compare. If the result is false, write out the contents of the objects and then call Assert.Fail()
There is a possibilty to wrap an assert as an Action in a try-catch. In the catch you can handle the additional compare:
public static void ExecuteAssert(Action assert)
{
if (assert == null) return;
try
{
assert();
}
catch (Exception ex)
{
// perform the compare
}
}
As remark: I use a similar method to continue test execution and avoid the entire test to stop, if some non-fatal checks fail. Actually I iterate through a number of actions:
private static void VerifyAll(params Action[] asserts)