How to do code management for AB testing?
In particular, how to organize the code (baseline and variant) in the project folder?
how to remove code after the experiment?
There is no "one" answer. You can do this in many ways different folders, different classes or do something as simple as if statement in your code.
I think what you are actually asking is how to find all these experiments in the code. A good solution for this is to make the "experiment definition" code with concrete classes so you can easily find it in your IDE and once you delete the experiment definition class your compilation will break and you will easily find all the places in the code that use it.
A good example of this pattern is used by the open source AB test framework called PETRI, which is promoting exactly that. You create an experiment spec (the AB test definition) as a concrete class and use it to conduct the the test.
#RequestMapping(value = "/conductExperimentWithSpecDefinition", method = RequestMethod.GET)
#ResponseBody
public String conductExperimentWithSpecDefinition( #RequestParam("fallback") String fallback) throws ClassNotFoundException {
return laboratory.conductExperiment(TestSpecDefinition.class, fallback, new StringConverter());
}
Related
I am trying to find a way to create custom suites of NUnit tests to target our wide variety of environments. The closest thing I found was this http://nunit.org/docs/2.5.6/suite.html which is exactly what I am looking for. Tying to implement this though, the [Suite] annotation doesnt even seem to exist.. Was this taken away? Is there a better solution now?
The SuiteAttribute was eliminated in NUnit 3. It never got a lot of use as most people simply organize their tests by namespace, which provides the same grouping of tests that the SuiteAttribute used to do.
FUN FACT: "Automatic namespace suites" were once a new cool thing!
If you want the ability to group tests in different ways, across namespace boundaries, you can use categories to do it. It's not as easy of course.
An alternative, if you are using the command-line console runner, is to list the fixtures you want to run in a file and use the --testlist option.
Building off of Charlies post from above - The way I was able to set this up was using the --testlist option.
First create a testlist.txt file and store it somewhere in your solution. Structure the file in such a way so if you have a class like.
namespace NamespaceA
{
class TestGroup
{
[Test]
public void TestOne()
{
}
[Test]
public void TestTwo()
{
}
}
}
the files contents would look like this..
NamespaceA.TestGroup.TestOne
or for both..
NamespaceA.TestGroup
Then just your standard consule runner command
"nunit-console.exe" "path/to/.dll" --testlist="path/to/testlist.txt"
:D)
In Cucumber (the ruby version) you can easily call steps from other steps and thus build hierarchical libraries of steps making it easy to write the Gherkin feature specifications in the most generic terms.
However it is not readily apparent how to do this in Cucumber-JVM and I have been unable to find documentation for it.
Let me be clear I am not interested in calling the step implementation function directly because I don't want to have to know what its signature is, nor to change the call every time the implementation changes.
Rather, I want to pass an arbitrary string that will go through the regex matcher and automatically find the matching step and execute it. Just as the engine runs all steps.
simple example of what I would expect syntax to look like to define synonym "logout":
When("user logs out") { () =>
d.executeScript("logout();")
}
When("logout") { () =>
Step("user logs out")
}
This functionality is not supported in Cucumber-JVM. (Note that the Cucumber Backgrounder document you link in your question describes using Steps within Steps as "an anti-pattern")
Essentially, we believe that Cucumber is a collaboration tool and that Gherkin is not a programming language.
You can see a longer discussion of how we arrived at this decision here
To call steps within step definitions, inherit cuke4duke.Steps in java
import cuke4duke.StepMother;
import cuke4duke.Steps;
import cuke4duke.annotation.I18n.EN.When;
public class CallingSteps extends Steps {
public CallingSteps(StepMother stepMother) {
super(stepMother);
}
#When("^I call another step$")
public void iCallAnotherStep() {
Given("it is magic"); // This will call a step defined somewhere else.
}
}
Example:
https://github.com/cucumber-attic/cuke4duke/blob/master/examples/java/src/test/java/simple/CallingSteps.java
Note: cuke4duke support scala as well
Calling steps within steps is a terrible anti-pattern that can easily be replaced by something much simpler.
Instead of one step calling another step, have both steps call the same helper method.
If you apply this pattern with rigour you and up with
step definitions that are all just single calls to helper methods
a suite of helper methods that collectively provide a test-api
The art of elegantly implementing your Cucumber scenarios now becomes a known programming problem as all your functionality is now directly in code in your programming language rather than being in some restrictive construct specific to Cucumber.
You can now
refactor your helper methods to provide cleaner interaces
use parameters to give methods greater power
use naming to give all your calls greater clarity
use a helper method as an entry point to a suite of extra functionality
use delegation to move functionality out of helper methods and into test service objects
...
Providing this separation can be initially challenging if you are not a programmer or not experienced in the particular programming language in use. However once you get past this initial hurdle the code you can and should produce will be much easier to work with than the tangled mess that inevitably occurs with step nesting.
In Cucumber each Step is a Method. That way, you can call other methods in any step that you want.
#When("^click on \"([^\"]*)\"$")
public void clickOn(String arg1) throws Throwable {
driver.findElement(By.linkText(arg1)).click();
}
#Then("^should see the static elements changing$")
public void shouldSeeTheStaticElementsChanging() throws Throwable {
clickOn();
}
I am using junit with eclipse to write function tests.
When running an individual test it runs in the order that I have them set within the class.
Eg.
testCreateUser
testJoinUserToRoom
testVerify
testDeleteUser
However when I run this test as part of a suite, (so in a package) the order is random.
It will for example do the verify, then delete user then joinuserToRoom then Createuser.
My tests within the suite are not dependent on each other. However each individual test within a test is dependent on them being run in the correct order.
Is there any way I can achieve this?
Thanks.
You can't guarantee the order of execution of test methods in JUnit.
The order of execution of test classes within a suite is guaranteed (if you're using Suite), but the order of execution if the test classes are found by reflection isn't (for instance, if you're running a package in Eclipse, or a set of tests from maven or ant). This may be definable by ant or maven, but it isn't defined by JUnit.
In general, JUnit executes the test methods in the order in which they are defined in the source file, but not all JVMs guarantee this (particulary with JVM 7). If some of the methods are inherited from an abstract base test class, then this may not hold either. (This sounds like your case, but I can't tell from your description).
For more information on this, see my answer to Has JUnit4 begun supporting ordering of test? Is it intentional?.
So what can you do to fix your problem? There are two solutions.
In your original example, you've actually only got one test (verify), but you've got 4 methods, two setup (createUser, joinUserToRoom) and one teardown (deleteUser). So your first option is to better define your test cases, using a TestRule, in particular ExternalResource. ExternalResource allows you to define before/after behaviour for a test, similar to #Before/#After. However, the advantage of ExternalResource is that you can factor this out of your test.
So, you would create/delete the user in your external resource:
public class UsesExternalResource {
#Rule
public ExternalResource resource= new ExternalResource() {
#Override
protected void before() throws Throwable {
// create user
};
#Override
protected void after() {
// destroy user
};
};
#Test
public void testJoinUserToRoom() {
// join user to room
// verify all ok
}
}
For me, this is simpler and easier to understand, and you get independent tests, which is a good thing. This is what I would do, but you will need to refactor your tests a bit. You can also stack these Rules using RuleChain.
Your second option, if you really want to introduce dependencies between your test methods, is to look at TestNG, in which you can define dependencies from one test to another.
If they have a 'correct' order, then they are not multiple tests, but a single test that you have incorrectly annotated as being multiple independent tests.
Best practise would to rewrite them in approved junit style (setup - act - verify), supported by #Before or #BeforeClass methods that did any required common setup.
Quick workaround would be to have a single #Test-annotated method that called the other test methods in sequence. That becomes something like the preferred alternative if you are using Junit not to do strict unit testing, but something more like scenario-driven systems testing. It's not necessarily the best tool for such use, but it does work perfectly well in some cases.
Then, what you would have so far is have a single test:
#Test public void testUserNominalLifeCycle(...
which could then, if you are feeling virtuous, be supplemented by extra new tests like
#Test public void testUserWhoNeverJoinsARoom(...
I have been using CodeDom to do some code generation. It works great, but I haven't found a way to include the generated source code files in a project. I started using T4 and the T4Toolbox to generate code because it supports integration with project files.
Does anyone know if CodeDom supports this functionality too? I'd consider taking a second look at CodeDom if it only supported this one feature.
Here is an example of how I make a source code file with CodeDom:
protected void CreateSourceFile(CodeCompileUnit codeCompileUnit,
string fileName,
out string fileNameWithExtension)
{
fileNameWithExtension = string.Format("{0}.{1}",
fileName,
CodeProvider.FileExtension);
var indentedTextWriter =
new IndentedTextWriter(new StreamWriter(fileNameWithExtension,
false),
TabString);
CodeProvider.GenerateCodeFromCompileUnit(codeCompileUnit,
indentedTextWriter,
new CodeGeneratorOptions());
indentedTextWriter.Close();
}
That works fine but it just outputs the file to the hard drive somewhere (probably bin folder).
Here is a second example of some code I use with T4, this one specifies the output as part of the project the template is transformed in:
public class RDFSClassGenerator : Generator
{
private readonly string rootNamespace;
private readonly string ontologyLocation;
public RDFSClassGenerator(
string rootNamespace,
string ontologyLocation)
{
this.rootNamespace = rootNamespace;
this.ontologyLocation = ontologyLocation;
}
protected override void RunCore()
{
XElement ontology = XElement.Load(ontologyLocation);
var service = new RDFSGeneratorService(ontology);
foreach (MetaClass metaClass in service.MetaClasses)
{
var rdfsClassTemplate = new RDFSClassTemplate(rootNamespace, metaClass);
rdfsClassTemplate.Output.File = "Domain/" + metaClass.Name + ".cs";
rdfsClassTemplate.Render();
}
}
}
So the T4 code will output the file into the "Domain" folder of my project. But the CodeGen stuff just outputs the file on disk and doesn't update the project file.
Here is a visual:
Yes, it can. Here is how: http://www.olegsych.com/2009/09/t4-and-codedom-better-together/
Short answer is no, but I could be wrong (ever try to prove a negative?)
Your question was a little confusing as CodeDom isn't exactly equitable with T4. T4 templates are a convenient way of generating code files the same way, for example, asp.net generates HTML files, mixing text and code that gets executed to generate a file that is then interpreted by something else (such as a compiler or a browser). CodeDom is usually used to generate assemblies at runtime rather than files, although you can do it (as you have discovered).
While T4 makes it easy to add files to the solution, you can do this with CodeDom as well. I don't believe it supports interaction with the solution directly, but you can manage this using EnvDTE, or the automation model for Visual Studio.
The problem with this is that the automation model isn't easy to work with. EnvDTE is a wrapper around COM classes, which is always fun to code against. Also, you have to be careful when attempting to get the object. The naive implementation will get the object from the first instance of Visual Studio loaded. You have to poll the Running Object Table to find the current instance. Once you have it, you must deal with searching through the dte for the location you're looking for, deal with source control, locked files, etc etc.
Working with it, you start to learn why T4 was created in the first place.
The question you have to ask yourself is, "Does CodeDom give me enough that T4 doesn't to make up for all its shortcomings?"
I'm trying to run codeanalysis on an assembly that contains an entity model (edmx file). In project properties I have checked the "Suppress results from generated code" option, but I am still getting a lot of CA errors pertaining to the auto-generated EF code.
Has anyone experienced this? And is there a work-around?
Just put the attribute on your class definition.
But how to do it, since your file can get overridden any time. Use a separate file, since all generated classes are partial classes. Open a separate file, and write something like:
[GeneratedCode("EntityModelCodeGenerator", "4.0.0.0")]
public partial class YourEntitiesContextName : ObjectContext
{
}
This will skip code analysis on your particular generated class. StyleCop for instance is more smart and doesn't touch files that have .designer/.generated part in their name or regions that have generated word in their name.
Well, "Suppress results from generated code" really means "Don't look at types with GeneratedCodeAttribute". EF's code generator hasn't added this, historically (though I've suggested it to the team). But you can add it if you use custom T4.