I am currently getting to understand jobLauncherTestUtils. I have read about it from multiple resources such as following:
https://docs.spring.io/spring-batch/docs/current/api/org/springframework/batch/test/JobLauncherTestUtils.html
https://livebook.manning.com/concept/spring/joblaunchertestutils
I wanted to understand when we call jobLauncherTestUtils.launchJob(), what does it mean by end-to-end testing of job. Does it actually launch the job. If so, then what's the point of testing the job without mocks? If not so, then how does it actually tests a job?
I wanted to understand when we call jobLauncherTestUtils.launchJob(), what does it mean by end-to-end testing of job.
End-to-End testing means testing the job as a black box based on the specification of its input and output. For example, let's assume your batch job is expected to read data from a database table and write it to a flat file.
And end-to-end test would:
Populate a test database with some sample records
Run your job
Assert that the output file contains the expected records
Without individually testing the inner steps of this job, you are testing its functionality from end (input) to end (output).
JobLauncherTestUtils is a utility class that allows you to run an entire job like this. It also allows you to test a single step from a job in isolation if you want.
Does it actually launch the job.
Yes, the job will be run as if it was run outside a test. JobLauncherTestUtils is just an utility class that uses a regular JobLauncher behind the scene. You can run your job in unit tests without this utility class.
If so, then what's the point of testing the job without mocks?
The point of testing a job without mocks is to ensure the job is working as expected with real resources it depends on or interact with. You can always mock a database or a message broker in your tests, but the mocking code could be buggy and does not reflect the real behaviour of a database or a message broker.
Related
could someone explain please how Flutter/Dart tests are executed using test runner?
Are the tests executed synchronously or asynchronously?
Does the testing framework execute every single test synchronously, meaning that only a single test and test suite is executed at any single time?
Or does the testing framework only execute a single test at a time within a test suite, but are able to execute multiple test suites at the same time?
Or testing framework run all tests and test suites completely independent of each other at the same time, completely asynchronously?
This is important because it has a direct impact on the way we are or aren't able to structure our tests, especially when it comes to the set up and tear downs of tests, and the way we assert functionality is working correctly.
Thanks!
In general, dart test will execute many tests in parallel (the parallelism level varies based on CPU core count), but you can disable this with a command line flag.
You should not write tests with any inter-dependence (i.e. one test should not rely on some global state set up by another test). For example, you may find that because your laptop has a different CPU configuration to your CI server, your tests might pass locally but fail in CI due to different ordering.
If you have some setup logic that is very expensive, and needs to be reused between multiple tests, you can use setUpAll() to run some code once before every test in a test group, however this is still discouraged. Personally, I prefer to just join the tests into one long test, to keep all tests self-contained.
This has some advantages. For example, you can use --total-shards and --shard-index to parallelize tests in CI (by creating multiple jobs to each run a different subset of the test suite).
You can also randomize the order of your tests with --test-randomize-ordering-seed, to make sure you aren't accidentally setting up such dependences between tests that might invalidate their results (i.e. perhaps test2 only passes if it happens after test1, randomizing the ordering will catch this).
TLDR
Many tests run in parallel. Try to keep tests self-contained, with no dependence on the order of tests. Extract setup logic into functions and pass it into setUp. If you really really need the performance, you can try setUpAll but avoid it if possible.
I have a situation (detailed below) in which I want to run one NUnit test after all the other tests have completed. I know that I can use the order attribute to start my tests in a certain order but in this case:
I want to attribute (or otherwise change) only one test out of several hundred.
I want this test to run last, not first.
I want this test to run after all other tests have completed, not after they've started.
I have experimented with OneTimeTearDown, but ideally this would run as a regular, named test and appear that way in the test results.
(Why)
I have several hundred named, hand-crafted tests that run against different folders of json test files. Non-programmers add files to these folders from time to time. The purpose of this final test is to introspect those folders and compare the contents on disk with the files for which a test has already been executed (these are recorded by each test). If this indicates that there are untested files that, itself, constitutes a test failure.
It's an interesting question. Basically you want a meta-test... one that tests how well you are testing. For that reason, it's logical for it to actually be a test.
Unfortunately, NUnit only supports this sort of "run after everything" in a OneTimeTearDown. Now, you can Perform assertions in a OneTimeTearDown, but any failures are treated as errors, i.e. unexpected exceptions. For some purposes, this may be workable, but it doesn't look quite the same as a failure.
If I were doing this, I think I'd make it a separate analytical step in my script, after the tests had been run.
I had a quick question about the workflow plugin. I'm trying to see if the plugin will be able to satisfy my use case:
We have a jenkins job that will build our app
We want to spin off a suite of test jobs that will perform various tests on the newly built app (unit, integration, etc). These will need to be run in parallel and we want to run them on more than one jenkins node for performance reasons
We'll take the aggregated output from all our test processes from step 2 and be able to decide whether or not we should deploy (everything's passed) or not
I was curious as to whether or not I'd be able to accomplish this within the plugin and if so if you had any tips/pointers to a start.
Thanks!
You can certainly run nodes inside parallel branches. If one branch fails, the parallel step as a whole fails. If you want the build to succeed, but behave differently depending on test results, you can capture them directly as Groovy variables in various ways.
If you are using JUnitArchiver, currently it does not provide a simple means of exposing the test results directly to the Pipeline script (JENKINS-26276), though if you just want to tell if there are some failures or none, you can inspect currentBuild.status.
If you have JUnit-format test results and wish to automatically split them amongst various nodes (especially helpful in case you have a large pool of machines and it would be unmaintainable to manually divide your tests), see this demo of the Parallel Test Executor plugin’s splitTests step.
I'm running JUnit 4 with AnyLogic. In one of my tests, I need access to the Experiment running the test. Is there any clean way to access it at runtime? E.g., is there a static method along the lines of Experiment.getRunningExperiment()?
There isn't a static method that I know of (and, if there was, it might be complicated by multi-run experiments which permit parallel execution, although perhaps not since there's still a single Experiment, though there'd be thread-safety issues).
However, you can use getEngine().getExperiment() from within a model. You probably need to explain more about your usage context. If you're using AnyLogic Pro and exporting the model to run standalone, then you should have access to the experiment instance anyway (as in the help "Running the model from outside without UI").
Are you trying to run JUnit tests from within an Experiment? If so, what's your general design? Obviously JUnit doesn't sit as well in that scenario since it 'expects' to be instantiating and running the thing to be tested. For my automated tests (where I can't export it standalone because I don't use AnyLogic Pro), I judged that it was easier to avoid JUnit (it's just a framework after all) and implement the tests 'directly' (by having my model components write outputs and, at the end of the run in the Experiment, having the Experiment compare the outputs to pre-prepared expected ones and flag if the test was passed or failed). With AnyLogic Pro, you could still export standalone and use JUnit to run the 'already-a-test' Experiments (with the JUnit test checking the Experiment for a testPassed Boolean being set at the end or whatever).
The fact that you want to get running experiments suggests that you're potentially doing this whilst runs are potentially executing. If so, could you explain a bit about your requirements?
I have a test suite of ~ 1500 tests and they generally run and finish within 'reasonable time'.
Recently, however, I've changed parts of the code to use threads -- and now my builds fail from time to time by simply timing out. I imagine that a thread refuses to die and the build waits until reaching the maximum build time.
My problem is how to detect which test is causing the problem?
Can I activate some logging that shows me that a test has started/finished? I can of course be done by inserting code in every single test method - or just the fixtures, but that is A LOT of work that I'd rather avoid.
I'd suggest upgrading to NUnit 2.5 and decorating your tests with Timeout attribute, specifying maximum per-test run time. For example, you can put this in your AssemblyInfo.cs:
[assembly: NUnit.Framework.Timeout(100)]
NUnit will now launch each test in a separate thread and cancell it if it exceeds its time slot. However, this might be costly, so it's probably better to identify long-running tests and then remove assembly-level attribute in favor of test-fixture time slots. You can also override this on individual tests, assigning them more time to run.
This way you move the timeout/hang detection from CruiseControl.Net to NUnit and get information inside the report on the tasks that did not complete properly. Unfortunately there's no way for CC.Net to get this information when it has to kill the process because of timeout.