In a Java project I'm working (alongside a team of 8 devs), we have a large backlog of features that lack automated tests. We are covering this backlog and we need to keep track of who wrote a JUnit test and when, plus we have to measure how many test we wrote as a team in a week/month/semester (as you may have figured out already, this information is for management purposes). We figured we'd do this by marking the tests with the information we need (author, creation date) and let Eclipse do the processing work, showing us tests we wrote, who wrote'em and how far we were from reaching our goals. How would you smart people go about this? What plugins would you use?
I tried to use Eclipse Custom Tags for this, but it's not the purpose of the feature, and the results I got were kind of brittle. I created a TEST tag that was supposed to mark a test method. It looks like this: (date is mm-dd-yyyy)
//TEST my.name 08-06-2011
Since Eclipse processes tag description by substringing (contains/doesn't contain), it's, as I said, very brittle. I can timestamp the tag, but it's just a string. Eclipse can't process it as a date, compare dates, filter by date interval, stuff like that.
I searched for plugins, but no dice.
My B-plan is to create an Annotation with the information we need and process our test packages using Eclipse Annotation Processing Tool. I haven't tried anything on this front yet, though, just an idea. Anyone knows a good plugin for this kind of Annotation processing? Or any starter tips for dealing with Eclipse APT.
Thanks a bunch, folks
I would not use Eclipse for this.
Your team should be checking the tests into a version control system such as Subversion, Git, Team Foundation Server, etc. From there it should a fairly straightforward matter to determine the owner and check-in time. You can and should do this sort of metrics calculation during every build. Better yet, be sure that your build script actually runs your tests and uses a tool like EMMA to instrument the code and determine the actual coverage.
As a fallback for measuring coverage, if you choose a naming convention then you may even be able to correlate the test classes by file name back to the feature under test.
Many modern build systems, such as CruiseControl, have integration for doing these sorts of things quite nicely.
Related
We use gradle as our build tool and use the idea plugin to be able to generate the project/module files. The process for a new developer on the project would look like this:
pull from source control.
run 'gradle idea'.
open idea and be able to develop without any further setup.
This all works nicely, but generally only gets exercised when a new developer joins or someone gets a new machine. I would really like to automate the testing of this more frequently in the same way we automate our unit/integration tests as part of our continuous integration process.
Does anyone know if this is possible and if there is any libraries for doing this kind of thing?
You can also substitue idea for eclipse as we have a similar process for those that prefer using eclipse.
The second step (with or without step one) is easy to smoke test (just execute the task as part of a CI build), the third one less so. However, if you are following best practices and regenerate IDEA files rather than committing them to source control, developers will likely perform both steps more or less regularly (e.g. every time a dependency changes).
As Peter noted, the real challenge is step #3. The first 2 ones are solved by your SCM plugin and gradle task. You could try automating the last task by doing something like this
identify the proper command line option, on your platform, that opens a specified intellij project from the command line
find a simple good enough scenario that could validate that the generated project is working as it should. E.g. make a clean then build. Make sure you can reproduce these steps using keyboard shortcuts only. Validation could be made by validating either produced artifacts or test result reports, etc
use an external library, like Robot, to program the starting of intellij and the running of your keyboards. Here's a simple example with Robot. Use a dynamic language with inbuilt console instead of pure Java for that, it will speed your scripting a lot...
Another idea would be to include a daemon plugin in intellij to pass back the commands from external CLI. Otherwise take contact with the intellij team, they may have something to ease your work here.
Notes:
beware of false negatives: any failure could be caused by external issues, like project instability. Try to make sure you only build from a validated working project...
beware of false positives: any assumption / unchecked result code could hide issues. Make sure you clean properly the workspace, installation, to have a repeatable state and standard scenario matching first use.
Final thoughts: while interesting from a theoretical angle, this automation exercise may not bring all the required results, i.e. the validation of the platform. Still it's an interesting learning experience and could serve as a material for a nice short talk, especially if you find out interesting stuff. Make it a beer challenger with your team when you have a few idle hours to try to see who can implement the fastest a working solution ;) Good luck!
Does anyone know of a built-in way to have Hudson keep track of how many files are changed, added or deleted in the source code repository in each build ? I'd like to plot the results in the same way that the JUnit test results graphs show the numbers of passing and failing tests for each build.
The Measurement Plots plugin and the Plot Plugin look like they might give me a starting point, but i'm wondering if there might be a more specific plugin or feature already available.
My SCM system is CVS, but I'd like a generic solution that would work with other SCM systems.
I don't believe there are any existing plugins that will do this directly.
If it doesn't need to be specifically tracked for each build (that is if you are really more interested in changes over time), then I would suggest setting up Sonar, which tracks daily changes from your builds and integrates fantastically with Hudson, or FishEye which connects directly to your SCM system.
But why not try to write the plugin for Hudson? Seems like the sort of thing that people might like to visualize as a per-build metric.
I think this question is more generic than specific to Hudson. You're probably going to have to write a little code by yourself. Unfortunately, I don't think any solution will be SCM agnostic because Hudson tends to use the SCM tools themselves to do the SCM bits.
I couldn't find any off-the-shelf solutions, so here's what I see would have to be done:
Find the SCM command that you are using (i.e., svn up, cvs -n).
Use wc -l or some other command to count the number of lines in output. This will give you an estimate to the number of changed/added/deleted files.
Parse the output if you want the names of individual files that have been added/changed/deleted.
Unfortunately, I don't think there's an SCM-agnostic way of doing this. Perhaps the best you could do is find a pure-Java CVS/SVN client implementation that you could modify to keep track of files as they come in from the SCM.
If I want to enable a new piece of functionality to a subset of known users first, is there any automated system of framework that exists to do this?
Perhaps not directly with version control - you might be interested to read how flickr goes about selectively deploying functionality: http://code.flickr.com/blog/page/2/
And this guy talks about implementing something similar in a rails app: http://www.alandelevie.com/2010/05/19/feature-flippers-with-rails/
Most programming languages have if statements.
I don't know what "switching between them at runtime" means. You usually don't check executable code into an SCM system. There's a separate process to check out, build, package, and deploy. That's the province of continuous integration and automated builds in agile techniques.
SCM systems like Subversion allow you to have tags and branches for parallel development. You're always free to build, package, and deploy those as you see fit.
As far as I know no...
If you wanted a revision control system that had multiple versions that you could switch between. Find a SCM you like and lookup branching.
But, it sounds like you want it to me able to switch versions in the SCM programmatically during runtime. The problem with that is, for a revision control system to be able to do that it would have to be aware of the language and how it's implemented.
It would have to know how load and run the next version. For example, if it was C code it would have to dynamically compile and run it on the fly. If it was PHP it would have to magically load the script in a sandbox http server that has PHP support. Etc... In which case, it isn't possible.
You can write an app to change the version in the scm by using the command line.
To do it during runtime, that functionality has to be part of the application itself.
The best (only) way I can think of doing it is to have one common piece of code that acts like a 'bootloader', which uses a system call to checkout the correct branch based on whatever your requirements are. It then (if necessary) compiles that code, and runs it.
It's not technically 'at runtime', but it appears that way if it works.
Your first other option is something that dynamically loads code, but that's very language-dependent, and you'd need to specify.
The other is to permanently have both in the working codebase (which doubles your size if it's a full duplication), and switch at runtime. You can save a good bit of space by using objects that are shared between both branches, and things like conditional compilation to use the same source files for both targets.
I'm evaluating the possibility of developing an Eclipse plugin to modify the source code of some Java files.
The Eclipse plugin should:
add one menu option or context menu option to launch the modification process.
add a key binding
only alter the UI in that way when an editor has been open on a Java file.
the modification process would not open a dialog, or maybe, a very simple one.
the modification process would traverse the AST of the Java file and would modify it.
Considering that we have no experience with Eclipse plugins and we need spend time in reading docs, how much time do you estimate in developing that plugin?
Thanks in advance.
It's really not that difficult at all... I had students in my design patterns class doing it for an assignment (adding/removing javabean getters and setters)
See http://help.eclipse.org/ganymede/topic/org.eclipse.jdt.doc.isv/guide/jdt_api_manip.htm
[EDIT: added the following article reference]
And a great article on it at http://www.eclipse.org/articles/article.php?file=Article-JavaCodeManipulation_AST/index.html (from 2006 -- there may be a few API changes since)
Yes, writing plugins takes a little getting used to, but so does any API.
And you can modify the AST -- see the page I reference above.
(I should note that the above link is from the eclipse help, which can also be accessed via Help->Help Contents inside Eclipse -- there's a lot of good info in there, but it's just a starting point)
You'll probably spend quite some time cursing the complexity of the eclipse plugin system. There are some example plugin development projects that can be very helpful if they cover the area you're working in.
I'd say you're looking at 2-4 days of work, spent mainly getting familiar with the platform - someone with a lot of experience writing eclipse plugins would probably take no more than an hour.
However, your step 5 could be tricky. I don't know how easy it is to access and change the Java AST; my experience is based on developing an editor plugin for an exotic file format rather than Java code.
Well, the four first points are easy to achieve, even by monkey coders that look at the eclipse PDE documentation shipped with Eclipse. These can be achieve in 1 day of work, maybe 2.
The hardest point is really the fifth one and the kind of modification you expect to do. Acting directly on the editor content is simple, accessing the editor internal AST and modifying it is really a bigger challenge and I doubt that it could be achieve in less than a week by unexperimented people (it can take longer, depending of what kind of modification you want to apply).
I've used the MIT Continuous testing plugin in the past, but it has long since passed out of date and is no longer compatible with anything approaching a modern release of Eclipse.
Does anyone have a good replacement? Free, naturally, is preferred.
I found that Infinitest now has an Eclipse plugin that seems to work pretty well.
There is a list in this Ben Rady article at Object Mentor: Continuous Testing Explained. Unfortunately the only Eclipse tool appears to be CT-Eclipse which is not currently maintained either.
There is also Fireworks for IntelliJ and Infinitest which is not IDE specific but also has some IntelliJ integration.
My experience is that continuous testing within the IDE can become unwieldy and distracting, so I prefer to use something like CruiseControl to do this kind of testing. One tool I have found very useful is EclEmma, which gives you a very fast coverage turnaround for your units, helping you to decide when you have finished testing a particular area of the code.
Infinitest decides what tests it wants to run. Often it runs the wrong ones. Green bar sometimes good, sometimes meaningless.
I've had good experience with infinitest on a small and simple project. I've not run into any issues with it and find it fast and helpful.
I also use Infinitest (and voted for one of its answers), but I wanted to add another approach, which relies on the build server. Whenever you want to implement something, create a branch in your VCS, do your changes, commit to your branch. If you have a build server configured, which runs unit tests on every checkin, your unit tests are then run on the build server without actually having polluted the trunk (or HEAD, whatever you call it) and without you waiting for the test run to finish.
I admit that this is not really continuous unit testing in the sense you asked the question, but for large projects or large test suites even a "normal" continuous test runner may slow you down way to much.
For small projects I also recommend Infinitest or CT Eclipse.