Currently, few people look at the nightly build results. This is kind of a waste as they can provide lots of useful information to both developers and users.
Usage
Usage scenarios for what someone can accomplish by looking at the nightly results. Users:
- see if their platform is currently working well
- see if a particular bit of functionality is working
- choose a good nightly build to use based on the above criteria
Developers:
- see if their recent changes work on platforms other than they develop on
- see if the or other recent changes broke anything
- figure out a way to get sufficient information to troubleshoot problems on other platforms
- ideally, add new development platforms to the build without too much difficulty (eg just setting up a cron job and copying over an ssh key)
Requirements
The biggest issue may be the lack of fine grained reporting of results without looking through an enormous, unstructured, file. Since it makes sense for most modules to have some failing tests most of the time (since there is always functionality that is being worked on), we need something better than simply reporting if everything is OK or not.
To facilitate the user scenarios, the page must do the following:
- allow quick determination of the status on a particular platform (eg Fedora 16, just having the platforms labeled better would work here)
- allow quick checking of a the results of a particular test (having a small file to look through per module, perhaps just test outputs, would probably suffice)
- easily determine which rpms, svn version or whatever corresponds to the nightly build (the putting the rpms in the same directory, so just adding constant links to the page template should be easy enough)
- move back in time to inspect earlier nightly builds (just saving the page each night and having a link to the cur num -1 page would be easy)
To facilitate the developer usages one must:
- get reliable status information for their module (an indicator for build success or failure and the number of passed/failed tests might suffice. This should be easy to get scons to produce).
- quickly locate what is wrong with their module (so reliably have a relatively small amount of text to look through to find the changes--a build log for just building or just testing their module would probably be good)
- compare the number of test failures between two builds (to see if something new is broken-being able to open two web pages side by side is probably good enough)
- figure out the relevant parameters for the failed platform (swig version, boost version etc, this can either be generated from scons or be done statically, the former is probably safe and pretty easy as it is already in scons)
- figure out if their is a quick way they can get to that build environment (eg a machine name with a config.py if there is an accessible machine, this would have to be added manually, but should rarely change and so maintenance would be easy)
Content for the page
To facilitate these the page probably should have:
- the svn version in a prominent place
- a link to the downloadable builds for each platform
- a link to the previous nights build
- a link to generated files for the build (swig wrappers, build log--probably per module is best)
- a list of information about each build platform
- an indicator about whether a given module could be compiled
- the number of tests for the module that passed and the number that failed for each platform
The build platform information should include:
- distribution (fedora 16 etc)
- boost version
- swig version
- compiler version
- a machine name if available with a link to a config.py (or, perhaps better, how to check it out of svn)
Really cool things, but that might be a bit tricky would include
- a list of what tests failed this day that didn't fail the day before
tagging of classes in the docs with results of related tests. We could annotate tests with class and function names of the things they are testing and, once we have the test results back in a per-test basis, output dox blocks that add lines to the class or function docs that show the test results. It is not clear how to annotate the tests, but the reporting would be straight forward through a TestRunner that extracted the TestResults and made dox pages in addition to printing the results to the screen.