Skip to content
This repository was archived by the owner on Aug 8, 2024. It is now read-only.

Test and Test Reporting Wish List

Jump to bottom
Beman edited this page Feb 4, 2011 · 7 revisions

Test and Test Reporting Wish List

This is Beman Dawes' initial attempt; it is unofficial and likely to evolve radically.

Repository organization

The use cases assume multiple Git repositories organized on the "Fork + Pull" model:

Boost Mothership Repo The "master" branch is equivalent to current SVN branches/release. Pull requests are processed automatically (see below). There is nothing equivalent to current SVN trunk, although the current trunk is preserved for historical purposes.

Library Public Repos One per library, cloned from the Boost Mothership repo. Roughly equivalent to the current SVN trunk for the library. Each library chooses its own model for how the lib's public repo is maintained, such as "Fork + Pull", "Shared", or whatever.

Library Developers Local Repos Cloned from the library's public repo.

Use cases

Boost mothership master testing

This is equivalent to the current branches/release testing. May be useful to separate into "official release" testers/compilers controlled by release managers, and "everyone welcome" testers/compilers. Primary reporting probably via public tables, similar to current system. Email notification of breakage would be very useful.

Pull request testing of a Library public repo

When a library's public repo sends a pull request to the Boost Mothership repo, a test of that library and all other libraries that depend on it is run. If all tests succeed (or are marked up) the pull request is automatically merged. An email is generated indicating success or failure, and pointing to where the test results are available. Note: The library's public repo needs to be up-to-date relative to the mothership master.

Explicit request for testing a library public repo

This is the "test on demand" scenario invoked, for example, when a developer thinks a library public repo is ready for a pull request to the mothership, but wants to be sure. An email is generated indicating success or failure, and pointing to where the test results are available. Variations include requests for partial tests with specific tests, specific compilers, etc.

Local testing with multiple local compilers

After tests are passing with the developer's primary compiler, the developer wants to test against multiple compilers installed locally.

Local testing with one local compiler

The developer wants a test on a specific locally installed compiler.

Testing

  • Each of the Use Cases is supported.

  • Choice of compilers falls into several categories:

    • Boost release compilers: Set by Boost policy; tests must pass or be marked up.
    • A given library's release compilers: Some libs may wish to add additional compilers as "test" and/or "release compilers".
    • Some compilers are just "test" compilers included for informational purposes; tests neither need to pass nor failures be marked up.

Test Reporting

The reporting wishes for many of the use cases, particularly those that test against a library's public or local repository, are likely to be very individualistic. Thus reporting might be offered in several standard forms, plus a well-defined file format that developers could use to generate their own favorite reports, emails, etc.

For each test, regardless of pass or fail, it would be useful to be able to click through to:

  • Git equivalent (with link) to the revision tested.
  • Git equivalent (with link) to the last known good revision; this is a surrogate for the revision that broke the test.
  • Compile elapsed time. Test elapsed time.
  • Compiler command line.
  • Link command line.
  • Run command line.
  • Compiler output, including both warnings and errors.
  • Linker output.
  • Run output, regardless of test success or failure.

Clone this wiki locally