Need way to run all tests in all packages

[jimka2001]

There does not seem to be a way to run all the tests even if they are in multiple packages.
It is common for a CL project to contain multiple packages, each potentially containing test cases.
It seems you can run the tests for each package by calling run-tests multiple times, each time with a different package designator. This is not really ideal.

I'm creating pull-request test-multiple-packages with an enhancement fixing this limitation.

[ThomasHermann]

There has been a proposal for this for the past 2 years (little embarrassed it's been that long).

http://markcox80.github.io/lisp-unit-proposal/

This proposal is what I'm working towards in the 'feature/summary' branch.

[jimka2001]

But just making the run-tests function take a list of package designators is pretty easy. Right? Or am I missing something?

[ThomasHermann]

You have to have some way to avoid name clashes. Different packages may have tests with the same name. Mark Cox has thought through this and has a couple prototype functions to implement, I just need to finish the integration.

[jimka2001]

Hmm, it is curious to me why two different packages would define tests of
the same name. Aren't names symbols? In which case package X may have a
test name X:test1 and package Y may have a test named Y:test1 so the names
will be different. Right?
On Jan 10, 2016 5:53 PM, "Thomas M. Hermann" notifications@github.com
wrote:

You have to have some way to avoid name clashes. Different packages may
have tests with the same name. Mark Cox has thought through this and has a
couple prototype functions to implement, I just need to finish the
integration.

—
Reply to this email directly or view it on GitHub
#35 (comment)
.

[jackcarrozzo]

Did anything end up happening with this? Would love the functionality to replace my ghetto mapcar.

[jimka2001]

I didn't understand why this would be a problem in the first place. I don't see why lisp-unit should
support two different tests with the same name. E.g., if package X defines a test names X::test1, and package Y also defines a test named X::test1. I don't see why lisp-unit should try to support that?

For me package X defines X::test1 and package Y defines Y::test1 and it works just fine.

[jackcarrozzo]

It seems like you probably only work in one package at a time. This is the use case for me at the moment, subject changed for fun:

We're writing a motion controller for a robot arm, and we're using modern lisp packaging standards (ie, most things in quicklisp). We touch the following files and begin working:

radbot.asd - main loader
src/radbot.lisp - defines "main" package radbot, exports external syms from others that we want visible
src/hw.lisp - defines package radbot.hw, contains code to talk to our drivers
src/math.lisp - defines radbot.math, contains motion planner code
src/ui.lisp - defines radbot.ui

Defining tests cases at the bottom of each file makes the most sense to me and is the easiest in my workflow, as I can easily see the funcs and the tests in the same file, as well as iterate the tests cases as I work on the code (vs having to open up another buffer for a test file and edit it in tandem).

So, we can run each package's tests with (run-tests :all :radbot.ui) or whatnot. I use a func in my main package to call each other package's tests as such, but it's inelegant in that each package reports its own summary and the map doesn't stop iterating packages on finding an error.

Regardless of whether you like/use this particular package setup or not, would you accept a PR to let us do so?

[heegaiximephoomeeghahyaiseekh]

The proposal is way overthinking it. No wonder it hasn't been done in 2 years.

The package below took an hour to develop, and it took maybe 15 minutes to do this write-up about it. It defines run-all-tests, which runs all the tests that Lisp-Unit knows about, and combines their separate results into a single result object. It works wonders for my project, and might even save me a little work.

The drawback comes if the libraries you depend on ship with Lisp-Unit tests. Then (run-all-tests) would test not only your project, but your dependencies that use Lisp-Unit, too.

A quick solution to that problem that won't take 2 years is simply to clear the hash table at lisp-unit::*test-db* after all your dependencies load and before your code defines any unit tests.

My project has 9 dependencies, most of them by Eitaro Fukamachi, who uses his own testing library,
so I'm not running into that problem.

(defpackage :lisp-unit-improvement
  (:documentation "Provides a fix for a severe deficiency in Lisp-Unit.")
  (:use :lisp-unit :common-lisp)
  (:export :run-all-tests))

(in-package :lisp-unit-improvement)

(defun merge-test-results (list-of-test-results)
  (let ((new-db (make-hash-table :test 'eq))
        (new-error-tests nil)
        (new-exerr 0)
        (new-fail 0)
        (new-failed-tests nil)
        (new-missing-tests nil)
        (new-pass 0)
        (new-results (make-instance 'lisp-unit::test-results-db)))    
                   
    (loop for test-result in list-of-test-results
       do
         (maphash (lambda (k v)
                    (setf (gethash k new-db)
                          v))
                  (slot-value test-result 'lisp-unit::database))
         (setf new-error-tests (append new-error-tests
                                       (slot-value test-result
                                                   'lisp-unit:error-tests)))
         (incf new-exerr (slot-value test-result 'lisp-unit::exerr))
         (incf new-fail (slot-value test-result 'lisp-unit::fail))
         (setf new-failed-tests
               (append new-failed-tests (slot-value test-result
                                                    'lisp-unit:failed-tests)))
         (setf new-missing-tests
               (append new-missing-tests (slot-value test-result
                                                     'lisp-unit:missing-tests)))
         (incf new-pass (slot-value test-result 'lisp-unit::pass)))
    (loop for (slot value) in
         `((lisp-unit::database ,new-db)
           (lisp-unit:error-tests ,new-error-tests)
           (lisp-unit::exerr ,new-exerr)
           (lisp-unit::fail ,new-fail)
           (lisp-unit:failed-tests ,new-failed-tests)
           (lisp-unit:missing-tests ,new-missing-tests)
           (lisp-unit::pass ,new-pass))
       do
         (setf (slot-value new-results slot) value))
    new-results))

(defun run-all-tests ()
  (let ((results nil))
    (maphash (lambda (package crap)
               (declare (ignore crap))
               (push (lisp-unit:run-tests :all package)
                     results))
             lisp-unit::*test-db*)
    (merge-test-results results)))