Add test selector support

Test all and test ns understand leiningen style test selectors, eg
metadata markers on `deftest`s. No support for namespace level metada
markers for now.

If test selector is specified only the test(s) run that have that
selector in their metadata.

If all loaded tests or project tests are invoked with prefix, ask for
test selector in the minibuffer. A command is available to run ns
tests with a selector.

Author: benedekfazekas

CHANGELOG.md

@@ -20,6 +20,7 @@
 * Add support for shadow-cljs to `cider-jack-in`.
 * [#2244](https://github.com/clojure-emacs/cider/issues/2244): Display the REPL type in the modeline.
 * [#2238](https://github.com/clojure-emacs/cider/pull/2238): Allow specifying predicates for entries in `cider-jack-in-lein-plugins` and `cider-jack-in-nrepl-middlewares`.
+* Add support for test selectors. If test all or all loaded is called with a prefix ask for a selector in the minibuffer and only run those tests in the project which are marked with the selector. Add variation of test namespace which asks for a selector the same way and only runs a subset of the namespace tests.
 
 ### Bugs Fixed
 

cider-repl.el

@@ -1409,6 +1409,8 @@ constructs."
 (declare-function cider-run "cider-interaction")
 (declare-function cider-refresh "cider-interaction")
 (declare-function cider-version "cider")
+(declare-function cider-test-run-loaded-tests "cider-test")
+(declare-function cider-test-run-project-tests "cider-test")
 (cider-repl-add-shortcut "clear-output" #'cider-repl-clear-output)
 (cider-repl-add-shortcut "clear" #'cider-repl-clear-buffer)
 (cider-repl-add-shortcut "clear-banners" #'cider-repl-clear-banners)
@@ -1425,6 +1427,9 @@ constructs."
 (cider-repl-add-shortcut "test-ns" #'cider-test-run-ns-tests)
 (cider-repl-add-shortcut "test-all" #'cider-test-run-loaded-tests)
 (cider-repl-add-shortcut "test-project" #'cider-test-run-project-tests)
+(cider-repl-add-shortcut "test-ns-with-selector" #'cider-test-run-ns-tests-with-selector)
+(cider-repl-add-shortcut "test-all-with-selector" (lambda () (cider-test-run-loaded-tests 'prompt-for-selector)))
+(cider-repl-add-shortcut "test-project-with-selector" (lambda () (cider-test-run-project-tests 'prompt-for-selector)))
 (cider-repl-add-shortcut "test-report" #'cider-test-show-report)
 (cider-repl-add-shortcut "run" #'cider-run)
 (cider-repl-add-shortcut "conn-info" #'cider-display-connection-info)

cider-test.el

@@ -141,6 +141,7 @@ Add to this list to have CIDER recognize additional test defining macros."
     (define-key map (kbd "C-t") #'cider-test-run-test)
     (define-key map (kbd "C-g") #'cider-test-rerun-test)
     (define-key map (kbd "C-n") #'cider-test-run-ns-tests)
+    (define-key map (kbd "C-s") #'cider-test-run-ns-tests-with-selector)
     (define-key map (kbd "C-l") #'cider-test-run-loaded-tests)
     (define-key map (kbd "C-p") #'cider-test-run-project-tests)
     (define-key map (kbd "C-b") #'cider-test-show-report)
@@ -149,6 +150,7 @@ Add to this list to have CIDER recognize additional test defining macros."
     (define-key map (kbd "t")   #'cider-test-run-test)
     (define-key map (kbd "g")   #'cider-test-rerun-test)
     (define-key map (kbd "n")   #'cider-test-run-ns-tests)
+    (define-key map (kbd "s")   #'cider-test-run-ns-tests-with-selector)
     (define-key map (kbd "l")   #'cider-test-run-loaded-tests)
     (define-key map (kbd "p")   #'cider-test-run-project-tests)
     (define-key map (kbd "b")   #'cider-test-show-report)
@@ -158,8 +160,11 @@ Add to this list to have CIDER recognize additional test defining macros."
   '("Test"
     ["Run test" cider-test-run-test]
     ["Run namespace tests" cider-test-run-ns-tests]
+    ["Run namespace tests with selector" cider-test-run-ns-tests-with-selector]
     ["Run all loaded tests" cider-test-run-loaded-tests]
+    ["Run all loaded tests with selector" (apply-partially cider-test-run-loaded-tests 'prompt-for-selector)]
     ["Run all project tests" cider-test-run-project-tests]
+    ["Run all project tests with selector" (apply-partially cider-test-run-project-tests 'prompt-for-selector)]
     ["Run tests after load-file" cider-auto-test-mode
      :style toggle :selected cider-auto-test-mode]
     "--"
@@ -186,6 +191,7 @@ Add to this list to have CIDER recognize additional test defining macros."
     ;; `f' for "run failed".
     (define-key map "f" #'cider-test-rerun-failed-tests)
     (define-key map "n" #'cider-test-run-ns-tests)
+    (define-key map "s" #'cider-test-run-ns-tests-with-selector)
     (define-key map "l" #'cider-test-run-loaded-tests)
     (define-key map "p" #'cider-test-run-project-tests)
     ;; `g' generally reloads the buffer.  The closest thing we have to that is
@@ -202,8 +208,11 @@ Add to this list to have CIDER recognize additional test defining macros."
         ["Rerun current test" cider-test-run-test]
         ["Rerun failed/erring tests" cider-test-rerun-failed-tests]
         ["Run all ns tests" cider-test-run-ns-tests]
+        ["Run all ns tests with selector" cider-test-run-ns-tests-with-selector]
         ["Run all loaded tests" cider-test-run-loaded-tests]
+        ["Run all loaded tests with selector" (apply-partially cider-test-run-loaded-tests 'prompt-for-selector)]
         ["Run all project tests" cider-test-run-project-tests]
+        ["Run all project tests with selector" (apply-partially cider-test-run-project-tests 'prompt-for-selector)]
         "--"
         ["Jump to test definition" cider-test-jump]
         ["Display test error" cider-test-stacktrace]
@@ -611,61 +620,67 @@ This uses the Leiningen convention of appending '-test' to the namespace name."
 (declare-function cider-emit-interactive-eval-output "cider-interaction")
 (declare-function cider-emit-interactive-eval-err-output "cider-interaction")
 
-(defun cider-test-execute (ns &optional tests silent)
+(defun cider-test-execute (ns &optional tests silent prompt-for-selector)
   "Run tests for NS, which may be a keyword, optionally specifying TESTS.
 
 This tests a single NS, or multiple namespaces when using keywords `:project',
 `:loaded' or `:non-passing'.  Optional TESTS are only honored when a single
 namespace is specified.  Upon test completion, results are echoed and a test
 report is optionally displayed.  When test failures/errors occur, their sources
 are highlighted.
-If SILENT is non-nil, suppress all messages other then test results."
+If SILENT is non-nil, suppress all messages other then test results.
+If PROMPT-FOR-SELECTOR is non-nil, prompt the user for a test selector.
+The selector will be used to filter the tests before running them."
   (cider-test-clear-highlights)
-  (cider-map-connections
-   (lambda (conn)
-     (unless silent
-       (if (and tests (= (length tests) 1))
-           ;; we generate a different message when running individual tests
-           (cider-test-echo-running ns (car tests))
-         (cider-test-echo-running ns)))
-     (cider-nrepl-send-request
-      `("op"     ,(cond ((stringp ns)         "test")
-                        ((eq :project ns)     "test-all")
-                        ((eq :loaded ns)      "test-all")
-                        ((eq :non-passing ns) "retest"))
-        "ns"     ,(when (stringp ns) ns)
-        "tests"  ,(when (stringp ns) tests)
-        "load?"  ,(when (or (stringp ns)
-                            (eq :project ns))
-                    "true"))
-      (lambda (response)
-        (nrepl-dbind-response response (summary results status out err)
-          (cond ((member "namespace-not-found" status)
-                 (unless silent
-                   (message "No test namespace: %s" (cider-propertize ns 'ns))))
-                (out (cider-emit-interactive-eval-output out))
-                (err (cider-emit-interactive-eval-err-output err))
-                (results
-                 (nrepl-dbind-response summary (error fail)
-                   (setq cider-test-last-summary summary)
-                   (setq cider-test-last-results results)
-                   (cider-test-highlight-problems results)
-                   (cider-test-echo-summary summary results)
-                   (if (or (not (zerop (+ error fail)))
-                           cider-test-show-report-on-success)
-                       (cider-test-render-report
-                        (cider-popup-buffer cider-test-report-buffer
-                                            cider-auto-select-test-report-buffer)
-                        summary results)
-                     (when (get-buffer cider-test-report-buffer)
-                       (with-current-buffer cider-test-report-buffer
-                         (let ((inhibit-read-only t))
-                           (erase-buffer)))
-                       (cider-test-render-report
-                        cider-test-report-buffer
-                        summary results))))))))
-      conn))
-   :clj))
+  (let ((selector (when prompt-for-selector (cider-read-from-minibuffer "Test selector (as string): "))))
+    (cider-map-connections
+     (lambda (conn)
+       (unless silent
+         (if (and tests (= (length tests) 1))
+             ;; we generate a different message when running individual tests
+             (cider-test-echo-running ns (car tests))
+           (cider-test-echo-running ns)))
+       (cider-nrepl-send-request
+        `("op"     ,(cond ((stringp ns)         "test")
+                          ((eq :project ns)     "test-all")
+                          ((eq :loaded ns)      "test-all")
+                          ((eq :non-passing ns) "retest"))
+          "selector" ,(when (stringp selector) selector)
+          "ns"     ,(when (stringp ns) ns)
+          "tests"  ,(when (stringp ns) tests)
+          "load?"  ,(when (or (stringp ns)
+                              (eq :project ns))
+                      "true"))
+        (lambda (response)
+          (nrepl-dbind-response response (summary results status out err)
+            (cond ((member "namespace-not-found" status)
+                   (unless silent
+                     (message "No test namespace: %s" (cider-propertize ns 'ns))))
+                  (out (cider-emit-interactive-eval-output out))
+                  (err (cider-emit-interactive-eval-err-output err))
+                  (results
+                   (nrepl-dbind-response summary (error fail)
+                     (setq cider-test-last-summary summary)
+                     (setq cider-test-last-results results)
+                     (cider-test-highlight-problems results)
+                     (cider-test-echo-summary summary results)
+                     (if (or (not (zerop (+ error fail)))
+                             cider-test-show-report-on-success)
+                         (cider-test-render-report
+                          (cider-popup-buffer
+                           cider-test-report-buffer
+                           cider-auto-select-test-report-buffer)
+                          summary
+                          results)
+                       (when (get-buffer cider-test-report-buffer)
+                         (with-current-buffer cider-test-report-buffer
+                           (let ((inhibit-read-only t))
+                             (erase-buffer)))
+                         (cider-test-render-report
+                          cider-test-report-buffer
+                          summary results))))))))
+        conn))
+     :clj)))
 
 (defun cider-test-rerun-failed-tests ()
   "Rerun failed and erring tests from the last test run."
@@ -677,27 +692,40 @@ If SILENT is non-nil, suppress all messages other then test results."
           (message "No prior failures to retest")))
     (message "No prior results to retest")))
 
-(defun cider-test-run-loaded-tests ()
-  "Run all tests defined in currently loaded namespaces."
-  (interactive)
-  (cider-test-execute :loaded))
+(defun cider-test-run-loaded-tests (prompt-for-selector)
+  "Run all tests defined in currently loaded namespaces.
 
-(defun cider-test-run-project-tests ()
-  "Run all tests defined in all project namespaces, loading these as needed."
-  (interactive)
-  (cider-test-execute :project))
+If PROMPT-FOR-SELECTOR is non-nil, prompt the user for a test selector to filter the tests with."
+  (interactive "P")
+  (cider-test-execute :loaded nil nil prompt-for-selector))
+
+(defun cider-test-run-project-tests (prompt-for-selector)
+  "Run all tests defined in all project namespaces, loading these as needed.
+
+If PROMPT-FOR-SELECTOR is non-nil, prompt the user for a test selector to filter the tests with."
+  (interactive "P")
+  (cider-test-execute :project nil nil prompt-for-selector))
 
-(defun cider-test-run-ns-tests (suppress-inference &optional silent)
+(defun cider-test-run-ns-tests-with-selector (suppress-inference)
+  "Run tests filtered by a selector for the current Clojure namespace context.
+
+With a prefix arg SUPPRESS-INFERENCE it will try to run the tests in the
+current ns."
+  (interactive "P")
+  (cider-test-run-ns-tests suppress-inference nil 't))
+
+(defun cider-test-run-ns-tests (suppress-inference &optional silent prompt-for-selector)
   "Run all tests for the current Clojure namespace context.
 
 If SILENT is non-nil, suppress all messages other then test results.
 With a prefix arg SUPPRESS-INFERENCE it will try to run the tests in the
-current ns."
+current ns.  If PROMPT-FOR-SELECTOR is non-nil, prompt the user for
+a test selector to filter the tests with."
   (interactive "P")
   (if-let* ((ns (if suppress-inference
                     (cider-current-ns t)
                   (funcall cider-test-infer-test-ns (cider-current-ns t)))))
-      (cider-test-execute ns nil silent)
+      (cider-test-execute ns nil silent prompt-for-selector)
     (if (eq major-mode 'cider-test-report-mode)
         (when (y-or-n-p (concat "Test report does not define a namespace. "
                                 "Rerun failed/erring tests?"))

doc/running_tests.md

@@ -13,12 +13,24 @@ test namespace inference logic (e.g. you have some tests in the implementation
 namespace that were defined with `clojure.test/with-test`)
 - in such cases you should use <kbd>C-u C-c C-t C-n</kbd>, which will simply run
 whatever tests are present in the currently visited/active namespace.
+You can also press <kbd>C-c C-t C-s</kbd> to run a subset of the tests defined in
+the namespace prefiltered by a test selector. CIDER will ask you for this selector
+in the minibuffer. If you call this command with a prefix (<kbd>C-u C-c C-t C-s</kbd>)
+you can suppress the inference logic as for run tests for the namespace.
 
 You can also run all loaded tests with <kbd>C-c C-t l</kbd> or <kbd>C-c C-t
 C-l</kbd> and all tests within a project with <kbd>C-c C-t p</kbd> or <kbd>C-c
 C-t C-p</kbd> (note that this will load **all** namespaces in your
-project). Using <kbd>C-c C-t t</kbd> or <kbd>C-c C-t C-t</kbd>, you can execute
-only the test at point.
+project). If you call these two with a prefix CIDER will ask for a test selector
+and only run those tests in the project which are marked with that selector. Using
+<kbd>C-c C-t t</kbd> or <kbd>C-c C-t C-t</kbd>, you can execute only the
+test at point.
+
+Test selector is originally a `leiningen` feature see `lein help test` for some explanation.
+People use them to define subsets of tests usually run together for different purposes. For
+example you can mark some of your tests with the `^:smoke` metadata marker others with
+`^:integration`. This enables you to run these tests separately in your build pipeline.
+This feature in CIDER helps you to run these subsets in your development environment.
 
 All test commands are available in REPL buffers as well. There you can also use
 <kbd>,</kbd> to invoke some of the testing commands.
@@ -30,6 +42,7 @@ Keyboard shortcut               | Description
 --------------------------------|-------------------------------
 <kbd>g</kbd>                    | Run test at point.
 <kbd>n</kbd>                    | Run tests for current namespace.
+<kbd>s</kbd>                    | Run tests for current namespace prefiltered with a selector.
 <kbd>l</kbd>                    | Run tests for all loaded namespaces.
 <kbd>p</kbd>                    | Run tests for all project namespaces. This loads the additional namespaces.
 <kbd>f</kbd>                    | Re-run test failures/errors.