`npm test` is (obviously) an aggregate of tests, but the result seems to be all or nothing (not a percentage)

[fuhrmanator]

I'm trying to figure out how to easily use testing in node without having to re-copy my test config to GitHub classroom. That is, it seems that using npm test (which normally runs all tests for a project) will result in an all or nothing grade, rather than some percentage.

Here's an example (Jest is the test framework):

We see that 46/48 tests passed, but since the npm test command (aggregate) failed, it's 0/1 tests passed in GitHub Classroom.

Of course, I could re-enter the 48 individual tests into GitHub classroom (instead of using the npm test aggregate), but who wants to do all that clicking?

Since there's a YAML feature, I asked CoPilot to write a shell script (bash) to generate the YAML based on the tests that jest finds (it's dynamic). It also tries to determine the number of tests inside each test file to adjust the max-points for each test (it's still no granular enough, but it's better than a big 0 or 1 for npm test). You don't need to use a test pattern, but I did (because my project has tests that already pass before my students start, and for which they don't get points).

I haven't fully tested this (because my course is live), but I will try it later. ETA: the first version didn't have the correct test command (fixed it)

#!/bin/bash

# Start of the YAML file
echo "# This file was generated by a bash script
name: Autograding Tests
'on':
- push
- workflow_dispatch
- repository_dispatch
permissions:
  checks: write
  actions: read
  contents: read
jobs:
  run-autograding-tests:
    runs-on: ubuntu-latest
    if: github.actor != 'github-classroom[bot]'
    steps:
    - name: Checkout code
      uses: actions/checkout@v4" > tests.yml

# Initialize total test count
total_test_count=0

# Flag to track if it's the first test
first_test=true

# For each test file
for testfile in $(npx jest --listTests --testPathPattern=lab0.test.ts); do
  # Extract the test name from the file path
  testname=$(basename $testfile .ts)

  # Count the number of tests in the file
  test_count=$(grep -Eo '(it|test)\s*\(' $testfile | wc -l)

  # If it's the first test, include the setup command
  if $first_test ; then
    echo "
    - name: $testname
      id: $testname
      uses: classroom-resources/autograding-command-grader@v1
      with:
        test-name: $testname
        setup-command: npm ci
        command: npx jest $testfile
        timeout: 10
        max-score: $test_count" >> tests.yml
    first_test=false
  else
    # If it's not the first test, don't include the setup command
    echo "
    - name: $testname
      id: $testname
      uses: classroom-resources/autograding-command-grader@v1
      with:
        test-name: $testname
        command: npx jest $testfile
        timeout: 10
        max-score: $test_count" >> tests.yml
  fi

  # Add to total test count
  ((total_test_count+=test_count))
done

# Output the estimated number of test blocks found
echo "Estimated number of test blocks found: $total_test_count"

[sierrajulietromeo]

I have a similar issue, and I wrote about it here on GitHub community.