Sort globbed tests for deterministic execution

[Hellenpac]

Describe the bug
Otter run returns grades for questions in a random order.
To Reproduce
Steps to reproduce the behavior:
On terminal, run: otter run -a autograder.zip assignment.ipynb

Expected behavior
Expected an ordered list of questions with their respective grades, like it used to be.
Versions
Python 3.13.5 and Otter v6.1.3

Additional context
Here is an example:
name score max_score
0 Public Tests NaN NaN
1 q3_2_3 3.0 3.0
2 q3_1_4 3.0 3.0
3 q3_3_2 3.0 3.0
4 q1_1_1 3.0 3.0
5 q3_1_1 3.0 3.0
6 q1_0 3.0 3.0
7 q1_1_4 3.0 3.0
8 q1_2_3 3.0 3.0
9 q3_2_2 3.0 3.0
10 q3_1_5 3.0 3.0
11 q2_1_2 3.0 3.0
12 q2_1_3 3.0 3.0
13 q3_1_8 3.0 3.0
14 q2_1_4 3.0 3.0
15 q2_1_5 3.0 3.0
16 q3_1_9 2.0 2.0
17 q4_1 0.0 3.0
18 q2_1_1 3.0 3.0
19 q1_1_3 3.0 3.0
20 q3_1_2 3.0 3.0
21 q3_2_1 3.0 3.0
22 q3_1_6 3.0 3.0
23 q1_2_1 3.0 3.0
24 q3_0 3.0 3.0
25 q3_3_1 3.0 3.0
26 q1_1_2 3.0 3.0
27 q3_1_3 3.0 3.0

[chrispyles]

Is this dataframe coming from otter run or otter grade? IIRC otter run doesn't print a dataframe. In either case, can you please send full steps to reproduce this dataframe? Feel free to PM me an autograder zip and submission file in the Otter Slack (link in README) if you don't want to post them publicly.

[Hellenpac]

It is coming from otter run.

project3-autograder.zip

project3partner12_344776_15096284_project3.ipynb

Thank you, --Hellen

[chrispyles]

The ordering of the results listed is stable so I do not think it is random. The results are intended to be listed in the order that each test is run, so it would seem that these are not being run in the intended order. The reason why this is happening is because the file is expected to be named project3.ipynb in the notebook but it has a different name:

When this happens, Otter is able to fall back to finding the tests manually, but they get run all at once at the end of executing the notebook (and in the order they are returned by glob.glob, hence the apparently-random ordering) instead of at the correct time indicated by grader.check calls.

I can update Otter to sort the tests when they get globbed, but the filename still should not be changed from what Otter expects it to be, as this can have difficult-to-debug side effects.

[chrispyles]

here is the glob that needs to be sorted.