Skip to content

Print all test case error messages together at the end #745

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 11 commits into from
Nov 30, 2024
Merged

Print all test case error messages together at the end #745

Show file tree
Hide file tree
Changes from 7 commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
af57648
fix #744
wilfonba Nov 24, 2024
22c8646
small formatting change
wilfonba Nov 24, 2024
059176c
add skipped cases
wilfonba Nov 24, 2024
c7f8e17
lint toolchain
wilfonba Nov 24, 2024
ea46a25
small fix
wilfonba Nov 24, 2024
2e9722d
Merge remote-tracking branch 'upstream/master' into testCases
wilfonba Nov 24, 2024
7a4c000
fix bug?
wilfonba Nov 24, 2024
1eda45a
passes locally with --no-mpi
wilfonba Nov 24, 2024
aeeb29a
remove junk file
wilfonba Nov 24, 2024
c28bbdf
Merge remote-tracking branch 'upstream/master' into testCases
wilfonba Nov 26, 2024
ab90416
Merge branch 'master' into testCases
sbryngelson Nov 30, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
237 changes: 237 additions & 0 deletions test4.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,237 @@
mfc: OK > (venv) Entered the Python 3.12.6 virtual environment (>= 3.9).

.=++*: -+*+=. | [email protected] [Darwin]
:+ -*- == =* . | --------------------------------------------
:*+ == ++ .+- |
:*##-.....:*+ .#%+++=--+=:::. | --jobs 8
-=-++-======#=--**+++==+*++=::-:. | --mpi --no-gpu --no-debug --no-gcov --no-unified
.:++=----------====+*= ==..:%..... |
.:-=++++===--==+=-+= +. := |
+#=::::::::=%=. -+: =+ *: | ----------------------------------------------------------
.*=-=*=.. :=+*+: -...-- | $ ./mfc.sh (build, run, test, clean, count, packer) --help

Build | syscheck and pre_process | Generic Build

Generating case.fpp.
Writing a (new) custom case.fpp file.
$ cmake --build /Users/benwilfong/Documents/software/MFC-Wilfong/build/staging/7d9b728a37 --target syscheck --parallel 8 --config Release

[100%] Built target syscheck

$ cmake --install /Users/benwilfong/Documents/software/MFC-Wilfong/build/staging/7d9b728a37

-- Install configuration: "Release"
-- Up-to-date: /Users/benwilfong/Documents/software/MFC-Wilfong/build/install/7d9b728a37/bin/syscheck

Generating case.fpp.
Writing a (new) custom case.fpp file.
$ cmake --build /Users/benwilfong/Documents/software/MFC-Wilfong/build/staging/9a4af0a3bd --target pre_process --parallel 8 --config Release

[100%] Built target pre_process

$ cmake --install /Users/benwilfong/Documents/software/MFC-Wilfong/build/staging/9a4af0a3bd

-- Install configuration: "Release"
-- Up-to-date: /Users/benwilfong/Documents/software/MFC-Wilfong/build/install/9a4af0a3bd/bin/pre_process

Build | syscheck and simulation | Generic Build

Generating case.fpp.
Writing a (new) custom case.fpp file.
$ cmake --build /Users/benwilfong/Documents/software/MFC-Wilfong/build/staging/7d9b728a37 --target syscheck --parallel 8 --config Release

[100%] Built target syscheck

$ cmake --install /Users/benwilfong/Documents/software/MFC-Wilfong/build/staging/7d9b728a37

-- Install configuration: "Release"
-- Up-to-date: /Users/benwilfong/Documents/software/MFC-Wilfong/build/install/7d9b728a37/bin/syscheck

Generating case.fpp.
Writing a (new) custom case.fpp file.
$ cmake --build /Users/benwilfong/Documents/software/MFC-Wilfong/build/staging/98998883b5 --target simulation --parallel 8 --config Release

[100%] Built target simulation

$ cmake --install /Users/benwilfong/Documents/software/MFC-Wilfong/build/staging/98998883b5

-- Install configuration: "Release"
-- Up-to-date: /Users/benwilfong/Documents/software/MFC-Wilfong/build/install/98998883b5/bin/simulation

Build | syscheck and pre_process | Generic Build

Generating case.fpp.
Writing a (new) custom case.fpp file.
$ cmake --build /Users/benwilfong/Documents/software/MFC-Wilfong/build/staging/9bf8c42b52 --target syscheck --parallel 8 --config Release

[100%] Built target syscheck

$ cmake --install /Users/benwilfong/Documents/software/MFC-Wilfong/build/staging/9bf8c42b52

-- Install configuration: "Release"
-- Up-to-date: /Users/benwilfong/Documents/software/MFC-Wilfong/build/install/9bf8c42b52/bin/syscheck

Generating case.fpp.
Writing a (new) custom case.fpp file.
$ cmake --build /Users/benwilfong/Documents/software/MFC-Wilfong/build/staging/617a1665e9 --target pre_process --parallel 8 --config Release

[100%] Built target pre_process

$ cmake --install /Users/benwilfong/Documents/software/MFC-Wilfong/build/staging/617a1665e9

-- Install configuration: "Release"
-- Up-to-date: /Users/benwilfong/Documents/software/MFC-Wilfong/build/install/617a1665e9/bin/pre_process

Build | syscheck and simulation | Generic Build

Generating case.fpp.
Writing a (new) custom case.fpp file.
$ cmake --build /Users/benwilfong/Documents/software/MFC-Wilfong/build/staging/9bf8c42b52 --target syscheck --parallel 8 --config Release

[100%] Built target syscheck

$ cmake --install /Users/benwilfong/Documents/software/MFC-Wilfong/build/staging/9bf8c42b52

-- Install configuration: "Release"
-- Up-to-date: /Users/benwilfong/Documents/software/MFC-Wilfong/build/install/9bf8c42b52/bin/syscheck

Generating case.fpp.
Writing a (new) custom case.fpp file.
$ cmake --build /Users/benwilfong/Documents/software/MFC-Wilfong/build/staging/9216688c41 --target simulation --parallel 8 --config Release

[100%] Built target simulation

$ cmake --install /Users/benwilfong/Documents/software/MFC-Wilfong/build/staging/9216688c41

-- Install configuration: "Release"
-- Up-to-date: /Users/benwilfong/Documents/software/MFC-Wilfong/build/install/9216688c41/bin/simulation


Test pre_process and simulation | Only 1D (79 tests)

tests/UUID (s) Summary

Failed test tests/2A5CFF02: 1D -> weno_order=3 after 1 attempt(s).
0FF51EDE 1.88 1D -> Acoustic Source -> Square -> Wavelength
32D0F235 1.88 1D -> 1 Fluid(s) -> riemann_solver=1 -> wave_speeds=2
F97573DB 1.89 1D -> 1 Fluid(s) -> riemann_solver=2 -> avg_state=1
A83CADB5 1.89 1D -> Phase Change model 5 -> 2 Fluid(s)
5DCF300C 1.89 1D -> Chemistry -> Perfect Reactor
63F2D0C0 1.92 1D -> Bubbles -> Polytropic -> bubble_model=2
5527832F 1.53 1D -> 1 Fluid(s) -> riemann_solver=1 -> mixture_err
677DAA82 1.50 1D -> weno_order=5 -> mapped_weno=T
5EC236F2 1.53 1D -> bc=-6
3A8359F6 1.58 1D -> 2 Fluid(s) -> riemann_solver=2 -> mixture_err
AED93D34 1.53 1D -> bc=-8
18BDCBC8 1.59 1D -> 1 Fluid(s) -> riemann_solver=2 -> mixture_err
9090FC4B 1.50 1D -> weno_order=5 -> wenoz=T
9EF19F0A 1.42 1D -> Phase Change model 5 -> 3 Fluid(s)
Failed test tests/F9850EA5: 1D -> weno_order=3 -> wenoz=T after 1 attempt(s).
3643454B 1.51 1D -> Bubbles -> Polytropic -> bubble_model=3
66693DCC 1.43 1D -> Acoustic Source -> Gaussian -> Dipole
BDE56A04 1.49 1D -> 2 Fluid(s) -> low_Mach=2
8E3D99E6 1.44 1D -> 1 Fluid(s) -> Viscous -> weno_Re_flux
105673CA 1.48 1D -> Bubbles -> adv_n=T
2AB32975 1.41 1D -> bc=-3
A60691E7 1.44 1D -> bc=-11
C5B79059 1.43 1D -> bc=-9
5E2265C4 1.56 1D -> Viscosity -> Bubbles -> Polytropic -> bubble_model=3
727F72ED 1.41 1D -> bc=-10
3053B44F 1.39 1D -> Acoustic Source -> Gaussian -> Sigma Dist
6B3AE48F 1.81 1D -> Chemistry -> Inert Shocktube -> Riemann Solver 2 -> Gamma Method 1
33813627 1.34 1D -> Acoustic Source -> Sine -> Wavelength
AE02324F 1.43 1D -> Bodyforces -> cfl_adap_dt=T
2A6136EF 1.42 1D -> 2 Fluid(s) -> Viscous -> weno_Re_flux -> weno_avg
11DAFD23 1.48 1D -> Bubbles -> nb=1
6C50C62A 1.44 1D -> 1 Fluid(s) -> low_Mach=2
D79C3E6F 1.39 1D -> bc=-1
729A4333 1.37 1D -> Viscosity -> Bubbles -> nb=1
4AEF478A 1.41 1D -> 1 Fluid(s) -> riemann_solver=1 -> avg_state=1
DEA0A0D6 1.34 1D -> Bodyforces
2DE548B7 1.31 1D -> Acoustic Source -> Square -> Number of Pulses
9DAC4DDC 1.40 1D -> 2 Fluid(s) -> riemann_solver=2 -> alt_soundspeed
02748F0F 1.35 1D -> 1 Fluid(s) -> Viscous -> weno_Re_flux -> weno_avg
3B3F8C68 1.58 1D -> Bubbles -> adv_n=T -> adap_dt=T
4D7D85B4 1.73 1D -> Chemistry -> Inert Shocktube -> Riemann Solver 1 -> Gamma Method 1
EE34D7DC 1.42 1D -> Acoustic Source -> Gaussian -> Sigma Time
2F58A83F 1.37 1D -> Acoustic Source -> Square -> Frequency
83EFC30C 1.40 1D -> 2 Fluid(s) -> riemann_solver=1 -> wave_speeds=2
7789B55A 1.38 1D -> weno_order=5
5DAF1F94 1.43 1D -> Acoustic Source -> Square -> Delay
48D5C130 1.39 1D -> weno_order=5 -> teno=T
1E738705 1.39 1D -> 2 Fluid(s) -> riemann_solver=1 -> mixture_err
ED728400 1.43 1D -> bc=-16
356A5C50 3.44 1D -> Bubbles -> QBMM -> Non-polytropic
461DCB09 1.49 1D -> 2 Fluid(s) -> riemann_solver=2 -> wave_speeds=2
9465D0F5 1.43 1D -> Viscosity -> Bubbles -> Polytropic -> bubble_model=2
3FC6FC4A 1.45 1D -> bc=-12
3AE495F4 1.37 1D -> bc=-5
F1E9F8BC 1.44 1D -> Acoustic Source -> Sine -> Delay
53E13D19 1.64 1D -> Viscosity -> Bubbles -> QBMM -> bubble_model=3
BE0B2040 1.56 1D -> Bubbles -> artificial_Ma
CE9DBA3F 3.35 1D -> Bubbles -> QBMM
2FD933A2 1.47 1D -> bc=-15
5576FEEA 2.25 1D -> Bubbles -> QBMM -> bubble_model=3 -> Non-polytropic
5D68B0F5 1.41 1D -> weno_order=7 -> mapped_weno=T
6784C02E 2.04 1D -> Viscosity -> Bubbles -> QBMM
4927BF4B 1.46 1D -> Bubbles -> low_Mach=2
23A75A10 1.50 1D -> 2 Fluid(s) -> low_Mach=1
0347811A 1.49 1D -> weno_order=7 -> wenoz=T
1CCA82F5 1.45 1D -> 2 Fluid(s) -> riemann_solver=1 -> mpp_lim
0879E062 1.43 1D -> 2 Fluid(s) -> riemann_solver=1 -> avg_state=1
1B0A7157 1.40 1D -> Acoustic Source -> Sine -> Frequency
B3E70A3A 1.44 1D -> weno_order=5 -> mp_weno=T
6D24B115 1.42 1D -> 2 Fluid(s) -> riemann_solver=2 -> avg_state=1
2F35A1FE 1.40 1D -> 1 Fluid(s) -> riemann_solver=2 -> model_eqns=3
37FA2CEF 1.43 1D -> Viscosity -> Bubbles -> QBMM -> bubble_model=3 -> cfl_const_dt=T
2BDE2018 1.75 1D -> Chemistry -> Inert Shocktube -> Riemann Solver 1 -> Gamma Method 2
1E288DE4 1.89 1D -> Bubbles -> QBMM -> bubble_model=3
C4907722 1.07 1D -> 2 Fluid(s) -> riemann_solver=2 -> mpp_lim
F8ADA51B 1.35 1D -> Chemistry -> Inert Shocktube -> Riemann Solver 2 -> Gamma Method 2
842C6FFC 21.51 1D -> Phase Change model 6 -> 2 Fluid(s)
786DE444 15.34 1D -> Phase Change model 6 -> 3 Fluid(s)


Test Summary: 77 passed, 2 failed, 0 skipped.

Failed Cases

Failed test tests/2A5CFF02: 1D -> weno_order=3 after 1 attempt(s).
Test tests/2A5CFF02: 1D -> weno_order=3: Variable n°1 (1-indexed) in D/cons.1.00.000050.dat is not within tolerance:
- Candidate: 0.99999999776016
- Golden: 1.00000000000027
- Error: abs: 2.24E-09, rel: 2.24E-09
- Tolerance: abs: 1.00E-12, rel: 1.00E-12

Failed test tests/F9850EA5: 1D -> weno_order=3 -> wenoz=T after 1 attempt(s).
Test tests/F9850EA5: 1D -> weno_order=3 -> wenoz=T: Variable n°1 (1-indexed) in D/cons.1.00.000050.dat is not within tolerance:
- Candidate: 0.99999999851747
- Golden: 1.00000000000049
- Error: abs: 1.48E-09, rel: 1.48E-09
- Tolerance: abs: 1.00E-12, rel: 1.00E-12

Skipped Cases

1D -> bc=-2
1D -> bc=-4
1D -> bc=-7
1D -> weno_order=3 -> mapped_weno=T
1D -> weno_order=7
1D -> weno_order=7 -> teno=T
1D -> 1 Fluid(s) -> riemann_solver=2 -> wave_speeds=2
1D -> 1 Fluid(s) -> low_Mach=1
1D -> 1 Fluid(s) -> Viscous
1D -> 2 Fluid(s) -> riemann_solver=2 -> model_eqns=3
1D -> 2 Fluid(s) -> Viscous
1D -> 2 Fluid(s) -> Viscous -> weno_Re_flux
1D -> 2 MPI Ranks
1D -> Acoustic Source -> Sine -> Number of Pulses
1D -> Bubbles -> bubble_model=2
1D -> Bubbles -> low_Mach=1
1D -> Hypoelasticity -> 1 Fluid(s)
1D -> Hypoelasticity -> 2 Fluid(s)
1D -> Viscosity -> Bubbles -> bubble_model=2
1D -> Viscosity -> Bubbles -> QBMM -> bubble_model=3 -> cfl_adap_dt=T

mfc: ERROR > main.py finished with a 2 exit code.
mfc: (venv) Exiting the Python virtual environment.
34 changes: 29 additions & 5 deletions toolchain/mfc/test/test.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,9 +19,12 @@
nFAIL = 0
nPASS = 0
nSKIP = 0
errors = []

def __filter(cases_) -> typing.List[TestCase]:
cases = cases_[:]
selected_cases = []
skipped_cases = []

# Check "--from" and "--to" exist and are in the right order
bFoundFrom, bFoundTo = (False, False)
Expand Down Expand Up @@ -53,14 +56,17 @@ def __filter(cases_) -> typing.List[TestCase]:
cases.remove(case)

if ARG("percent") == 100:
return cases
return cases, skipped_cases

return sample(cases, k=int(len(cases)*ARG("percent")/100.0))
selected_cases = sample(cases, k=int(len(cases)*ARG("percent")/100.0))
skipped_cases = [item for item in cases if item not in selected_cases]

return selected_cases, skipped_cases

def test():
# pylint: disable=global-statement, global-variable-not-assigned
global nFAIL, nPASS, nSKIP
global errors

cases = list_cases()

Expand All @@ -75,7 +81,8 @@ def test():

return

cases = [ _.to_case() for _ in __filter(cases) ]
cases, skipped_cases = __filter(cases)
cases = [ _.to_case() for _ in cases ]

if ARG("list"):
table = rich.table.Table(title="MFC Test Cases", box=rich.table.box.SIMPLE)
Expand Down Expand Up @@ -125,9 +132,23 @@ def test():
[ sched.Task(ppn=case.ppn, func=handle_case, args=[case], load=case.get_cell_count()) for case in cases ],
ARG("jobs"), ARG("gpus"))

nSKIP = len(skipped_cases)
cons.print()
cons.unindent()
cons.print(f"\nTest Summary: [bold green]{nPASS}[/bold green] passed, [bold red]{nFAIL}[/bold red] failed, [bold yellow]{nSKIP}[/bold yellow] skipped.")
cons.print(f"\nTest Summary: [bold green]{nPASS}[/bold green] passed, [bold red]{nFAIL}[/bold red] failed, [bold yellow]{nSKIP}[/bold yellow] skipped.\n")

# Print a summary of all errors at the end if errors exist
if len(errors) != 0:
cons.print(f"[bold red]Failed Cases[/bold red]\n")
for e in errors:
cons.print(e)

# Print the list of skipped cases
if len(skipped_cases) != 0:
cons.print("[bold yellow]Skipped Cases[/bold yellow]\n")
for c in skipped_cases:
cons.print(f"[bold yellow]{c.trace}[/bold yellow]")

exit(nFAIL)


Expand Down Expand Up @@ -216,6 +237,7 @@ def _handle_case(case: TestCase, devices: typing.Set[int]):
def handle_case(case: TestCase, devices: typing.Set[int]):
# pylint: disable=global-statement, global-variable-not-assigned
global nFAIL, nPASS, nSKIP
global errors

nAttempts = 0

Expand All @@ -228,9 +250,11 @@ def handle_case(case: TestCase, devices: typing.Set[int]):
except Exception as exc:
if nAttempts < ARG("max_attempts"):
cons.print(f"[bold yellow] Attempt {nAttempts}: Failed test {case.get_uuid()}. Retrying...[/bold yellow]")
errors.append(f"[bold yellow] Attempt {nAttempts}: Failed test {case.get_uuid()}. Retrying...[/bold yellow]")
continue
nFAIL += 1
cons.print(f"[bold red]Failed test {case} after {nAttempts} attempt(s).[/bold red]")
cons.print(f"{exc}")
errors.append(f"[bold red]Failed test {case} after {nAttempts} attempt(s).[/bold red]")
errors.append(f"{exc}")

return
Loading