add a test driver script by kilograham · Pull Request #2850 · raspberrypi/pico-sdk

kilograham · 2026-03-05T21:19:25Z

No description provided.

will-v-pi · 2026-03-06T09:40:39Z

tools/run_test_matrix.py

+            # ====================== START MINICOM EARLY FOR FULL UART CAPTURE ======================
+            print(f"   Starting minicom capture early on {args.serial_port} @ {args.baud} baud...")


Suggested change

# ====================== START MINICOM EARLY FOR FULL UART CAPTURE ======================

print(f" Starting minicom capture early on {args.serial_port} @ {args.baud} baud...")

# ====================== START GRABSERIAL EARLY FOR FULL UART CAPTURE ======================

print(f" Starting grabserial capture early on {args.serial_port} @ {args.baud} baud...")

will-v-pi · 2026-03-06T09:41:02Z

tools/run_test_matrix.py

+                )
+                print(f"   grabserial started (PID {logger_proc.pid}) — capturing from now on")
+
+                # Give minicom a moment to open the port and settle


Suggested change

# Give minicom a moment to open the port and settle

# Give grabserial a moment to open the port and settle

will-v-pi · 2026-03-06T09:57:12Z

tools/run_test_matrix.py

+
+                gdb_cmd = [
+                    args.gdb, "--batch", "--nx", full_path,
+                    "-ex", "target extended-remote localhost:3333",


Could you add an argument to support passing a different OpenOCD server, eg

Suggested change

"-ex", "target extended-remote localhost:3333",

"-ex", f"target extended-remote {args.openocd_server}",

peterharperuk

I like the idea of this. I suggest using it is the best way to find issues.

The pico test utility still generates passes after some failures have occurred which is a bit annoying. I should push some fixes.

lurch · 2026-03-06T15:36:42Z

tools/run_test_matrix.py

+import signal
+
+def main():
+    parser = argparse.ArgumentParser(description="RP2350 ELF test runner with OpenOCD + pyserial UART capture")


Suggested change

parser = argparse.ArgumentParser(description="RP2350 ELF test runner with OpenOCD + pyserial UART capture")

parser = argparse.ArgumentParser(description="RP2xxx ELF test runner with OpenOCD + grabserial UART capture")

I guess there's no reason why this wouldn't work with RP2040 too?

lurch · 2026-03-06T15:39:30Z

tools/run_test_matrix.py

+        custom_float_funcs_test_pico_vfp.elf \
+    --path-postfix \
+       test/pico_float_test \
+    -- out_dir


Might be worth splitting this into

Suggested change

-- out_dir

--

out_dir

so that it look less like a typo of --out_dir ?

lurch · 2026-03-06T15:49:07Z

tools/run_test_matrix.py

+
+    for seg in args.path_segments:
+        for elf in args.elf_filenames:
+            full_path = f"{args.path_prefix}/{seg}/{args.path_postfix}/{elf}"


Suggested change

full_path = f"{args.path_prefix}/{seg}/{args.path_postfix}/{elf}"

full_path = os.path.join(args.path_prefix, seg, args.path_postfix, elf)

--path-postfix is an optional arg, so using os.path.join means that you won't end up with // in the path.

lurch · 2026-03-06T15:53:15Z

tools/run_test_matrix.py

+
+    args = parser.parse_args()
+
+    os.makedirs(args.output_dir, exist_ok=True)


Should it delete and then recreate output_dir, so that we don't get into a confusing state of output_dir containing both "old logs" and "new logs"? (You can use https://docs.python.org/3/library/shutil.html#shutil.rmtree for that)

lurch · 2026-03-06T16:04:11Z

tools/run_test_matrix.py

+                continue
+
+            attempted += 1
+            sanitized_out_file = f"{seg}.{elf}.out".replace("/",".")


The .replace("/",".") doesn't seem necessary, because the help-text for --path-segments says "(no slashes)" ? (Or does the help-text need updating?)

Ahhh. but the examples at the top of this file shows that a slash is allowed in the ELF-filenames.

lurch · 2026-03-06T16:11:40Z

tools/run_test_matrix.py

+Which will run the following if they exist, storing the output in out_dir/
+   gcc_build/test/pico_float_test/custom_float_funcs_test_compiler.elf
+   gcc_build/test/pico_float_test/custom_float_funcs_test_pico_dcp.elf
+   gcc_build/test/pico_float_test/custom_float_funcs_test_pico.elf
+   gcc_build/test/pico_float_test/custom_float_funcs_test_pico_vfp.elf
+   clang_build/test/pico_float_test/custom_float_funcs_test_compiler.elf
+   clang_build/test/pico_float_test/custom_float_funcs_test_pico_dcp.elf
+   clang_build/test/pico_float_test/custom_float_funcs_test_pico.elf
+   clang_build/test/pico_float_test/custom_float_funcs_test_pico_vfp.elf


Might be worth clarify the style of the output filenames?

Suggested change

Which will run the following if they exist, storing the output in out_dir/

gcc_build/test/pico_float_test/custom_float_funcs_test_compiler.elf

gcc_build/test/pico_float_test/custom_float_funcs_test_pico_dcp.elf

gcc_build/test/pico_float_test/custom_float_funcs_test_pico.elf

gcc_build/test/pico_float_test/custom_float_funcs_test_pico_vfp.elf

clang_build/test/pico_float_test/custom_float_funcs_test_compiler.elf

clang_build/test/pico_float_test/custom_float_funcs_test_pico_dcp.elf

clang_build/test/pico_float_test/custom_float_funcs_test_pico.elf

clang_build/test/pico_float_test/custom_float_funcs_test_pico_vfp.elf

Which will run each of the following if they exist:

gcc_build/test/pico_float_test/custom_float_funcs_test_compiler.elf

gcc_build/test/pico_float_test/custom_float_funcs_test_pico_dcp.elf

gcc_build/test/pico_float_test/custom_float_funcs_test_pico.elf

gcc_build/test/pico_float_test/custom_float_funcs_test_pico_vfp.elf

clang_build/test/pico_float_test/custom_float_funcs_test_compiler.elf

clang_build/test/pico_float_test/custom_float_funcs_test_pico_dcp.elf

clang_build/test/pico_float_test/custom_float_funcs_test_pico.elf

clang_build/test/pico_float_test/custom_float_funcs_test_pico_vfp.elf

storing their UART output in:

out_dir/gcc_build.custom_float_funcs_test_compiler.elf.out

out_dir/gcc_build.custom_float_funcs_test_pico_dcp.elf.out

out_dir/gcc_build.custom_float_funcs_test_pico.elf.out

out_dir/gcc_build.custom_float_funcs_test_pico_vfp.elf.out

out_dir/clang_build.custom_float_funcs_test_compiler.elf.out

out_dir/clang_build.custom_float_funcs_test_pico_dcp.elf.out

out_dir/clang_build.custom_float_funcs_test_pico.elf.out

out_dir/clang_build.custom_float_funcs_test_pico_vfp.elf.out

lurch · 2026-03-06T16:15:47Z

tools/run_test_matrix.py

+   clang_build/test/pico_float_test/custom_float_funcs_test_pico.elf
+   clang_build/test/pico_float_test/custom_float_funcs_test_pico_vfp.elf
+
+This would run the same tests:


Suggested change

This would run the same tests:

This would run the same tests:

(but store the output in e.g out_dir/gcc_build.pico_float_test.custom_float_funcs_test_compiler.elf.out )

lurch · 2026-03-06T17:35:29Z

tools/run_test_matrix.py

+                gdb_result = subprocess.run(
+                    gdb_cmd,
+                    check=False,
+                    timeout=60,


Maybe it's worth adding an optional-argument for this timeout, in case you have a particularly long-running ELF file? 🤔

lurch · 2026-03-06T17:37:36Z

tools/run_test_matrix.py

+                        except:
+                            pass
+
+            # ====================== CHECK LAST LINE ======================


Suggested change

# ====================== CHECK LAST LINE ======================

# ====================== CHECK WE CAPTURED SOME OUTPUT ======================

lurch · 2026-03-06T17:40:05Z

tools/run_test_matrix.py

+                print(f"Error reading output file: {e}")
+
+            print("="*60 + "\n")
+            try:


Suggested change

try:

# ====================== CHECK LAST LINE ======================

try:

lurch · 2026-03-06T17:41:34Z

tools/run_test_matrix.py

+
+            print("="*60 + "\n")
+            try:
+                with open(outfile, "r", encoding="utf-8", errors="ignore") as f:


No ned to open outfile as second time - you could just look at the last line of captured_content ?

lurch · 2026-03-06T17:44:49Z

tools/run_test_matrix.py

+    if attempted == 0:
+        print("No ELFs were found to test.")
+        sys.exit(0)
+
+    if succeeded == attempted:
+        print("All tests passed!")
+        sys.exit(0)
+    else:
+        print("Some tests failed.")
+        sys.exit(1)


I guess this could be shortened to:

Suggested change

if attempted == 0:

print("No ELFs were found to test.")

sys.exit(0)

if succeeded == attempted:

print("All tests passed!")

sys.exit(0)

else:

print("Some tests failed.")

sys.exit(1)

if attempted == 0:

print("No ELFs were found to test.")

elif succeeded == attempted:

print("All tests passed!")

else:

print("Some tests failed.")

sys.exit(1)

add a test driver script

824d9cb

kilograham added this to the 2.2.1 milestone Mar 5, 2026

kilograham requested a review from peterharperuk March 5, 2026 21:19

kilograham added 3 commits March 5, 2026 15:20

oops; pyserial no longer needed

519de70

clean up output slightly

b425b52

clean up output slightly

f8e6757

kilograham requested review from lurch and will-v-pi March 5, 2026 21:25

will-v-pi reviewed Mar 6, 2026

View reviewed changes

peterharperuk approved these changes Mar 6, 2026

View reviewed changes

lurch reviewed Mar 6, 2026

View reviewed changes

		# ====================== START MINICOM EARLY FOR FULL UART CAPTURE ======================
		print(f" Starting minicom capture early on {args.serial_port} @ {args.baud} baud...")

	# Give minicom a moment to open the port and settle
	# Give grabserial a moment to open the port and settle

	"-ex", "target extended-remote localhost:3333",
	"-ex", f"target extended-remote {args.openocd_server}",

	parser = argparse.ArgumentParser(description="RP2350 ELF test runner with OpenOCD + pyserial UART capture")
	parser = argparse.ArgumentParser(description="RP2xxx ELF test runner with OpenOCD + grabserial UART capture")

	full_path = f"{args.path_prefix}/{seg}/{args.path_postfix}/{elf}"
	full_path = os.path.join(args.path_prefix, seg, args.path_postfix, elf)


		args = parser.parse_args()

		os.makedirs(args.output_dir, exist_ok=True)

	This would run the same tests:
	This would run the same tests:
	(but store the output in e.g out_dir/gcc_build.pico_float_test.custom_float_funcs_test_compiler.elf.out )

	# ====================== CHECK LAST LINE ======================
	# ====================== CHECK WE CAPTURED SOME OUTPUT ======================

	try:
	# ====================== CHECK LAST LINE ======================
	try:

Conversation

kilograham commented Mar 5, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

peterharperuk left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants