WIP: experiment generating kinds in parallel (multiprocess) #717
firefoxci-taskcluster / test-unit-py310
failed
Jul 4, 2025 in 2m 58s
FirefoxCI (pull_request)
Run unit tests with py310 (Treeherder push)
Details
View task in Taskcluster | View logs in Taskcluster | View task group in Taskcluster
Task Status
Started: 2025-07-04T14:11:16.803Z
Resolved: 2025-07-04T14:12:29.760Z
Task Execution Time: 1 minute, 12 seconds, 957 milliseconds
Task Status: failed
Reason Resolved: failed
RunId: 0
Artifacts
- public/coverage.py310
- public/logs/live_backing.log
- public/logs/live.log
[taskcluster 2025-07-04 14:11:16.838Z] Task ID: NvQZdiUhSgShbWomcO1VsQ
[taskcluster 2025-07-04 14:11:16.838Z] Worker ID: 4178164250407755339
[taskcluster 2025-07-04 14:11:16.838Z] Worker Group: us-west1-b
[taskcluster 2025-07-04 14:11:16.838Z] Worker Node Type: projects/887720501152/machineTypes/c2-standard-4
[taskcluster 2025-07-04 14:11:16.838Z] Worker Pool: taskgraph-t/linux-gcp
[taskcluster 2025-07-04 14:11:16.838Z] Worker Version: 38.0.5
[taskcluster 2025-07-04 14:11:16.838Z] Public IP: 35.185.196.49
[taskcluster 2025-07-04 14:11:16.838Z] Hostname: taskgraph-t-linux-gcp-nn82cb5sttmvustnehxrpw
[taskcluster 2025-07-04 14:11:16.838Z] using cache "taskgraph-level-1-uv-v3-f59e202b9f4c189775df-LUZHGlaCTaeKGo37LLFKnw" -> /builds/worker/.task-cache/uv
[taskcluster 2025-07-04 14:11:16.838Z] using cache "taskgraph-level-1-checkouts-v3-f59e202b9f4c189775df-LUZHGlaCTaeKGo37LLFKnw" -> /builds/worker/checkouts
[taskcluster 2025-07-04 14:11:20.104Z] Downloading artifact "public/image.tar.zst" from task ID: LUZHGlaCTaeKGo37LLFKnw.
[taskcluster 2025-07-04 14:11:25.106Z] Download Progress: 78.33%
[taskcluster 2025-07-04 14:11:26.599Z] Downloaded artifact successfully.
[taskcluster 2025-07-04 14:11:26.599Z] Downloaded 255.333 mb
[taskcluster 2025-07-04 14:11:26.600Z] Decompressing downloaded image
[taskcluster 2025-07-04 14:11:28.751Z] Loading docker image from downloaded archive.
[taskcluster 2025-07-04 14:11:49.704Z] Image 'public/image.tar.zst' from task 'LUZHGlaCTaeKGo37LLFKnw' loaded. Using image ID sha256:d155ada461c10aa23879a984fb5ba8741435bd7d5423017d32fcd49c26d1efe8.
[taskcluster 2025-07-04 14:11:52.079Z] === Task Starting ===
[setup 2025-07-04T14:11:54.310+00:00] run-task started in /
...(1720 lines hidden)...
[task 2025-07-04T14:12:28.482+00:00] File "/builds/worker/.local/share/uv/python/cpython-3.10.16-linux-x86_64-gnu/lib/python3.10/multiprocessing/reduction.py", line 51, in dumps
[task 2025-07-04T14:12:28.482+00:00] cls(buf, protocol).dump(obj)
[task 2025-07-04T14:12:28.482+00:00] AttributeError: Can't pickle local object 'fake_load_graph_config.<locals>.<lambda>'
[task 2025-07-04T14:12:28.482+00:00] _______________________________ test_tasks_regex _______________________________
[task 2025-07-04T14:12:28.482+00:00]
[task 2025-07-04T14:12:28.482+00:00] run_taskgraph = <function run_taskgraph.<locals>.inner at 0x7fbbc3ac2dd0>
[task 2025-07-04T14:12:28.482+00:00] capsys = <_pytest.capture.CaptureFixture object at 0x7fbbc3c99f90>
[task 2025-07-04T14:12:28.482+00:00]
[task 2025-07-04T14:12:28.482+00:00] def test_tasks_regex(run_taskgraph, capsys):
[task 2025-07-04T14:12:28.482+00:00] run_taskgraph(["full", "--tasks=_.*-t-1"])
[task 2025-07-04T14:12:28.482+00:00] out, _ = capsys.readouterr()
[task 2025-07-04T14:12:28.482+00:00] > assert out.strip() == "_fake-t-1"
[task 2025-07-04T14:12:28.482+00:00] E AssertionError: assert '' == '_fake-t-1'
[task 2025-07-04T14:12:28.482+00:00] E
[task 2025-07-04T14:12:28.482+00:00] E - _fake-t-1
[task 2025-07-04T14:12:28.482+00:00]
[task 2025-07-04T14:12:28.482+00:00] test/test_main.py:109: AssertionError
[task 2025-07-04T14:12:28.482+00:00] ------------------------------ Captured log call -------------------------------
[task 2025-07-04T14:12:28.482+00:00] ERROR concurrent.futures:_base.py:344 exception calling callback for <Future at 0x7fbbc3bda860 state=finished raised AttributeError>
[task 2025-07-04T14:12:28.482+00:00] concurrent.futures.process._RemoteTraceback:
[task 2025-07-04T14:12:28.482+00:00] """
[task 2025-07-04T14:12:28.482+00:00] Traceback (most recent call last):
[task 2025-07-04T14:12:28.482+00:00] File "/builds/worker/.local/share/uv/python/cpython-3.10.16-linux-x86_64-gnu/lib/python3.10/multiprocessing/queues.py", line 244, in _feed
[task 2025-07-04T14:12:28.482+00:00] obj = _ForkingPickler.dumps(obj)
[task 2025-07-04T14:12:28.482+00:00] File "/builds/worker/.local/share/uv/python/cpython-3.10.16-linux-x86_64-gnu/lib/python3.10/multiprocessing/reduction.py", line 51, in dumps
[task 2025-07-04T14:12:28.482+00:00] cls(buf, protocol).dump(obj)
[task 2025-07-04T14:12:28.482+00:00] AttributeError: Can't pickle local object 'fake_load_graph_config.<locals>.<lambda>'
[task 2025-07-04T14:12:28.482+00:00] """
[task 2025-07-04T14:12:28.482+00:00]
[task 2025-07-04T14:12:28.482+00:00] The above exception was the direct cause of the following exception:
[task 2025-07-04T14:12:28.482+00:00]
[task 2025-07-04T14:12:28.482+00:00] Traceback (most recent call last):
[task 2025-07-04T14:12:28.482+00:00] File "/builds/worker/.local/share/uv/python/cpython-3.10.16-linux-x86_64-gnu/lib/python3.10/concurrent/futures/_base.py", line 342, in _invoke_callbacks
[task 2025-07-04T14:12:28.482+00:00] callback(self)
[task 2025-07-04T14:12:28.482+00:00] File "/builds/worker/checkouts/vcs/src/taskgraph/generator.py", line 273, in add_new_tasks
[task 2025-07-04T14:12:28.482+00:00] for task in future.result():
[task 2025-07-04T14:12:28.482+00:00] File "/builds/worker/.local/share/uv/python/cpython-3.10.16-linux-x86_64-gnu/lib/python3.10/concurrent/futures/_base.py", line 451, in result
[task 2025-07-04T14:12:28.482+00:00] return self.__get_result()
[task 2025-07-04T14:12:28.483+00:00] File "/builds/worker/.local/share/uv/python/cpython-3.10.16-linux-x86_64-gnu/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
[task 2025-07-04T14:12:28.483+00:00] raise self._exception
[task 2025-07-04T14:12:28.483+00:00] File "/builds/worker/.local/share/uv/python/cpython-3.10.16-linux-x86_64-gnu/lib/python3.10/multiprocessing/queues.py", line 244, in _feed
[task 2025-07-04T14:12:28.483+00:00] obj = _ForkingPickler.dumps(obj)
[task 2025-07-04T14:12:28.483+00:00] File "/builds/worker/.local/share/uv/python/cpython-3.10.16-linux-x86_64-gnu/lib/python3.10/multiprocessing/reduction.py", line 51, in dumps
[task 2025-07-04T14:12:28.483+00:00] cls(buf, protocol).dump(obj)
[task 2025-07-04T14:12:28.483+00:00] AttributeError: Can't pickle local object 'fake_load_graph_config.<locals>.<lambda>'
[task 2025-07-04T14:12:28.483+00:00] _______________________________ test_output_file _______________________________
[task 2025-07-04T14:12:28.483+00:00]
[task 2025-07-04T14:12:28.483+00:00] run_taskgraph = <function run_taskgraph.<locals>.inner at 0x7fbbc3b17250>
[task 2025-07-04T14:12:28.483+00:00] tmpdir = local('/tmp/pytest-of-worker/pytest-0/test_output_file0')
[task 2025-07-04T14:12:28.483+00:00]
[task 2025-07-04T14:12:28.483+00:00] def test_output_file(run_taskgraph, tmpdir):
[task 2025-07-04T14:12:28.483+00:00] output_file = tmpdir.join("out.txt")
[task 2025-07-04T14:12:28.483+00:00] assert not output_file.check()
[task 2025-07-04T14:12:28.483+00:00]
[task 2025-07-04T14:12:28.483+00:00] run_taskgraph(["full", f"--output-file={output_file.strpath}"])
[task 2025-07-04T14:12:28.483+00:00] assert output_file.check()
[task 2025-07-04T14:12:28.483+00:00] > assert output_file.read_text("utf-8").strip() == "\n".join(
[task 2025-07-04T14:12:28.483+00:00] ["_fake-t-0", "_fake-t-1", "_fake-t-2"]
[task 2025-07-04T14:12:28.483+00:00] )
[task 2025-07-04T14:12:28.483+00:00] E AssertionError: assert '' == '_fake-t-0\n_fake-t-1\n_fake-t-2'
[task 2025-07-04T14:12:28.483+00:00] E
[task 2025-07-04T14:12:28.483+00:00] E - _fake-t-0
[task 2025-07-04T14:12:28.483+00:00] E - _fake-t-1
[task 2025-07-04T14:12:28.483+00:00] E - _fake-t-2
[task 2025-07-04T14:12:28.483+00:00]
[task 2025-07-04T14:12:28.483+00:00] test/test_main.py:118: AssertionError
[task 2025-07-04T14:12:28.483+00:00] ------------------------------ Captured log call -------------------------------
[task 2025-07-04T14:12:28.483+00:00] ERROR concurrent.futures:_base.py:344 exception calling callback for <Future at 0x7fbbc3eb11e0 state=finished raised AttributeError>
[task 2025-07-04T14:12:28.483+00:00] concurrent.futures.process._RemoteTraceback:
[task 2025-07-04T14:12:28.483+00:00] """
[task 2025-07-04T14:12:28.483+00:00] Traceback (most recent call last):
[task 2025-07-04T14:12:28.483+00:00] File "/builds/worker/.local/share/uv/python/cpython-3.10.16-linux-x86_64-gnu/lib/python3.10/multiprocessing/queues.py", line 244, in _feed
[task 2025-07-04T14:12:28.483+00:00] obj = _ForkingPickler.dumps(obj)
[task 2025-07-04T14:12:28.483+00:00] File "/builds/worker/.local/share/uv/python/cpython-3.10.16-linux-x86_64-gnu/lib/python3.10/multiprocessing/reduction.py", line 51, in dumps
[task 2025-07-04T14:12:28.483+00:00] cls(buf, protocol).dump(obj)
[task 2025-07-04T14:12:28.483+00:00] AttributeError: Can't pickle local object 'fake_load_graph_config.<locals>.<lambda>'
[task 2025-07-04T14:12:28.483+00:00] """
[task 2025-07-04T14:12:28.483+00:00]
[task 2025-07-04T14:12:28.483+00:00] The above exception was the direct cause of the following exception:
[task 2025-07-04T14:12:28.483+00:00]
[task 2025-07-04T14:12:28.483+00:00] Traceback (most recent call last):
[task 2025-07-04T14:12:28.483+00:00] File "/builds/worker/.local/share/uv/python/cpython-3.10.16-linux-x86_64-gnu/lib/python3.10/concurrent/futures/_base.py", line 342, in _invoke_callbacks
[task 2025-07-04T14:12:28.483+00:00] callback(self)
[task 2025-07-04T14:12:28.483+00:00] File "/builds/worker/checkouts/vcs/src/taskgraph/generator.py", line 273, in add_new_tasks
[task 2025-07-04T14:12:28.483+00:00] for task in future.result():
[task 2025-07-04T14:12:28.483+00:00] File "/builds/worker/.local/share/uv/python/cpython-3.10.16-linux-x86_64-gnu/lib/python3.10/concurrent/futures/_base.py", line 451, in result
[task 2025-07-04T14:12:28.483+00:00] return self.__get_result()
[task 2025-07-04T14:12:28.483+00:00] File "/builds/worker/.local/share/uv/python/cpython-3.10.16-linux-x86_64-gnu/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
[task 2025-07-04T14:12:28.483+00:00] raise self._exception
[task 2025-07-04T14:12:28.483+00:00] File "/builds/worker/.local/share/uv/python/cpython-3.10.16-linux-x86_64-gnu/lib/python3.10/multiprocessing/queues.py", line 244, in _feed
[task 2025-07-04T14:12:28.483+00:00] obj = _ForkingPickler.dumps(obj)
[task 2025-07-04T14:12:28.483+00:00] File "/builds/worker/.local/share/uv/python/cpython-3.10.16-linux-x86_64-gnu/lib/python3.10/multiprocessing/reduction.py", line 51, in dumps
[task 2025-07-04T14:12:28.483+00:00] cls(buf, protocol).dump(obj)
[task 2025-07-04T14:12:28.483+00:00] AttributeError: Can't pickle local object 'fake_load_graph_config.<locals>.<lambda>'
[task 2025-07-04T14:12:28.483+00:00] =========================== short test summary info ============================
[task 2025-07-04T14:12:28.483+00:00] FAILED test/test_generator.py::test_kind_ordering - AssertionError: assert [] == ['_fake1', '_fake2', '_fake3']
[task 2025-07-04T14:12:28.483+00:00]
[task 2025-07-04T14:12:28.483+00:00] Right contains 3 more items, first extra item: '_fake1'
[task 2025-07-04T14:12:28.483+00:00]
[task 2025-07-04T14:12:28.483+00:00] Full diff:
[task 2025-07-04T14:12:28.483+00:00] + []
[task 2025-07-04T14:12:28.483+00:00] - [
[task 2025-07-04T14:12:28.483+00:00] - '_fake1',
[task 2025-07-04T14:12:28.483+00:00] - '_fake2',
[task 2025-07-04T14:12:28.483+00:00] - '_fake3',
[task 2025-07-04T14:12:28.483+00:00] - ]
[task 2025-07-04T14:12:28.483+00:00] FAILED test/test_generator.py::test_full_task_set - AssertionError: assert Graph(nodes=frozenset(), edges=frozenset()) == Graph(nodes={'_fake-t-2', '_fake-t-0', '_fake-t-1'}, edges=set())
[task 2025-07-04T14:12:28.483+00:00]
[task 2025-07-04T14:12:28.483+00:00] Matching attributes:
[task 2025-07-04T14:12:28.483+00:00] ['edges']
[task 2025-07-04T14:12:28.483+00:00] Differing attributes:
[task 2025-07-04T14:12:28.483+00:00] ['nodes']
[task 2025-07-04T14:12:28.483+00:00]
[task 2025-07-04T14:12:28.483+00:00] Drill down into differing attribute nodes:
[task 2025-07-04T14:12:28.483+00:00] nodes: frozenset() != {'_fake-t-2', '_fake-t-0', '_fake-t-1'}
[task 2025-07-04T14:12:28.483+00:00] Extra items in the right set:
[task 2025-07-04T14:12:28.483+00:00] '_fake-t-2'
[task 2025-07-04T14:12:28.483+00:00] '_fake-t-0'
[task 2025-07-04T14:12:28.483+00:00] '_fake-t-1'
[task 2025-07-04T14:12:28.483+00:00]
[task 2025-07-04T14:12:28.483+00:00] Full diff:
[task 2025-07-04T14:12:28.483+00:00] + frozenset()
[task 2025-07-04T14:12:28.483+00:00] - {
[task 2025-07-04T14:12:28.483+00:00] - '_fake-t-0',
[task 2025-07-04T14:12:28.483+00:00] - '_fake-t-1',
[task 2025-07-04T14:12:28.483+00:00] - '_fake-t-2',
[task 2025-07-04T14:12:28.483+00:00] - }
[task 2025-07-04T14:12:28.483+00:00] FAILED test/test_generator.py::test_full_task_graph - AssertionError: assert Graph(nodes=frozenset(), edges=frozenset()) == Graph(nodes={'_fake-t-2', '_fake-t-0', '_fake-t-1'}, edges={('_fake-t-1', '_fake-t-0', 'prev'), ('_fake-t-2', '_fake-t-1', 'prev')})
[task 2025-07-04T14:12:28.483+00:00]
[task 2025-07-04T14:12:28.483+00:00] Differing attributes:
[task 2025-07-04T14:12:28.483+00:00] ['nodes', 'edges']
[task 2025-07-04T14:12:28.483+00:00]
[task 2025-07-04T14:12:28.483+00:00] Drill down into differing attribute nodes:
[task 2025-07-04T14:12:28.483+00:00] nodes: frozenset() != {'_fake-t-2', '_fake-t-0', '_fake-t-1'}
[task 2025-07-04T14:12:28.483+00:00] Extra items in the right set:
[task 2025-07-04T14:12:28.483+00:00] '_fake-t-2'
[task 2025-07-04T14:12:28.483+00:00] '_fake-t-0'
[task 2025-07-04T14:12:28.483+00:00] '_fake-t-1'
[task 2025-07-04T14:12:28.483+00:00]
[task 2025-07-04T14:12:28.483+00:00] Full diff:
[task 2025-07-04T14:12:28.483+00:00] + frozenset()
[task 2025-07-04T14:12:28.483+00:00] - {
[task 2025-07-04T14:12:28.483+00:00] - '_fake-t-0',
[task 2025-07-04T14:12:28.483+00:00] - '_fake-t-1',
[task 2025-07-04T14:12:28.484+00:00] - '_fake-t-2',
[task 2025-07-04T14:12:28.484+00:00] - }
[task 2025-07-04T14:12:28.484+00:00]
[task 2025-07-04T14:12:28.484+00:00] Drill down into differing attribute edges:
[task 2025-07-04T14:12:28.484+00:00] edges: frozenset() != {('_fake-t-1', '_fake-t-0', 'prev'), ('_fake-t-2', '_fake-t-1', 'prev')}
[task 2025-07-04T14:12:28.484+00:00] Extra items in the right set:
[task 2025-07-04T14:12:28.484+00:00] ('_fake-t-1', '_fake-t-0', 'prev')
[task 2025-07-04T14:12:28.484+00:00] ('_fake-t-2', '_fake-t-1', 'prev')
[task 2025-07-04T14:12:28.484+00:00]
[task 2025-07-04T14:12:28.484+00:00] Full diff:
[task 2025-07-04T14:12:28.484+00:00] + frozenset()
[task 2025-07-04T14:12:28.484+00:00] - {
[task 2025-07-04T14:12:28.484+00:00] - (
[task 2025-07-04T14:12:28.484+00:00] - '_fake-t-1',
[task 2025-07-04T14:12:28.484+00:00] - '_fake-t-0',
[task 2025-07-04T14:12:28.484+00:00] - 'prev',
[task 2025-07-04T14:12:28.484+00:00] - ),
[task 2025-07-04T14:12:28.484+00:00] - (
[task 2025-07-04T14:12:28.484+00:00] - '_fake-t-2',
[task 2025-07-04T14:12:28.484+00:00] - '_fake-t-1',
[task 2025-07-04T14:12:28.484+00:00] - 'prev',
[task 2025-07-04T14:12:28.484+00:00] - ),
[task 2025-07-04T14:12:28.484+00:00] - }
[task 2025-07-04T14:12:28.484+00:00] FAILED test/test_generator.py::test_target_task_set - KeyError: '_fake-t-1'
[task 2025-07-04T14:12:28.484+00:00] FAILED test/test_generator.py::test_target_task_graph - KeyError: '_fake-t-1'
[task 2025-07-04T14:12:28.484+00:00] FAILED test/test_generator.py::test_always_target_tasks - KeyError: '_ignore-t-1'
[task 2025-07-04T14:12:28.484+00:00] FAILED test/test_generator.py::test_optimized_task_graph - KeyError: '_fake-t-2'
[task 2025-07-04T14:12:28.484+00:00] FAILED test/test_generator.py::test_load_tasks_for_kind - AssertionError: assert ('t-1' in {})
[task 2025-07-04T14:12:28.484+00:00] FAILED test/test_generator.py::test_kind_load_tasks[load transform] - AttributeError: 'list' object has no attribute 'values'
[task 2025-07-04T14:12:28.484+00:00] FAILED test/test_generator.py::test_kind_load_tasks[load transform no object] - AttributeError: 'list' object has no attribute 'values'
[task 2025-07-04T14:12:28.484+00:00] FAILED test/test_main.py::test_show_taskgraph_attr[tasks-expected0] - AssertionError: assert '' == '_fake-t-0\n_fake-t-1\n_fake-t-2'
[task 2025-07-04T14:12:28.484+00:00]
[task 2025-07-04T14:12:28.484+00:00] - _fake-t-0
[task 2025-07-04T14:12:28.484+00:00] - _fake-t-1
[task 2025-07-04T14:12:28.484+00:00] - _fake-t-2
[task 2025-07-04T14:12:28.484+00:00] FAILED test/test_main.py::test_show_taskgraph_attr[full-expected1] - AssertionError: assert '' == '_fake-t-0\n_fake-t-1\n_fake-t-2'
[task 2025-07-04T14:12:28.484+00:00]
[task 2025-07-04T14:12:28.484+00:00] - _fake-t-0
[task 2025-07-04T14:12:28.484+00:00] - _fake-t-1
[task 2025-07-04T14:12:28.484+00:00] - _fake-t-2
[task 2025-07-04T14:12:28.484+00:00] FAILED test/test_main.py::test_show_taskgraph_attr[target-expected2] - assert 1 == 0
[task 2025-07-04T14:12:28.484+00:00] FAILED test/test_main.py::test_show_taskgraph_attr[target-graph-expected3] - assert 1 == 0
[task 2025-07-04T14:12:28.484+00:00] FAILED test/test_main.py::test_show_taskgraph_attr[optimized-expected4] - assert 1 == 0
[task 2025-07-04T14:12:28.484+00:00] FAILED test/test_main.py::test_show_taskgraph_attr[morphed-expected5] - assert 1 == 0
[task 2025-07-04T14:12:28.484+00:00] FAILED test/test_main.py::test_tasks_regex - AssertionError: assert '' == '_fake-t-1'
[task 2025-07-04T14:12:28.484+00:00]
[task 2025-07-04T14:12:28.484+00:00] - _fake-t-1
[task 2025-07-04T14:12:28.484+00:00] FAILED test/test_main.py::test_output_file - AssertionError: assert '' == '_fake-t-0\n_fake-t-1\n_fake-t-2'
[task 2025-07-04T14:12:28.484+00:00]
[task 2025-07-04T14:12:28.484+00:00] - _fake-t-0
[task 2025-07-04T14:12:28.484+00:00] - _fake-t-1
[task 2025-07-04T14:12:28.484+00:00] - _fake-t-2
[task 2025-07-04T14:12:28.484+00:00] ============ 18 failed, 505 passed, 6 skipped, 3 xfailed in 28.64s =============
[taskcluster 2025-07-04 14:12:29.092Z] === Task Finished ===
[taskcluster 2025-07-04 14:12:29.323Z] Unsuccessful task run with exit code: 1 completed in 72.487 seconds
Loading