Add custom_op executor to the default executors to avoid hidden registration through thunder.torch.custom_op._register_custom_op

[Copilot]

As per title. Currently thunder.torch.custom_op._register_custom_op adds custom_op_ex to the default executor list.

I chose this way because I found it not great to see an executor in the default even when that executor does nothing. But now I'd find it worse to have the hidden behavior than having an empty executor in the default list. Also this hidden automatic registration might've caused some test failures of #2710

[crcrpar]

@Copilot could you change how we add custom_op_ex to the default executors? I mean, change that so that custom_op_ex is always included in the default executors even when _register_custom_op is not used at all.

Ref:

lightning-thunder/thunder/torch/custom_op.py

Lines 415 to 419 in f946af6

	# NOTE: `thunder.extend.add_default_executor` basically does `lst.insert(ex, 0)`.
	if custom_op_ex not in get_default_executors():
	default_executors = get_default_executors()
	new_default_executors = add_executor_lists(default_executors, [custom_op_ex])
	set_default_executors(new_default_executors)

[kiya00]

it seems the Lit Job has error:

=========================== short test summary info ============================
FAILED thunder/tests/test_recipes.py::test_plugins_composition - AssertionError: assert 'custom_op' in ['__ad_hoc_executor_129236491690128', 'transformer_engine_v1', 'nvfuser']
 +  where 'custom_op' = thunder.extend.OperatorExecutor('custom_op').name
FAILED thunder/tests/test_recipes.py::test_plugins_basics - AssertionError: assert 'custom_op' in ['__ad_hoc_executor_129236439815024', '__ad_hoc_executor_129236439825584', 'nvfuser', 'torch', 'python']
 +  where 'custom_op' = thunder.extend.OperatorExecutor('custom_op').name
= 2 failed, 2212 passed, 207 skipped, 21 xfailed, 7 xpassed, 46774 warnings in 879.83s (0:14:39) =

is it related?

[kshitij12345]

LGTM, thanks @crcrpar

NOTE to reviewers: Also tested with

lightning-thunder/thunder/tests/test_extend.py

Lines 123 to 129 in bd2f0ed

	def test_get_all_executors_includes_all_native_executors():
	executors = get_all_executors()
	actual = {e.name for e in executors}
	# apex and transformer_engine register the executor even if the external library they rely on is not available.
	expected = {
	"apex",
	"custom_op",

[kiya00]

LGTM, thank you!

[t-vi]

Thank you @crcrpar @kiya00 @kshitij12345