Auto collect dynamic SUT factories

[bkorycki]

No description provided.

[github-actions]

MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅

[superdosh]

This looks good to me overall, though I don't love the name.

[wpietri]

I like the theory, but

1. How do we know it works? I'd like to see a test that at least N factories are found under normal run conditions, and
2. How do we make sure it keeps working? That is, if somebody adds a new factory, are we sure they'll do the right thing to make sure it shows up?

[wpietri]

My recollection is that in Python this only works for code that has already been imported. With the explicit imports gone, how are we sure that this will actually get everything?

[bkorycki]

I believe load_namespace("suts") takes care of that.

[bkorycki]

1. A unit test for the actual collection is a great idea. I just added that.
2. All it takes for a new factory to show up is to have it inherit from DynamicDriverSUTFactory. And I would expect that every time someone adds a new dynamic factory that they test that it works with some CLI command as a part of their normal workflow. We could also add a unit test for every dynamic factory to ensure it gets loaded. Though that requires people to remember to add that same unit test every time they add a new dynamic factory, so I don't know how much that actually accomplishes.