feature: deferred loading and requirement pruning #1199

leondz · 2025-05-07T11:00:18Z

Support construction-time loading of optional modules. Includes

many generators now using this pattern
pattern for loading modules at run-time and failing if absent
optional requirements moved to pyproject.toml options and pruned from requirements.txt - deferred to feature: disable optional imports by default #1475
_load_deps and _clear_deps pattern, used in generator constructor and _load_client / _clear_client
tests to check that optional deps are in the right place and not in the wrong place

Todo / in scope:

Not done:

load/clear deps for plugin types other than generators
gh actions for testing optional components
deeper validation

Out of scope:

handling of versioning outside of pyproject.toml

Resolves #101

…t checks via plugin cache

garak/generators/nemo.py

…lugins

Signed-off-by: Leon Derczynski <[email protected]>

jmartin-tech

Final testing in progress, landing this may present some risk as testing resources for every generator changed is not feasible.

jmartin-tech · 2025-12-01T20:40:00Z

garak/generators/replicate.py

+        self.client = self.replicate

    def _clear_client(self):
+        self._clear_deps()


Will need to watch these for #1471 related issues.

jmartin-tech · 2025-12-01T20:43:30Z

garak/probes/audio.py

-            except ImportError as e:
-                logging.critical("Missing libraries for audio modules.", exc_info=e)
-                raise GarakException("Missing Libraries for audio modules.")
+            from datasets import load_dataset


There is a change in behavior here, with extra_dependency_names loading soundfile it becomes a required dependency for this probe. Before this PR it was only required if no user provided files already exist in the data_path.

Not a blocker, just noting for awareness.

jmartin-tech · 2025-12-01T20:48:36Z

garak/_plugins.py

+    quoted_module_list = "'" + "', '".join(absent_modules) + "'"
+    module_list = " ".join(absent_modules)
+    msg = f"⛔ Plugin '{calling_module}' requires Python modules which aren't installed/available: {quoted_module_list}"
+    hint = f"💡 Try 'pip install {module_list}' to get missing module."


Just noting, it might be nice to have this reference a group in the pyproject.toml. This may be added in #1475 when the as these groups are introduced.

garak/_plugins.py

jmartin-tech · 2025-12-03T20:18:00Z

garak/_plugins.py

+                    ".".join(dependency_module_name.split(".")[: n + 1])
+                    for n in range(dependency_module_name.count(".") + 1)
+                ]:
+                    if importlib.util.find_spec(dependency_path) is None:


Testing shows find_spec needs to be provided the runtime package name, currently dependency_path entries are the pypi package name.

This can be tested by attempting to load generators.huggingface.LLaVA.

⛔ Plugin 'generators.huggingface.LLaVA' requires Python modules which aren't installed/available: 'pillow' 💡 Try 'pip install pillow' to get missing module. Traceback (most recent call last): File "<frozen runpy>", line 198, in _run_module_as_main File "<frozen runpy>", line 88, in _run_code File "/home/jemartin/Projects/nvidia/garak/garak/__main__.py", line 14, in <module> main() File "/home/jemartin/Projects/nvidia/garak/garak/__main__.py", line 9, in main cli.main(sys.argv[1:]) File "/home/jemartin/Projects/nvidia/garak/garak/cli.py", line 596, in main generator = _plugins.load_plugin( ^^^^^^^^^^^^^^^^^^^^^ File "/home/jemartin/Projects/nvidia/garak/garak/_plugins.py", line 428, in load_plugin _import_failed(absent_modules, full_plugin_name) File "/home/jemartin/Projects/nvidia/garak/garak/_plugins.py", line 493, in _import_failed raise ModuleNotFoundError(msg) ModuleNotFoundError: ⛔ Plugin 'generators.huggingface.LLaVA' requires Python modules which aren't installed/available: 'pillow'

Looking at this code in particular, I don't think this validation is needed here at this time. It may be more appropriate have handle a missing import exception around the module instantiation call.

Looking at this code in particular, I don't think this validation is needed here at this time. It may be more appropriate have handle a missing import exception around the module instantiation call.

I believe the intent here is to summarise in one pass a list of all missing module names, which is determined using find_spec.

This is simply not the right place for this, load_plugin is called at instantiation so this is only evaluating for this plugin which is fully processed by _load_deps so not needed here.

In the next iteration PR #1475 we might want to add an early preprocessor that takes a comprehensive look at the full run config to determine if all dependencies required for the full run are available however I am thinking that might turn out to be an overly complex goal that may get deferred or shelved in favor of allowing the run to skip probes that happen to be missing dependencies instead for blocking start of the run. More discussion of that can happen in that PR in later.

Alright, this was extra code added after initial PR was made when a feature was requested to list all missing modules rather than one at a time. Is that additional feature no longer a requirement to land?

jmartin-tech

Suggestion offered removes the early extra deps check and defers the load and exception to the actual instance creation step.

This pattern is highly coupled to the current extra_dependency_names design, however no more so than the original AFAICT. One drawback is this always reports all extra_dependency_names in the hint vs only the specific one that triggered the raise.

jmartin-tech · 2025-12-03T21:26:56Z

garak/_plugins.py

+    if category in PLUGIN_TYPES:
+        extra_dependency_names = PluginCache.instance()[category][full_plugin_name][
+            "extra_dependency_names"
+        ]
+        if len(extra_dependency_names) > 0:
+            absent_modules = []
+            for dependency_module_name in extra_dependency_names:
+                for (
+                    dependency_path
+                ) in [  # support both plain names and also multi-point names e.g. langchain.llms
+                    ".".join(dependency_module_name.split(".")[: n + 1])
+                    for n in range(dependency_module_name.count(".") + 1)
+                ]:
+                    if importlib.util.find_spec(dependency_path) is None:
+                        absent_modules.append(dependency_module_name)
+            if len(absent_modules):
+                _import_failed(absent_modules, full_plugin_name)


Since the goal of load_plugin is to return an instance no need to check the dependencies early just let the instance creation raise:

Suggested change

if category in PLUGIN_TYPES:

extra_dependency_names = PluginCache.instance()[category][full_plugin_name][

"extra_dependency_names"

]

if len(extra_dependency_names) > 0:

absent_modules = []

for dependency_module_name in extra_dependency_names:

for (

dependency_path

) in [ # support both plain names and also multi-point names e.g. langchain.llms

".".join(dependency_module_name.split(".")[: n + 1])

for n in range(dependency_module_name.count(".") + 1)

]:

if importlib.util.find_spec(dependency_path) is None:

absent_modules.append(dependency_module_name)

if len(absent_modules):

_import_failed(absent_modules, full_plugin_name)

Is this in tension with the requested feature of listing all missing modules at once rather than piecemeal? I realise the granularity is different, but don't we want to cause the minimum number of user round trips between execution and dep installation?

Not really, load_plugin is only called when actually instantiating a plugin, testing in this location we will not evaluate all plugins required for the run. Since generators are the primary plugin type using this pattern just removing this is acceptable for now.

We should

merge these threads

get clear about reqs for this PR

I don't mind if we advise all missing modules at once or piecemeal. The latter has better UX. Agree this feature should belong in the right PR and code location, if the feature is going to manifest.

Understood, based on the targeted changes in scope this PR should remove this block.

Also the revisions already to made to load_deps will provide a consistent verbose list of all packages required for the plugin that is attempting to load. This covers the same scope as what this block does with added context available.

garak/_plugins.py

Co-authored-by: Jeffrey Martin <[email protected]> Signed-off-by: Leon Derczynski <[email protected]>

Signed-off-by: Jeffrey Martin <[email protected]>

leondz and others added 15 commits May 2, 2025 14:11

draft postponed import pattern for cohere generator

3000d4c

move extra dependency requirements into classdefs, mediate requiremen…

757e0f3

…t checks via plugin cache

actually do the plugin dep load

9310d0a

migrate generators to 'extra dependencies' pattern

dac569e

prune dupe lazyload

35e93fc

extra_dependency_names in all plugins

bf7f36b

active must be False for Probes using extra modules

6a39b0c

make PIL optional in generators.huggingface.LLaVA

56c6182

move optional load fail to ModuleNotFoundError

3657e04

add _load/_clear_deps() into base generator and _load/_clear client

865d604

put the MNFE where it belongs

d61957d

backoff exception placeholder must inherit base exception

8a7051e

test for reqs presence in pyproject.toml, requirements.txt

60775f6

handle hyphen in pypi pkg names

31e98d4

rm optional plugin deps

75babb7

leondz requested a review from jmartin-tech May 7, 2025 11:00

leondz added the architecture Architectural upgrades label May 7, 2025

leondz requested review from jmartin-tech and removed request for jmartin-tech May 7, 2025 11:04

leondz added this to the 0.11.0 milestone May 8, 2025

leondz added 5 commits May 8, 2025 14:19

skip generator tests if optional deps absent

83f551a

support sub-package deps

dd51196

scope optimum to nvidia

b33a46c

move import function to _load_deps

de5b3f1

rm import handling in langchain

19c31fe

jmartin-tech reviewed May 8, 2025

View reviewed changes

garak/generators/nemo.py Outdated Show resolved Hide resolved

leondz self-assigned this May 8, 2025

leondz removed this from the release 0.11.0 milestone May 8, 2025

leondz added 2 commits May 8, 2025 17:39

amend optimum to be nvidia flavour

54fabc5

dry - use garak._plugins.PLUGIN_TYPES as canonical def of 1st class p…

ffac714

…lugins

leondz added 6 commits November 13, 2025 13:29

handle optimum package naming

b4b7504

invert exception selection

0e074dc

make dep name processing consistent

c60d814

refresh plugin cache

309a246

Merge branch 'main' into update/optional_imports

74ff9a3

Signed-off-by: Leon Derczynski <[email protected]>

merge main

677eacb

leondz requested review from erickgalinkin and jmartin-tech November 21, 2025 15:05

leondz and others added 7 commits November 21, 2025 16:10

rm git merge notations in generated XDG cache plugin json

6da2389

remove nv optimum / EOL

27119e3

Merge branch 'main' into update/optional_imports

b06a634

Signed-off-by: Leon Derczynski <[email protected]>

rm more Optimum

07de920

add depnames for apikey probes, detector

143f84d

Merge branch 'main' into update/optional_imports

69244dc

migrate bedrock generator to extra_dependency_names pattern

53597f0

leondz assigned jmartin-tech and unassigned leondz Dec 1, 2025

jmartin-tech approved these changes Dec 1, 2025

View reviewed changes

update param names, types in _load and _clear deps

5daff3d

jmartin-tech reviewed Dec 3, 2025

View reviewed changes

jmartin-tech requested changes Dec 3, 2025

View reviewed changes

leondz and others added 3 commits December 4, 2025 12:56

handle MNFE exceptions during _load_deps

957cc75

Co-authored-by: Jeffrey Martin <[email protected]> Signed-off-by: Leon Derczynski <[email protected]>

cut _plugins.load_optional_module

621bfaf

defer extra dependency checks to plugin class

0be265b

Signed-off-by: Jeffrey Martin <[email protected]>

jmartin-tech added a commit to jmartin-tech/garak that referenced this pull request Dec 12, 2025

feature: deferred loading and requirement pruning (NVIDIA#1199)

b00a2cc

jmartin-tech force-pushed the update/optional_imports branch from 87b69a2 to 0be265b Compare December 12, 2025 21:51

Merge 'main' into update/optional_imports

05b9f5f

jmartin-tech approved these changes Dec 12, 2025

View reviewed changes

jmartin-tech merged commit c360fee into NVIDIA:main Dec 12, 2025
15 checks passed

github-actions bot locked and limited conversation to collaborators Dec 12, 2025

feature: deferred loading and requirement pruning #1199

feature: deferred loading and requirement pruning #1199

Uh oh!

Conversation

leondz commented May 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

jmartin-tech left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jmartin-tech left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

leondz commented May 7, 2025 •

edited

Loading

jmartin-tech left a comment •

edited

Loading