Fix core cfg listed twice in report in default run

[arjun-krishna1]

• Reset config_files list at start of load_config() since it already includes garak.core.yaml in its settings_files
• Prevents duplication when load_base_config() is called before load_config() in the CLI flow
• bug: core cfg listed twice in report in default run #1249

Verification

Here's a verification checklist for this fix:

Verification

List the steps needed to make sure the duplicate config_files fix works

• Run a basic garak scan and check the report:

  python -m garak -t test.Blank -p test.Test
  head -1 ~/.local/share/garak/garak_runs/garak.*.report.jsonl | python -c "import sys, json; d = json.load(sys.stdin); print('config_files:', d.get('_config.config_files'))"

[jmartin-tech]

Clearing a global variable does not remove the impacts the previous file may have already had on state. This method does not clear all package configurable state before processing.

At this time _config package global values are treated as common state and the files loaded are stored in config_files. A solution to duplicate entries in config_files would need to account for this. I would suggest some set logic and guards to avoid duplicate processing of files may be needed.

[arjun-krishna1]

Clearing a global variable does not remove the impacts the previous file may have already had on state. This method does not clear all package configurable state before processing.

At this time _config package global values are treated as common state and the files loaded are stored in config_files. A solution to duplicate entries in config_files would need to account for this. I would suggest some set logic and guards to avoid duplicate processing of files may be needed.

Understood @jmartin-tech , updated to stop duplicate processing of config files instead

[leondz]

lgtm - strange test failure though

[jmartin-tech]

Looking closer at the test failure, test_function.py is failing due to retained global state of _config across tests in that file. A short term fix to would be to add a fixture similar to the one recently added in test_config.py

garak/tests/test_config.py

Lines 98 to 108 in 7849f3b

	@pytest.fixture(autouse=True)
	def reload_config(request):
	def reload():
	if _config.transient.reportfile is not None:
	_config.transient.reportfile.close()
	if os.path.exists(_config.transient.report_filename):
	os.remove(_config.transient.report_filename)
	importlib.reload(_config)
	request.addfinalizer(reload)
	reload()

This fixture clears and reimports _config. This further exposes that design of the _config package needs some work if garak is used in an environment that will need to maintain state for multiple runs. The cli usage pattern masks this issue at this time.

[jmartin-tech]

#1549 addresses the global state issues that cause the test failure in this PR. Once that PR lands merging main into this branch should enable passing tests.

[arjun-krishna1]

#1549 addresses the global state issues that cause the test failure in this PR. Once that PR lands merging main into this branch should enable passing tests.

@jmartin-tech are we good to re-run the tests? I can't seem to do it on my end

[leondz]

Re-running

[jmartin-tech]

A commit is required to get tests to update to use the latest code, this can simply be a merge of main to the PR branch or a rebase.