More friendly output formats

[dxoigmn]

The primary way we inspect outputs is via Tensorboard. However, most other tools cannot consume Tensorboard summaries (protobufs). Your job is to research output formats that can feed into other tools (e.g., Inspect, HarmBench, etc.), and add support for these formats.

This is probably best done in conjunction with #12.

[ajithraj-intel]

@dxoigmn I wanted your help in clarifying some questions.

Q1. Is the intention of this issue to make the output of llmart look more like tools like Inspect and Harmbench - Interface with adversarial prompts generated and their response from custom models like this ?

Q2. Or is the intention to add support for more trackers like WandB, CometML, Aim, MLFlow, ClearML, DVCLive by modifying the section below.

LLMart/src/llmart/attack.py

Lines 43 to 51 in 0a49cd9

	# Setup Tensorboard
	accelerator = Accelerator(
	log_with="tensorboard",
	project_dir=cfg.output_dir,
	dataloader_config=DataLoaderConfiguration(
	split_batches=cfg.data.split_batches
	if cfg.data.split_batches is not None
	else (True if cfg.data.n_train > 1 else False)
	),

Q3. Or is the intention to convert the protobuf output to more universal outputs like json, hdf5, csv, etc.

Q4. Could you please explain the relation of this issue with #12 and how you want the solution which interlinks and fixes both of them ?

[dxoigmn]

@dxoigmn I wanted your help in clarifying some questions.

Q1. Is the intention of this issue to make the output of llmart look more like tools like Inspect and Harmbench - Interface with adversarial prompts generated and their response from custom models like this ?

It would be nice if our tool could output a dataset that is consumable by those tools. HarmBench, for example, has an evaluation script. Our tool should output a file that we can then pass as --completions_path for example. I haven't looked to closely to see if inspect has an analogous tool.

Q2. Or is the intention to add support for more trackers like WandB, CometML, Aim, MLFlow, ClearML, DVCLive by modifying the section below.

LLMart/src/llmart/attack.py

Lines 43 to 51 in 0a49cd9

Setup Tensorboard

accelerator = Accelerator(
log_with="tensorboard",
project_dir=cfg.output_dir,
dataloader_config=DataLoaderConfiguration(
split_batches=cfg.data.split_batches
if cfg.data.split_batches is not None
else (True if cfg.data.n_train > 1 else False)
),

No, we are happy with Tensorboard for now.

Q3. Or is the intention to convert the protobuf output to more universal outputs like json, hdf5, csv, etc.

Yes, one option is to write a separate tool that can consume tensorboard outputs and turn them into something consumable by HarmBench's eval script.

Q4. Could you please explain the relation of this issue with #12 and how you want the solution which interlinks and fixes both of them ?

The relationship is that whatever this new tool outputs should be the attack with the lowest loss and not just the last attack.