Merge pull request #4 from neuralmagic/model-centric-layout

model centric layout

Author: andy-neuma

.github/pull_request_template.md

@@ -0,0 +1,5 @@
+SUMMARY:
+"please provide a brief summary"
+
+TEST PLAN:
+"please outline how the changes were tested"

README.md

@@ -2,15 +2,13 @@
 
 This repository contains configurations for model validation.
 
-The `accuracy` folder contains YAML files for each model that configures information needed for the model to be validated through the [llm-eval-test](https://github.com/openshift-psap/llm-eval-test). There are 4 config files for each model:
+The configs are organized by model. Each top level folder should be named after the model stub and contain `accuracy` and `performance` sub-folders. Both folders contain configs specific to the activity. Shared configs, e.g. `storage.yml` should be in the common area.
 
+The `accuracy` folder contains YAML configs needed for the model to be validated through the [llm-eval-test](https://github.com/openshift-psap/llm-eval-test). There are 3 config files for each model:
 * server.yml: contains settings to start a vllm server with the model
 * client.yml: contains settings for the llm-eval-test harness for the model
 * accuracy.yml: contains evaluation tasks and accuracy expectations for the model
-* storage.yml: specifies where mode and dataset is located
-
-The `benchmark` folder contains YAML files for each model that configures information needed for the model to be validated through the [guidellm](https://github.com/neuralmagic/guidellm). There are  config files for each model:
 
+The `benchmark` folder contains YAML configs needed for the model to be validated through the [guidellm](https://github.com/neuralmagic/guidellm). There are 3 config files for each model:
 * server.yml: contains settings to start a vllm server with the model
 * client.yml: contains settings for the guidellm for the model
-* storage.yml: specifies where mode and dataset is located