feat: init model experiments, add doc reader #38
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
feat(experiments): Add document reader experiments
This pull request introduces a new framework for conducting and evaluating document reader experiments. This allows for standardized testing of various extractive Question Answering models on the QASports dataset and other popular QA datasets, leveraging the Haystack framework.
Key Changes:
New Experiments Module:
experimentsdirectory containing the logic for running the evaluations.experiments/doc_reader.py: A script to set up and run a Haystack pipeline for document reading tasks. It evaluates models and prints performance metrics.experiments/module.py: A supporting module that encapsulates dataset handling and model selection. It includes:Dataset,DocReader, andSports.QASports,SQuAD,AdversarialQA, andDuoRC.leomaurodesenv/QASports2dataset from the Hugging Face Hub.Dynamic Experiment Configuration:
doc_reader.pyscript now usesargparseto allow dynamic selection of the dataset, model, and specific sport (for QASports) via command-line arguments. This enhances flexibility and reproducibility.Improved Documentation:
README.mdhas been updated with a new "Performing Experiments" section, providing clear instructions on how to set up the environment and run the new experiment scripts.How to Run the Experiments:
To run the new experiments, follow these steps: