Introduce new utils repo

Signed-off-by: Ali Maredia <[email protected]>

Author: alimaredia

docs/utils/utils-repo.md

@@ -0,0 +1,32 @@
+# Create Separate Repo for User Utilities
+
+## Idea Overview
+
+Create a separate repository within the `instructlab` GitHub org to house scripts that aid in the data preparation necessary on documents before they are used in the Instructlab Synthetic Data Generation (SDG) process.
+
+Scripts in this repository may become features or incorporated in the Instructlab core repository after use and review by users and developers.
+
+## Motivation for this Proposal
+
+Instructlab does an excellent job creating data samples based on a document in the SDG process.
+The SDG process ingests qna.yaml files, and documents. After ingestion, SDG chunks those documents up, and creates samples based on those chunks.
+
+Qna.yaml file generation is currently done by hand and is a major pain point from users.
+Additionally there is very high variability in the content inside of source documents. Documents can have graphs, mathematical notation, pictures, tables, and other symbols that provide information users want to teach models.
+Even after documents are converted by a document conversion tool like [docling](https://github.com/DS4SD/docling), some information is not rendered correctly and need to be fixed manually.
+In large documents, manual fixes are very labor intensive.
+Creating great samples hinges on the documents being converted properly, thus to push the boundaries of the utility of Instructlab, ingesting large complex documents is a must.
+
+This repository is a place where the community can submit personal scripts that aid the data preparation before the SDG step and other parts of the Instructlab processes.
+These scripts maybe be related to accessing document readiness, document conversation for certain types of documents, automated document review and fixing, inspecting generated data, and automated qna.yaml creation.
+Many users and community members already have scripts they use day to day when using Instructlab. The `utils` repo would be a place where the maintainers of the project can collect and curate them for the benefit of the community.
+
+## Additional Info
+
+A few areas of focus for scripts that will be added to the repository are:
+
+- Assessing document readiness knowing the limitations of docling
+- Automating qna.yaml creation
+- Automating document review and fixing
+
+The docs, testing, CI, and release cadence, for this repository will be determined in an iterative manner.