Add illuminator notebook

[alinaryan]

This change adds the Illuminator tool’s core functions to the instructlab-knowledge notebook for analyzing a converted document and summarizing merged table cell issues for each table.

Also refactor's the illuminator to accept json as input and adjusts some imports to be relative

[alimaredia]

Might be nice to rename this to `convert_to_docling_document since it's not pdf specific anymore. It would also be nice for this function to have an argument on whether or not the markdown should be saved or not. I'm not sure in every case we want that.

[alimaredia]

@alinaryan This PR title mentions adding an illuminator notebook but there isn't on in here. Maybe the title should change to reflect the refactoring you're doing.

Also if you added the illuminator into the instructlab-knowledge notebook do you have an idea of what the calls the notebook would make might look like?

[alinaryan]

This PR title mentions adding an illuminator notebook but there isn't on in here. Maybe the title should change to reflect the refactoring you're doing.
Also if you added the illuminator into the instructlab-knowledge notebook do you have an idea of what the calls the notebook would make might look like?

@alimaredia
Added a commit that adds the notebook and reworded the PR desc. It currently outputs the illuminator results to the notebook output AND to an output file. LMK what you think about this approach

[iamemilio]

I think its worth going through this and clearing all the outputs from the instructlab-knowledge.ipynb notebook

[iamemilio]

I like the little emoji's in the console output

[alinaryan]

@JustinXHale LMK what you think about the UX design here

[JustinXHale]

This looks great @alinaryan!

UX Review of Data Pre-Processing: From source PDF to SDG-ready

Guideline	Status	Suggestion
Header	✅	Clear and concise title.
Goal/Objective Present	✅	The notebook provides a brief summary of what it does and why at the beginning. The numbered list (1, 2, 3, 4) clearly outlines the main steps users will follow in the notebook.
Setup & Prerequisites	✅	Prerequisites (like !pip install commands) are placed within each relevant major section, which supports a modular and step-by-step approach.
Markdown Before Code	🔹	Most code cells are preceded by markdown cells explaining their purpose. However, the initial code cell defining Path variables and directory creation lacks a clear introductory markdown explaining its role (e.g., "Initialize Workspace and Define Output Paths"). The empty code cell before "Read generated QAs and restructure" also lacks a preceding markdown cell, its probably left there by accident. Improvement to the "Why" before each section could be improved.
Handle Errors or Outputs	✅	The notebook includes print statements to show expected outputs and also notes a warning regarding single file support ("***** WARNING! Only one file at a time is supported at this time."). It also provides guidance if QA generation fails to meet the required number of pairs.
Hardcoding Avoided	✅	File paths are managed using pathlib.Path and defined variables (WORKSPACE_ROOT, SOURCE_DOCUMENT_DIR, etc.).
Concise Cells	✅	Code logic is generally broken into manageable chunks, avoiding overly long or dense cells. This is really good!
Code Commenting	✅	Inline comments are present in relevant sections to clarify logic.
End Wrap-Up & Handoff	✅	The notebook concludes with a "Summary" section that recaps the completed steps and clearly outlines the next steps, including a link to a relevant follow-up notebook.

Reviewer Notes

The notebook provides a clear and well-structured workflow for data pre-processing. Improving the initial setup instructions and adding more descriptive inline comments in key code sections could further enhance its user experience. The use of markdown headers and logical code separation is generally good. The numbered list in the introduction serves well as a table of contents, guiding the user through the notebook's flow. Improvement to the "Why/goal" before each section could be improved, similar to what is done in chunking.

[alimaredia]

@alinaryan when I ran through the notebook I wasn't seeing any output in the illuminator_readable_summary.txt. Any idea why this might be happening?

[alinaryan]

I updated, the full output now prints to the summary file and the notebook cell output

[anastasds]

Why is this removed?

[alinaryan]

I added it back!