Add an action that measures the doc usefulness using LLMs - #357
Draft
lebrice wants to merge 13 commits into
Draft
Conversation
lebrice
force-pushed
the
add_doc_metrics
branch
2 times, most recently
from
March 26, 2026 20:28
5949b2f to
ebc90b4
Compare
1 task
lebrice
added a commit
to mila-iqia/skills
that referenced
this pull request
Mar 28, 2026
Ports the four skills from satyaog's mila-iqia/mila-docs#364 into this marketplace. Each skill covers one step of the cluster onboarding sequence: account & MFA setup, local tooling (WSL/uv/milatools), SSH connection, and running interactive/batch jobs. Each skill's quiz.yaml is a filtered subset of the questions from lebrice's mila-iqia/mila-docs#357, restricted to questions relevant to that skill's scope. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
3 tasks
lebrice
added a commit
to lebrice/mila-docs
that referenced
this pull request
Apr 8, 2026
It seems likely that this file needs to be present in the master branch before the comment from docmetrics (see PR mila-iqia#358) can properly show up in the PRs. I don't know if there is another workaround, I have not done that much investigation. Hopefully this works, and worst case I'll make another PR to revert this change. Signed-off-by: Fabrice Normandin <normandf@mila.quebec>
lebrice
changed the title
lebrice
added a commit
that referenced
this pull request
Apr 10, 2026
It seems likely that this file needs to be present in the master branch before the comment from docmetrics (see PR #358) can properly show up in the PRs. I don't know if there is another workaround, I have not done that much investigation. Hopefully this works, and worst case I'll make another PR to revert this change. Signed-off-by: Fabrice Normandin <normandf@mila.quebec>
Signed-off-by: Fabrice Normandin <normandf@mila.quebec>
Signed-off-by: Fabrice Normandin <normandf@mila.quebec>
Signed-off-by: Fabrice Normandin <normandf@mila.quebec>
Signed-off-by: Fabrice Normandin <normandf@mila.quebec>
Signed-off-by: Fabrice Normandin <normandf@mila.quebec>
Signed-off-by: Fabrice Normandin <normandf@mila.quebec>
Signed-off-by: Fabrice Normandin <normandf@mila.quebec>
Signed-off-by: Fabrice Normandin <normandf@mila.quebec>
Signed-off-by: Fabrice Normandin <normandf@mila.quebec>
Signed-off-by: Fabrice Normandin <normandf@mila.quebec>
Signed-off-by: Fabrice Normandin <normandf@mila.quebec>
Switch to v0.0.5 of the docmetrics action which supports post-comment: 'false'. The evaluation still runs on the self-hosted runner as before, but results are saved as an artifact instead of posting the comment directly. A separate workflow_run workflow then picks up the artifact and posts the comment using the base repo's GITHUB_TOKEN, which has pull-requests: write access even for fork PRs. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Signed-off-by: Fabrice Normandin <normandf@mila.quebec>
lebrice
force-pushed
the
add_doc_metrics
branch
from
April 10, 2026 18:40
52e97f7 to
ffac92f
Compare
lebrice
added a commit
that referenced
this pull request
Apr 13, 2026
lebrice
added a commit
to mila-iqia/skills
that referenced
this pull request
Apr 14, 2026
* Add mila-tools plugin with Mila cluster quickstart skills Ports the four skills from satyaog's mila-iqia/mila-docs#364 into this marketplace. Each skill covers one step of the cluster onboarding sequence: account & MFA setup, local tooling (WSL/uv/milatools), SSH connection, and running interactive/batch jobs. Each skill's quiz.yaml is a filtered subset of the questions from lebrice's mila-iqia/mila-docs#357, restricted to questions relevant to that skill's scope. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * Update skills * Update skills and quiz * Run docmetrics with --num-candidates 3 and show per-question pass counts Each question is now evaluated three times per run so noisy single-shot outcomes don't dominate the reported score. The PR comment surfaces this by rendering per-question cells as "x/N" pass counts (e.g. 2/3), with a ↑/↓ delta column, and summary rows as "pct% ±std pp". The previous formatter compared answer dicts against True/False and silently rendered ❓ for every row; the new answer_cell handles both the single- and multi-candidate JSON shapes emitted by docmetrics. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * Use gemma4:e2b instead of gpt-oss:120b Signed-off-by: Fabrice Normandin <normandf@mila.quebec> * Apply suggestion from @lebrice Co-authored-by: Fabrice Normandin <fabrice.normandin@gmail.com> * Update mila-tools/skills/mila-base/quiz.yaml --------- Signed-off-by: Fabrice Normandin <normandf@mila.quebec> Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> Co-authored-by: Satya Ortiz-Gagne <satya.ortiz-gagne@mila.quebec> Co-authored-by: satyaog <satyaog@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
quiz.yamlwith a multiple-choice quiz that the docs should help answer.Signed-off-by: Fabrice Normandin normandf@mila.quebec