Skip to content

Commit 06b6789

Browse files
authored
Refactor GSM8KReasoning (#34)
remove _get_fewshot_target_text from GSM8KReasoning to make it consistent with other benchmarks that inherit from MATHReasoning and have no few-shot capability such as AIME2024 or MATH500 <!-- For Work In Progress Pull Requests, please use the Draft PR feature, see https://github.blog/2019-02-14-introducing-draft-pull-requests/ for further details. For a timely review/response, please avoid force-pushing additional commits if your PR already received reviews or comments. Before submitting a Pull Request, please ensure you've done the following: - 👷‍♀️ Create small PRs. In most cases this will be possible. - ✅ Provide tests for your changes. - 📝 Use descriptive commit messages. - 📗 Update any related documentation and include any relevant screenshots. - 📗 Reflect the changes you made in the changelog. --> ## PR Checklist - [x] Use descriptive commit messages. - [ ] Provide tests for your changes. - [ ] Update any related documentation and include any relevant screenshots. - [x] Check if changes need to be made to docs (README or any guides in `/docs/`). - [x] Reflect the changes you made in the changelog. ## What type of PR is this? (check all applicable) - [x] Refactor - [ ] Feature - [ ] Bug Fix - [ ] Optimization - [ ] Documentation Update ## Description ## Related Tickets & Documents <!-- For pull requests that relate or close an issue, please include them below. We like to follow [Github's guidance on linking issues to pull requests](https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue). For example having the text: "closes #1234" would connect the current pull request to issue 1234. And when we merge the pull request, Github will automatically close the issue. --> - Related Issue # - Closes # ## QA Instructions, Screenshots, Recordings _Please replace this line with instructions on how to test your changes, a note on the hardware and config this has been tested on, as well as any relevant additional information._ ## Added/updated tests? - [ ] Yes - [x] No, and this is why: _please replace this line with details on why tests have not been included_ - [ ] I need help with writing tests ## [optional] Are there any post deployment tasks we need to perform?
1 parent 5d65f32 commit 06b6789

File tree

1 file changed

+0
-4
lines changed

1 file changed

+0
-4
lines changed

src/eval_framework/tasks/benchmarks/math_reasoning.py

Lines changed: 0 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -567,9 +567,5 @@ def _extract_answer_with_fallback(self, completion: str) -> str:
567567
def _get_instruction_text(self, item: dict[str, Any]) -> str:
568568
return self.QUERY_TEMPLATE.format(question=item["question"])
569569

570-
def _get_fewshot_target_text(self, item: dict[str, Any]) -> str:
571-
# This should not be called since we're zero-shot only
572-
raise NotImplementedError("GSM8K Reasoning is zero-shot only")
573-
574570
def _get_ground_truth(self, item: dict[str, Any]) -> str | None:
575571
return self._extract_answer_fallback(item["answer"])

0 commit comments

Comments
 (0)