AI assisted review of Lectures (Code) and current Issues

[mmcky]

Description

@oyamad has demonstrated how AI could be used to improve the QuantEcon lectures code. A good example of this is QuantEcon/lecture-python.myst#463 where it was suggested to update some numpy code that would be more efficient and an improvement in syntax.

QuantEcon would like to adopt this strategy to undertake a lecture wide review using a range of AI tools including ChatGPT and Claude (Anthropic). The aim of this project is two-fold. First, we would like to use AI to improve the code contained in our lectures. Second, we would like to learn how best to make use of current AI tools to reduce our workload in finding solutions to current open issues. This involves documenting useful approaches to raising queries for ChatGPT and or Claude and writing a guide to using AI that we can write up as a blog post later this year.

Current Status: Assembling a team (please register your interest in joining this team by adding your name below).

Team

First Name	GitHub ID	Role
Matt	@mmcky	Project Admin and Reviewer

Editorial Board

First Name	GitHub ID	Role
John	@jstac	Final Reviewer (for complex change sets) at request of @mmcky

Tasks

1. Evaluate the use of ChatGPT and Claude in finding useful improvements for the lecture code.
2. Document the use of AI in making improvements to the lectures, collecting workflows and communicating across the team how to get the most useful results. How have you prompted the LLM?, is there a good template script (set of instructions) for giving a lecture and getting a useful response?

Example of asking claude to figure out a solution to an issue

Can you help me fix this lecture https://python.quantecon.org/cass_koopmans_1.html to solve this GitHub issue https://github.com/QuantEcon/lecture-python.myst/issues/457

produced QuantEcon/lecture-python.myst#457 (comment)

3. Use AI assistance to evaluate current open issues and using the results using your domain knowledge to evaluate the output, and modify as required. If there is a confusing issue or one that has no obvious bug reported, we can close those bug-related issues. Please mark @mmcky to review.
4. Write a blog post about the use of AI to improve our lectures (@mmcky)

Success Metrics

1. GitHub PRs
2. GitHub Issue Closures
3. Execution Time

Related Projects

To address Success Metric (3), it would be useful to find a tool that we can use to benchmark execution time at various points in a git repo history. There used to be a tool called vbench (written by Wes McKinney -- author of pandas) but that is long ago. @kp992 would you have any good ideas or suggestions of tools that exist, or tools we can create to facilitate this. It would be helpful to have metrics at the lecture level (or even lecture / code-cell level).

[kp992]

Thanks @mmcky for this exciting idea. Building some tooling around existing LLM is going to be super useful and fun.
I am happy to work on building some automation tooling/scripts.

There are 3 major problems that we can work on:

1. Review Agent: This LLM agent is resposible for reviewing the lecture(s) and find bugs, typos, code-improvement suggestions, etc.
2. Fix Agent: This LLM agent should either take a part of the Review Agent's output or the github issue as the input and automatically edits the files. Fix Agent and Review Agent should be able to communicate and arrive at a conclusion in a couple of back-and-forth iterations.
3. Verification script: This will run some necessary tests, execution speed bench-marking all under the single script to make it easy for the developers.

How does this sound? I am happy to jump on a call discuss in more details.