|
221 | 221 |
|
222 | 222 | % Abstract - REVISED: Curriculum design focus |
223 | 223 | \begin{abstract} |
224 | | - Machine learning education faces a fundamental gap: students learn algorithms without understanding the systems that execute them. They study gradient descent without measuring memory, attention mechanisms without analyzing $O(N^2)$ scaling, optimizer theory without knowing why Adam requires $3\times$ the optimizer-related memory of SGD (gradients plus two state buffers). This \emph{algorithm-systems divide} produces practitioners who can train models but cannot debug memory failures, optimize inference latency, or reason about deployment trade-offs—the very skills industry demands as ``ML systems engineering.'' We present TinyTorch, a 20-module curriculum that closes this gap through \emph{implementation-based systems pedagogy}: students construct PyTorch's core components (tensors, autograd, optimizers, CNNs, transformers) in pure Python, building a complete framework where every operation they invoke is code they wrote. The design employs three patterns: \emph{progressive disclosure} of complexity, \emph{systems-first integration} of profiling from the first module, and \emph{build-to-validate milestones} recreating 67 years of ML breakthroughs—from Perceptron (1958) through Transformers (2017) to MLPerf-style benchmarking. Requiring only 4GB RAM and no GPU, TinyTorch demonstrates that deep ML systems understanding is achievable without specialized hardware. The curriculum is available open-source at \texttt{mlsysbook.ai/tinytorch}. |
| 224 | + Machine learning education faces a fundamental gap: students learn algorithms without understanding the systems that execute them. They study gradient descent without measuring memory, attention mechanisms without analyzing $O(N^2)$ scaling, optimizer theory without knowing why Adam requires $3\times$ the optimizer-related memory of SGD (gradients plus two state buffers). This \emph{algorithm-systems divide} produces practitioners who can train models but cannot debug memory failures, optimize inference latency, or reason about deployment trade-offs---the very skills industry demands as ``ML systems engineering.'' We present TinyTorch, a 20-module curriculum that closes this gap through \emph{implementation-based systems pedagogy}: students construct PyTorch's core components (tensors, autograd, optimizers, CNNs, transformers) in pure Python, building a complete framework where every operation they invoke is code they wrote. The design employs three patterns: \emph{progressive disclosure} of complexity, \emph{systems-first integration} of profiling from the first module, and \emph{build-to-validate milestones} recreating 67 years of ML breakthroughs---from Perceptron (1958) through Transformers (2017) to MLPerf-style benchmarking. Requiring only 4GB RAM and no GPU, TinyTorch demonstrates that deep ML systems understanding is achievable without specialized hardware. The curriculum is available open-source at \texttt{mlsysbook.ai/tinytorch}. |
225 | 225 | \end{abstract} |
226 | 226 |
|
227 | 227 | % Main content |
@@ -844,7 +844,7 @@ \section{Course Deployment} |
844 | 844 | Translating curriculum design into effective classroom practice requires addressing integration models, infrastructure accessibility, and student support structures. This section presents deployment patterns designed for diverse institutional contexts. |
845 | 845 |
|
846 | 846 | \textbf{Textbook Integration.} |
847 | | -TinyTorch serves as the hands-on implementation companion to the \emph{Machine Learning Systems} textbook~\citep{mlsysbook2025} (\texttt{mlsysbook.ai}), creating synergy between theoretical foundations and systems engineering practice. While the textbook covers the full ML lifecycle—data engineering, training architectures, deployment monitoring, robust operations, and sustainable AI—TinyTorch provides the complementary experience of building core infrastructure from first principles. This integration enables a complete educational pathway: students study production ML systems architecture in the textbook (Chapter 4: distributed training patterns, Chapter 7: quantization strategies), then implement those same abstractions in TinyTorch (Module 06: autograd for backpropagation, Module 15: INT8 quantization). The two resources address different aspects of the same educational gap: understanding both \emph{how production systems work} (textbook's systems architecture perspective) and \emph{how to build them yourself} (TinyTorch's implementation depth). |
| 847 | +TinyTorch serves as the hands-on implementation companion to the \emph{Machine Learning Systems} textbook~\citep{mlsysbook2025} (\texttt{mlsysbook.ai}), creating synergy between theoretical foundations and systems engineering practice. While the textbook covers the full ML lifecycle---data engineering, training architectures, deployment monitoring, robust operations, and sustainable AI---TinyTorch provides the complementary experience of building core infrastructure from first principles. This integration enables a complete educational pathway: students study production ML systems architecture in the textbook (Chapter 4: distributed training patterns, Chapter 7: quantization strategies), then implement those same abstractions in TinyTorch (Module 06: autograd for backpropagation, Module 15: INT8 quantization). The two resources address different aspects of the same educational gap: understanding both \emph{how production systems work} (textbook's systems architecture perspective) and \emph{how to build them yourself} (TinyTorch's implementation depth). |
848 | 848 |
|
849 | 849 | \subsection{Integration Models} |
850 | 850 | \label{subsec:integration} |
@@ -996,7 +996,7 @@ \subsection{Teaching Assistant Support} |
996 | 996 |
|
997 | 997 | Effective deployment requires structured TA support beyond instructor guidance. |
998 | 998 |
|
999 | | -\textbf{TA Preparation}: TAs should develop deep familiarity with critical modules where students commonly struggle—Modules 06 (Autograd), 09 (CNNs), and 13 (Transformers)—by completing these modules themselves and intentionally introducing bugs to understand common error patterns. The \texttt{INSTRUCTOR.md} file documents frequent student errors (gradient shape mismatches, disconnected computational graphs, broadcasting failures) and debugging strategies. |
| 999 | +\textbf{TA Preparation}: TAs should develop deep familiarity with critical modules where students commonly struggle---Modules 06 (Autograd), 09 (CNNs), and 13 (Transformers)---by completing these modules themselves and intentionally introducing bugs to understand common error patterns. The \texttt{INSTRUCTOR.md} file documents frequent student errors (gradient shape mismatches, disconnected computational graphs, broadcasting failures) and debugging strategies. |
1000 | 1000 |
|
1001 | 1001 | \textbf{Office Hour Demand Patterns}: Student help requests are expected to cluster around conceptually challenging modules, with autograd (Module 06) likely generating higher office hour demand than foundation modules. Instructors should anticipate demand spikes by scheduling additional TA capacity during critical modules, providing pre-recorded debugging walkthroughs, and establishing async support channels (discussion forums with guaranteed response times). |
1002 | 1002 |
|
|
0 commit comments