Add Homework 1: Q-iteration for MountainCar

Includes:
- Theory PDF with written problems
- Problem 1: Tabular Q-iteration implementation
- Precomputed transition tables (200x200 discretization)
- Submission instructions and leaderboard link
- Policy template for students to implement

Author: eugenevinitsky

.gitignore

@@ -219,3 +219,7 @@ tensorboard/
 resources
 server/leaderboard.db
 *.db
+
+# Homework solutions (keep private)
+*solutions*
+CLAUDE.md

README.md

@@ -13,11 +13,6 @@ Reinforcement Learning course project using [PufferLib](https://github.com/Puffe
 curl -LsSf https://astral.sh/uv/install.sh | sh
 ```
 
-**Windows:**
-```powershell
-powershell -ExecutionPolicy ByPass -c "irm https://astral.sh/uv/install.ps1 | iex"
-```
-
 **Homebrew (macOS):**
 ```bash
 brew install uv

homeworks/homework_1/Homework_1_theory.pdf

@@ -0,0 +1,58 @@
+# Homework 1: Value-Based Reinforcement Learning
+
+This homework covers tabular Q-learning methods.
+
+## Structure
+
+- `Homework_1_theory.pdf` - Theory questions (submit on Gradescope)
+- `problem_1/` - Tabular Q-iteration for MountainCar
+
+## Submission Instructions
+
+### Theory (Gradescope)
+Submit your answers to the theory questions as a PDF on Brightspace.
+
+### Programming Problems (Leaderboard)
+
+For each programming problem, submit to the course leaderboard:
+
+**Problem 1: MountainCar Q-Iteration**
+- Submit: `policy.py` and `checkpoint.pt`
+- Your policy must achieve mean reward > -150
+
+### How to Submit to Leaderboard
+
+1. Go to the course leaderboard: https://eval-server-production-c3fe.up.railway.app/
+2. Select MountainCar
+3. Upload your `policy.py` and `checkpoint.pt` files
+4. Wait for evaluation results
+
+### File Requirements
+
+Your `policy.py` must contain:
+- A `Policy` class with a `forward(obs)` method that returns an action
+- A `load_policy(checkpoint_path)` function that returns a Policy instance
+
+Your `checkpoint.pt` must be loadable by your `load_policy` function.
+
+## Getting Started
+
+First, follow the setup instructions in the main [README](../../README.md) to install dependencies using `uv sync`.
+
+Then:
+```bash
+# Activate the virtual environment
+source .venv/bin/activate
+
+# Problem 1: Run Q-iteration
+cd homeworks/homework_1/problem_1
+python problem_1.py
+
+# Test your policy locally
+python policy.py
+```
+
+## Grading
+
+- Theory questions: See Brightspace rubric
+- Programming problems: Pass/fail based on leaderboard performance thresholds

homeworks/homework_1/README.md

@@ -0,0 +1,129 @@
+# Problem 1: Tabular Q-Iteration for MountainCar
+
+In this problem, you will implement Q-iteration (dynamic programming) for the MountainCar environment. The continuous observation space has been discretized into a 200x200 grid.
+
+## Environment
+
+**MountainCar** has a car stuck in a valley that must build momentum to reach the goal on the right.
+
+**Observation space (2 dimensions):**
+- Position: range [-1.2, 0.6], goal at position >= 0.5
+- Velocity: range [-0.07, 0.07]
+
+**Actions:**
+- 0: Push left
+- 1: No push
+- 2: Push right
+
+**Reward:** -1 for each timestep (encourages reaching the goal quickly)
+
+## Transition Tables
+
+We provide precomputed transition tables that describe the discretized environment dynamics.
+
+> **WARNING:** Do NOT modify or regenerate the transition tables (`.npy` files). The grading server uses the same tables - changing them will cause your submission to fail.
+
+### State Space Discretization
+
+The state space is discretized into a 200x200 grid:
+
+- **s0 (first index):** Position index, ranging from 0 to 199
+  - Index 0 = position -1.2 (leftmost)
+  - Index 199 = position 0.6 (rightmost)
+  - Goal region (position >= 0.5) is roughly indices 189-199
+
+- **s1 (second index):** Velocity index, ranging from 0 to 199
+  - Index 0 = velocity -0.07 (moving left fastest)
+  - Index 199 = velocity 0.07 (moving right fastest)
+  - Index ~100 = velocity ~0 (stationary)
+
+### Table Formats
+
+**P (transition_next_states.npy):** shape `(200, 200, 3, 2)`
+
+```python
+P[s0, s1, a] = [s0', s1']  # numpy array of 2 integers
+```
+
+Given current state indices (s0, s1) and action a, returns the next state indices.
+
+Example:
+```python
+next_state = P[100, 100, 2]  # State (100,100), action 2 (push right)
+s0_next, s1_next = next_state[0], next_state[1]
+# Now you can look up Q[s0_next, s1_next, :] to get Q-values at next state
+```
+
+**R (transition_rewards.npy):** shape `(200, 200, 3)`
+
+```python
+R[s0, s1, a] = reward  # single float, always -1.0
+```
+
+The immediate reward for taking action a in state (s0, s1).
+
+**D (transition_dones.npy):** shape `(200, 200, 3)`
+
+```python
+D[s0, s1, a] = done  # boolean: True or False
+```
+
+Whether the episode terminates after taking action a. True only when the car reaches the goal.
+
+## Your Task
+
+### 1. Implement `q_iteration()` in `problem_1.py`
+
+Implement Q-iteration using the Bellman optimality equation:
+
+```
+Q(s, a) = R(s, a) + gamma * (1 - D(s, a)) * max_a' Q(s', a')
+```
+
+where s' = P(s, a) is the next state.
+
+**Algorithm:**
+1. Initialize Q-table to zeros: shape (200, 200, 3)
+2. Repeat until convergence or max_iterations:
+   - For each state-action pair (s0, s1, a):
+     - Look up next state: `s0', s1' = P[s0, s1, a]`
+     - Look up reward: `r = R[s0, s1, a]`
+     - Look up done: `d = D[s0, s1, a]`
+     - Update: `Q_new[s0, s1, a] = r + gamma * (1 - d) * max_a' Q[s0', s1', a']`
+   - If max|Q_new - Q| < theta: converged, stop
+   - Q = Q_new
+3. Return the converged Q-table
+
+### 2. Implement `forward()` in `policy.py`
+
+Implement the action selection method:
+1. Discretize the observation using `self.discretize_state(obs)`
+2. Look up the Q-values for that state in `self.q_table`
+3. Return the action with the highest Q-value
+
+## Running Your Solution
+
+```bash
+python problem_1.py
+```
+
+This will:
+1. Load the transition tables
+2. Run your Q-iteration implementation
+3. Save the Q-table to `checkpoint.pt`
+4. Evaluate your policy
+
+## Submission
+
+Submit two files to the leaderboard:
+- `checkpoint.pt` - Your Q-table saved with torch.save (shape: 200x200x3)
+- `policy.py` - With your implementation of the `forward()` method
+
+## Grading
+
+Your policy must achieve a **mean reward better than -150** to pass.
+
+A well-implemented solution should:
+- Consistently reach the goal (position >= 0.5)
+- Complete episodes in fewer than 150 steps on average
+- Achieve 100% success rate

homeworks/homework_1/problem_1/README.md

@@ -0,0 +1,87 @@
+"""
+Script to build the transition table for MountainCar Q-iteration.
+Run this once to generate the transition files for students.
+
+WARNING: Students should NOT run this script or modify the transition tables.
+The .npy files are provided as-is and will be used by the grading server.
+If you regenerate them, your solution may not work correctly during evaluation.
+"""
+
+import numpy as np
+from tqdm import tqdm
+import gymnasium as gym
+
+
+N_BINS = 200
+N_ACTIONS = 3  # 0: push left, 1: no push, 2: push right
+
+# MountainCar observation bounds
+STATE_BOUNDS = [
+    (-1.2, 0.6),    # Position
+    (-0.07, 0.07),  # Velocity
+]
+
+
+def discretize_state(observation):
+    if observation.ndim > 1:
+        observation = observation[0]
+    indices = []
+    for i, (low, high) in enumerate(STATE_BOUNDS):
+        val = np.clip(observation[i], low, high)
+        scaled = (val - low) / (high - low) * (N_BINS - 1)
+        idx = int(np.clip(np.round(scaled), 0, N_BINS - 1))
+        indices.append(idx)
+    return tuple(indices)
+
+
+def undiscretize_state(indices):
+    obs = []
+    for i, (low, high) in enumerate(STATE_BOUNDS):
+        val = low + (indices[i] / (N_BINS - 1)) * (high - low)
+        obs.append(val)
+    return np.array(obs)
+
+
+def build_transition_table():
+    print("Building transition table for MountainCar...")
+    env = gym.make("MountainCar-v0")
+
+    # Shape: (position_bins, velocity_bins, actions, next_state_dims)
+    next_states = np.zeros((N_BINS, N_BINS, N_ACTIONS, 2), dtype=np.int32)
+    rewards = np.zeros((N_BINS, N_BINS, N_ACTIONS), dtype=np.float32)
+    dones = np.zeros((N_BINS, N_BINS, N_ACTIONS), dtype=bool)
+
+    for s0 in tqdm(range(N_BINS), desc="Building transitions"):
+        for s1 in range(N_BINS):
+            continuous_state = undiscretize_state((s0, s1))
+
+            for action in range(N_ACTIONS):
+                # Reset and set state directly
+                env.reset()
+                env.unwrapped.state = continuous_state.copy()
+
+                obs, reward, terminated, truncated, _ = env.step(action)
+                next_state_indices = discretize_state(obs)
+
+                next_states[s0, s1, action] = next_state_indices
+                rewards[s0, s1, action] = reward
+                dones[s0, s1, action] = terminated  # Don't include truncated
+
+    env.close()
+
+    np.save("transition_next_states.npy", next_states)
+    np.save("transition_rewards.npy", rewards)
+    np.save("transition_dones.npy", dones)
+    print("Saved: transition_next_states.npy, transition_rewards.npy, transition_dones.npy")
+
+    # Print some stats
+    print(f"\nTransition table stats:")
+    print(f"  Shape P: {next_states.shape}")
+    print(f"  Shape R: {rewards.shape}")
+    print(f"  Shape D: {dones.shape}")
+    print(f"  Terminal states: {dones.sum()} / {dones.size} ({100*dones.sum()/dones.size:.1f}%)")
+    print(f"  Reward range: [{rewards.min()}, {rewards.max()}]")
+
+
+if __name__ == "__main__":
+    build_transition_table()