diff --git a/docs.json b/docs.json index a30ace3..43fc711 100644 --- a/docs.json +++ b/docs.json @@ -55,7 +55,8 @@ "staff/assignments/handgrading", "staff/assignments/autograding", "staff/assignments/rubrics", - "staff/assignments/testing" + "staff/assignments/testing", + "staff/assignments/test-insights" ] }, { @@ -73,7 +74,8 @@ "staff/discussion", "staff/gradebook", "staff/surveys", - "staff/office-hours/intro" + "staff/office-hours/intro", + "staff/office-hours/ai-assistance" ] } ] diff --git a/staff/assignments/autograding.mdx b/staff/assignments/autograding.mdx index dde886a..37e8e11 100644 --- a/staff/assignments/autograding.mdx +++ b/staff/assignments/autograding.mdx @@ -14,6 +14,16 @@ This documentation is under construction. Check back soon for detailed autograde The autograder system uses an "overlay grader" architecture that allows flexible grading configurations. +### Empty Submission Detection + +Pawtograder automatically detects empty submissions to help identify students who may need assistance. A submission is considered empty if: + +- The repository contains no files beyond the initial starter code +- No meaningful changes have been made to the starter files +- The submission appears to be unmodified from the template + +Empty submissions are flagged in the grading interface, allowing instructors to quickly identify students who may not have started the assignment or encountered issues with their repository setup. + *Detailed documentation coming soon.* ## Quickstart: Java and Python diff --git a/staff/assignments/handgrading.mdx b/staff/assignments/handgrading.mdx index 3c0f74c..34ec5f3 100644 --- a/staff/assignments/handgrading.mdx +++ b/staff/assignments/handgrading.mdx @@ -10,6 +10,10 @@ Instructors can define multiple handgrading rubrics for each assignment, althoug Once all required rubric checks are done, the grader can click the "Complete Review" button to confirm that they have completed grading. This does **not** release the review to the student, but indicates to the instructor that it **could** be released (at the instructor's discretion). + +TAs can edit or delete rubric scores even after completing a review. This allows for corrections or adjustments without needing to "uncomplete" the review. + + ## Grading Submissions That You Have Been Assigned When you have submissions assigned to grade, you will see a summary of them on the assignment's landing page. @@ -22,6 +26,10 @@ Click on any submission to view the grading interface. Click on the "Files" tab Instructors can assign graders to entire submissions (grading the whole rubric on each submission), or can assign graders to only parts of each rubric for each submission. The grading interface will show you the rubric parts that you have been assigned to grade, and you can grade each part individually. If you are assigned to grade only a part of the rubric, you can also click on "View + Grade Full Rubric" to have access to grade the full rubric. + +When reassigning a rubric part to a TA who already completed grading a different rubric part for the same submission, the new work will now properly appear in their pending work list. Previously, the existing review assignment's completed timestamp was preserved, causing the new work to not appear. + + Grading interface with rubric sidebar and code view diff --git a/staff/assignments/test-insights.mdx b/staff/assignments/test-insights.mdx new file mode 100644 index 0000000..ec0f517 --- /dev/null +++ b/staff/assignments/test-insights.mdx @@ -0,0 +1,122 @@ +--- +title: "Test Insights and Error Explorer" +description: "Analyze test failures, group errors, and regrade submissions efficiently" +icon: "magnifying-glass-chart" +--- + +# Test Insights and Error Explorer + +Pawtograder provides powerful tools to help instructors understand patterns in test failures and efficiently regrade submissions when issues are identified. + +## Error Explorer + +The error explorer helps you identify and analyze common test failures across all student submissions. This feature groups similar errors together, making it easy to: + +- Identify widespread issues that may indicate problems with the assignment or autograder +- Find students affected by specific errors +- Quickly regrade submissions after fixing autograder issues + +### Viewing Error Groups + +Navigate to the assignment's autograder page to access the error explorer. Errors are automatically grouped by: + +- Test name +- Error message +- Stack trace patterns + +Each error group shows: + +- Number of affected submissions +- Representative error message +- List of affected students + +### Copying Student Emails + +You can view and copy the email addresses of all students affected by a specific error group. This is useful for: + +- Notifying students about known issues +- Communicating fixes or workarounds +- Following up with students who need assistance + +To copy affected students' emails: + +1. Click on an error group in the error explorer +2. Click the "Copy Emails" button +3. Paste the email list into your email client + +## Regrade Workflow + +When you identify an issue with your autograder or tests, Pawtograder provides a streamlined workflow to regrade affected submissions. + +### Launching Regrade from Error Explorer + +1. **Identify the Issue**: Use the error explorer to find the error group you want to regrade +2. **Launch Regrade**: Click the "Regrade" button from the error explorer +3. **Preselect Submissions**: The system automatically preselects all submissions affected by that error +4. **Choose Commits**: Select which commits to regrade in the regrade dialog + +### Regrade Dialog Options + +The regrade dialog provides several options: + +- **Regrade Latest Commits**: Regrade each student's most recent submission +- **Regrade Specific SHA**: Enter a specific commit SHA to regrade +- **Auto-Promote**: Automatically promote the new submission if it scores higher than the current active submission + + +Auto-promote is useful when fixing autograder bugs that incorrectly penalized students. It ensures students automatically receive the benefit of the corrected grading without manual intervention. + + +### Conflict Detection + +The regrade system includes conflict detection to prevent issues: + +- Warns if students have made new commits since the error occurred +- Alerts if submissions are already being graded +- Provides clear validation messages before executing the regrade + +## Best Practices + +### Analyzing Test Failures + +1. **Review error patterns early**: Check the error explorer shortly after the assignment deadline +2. **Look for clusters**: Large error groups may indicate assignment or autograder issues +3. **Compare across submissions**: Use the error explorer to see if errors are consistent or varied + +### Regrading Submissions + +1. **Test your fix first**: Use the test assignment feature to verify your autograder fix works correctly +2. **Document the issue**: Keep notes on what was fixed for future reference +3. **Communicate with students**: Let affected students know about the regrade and any changes +4. **Use auto-promote carefully**: Only enable auto-promote when you're confident the new grades are correct + +### Office Hours Integration + +When students request help with test failures: + +1. Use the error explorer to see if their issue is widespread +2. Check if other students have the same error pattern +3. Direct students to existing help requests or discussions if the issue is common +4. Use the AI assistance feature to help diagnose unique errors + +## Technical Details + +### Error Grouping Algorithm + +Errors are grouped using: + +- Exact test name matching +- Fuzzy matching on error messages (ignoring line numbers and variable names) +- Stack trace similarity analysis + +### Regrade Performance + +- Regrades are processed asynchronously in the background +- Students receive notifications when their regrade completes +- The system prevents duplicate regrades for the same commit + +### Data Retention + +- Error groups are updated in real-time as new submissions arrive +- Historical error data is preserved for analysis +- Regrade history is maintained for audit purposes diff --git a/staff/gradebook.mdx b/staff/gradebook.mdx index 6ebd1ae..093e87c 100644 --- a/staff/gradebook.mdx +++ b/staff/gradebook.mdx @@ -125,6 +125,10 @@ return (100 * validValues.reduce((a, b) => a + (b && b.score ? b.score / b.max_s In either case, grades that are missing will count as 0 points, *unless* the grade is marked as `excused`, in which case it is excluded from the average. + +The mean calculation now tolerates undefined values in the input array, automatically filtering them out before computing the average. This prevents errors when some gradebook columns have not yet been populated. + + #### `drop_lowest(array, count)` Removes the lowest scoring droppable items: diff --git a/staff/office-hours/ai-assistance.mdx b/staff/office-hours/ai-assistance.mdx new file mode 100644 index 0000000..04a01b2 --- /dev/null +++ b/staff/office-hours/ai-assistance.mdx @@ -0,0 +1,96 @@ +--- +title: "AI Assistance for Office Hours" +description: "Use AI tools to help support students with programming assignments through the Model Context Protocol" +icon: "sparkles" +--- + +# AI Assistance for Office Hours + +Pawtograder includes AI assistance tools that enable instructors and TAs to use AI assistants (like Claude) to help support students who are struggling with errors in their programming assignments. + +## Overview + +The AI assistance feature uses the Model Context Protocol (MCP) to provide AI assistants with secure, read-only access to relevant course data. This allows AI tools to: + +- Fetch help requests and discussion threads +- Review student submissions and test results +- Analyze build output and error messages +- Provide context-aware suggestions for helping students + + +AI assistance is only available to instructors and graders. Students cannot access these tools directly. All access is authenticated through Supabase OAuth. + + +## Features + +### AI Help Button + +When viewing a help request chat or discussion thread, instructors and TAs will see an **AI Help** button. Clicking this button: + +- Generates an MCP context for the current conversation +- Provides a connection string that can be used with any MCP-compatible client +- Includes relevant assignment context, submission data, and test results + +### Available Tools + +The MCP server provides several tools for AI assistants: + +- **Fetch help requests**: Retrieve details about student help requests +- **View discussion threads**: Access course discussion posts and replies +- **Review submissions**: Examine student code submissions +- **Analyze test results**: Review autograder output and test failures +- **Check build output**: Inspect compilation errors and build logs + +### Privacy Protection + +The AI assistance system is designed with privacy in mind: + +- Never exposes the users table or private profile information +- Only provides access to course-related data +- Requires instructor or grader authentication +- All access is logged and auditable + +## Setting Up AI Assistance + +### Adding Assignment Context + +To help AI assistants provide better support, you can add a `handout_url` to your assignments. This URL should point to the assignment handout or instructions, which provides important context for the AI assistant. + +To add a handout URL: + +1. Navigate to the assignment settings +2. Add the URL to the assignment handout in the `handout_url` field +3. Save the assignment + +The AI assistant will use this context when analyzing student submissions and providing suggestions. + +## Using AI Assistance + +### With Claude Desktop + +1. Click the **AI Help** button in a help request or discussion thread +2. Copy the MCP connection string +3. Add the connection to your Claude Desktop MCP configuration +4. Claude will have access to the relevant course context + +### With Other MCP Clients + +Any MCP-compatible client can connect to the Pawtograder MCP server using the connection string provided by the AI Help button. + +## Best Practices + +- **Review AI suggestions**: Always review and verify AI-generated suggestions before sharing with students +- **Use for context**: AI assistance is most helpful for understanding error patterns and suggesting debugging approaches +- **Maintain privacy**: Don't share AI-generated content that includes sensitive student information +- **Provide feedback**: If AI suggestions are unhelpful, use that as an opportunity to improve assignment documentation + +## Technical Details + +The MCP server is built into Pawtograder and uses: + +- Supabase OAuth for authentication +- Read-only database access +- Secure token-based authorization +- Standard MCP protocol for tool invocation + +For more technical details about the MCP implementation, see the [Developers documentation](/developers/intro). diff --git a/staff/office-hours/intro.mdx b/staff/office-hours/intro.mdx index 43b2311..4ea4db2 100644 --- a/staff/office-hours/intro.mdx +++ b/staff/office-hours/intro.mdx @@ -65,9 +65,42 @@ When a student has an active help request, it follows them around the site. They This ensures students never miss updates, even while browsing the discussion board or other course pages. +## Get Help Button + +Students can access office hours from anywhere in the course through a persistent **Get Help** button that appears site-wide. This button: + +- Opens a help drawer showing office hours status and available queues +- Displays which queues are currently active ("Happening now") +- Shows the weekly queue schedule with live status highlights +- Provides quick access to create new help requests +- Shows demo queues for testing (if configured) + +The help drawer includes: + +- **Office Hours Status Card**: Overview of current office hours availability +- **Per-Queue Cards**: Individual cards for each help queue showing status and schedule +- **Weekly Schedule**: Visual calendar showing when each queue is staffed + +## Notifications + +Students receive comprehensive notifications about their help requests: + +- **Browser Notifications**: Desktop notifications when staff responds or starts a video call +- **Title Bar Updates**: Browser tab title shows unread message counts +- **Sound Alerts**: Optional audio notifications for new messages +- **Favicon Badge**: Visual indicator in browser tab for unread messages + +Students can test notifications and manage preferences from the help request interface. + ## Resolving Requests -A help request can be marked as "Resolved" by either a staff member or a student. +Students can resolve their own help requests from either the floating banner or the main chat view. When resolving, they select a resolution status: + +- **Resolved - Got help**: Student received assistance +- **Resolved - Figured it out**: Student solved the problem independently +- **Resolved - No longer needed**: Request is no longer relevant + +Staff members can also mark requests as resolved. ## Feedback and Analytics diff --git a/students/assignments/create-submission.mdx b/students/assignments/create-submission.mdx index 09500b4..f81c504 100644 --- a/students/assignments/create-submission.mdx +++ b/students/assignments/create-submission.mdx @@ -25,4 +25,16 @@ The "Files" tab shows the files that your instructor configured for the submissi Files tab displaying submitted code files +### File Preview + +Pawtograder supports previewing various file types in your submission: + +- **Code files**: Syntax-highlighted display of source code +- **Markdown files**: Rendered markdown preview with formatting +- **Binary files**: Supported for submission (images, PDFs, etc.) + + +Markdown files (`.md`, `.markdown`) are automatically rendered with formatting, making it easy to review documentation or README files in your submission. + + These pages also have a link to the specific GitHub commit that was used to create the submission, and a link to download that complete repository as a zip file. diff --git a/students/office-hours.mdx b/students/office-hours.mdx index 9e6e241..9d71437 100644 --- a/students/office-hours.mdx +++ b/students/office-hours.mdx @@ -86,6 +86,17 @@ Once you submit a help request: Your active help request follows you around the site! You'll see status updates and video call notifications even while browsing the discussion board or other pages. +### Notifications + +Pawtograder provides multiple notification options to ensure you never miss updates about your help request: + +- **Browser Notifications**: Desktop notifications when staff responds or starts a video call (requires permission) +- **Title Bar Updates**: Your browser tab title shows unread message counts +- **Sound Alerts**: Optional audio notifications for new messages +- **Favicon Badge**: Visual indicator in your browser tab when you have unread messages + +You can test notifications and manage your preferences from the help request interface. + ## Help Request Status Your help request will show different statuses: @@ -94,4 +105,17 @@ Your help request will show different statuses: - **In Progress**: A staff member is actively helping you - **Resolved**: Your request has been marked as complete +## Resolving Your Request + +When you're ready to close your help request, you can resolve it yourself from either: + +- The floating help request banner that follows you around the site +- The main chat view in your help request + +You'll be prompted to select a resolution status to help course staff understand the outcome: + +- **Resolved - Got help**: You received assistance and your issue was addressed +- **Resolved - Figured it out**: You solved the problem on your own +- **Resolved - No longer needed**: The request is no longer relevant + After receiving help, you can provide feedback on the assistance you received. This feedback is visible only to instructors and helps improve the office hours experience.