Refactor OutputsManager to align with default Jupyter behavior

[ellisonbg]

Summary

This PR refactors the notebook output handling system to align with standard Jupyter behavior while maintaining the performance benefits of the collaborative server architecture.

Key Improvements

OutputsManager Now Follows Default Jupyter Behavior

The refactored OutputsManager now works exactly like standard Jupyter:

• Full outputs saved to notebook files: All cell outputs are written to .ipynb files on disk, ensuring compatibility with standard Jupyter workflows
• Fast autosaving preserved: Outputs are still kept out of the YDoc using lightweight placeholders, maintaining the performance advantage of server-side collaboration
• No metadata flags required: Works transparently without special notebook metadata settings

This means users get the best of both worlds:

• ✅ Standard Jupyter file format with full outputs on disk
• ✅ Fast collaborative editing with minimal YDoc synchronization overhead
• ✅ No breaking changes to existing workflows

Dynamic Autosave Intervals

The YRoomFileAPI now implements adaptive autosave timing to optimize performance across different file sizes and I/O environments:

• Adaptive timing: Poll interval automatically adjusts based on how long saves take
  • Small files with fast I/O → shorter intervals → more responsive autosaving
  • Large files or slower I/O → longer intervals → reduces unnecessary polling overhead
• Configurable parameters:
  • min_poll_interval (default: 0.5s) - minimum autosave interval
  • poll_interval_multiplier (default: 5.0) - multiplier applied to save duration to calculate next interval
• Example: If a save takes 2 seconds, the next poll interval becomes 2s × 5 = 10s, scaling appropriately to the file size and I/O performance

This ensures the autosave system adapts to realistic file I/O conditions, providing optimal responsiveness for small files while avoiding excessive polling overhead for large files or slower storage systems.

Cleaner, Better-Documented Code

• Comprehensive documentation: All methods now have detailed docstrings explaining purpose, parameters, and return values
• Better code organization: Extracted utility functions (_create_output_url, _create_output_placeholder) and helper methods (_upgrade_notebook_format, _ensure_cell_id, _process_outputs_from_cell)
• Simplified logic: Removed complex conditional paths, making the code easier to understand and maintain
• Improved error handling: Graceful fallbacks when outputs aren't found

Expanded Test Coverage

• Comprehensive test suite (857 lines) covering all output types
• Tests for display ID tracking, notebook loading/saving workflows
• Edge case and error handling validation

Experimental Features

⚠️ Note: This PR also includes an experimental OptimizedOutputsManager that supports excluding outputs from saved notebook files via an exclude_outputs metadata flag. This feature is disabled by default and not recommended for production use.

Migration Impact

No migration needed - existing notebooks and workflows continue to work unchanged. The refactored OutputsManager is a drop-in replacement with improved code quality and documentation.

[dlqqq]

@ellisonbg Thank you for making these changes! I've reviewed the portion of changes concerning the new YRoomFileAPI auto-save behavior. Left some (non-blocking) feedback below.

@3coins can help review the changes related to outputs.

[dlqqq]

This instance attribute doesn't seem necessary, since it is only bound to the local variable save_duration in the save() method.

[dlqqq]

The poll_interval configurable trait is now only used to set the initial value of self._adaptive_poll_interval, which is reset after the very first auto-save.

I recommend removing poll_interval to avoid leading developers to confuse this with min_poll_interval. The initial value of self._adaptive_poll_interval can be set to self.min_poll_interval.

[dlqqq]

Minor suggestion: It may be worth adding validation to ensure that min_poll_interval > 0 and poll_interval_multiplier > 0. traitlets provides the @validate decorator for this: https://traitlets.readthedocs.io/en/stable/using_traitlets.html#basic-example-validating-the-parity-of-a-trait

For example:

# outside YRoomFileAPI
DEFAULT_MIN_POLL_INTERVAL = 0.5

# within YRoomFileAPI
@validate('min_poll_interval')
def _validate_min_poll_interval(self, proposal):
    if proposal['value'] <= 0:
        self.log.warning("The configured min_poll_interval cannot be <=0. Using default value instead.")
        return DEFAULT_MIN_POLL_INTERVAL
    return proposal['value']

# ... similarly for poll_interval_multiplier

[3coins]

@ellisonbg
Tested the outputs behavior which works as expected, but have some concerns about the adaptive saves.

1. If a save takes 10s with a multiplier of default 5.0, the next interval becomes 50s. For slow saves, this could result in intervals of minutes or longer. Should we introduce a max_poll_interval to avoid this?
2. The algorithm assumes save time is predictive of future save time in a linear way, which may not hold for notebook size changes (adding/removing large outputs), network conditions, system load variations or different file types. Should we add an exponential moving average?

   # Instead of using only last save duration
   if self._last_save_duration is not None:
       # Smooth with exponential moving average
       alpha = 0.3  # Smoothing factor
       smoothed_duration = (alpha * save_duration +
                           (1 - alpha) * self._last_save_duration)
   else:
       smoothed_duration = save_duration
   
   new_interval = smoothed_duration * self.poll_interval_multiplier

3. If saves fail, the adaptive interval isn't updated, should we add failure backoff?

# Exponential backoff on save failures
      self._adaptive_poll_interval = min(
          self.max_poll_interval,
          self._adaptive_poll_interval * 2
      )
      self.log.error("Save failed, backing off poll interval")

[3coins]

Here is a comparison of current algorithm with exponential moving average for some hypothetical notebook save times.

[3coins]

After speaking with @ellisonbg, it seems like he has done a few experiments, and the save times for very large notebooks (with 100s of plots) on modern hardware is <1s. The configurable multiplier does provide a lever to adjust this, which should help with containing very high save intervals. Some of the feedback that @dlqqq provided still makes sense, but overall looks good.

[Zsailer]

I'd like to get this merged today, if possible. @ellisonbg are you able to make @dlqqq's changes?

I have some changes coming, and I want to avoid a painful rebase 😅

[dlqqq]

Suggestions implemented in #169, which can be merged on top of this.

Approving to unblock. Proceeding to merge as we have 3 approvals.

[dlqqq]

@Zsailer This PR is now merged, so you're clear to open your PR! 🎉

[ellisonbg]

Thanks everyone!