Skip to content

Comments

feat: improve upload management: FIFO slicing + session-aware adaptiv…#195

Merged
paul-fresquet merged 6 commits intofeature/improve-upload-managementfrom
feature/improve-upload-management-fifo-slicing
Sep 5, 2025
Merged

feat: improve upload management: FIFO slicing + session-aware adaptiv…#195
paul-fresquet merged 6 commits intofeature/improve-upload-managementfrom
feature/improve-upload-management-fifo-slicing

Conversation

@Ghass-M
Copy link
Contributor

@Ghass-M Ghass-M commented Aug 27, 2025

Summary

Introduces a centralized slicing queue to enforce FIFO ordering and bounded concurrency across uploads, and resets adaptive upload parameters on session changes.

Key changes

  • New UploadSlicingManager: Adds IUploadSlicingManager and UploadSlicingManager using a Channel<Func<Task>>, FIFO processing, and max concurrency of 2; ignores stale work after session resets.
  • Session-aware adaptive reset: AdaptiveUploadController subscribes to ISessionService.SessionObservable and fully resets chunk size, parallelism, and window.
  • DI and factory wiring: Registers AdaptiveUploadController and UploadSlicingManager in SingletonsModule; resolves and injects IUploadSlicingManager in FileUploadProcessorFactory.
  • Processor integration: FileUploadProcessor enqueues slicing via IUploadSlicingManager instead of starting it directly to ensure ordered execution.
  • Tests updated: Adjusts unit tests to use SliceAndEncryptAdaptiveAsync, injects and stubs IUploadSlicingManager, and wires ISessionService mock for the controller.

Rationale

  • Problem: Parallel slicers could interleave across files, causing contention and non-deterministic ordering; adaptive state leaked across sessions.
  • Solution: Centralize slicing with FIFO + bounded concurrency and reset adaptive state on session change.

Notes

  • Backward compatibility: No public API changes; behavior is more predictable under load.
  • Configurability: MAX_CONCURRENT_SLICES is set to 2 and can be tuned later.

@Ghass-M Ghass-M requested a review from paul-fresquet August 28, 2025 15:30
@paul-fresquet paul-fresquet requested a review from Copilot August 29, 2025 07:29

This comment was marked as outdated.

Copy link
Contributor

@paul-fresquet paul-fresquet left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good job 👍 Here are a few comments


public interface IUploadSlicingManager
{
Task Enqueue(Func<Task> startSlicingAsync);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

note: the signature might be: Task<UploadProgressState> Enqueue(SharedFileDefinition sharedFileDefinition).

UploadSlicingManager would then instanciate the FileSlicer, the UploadProgressState and return the UploadProgressState.

@sonarqubecloud
Copy link

Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR introduces a centralized upload slicing queue to enforce FIFO ordering and bounded concurrency across uploads, while also implementing session-aware adaptive parameter resets to improve upload management predictability and consistency.

  • Adds UploadSlicingManager to centralize slice processing with FIFO ordering and configurable concurrency limits
  • Implements session-aware adaptive upload controller that resets parameters when sessions change
  • Refactors upload architecture to use the centralized slicing manager instead of direct slice processing

Reviewed Changes

Copilot reviewed 18 out of 18 changed files in this pull request and generated 4 comments.

Show a summary per file
File Description
UploadSlicingManager.cs New centralized manager for FIFO slice processing with bounded concurrency
AdaptiveUploadController.cs Enhanced with session-aware reset capability and improved downscaling logic
FileUploadProcessor.cs Refactored to use slicing manager instead of direct FileSlicer dependency
FileSlicer.cs Added backpressure control to prevent excessive pending slices
FileUploadCoordinator.cs Changed from bounded to unbounded channel for slice queueing
Multiple test files Updated tests to accommodate new architecture and dependencies

Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.

@paul-fresquet paul-fresquet merged commit c87dd0c into feature/improve-upload-management Sep 5, 2025
39 of 41 checks passed
@paul-fresquet paul-fresquet deleted the feature/improve-upload-management-fifo-slicing branch September 5, 2025 04:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants