-
Notifications
You must be signed in to change notification settings - Fork 0
Open
Labels
enhancementNew feature or requestNew feature or request
Description
Problem
DataSourcePipeline fetches data sources sequentially (lines 50-71), causing total sync time to equal the sum of all fetch times.
Current implementation:
do {
restoreImages = try await fetchRestoreImages(options: options)
} catch { throw error }
do {
xcodeVersions = try await fetchXcodeVersions(options: options)
} catch { throw error }Proposed Solution
Use structured concurrency to fetch independent sources in parallel:
async let restoreImages = fetchRestoreImages(options: options)
async let xcodeVersions = fetchXcodeVersions(options: options)
async let swiftVersions = fetchSwiftVersions(options: options)
let (images, xcode, swift) = try await (restoreImages, xcodeVersions, swiftVersions)Additionally:
- Add size limits for in-memory deduplication to prevent excessive memory usage
- Consider streaming/chunking for large datasets
Impact
Performance: Reduce sync time from sum(all sources) to max(source time)
Example: If 3 sources take 5s, 8s, and 3s:
- Current: 16 seconds total
- Parallel: 8 seconds total (50% faster)
Files Affected
- Examples/Bushel/Sources/BushelImages/DataSources/DataSourcePipeline.swift:50-71
- Deduplication methods (potential memory issues)
References
- PR update from writing blog post and examples MistKit#129 Review: update from writing blog post and examples MistKit#129 (comment)
- Section: Performance Considerations
- Related: Add comprehensive test suite for Bushel demo MistKit#136
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
enhancementNew feature or requestNew feature or request