[GSoC 2025] Optimize Data Generation Pipeline for DIA-MS Diffusion Model

[singjc]

Background

Our project aims to train a diffusion model to deconvolute complex DIA-MS/MS data. A critical bottleneck in our current workflow is the data generation step that prepares input for model training.

Current Challenges

• The current data generation process takes approximately 1.5 days and consumes ~800GB RAM for processing a single DIA isolation window
• The process extracts MS1 and MS2 spectra within an isolation window across a sliding window of N overlapping retention time points
• This creates unnecessarily dense data which may be contributing to the performance issues. (However, it's done this way to ensure that at least one of the input training samples will contain the elution peak profile of an analyte)

Task Objectives

• Analyze the current data generation pipeline to identify inefficiencies
• Implement and test optimizations to reduce memory usage and processing time
• Evaluate alternative approaches such as:
  • Using sequential (non-overlapping) retention time windows instead of sliding windows
  • Considering strategies to handle peak cutoffs at window boundaries
  • Identifying redundant data that can be safely discarded

Deliverables

• Modified code with documented optimizations
• Performance comparison between original and optimized implementations
• Analysis of any trade-offs in data quality vs. performance
• Recommendations for future improvements

Resources

• Example data and processing notebook: nbs/hela_2018.ipynb
• Background literature for understanding the structure of DIA-MS/MS data DIA-MS data in our wiki

Difficulty

• Beginner to Intermediate - This task provides a good entry point to understand our data pipeline while making meaningful contributions to the project's efficiency.

[nithin200417]

Hi @singjc,

I’m working on optimizing the data generation pipeline and need some input to proceed:

1. Could you share an example input file (e.g., .sqmass) and typical parameters (e.g., window_size, sliding_step) for data_generation.py?

2. I’m testing alternatives like sequential (non-overlapping) windows and handling peak cutoffs to reduce memory/time. Are there constraints (e.g., minimum overlap) or metrics to evaluate trade-offs?

Once I have the input, I’ll run the current script for a baseline, test optimizations, and share results. Let me know if there’s anything else to consider!

Thanks!

[singjc]

Sorry for the late reply

Could you share an example input file (e.g., .sqmass) and typical parameters (e.g., window_size, sliding_step) for data_generation.py?

I have uploaded the raw data here

For parameters used for the current data being generated, see here

I’m testing alternatives like sequential (non-overlapping) windows and handling peak cutoffs to reduce memory/time. Are there constraints (e.g., minimum overlap) or metrics to evaluate trade-offs?

We initially tried sequential windows, but we ran into issues where the peak would potentially get cutoff, which is why we tried to switch to tiling overlapping windows, but this results in a lot of extra data and slow processing. There doesn't necessarily need to be overlaps between the data, we just want to ensure that the slice of data that we have and train on actually contains a valid peak in it. One thing we could do, is perform a preliminarily peak peaking search and use those seeds to extract data around, which would ensure that there is at least a peak in each slice of the data.