Adapting OpenTSLM-Flamingo for Predictive Maintenance (Multi-Sensor Engine Data)

[aiesha-khawaja]

What Stanford Biodesign Digital Health Group-related project challenge related to?

None

Reproduction

I’m a PhD researcher working on predictive maintenance for aircraft engines and exploring OpenTSLM-Flamingo as the core model. My dataset includes 13 continuous sensor channels (CHT 1–6, EGT 1–6, Manifold Pressure).

I am trying to adapt OpenTSLM to:

1. Take a time window of multi-sensor data as input.
2. Generate a human-readable diagnostic rationale (CoT).
3. Classify the engine status (Normal / Maintenance Required / Impending Failure).

My adaptation plan:

• Create a labeled CoT dataset (256–512 timestamps per window).
• Build MaintenanceCoTQADataset.py (based on ECGQACoTQADataset.py).
• Add maintenance_loader.py similar to har_cot_loader.py.
• Add a new training stage stage_maintenance_cot in curriculum_learning.py.
• Fine-tune from stage5_ecg_cot.ckpt on the new dataset.

Expected behavior

I expected the model to process 13-channel time-series input, generate meaningful CoT rationales, and correctly classify engine status after fine-tuning, similar to the ECG-QA or HAR CoT setups.

Additional context

I’m adapting OpenTSLM-Flamingo for academic research in predictive maintenance. Since OpenTSLM integrates time-series and text reasoning using gated cross-attention and CoT training, it seems ideal for this domain.

I’d appreciate guidance on:

• Whether this adaptation workflow matches your intended approach.
• Any architectural or preprocessing considerations for 13 multivariate channels.
• Best practices for fine-tuning OpenTSLM-Flamingo on smaller domain-specific CoT datasets.

Code of Conduct

• I agree to follow this projects's Code of Conduct and Contributing Guidelines

[PSchmiedmayer]

@aiesha-khawaja Thank you for reaching out!
Tagging @ThomasKaar, @RealLast, and @max-rosenblattl here; they might have some first thoughts 👍

[aiesha-khawaja]

Thank you @PSchmiedmayer for the response. Looking forward to their insights.

[mltrinhammer]

Hi @aiesha-khawaja,
Taking the liberty of chiming in here, even though I am not part of the openTSLM author team.

I am working on a fine-tuning use-case which matches yours very much from a data engineering point of view (though very different thematically, see https://github.com/mlt94/synchrony)

To your questions:

1. I am practically using the same workflow as you describe.
2. I believe the openTSLM-Flamingo or SP frameworks should be able to handle 13 time-series inputs
3. I am planning to run the curriculum_learning.py on the first two stages ( "stage1_mcq", "stage2_captioning") before training my custom stage. I have tried using the flamingo-ECG model to run inference on my own time-series data, assuming that the most-trained model would be best; however, this turned out not to be the case, as this model would just output rationales reflecting the ECG domain. I am therefore opting for the earlier stages.

I wonder how did you did the rationale generation part (Generate a human-readable diagnostic rationale (CoT)). My data is very sensitive, so I couldnt use GPT-4o like the authors. A multimodal Gemma 3 27B turned out to be more than sufficient for my use-case, and could run on my university hardware. I would simply ask it to describe the time-series data that I would offer in a heatmap (over a subset of my features, not all 17), and it gave very satisfactory results. I found the performance to be markedly improve by using heatmaps with 8 bins over plots, as my time-series data fluctuates a lot; the model had a much easier time describing a peak in bins 5-7 than pinpointing same movement on a plot.

Feel free to reach out if you want to share further thoughts. Good luck with your research!