Paper: MOAB Fusion Network Experiment

[Sukh-P]

Arxiv/Blog/Paper Link

Paper from Queen Mary's University (@omniaalwazzan) can be found here and code related to the paper here

Detailed Description

This paper from Queen Mary's University DERI group presents a new type of fusion architecture (Multi-Modal Outer Arithmetic Block, MOAB) which is fusion network which uses outer arithmetic (e.g. outer product, addition etc) to create interaction vectors, this is applied to a problem which has similarities to PVNet e.g. multimodal inputs and separate encodings before fusion of the 1d vectors.

Would be an interesting experiment to implement and train a MOAB style fusion network instead of the concatenation fusion network we currently use to compare forecasting results (would be good to compare other fusion networks too).

[Sukh-P]

Was able to get something running for this https://wandb.ai/openclimatefix/moab_experiments/runs/haq0wc07 but there was a bug so it didn't finish it's run fully, wasn't better than baseline initially but want to try to run it again

[omniaalwazzan]

Hi Sukh-P,

Thanks for sharing the run and config. I took a thorough look at the plots. A few things stood out:

The learning rate stayed flat at 1e-4 throughout the run. That’s expected with your current setup, since you’re using ReduceLROnPlateau and the validation loss kept improving, so the scheduler never actually kicked in. So the flat curve isn’t a bug; it’s just how that scheduler works. That said, it might also explain why the model didn’t really pull ahead of the baseline, since the LR never decayed. It could be worth trying a different schedule (e.g., cosine or step decay) to see if that gives the model more of an edge.

On the validation MAE: compared to persistence (~0.12), the model looks good, it gets down to ~0.028, so it’s clearly better. When comparing your baseline run and the MOAB experiment side by side, though, they’re almost identical, with the baseline a touch ahead at the end (the gap is tiny).

The quantile losses are coming down smoothly, and the calibration plots look solid (median and lower quantiles line up nicely; only the 0.98 quantile is a bit under). That weird drop to zero at the very end is just a logging artifact from the crash.

Across horizons, MSE and MAE both trend downward, with longer horizons naturally being a bit harder. And from the sample forecasts, the model is definitely capturing the patterns; some are a bit jagged, but overall they make sense.

A couple of tweaks before re-running might help:

• Log validation metrics once per epoch instead of every step (to avoid the final zero issue), right now the validation metrics are being logged every batch during validation. That means if the last batch is empty or produces NaNs, W&B records that as the final value, which is why the summary sometimes shows zeros or NaNs. Instead, it’s better to collect the metrics across the whole validation set and then log the averaged result once per epoch. That way, the summary reflects the actual performance for the epoch, not just whatever happened in the final batch.
• Try lowering the AdamW weight decay (0.25 feels heavy, maybe 0.01–0.05).

On the checkpoint path:

Right now, the config sets dirpath:
PLACEHOLDER/${model_name}.

Since PLACEHOLDER isn’t replaced, Lightning literally writes into a folder called PLACEHOLDER.

That’s why you see warnings like:

Checkpoint directory /home/.../PLACEHOLDER/haq0wc07 exists and is not empty.

Training still works, but this can cause crashes or overwrite issues.

A safer option is:

dirpath: checkpoints/${model_name}

This was actually really interesting to think through. I had to stretch my brain a bit, which I enjoyed. Please don’t hesitate to send me any issues you run into. I’d be more than happy to help out. I hope this helps, and I'm looking forward to seeing how your next run goes!

Cheers,
Omnia

[Sukh-P]

Hey @omniaalwazzan, thanks so much for your message and for taking a look! This was a quick and dirty implementation but I wanted to get some results to see how it was doing, I used the MOAB technique on our two main inputs which are a numerical weather prediction and satellite images as they are the two most different types of modalities we pass in, they then combine with some other more minor inputs id embeddings and some sun position encodings.

Yes the learning rate drop didn't kick in as I had not set the early stopping patience > the patience on the LR scheduler, need to change that for the next run.

Yes the weight decay is heavy, we have had good results from using this but I could play with that hyperparameter. And yes I didn't overwrite some placeholder directories but actually I think that warning is unrelated as even when I have changed it it appears, one to look into but not impacting things currently as far as I can see.

Thanks for your kind offer, when I get a bit of time to revisit and rerun this I will update this issue and let if you know if I have any issues/blockers!

[omniaalwazzan]

Thanks for explaining, that makes sense. Good point about the patience settings, too; adjusting that so the LR scheduler actually kicks in might help in the next run. I get you on the weight decay; if it’s worked well in your past setups, it’s fair to start there, though it could still be interesting to sweep it alongside the scheduler change. Makes sense for the placeholder! This really does seem like an exciting project. I’m always happy to hear about fusion. You’ve got nice complementary views that could share strong signals, and I strongly hope MOAB can help here 🙂. This was really fun to think through, so thanks again for sharing, and of course, if you hit any issues or blockers, please do send them my way. I'd be happy to help out. Best, Omnia

…

On Fri, Sep 19, 2025 at 10:57 AM Sukhil Patel ***@***.***> wrote: *Sukh-P* left a comment (openclimatefix/pvnet#272) <#272 (comment)> Hey @omniaalwazzan <https://github.com/omniaalwazzan>, thanks so much for your message and for taking a look! This was a quick and dirty implementation but I wanted to get some results to see how it was doing, I used the MOAB technique on our two main inputs which are a numerical weather prediction and satellite images as they are the two most different types of modalities we pass in, they then combine with some other more minor inputs id embeddings and some sun position encodings. Yes the learning rate drop didn't kick in as I had not set the early stopping patience > the patience on the LR scheduler, need to change that for the next run. Yes the weight decay is heavy, we have had good results from using this but I could play with that hyperparameter. And yes I didn't overwrite some placeholder directories but actually I think that warning is unrelated as even when I have changed it it appears, one to look into but not impacting things currently as far as I can see. Thanks for your kind offer, when I get a bit of time to revisit and rerun this I will update this issue and let if you know if I have any issues/blockers! — Reply to this email directly, view it on GitHub <#272 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AI4WPZFN3ULPQSK6YDJ4TSL3TQKUVAVCNFSM6AAAAABREZOL3OVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTGMJSGUZTINJXGI> . You are receiving this because you were mentioned.Message ID: ***@***.***>