Simplified Wandb View/pruning

[Jack-Khuu]

Love all the charts we have, but it is a tad overwhelming (and some that look worse that it actually is, just because of context).

Should we look into either (a) documentation (b) simplified view or (c) reduce the number of charts?

Thoughts @felipemello1?

[felipemello1]

yes! I think you are the 4th person to say that haha. Lets prune. Allen already opened an issue: #372

What do you mean by "(and some that look worse that it actually is, just because of context)."?

Did you have any specific solution in mind? Like:
a) are the graph useless and should just be deleted?
b) should we have a flag for "enable_perf_metrics", and if false, removes everything "_perf"?
c) anything else?

Docs are here: #380

[Jack-Khuu]

(and some that look worse that it actually is, just because of context).

This came out more negative than i intended 😅. I was mostly highlighting red herrings, where a graph looks active (e.g. std of rewards), but is actually normal given the metric or irrelevant to the current change

a) are the graph useless and should just be deleted?

For example metrics that are usually flatline, it just takes up visual real estate. Ofc ignore me if these are actually useful and I'm just a noob

Another would be count_weight_updates

b) should we have a flag for "enable_perf_metrics", and if false, removes everything "_perf"?

Actually now that you mention it, yes! This would easily make things super clean

c) anything else?

This is just a wish list (probably overkill): within subsections there's charts that provide signal, but aren't the first line signal.
For example:

• I always look at sum_MathReward_reward, but rarely do more than glance at std_MathReward_reward
• Under rl_trainer there's 3 graphs:- I only ever look at avg_grpo_loss

I'd want to have the other graphs when i need them, but I don't necessarily need them to be the same weight

[felipemello1]

a) lr could change if we have a scheduler. Training steps could be different if we do grad_acc. Count weight updates is more of a sanity check. But i see, some graphs can be flat and we should disable in this case. I agree!

b) cool

c) that makes sense. I am not sure how to handle it. Here is what comes to my mind:
- 1) Tag metrics and let users filter on those, similar to logging debugging, info, warn etc.
- 2) Let users add custom filtering metrics, eg: MetricCollector(filter_fn=my_filter_fn)

It sounds like (1) would be a good one. But the graphs wouldnt be there for when you need then. Maybe there is room for better metric breakdown? e.g. reward/ , reward_details/, and users can minimize reward_details?

[Jack-Khuu]

But the graphs wouldnt be there for when you need then

Not familiar, but i wonder if there's some magic we can do with wandb's Report feature (torchforge/grpo-training/reports/Test-Report--Vmlldzo4NTA3). It let's us collect whatever we want in terms of metrics, but only shows the ones we care about

I'll DM you an example

[Jack-Khuu]

API actually looks pretty straight forward: https://github.com/wandb/wandb-workspaces