CHIP : Incremental feature aggregation

[kambstreat]

Summary

Proposal to support incremental aggregations.

Why / Goal

Test Plan

• Added Unit Tests
• Covered by existing CI
• Integration tested

Checklist

• Documentation update

Reviewers

[blrnw3]

An alternative approach is to store an intermediate 'tiled' representation: each day store the aggregate for just that day, then compute the longer windows from the intermediate.
e.g. For the above example, store the count of inp_col each day, then your 3 and 10 day windows just need to sum those intermediate counts to get the final values.
The benefit here is it works for almost any kind of aggregation, including max, min etc.
I'm fairly sure this is how the 'tiled architecture' works for the online flow

[kambstreat]

@blrnw3 Yes. I am going to change the architecture. Going to get the daily aggregations and store it in table. The only change would be the way we store the IRs. For example, for avg, we need to store both sum/count.