Skip to content

WIP: Support avg time_series aggregation #4298

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

simonswine
Copy link
Contributor

@simonswine simonswine commented Jul 14, 2025

This implements an avg time series aggregation. Achieving an avg
aggration is not trivial:

  • On ingest we split each profile series into smaller smaller ones,
    by using the dynamic labels.

This PR will reaassemble profiles matching with matching ID/timestamp
and build an avg per series and the aggregate the averages per sum.

So lets say we have a 3 replica deployment with each using exactly 1GiB
of in use memory.

We would be expect each pods series to be a flat 1GiB (no matter how
many points fall into the step size) and when those 3 pods are combined,
we would expect a flat 3GiB line.

Drawing:


                                                       │                                                                                    
                     Step A: 0 - 30s                   │         Step B: 30 - 60s                                                           
                                                       │                                                                                    
──────────────x───────────────────────────x────────────┼──────────────────x───────────────────────────x──        Pod A (scape interval 15s) 
              2 GiB                       1.8 GiB      │                  2.4 GiB                     2 GiB                                 
                                                       │                                                                                    
                                                       │                                                                                    
                                                       │                                                                                    
                                                       │                                                                                    
─x──────────────────────────x──────────────────────────┼───x──────────────────────────────x──────────────        Pod B (scrape interval 15s)
 32GiB                      40GiB                      │   41 GiB                         42 GiB                                            
                                                       │                                                                                    
                                                       │                                                                                    
                                                       │                                                                                    
─────────x─────────────────────────────────────────────┼──────────x──────────────────────────────────────        Pod C (scrape interval 30s)
         0.5GiB                                        │          0.5GiB                                                                    
                                                       │                                                                                    
                                                       │                                                                                    
                                                       │                                                                                    
      Avg aggreation:                                  │                                                                                    
                                                       │                                                                                    
       Pod A :   1.9 GiB                               │     Pod A :   2.2 GiB                                                              
       Pod B :  36.0 GiB                               │     Pod B :  41.5 GiB                                                              
       Pod C :   0.5 GiB                               │     Pod C :   0.5 GiB                                                              
       -----------------                               │     ----------------                                                               
                38.4 GiB                               │                                                                                    

This implements an avg time series aggregation. Achieving an avg
aggration is not trivial:

- On ingest we split each profile series into smaller smaller ones,
  by using the dynamic labels.

This PR will reaassemble profiles matching with matching ID/timestamp
and build an avg per series and the aggregate the averages per sum.

So lets say we have a 3 replica deployment with each using exactly 1GiB
of in use memory.

We would be expect each pods series to be a flat 1GiB (no matter how
many points fall into the step size) and when those 3 pods are combined,
we would expect a flat 3GiB line.
@simonswine simonswine self-assigned this Jul 15, 2025
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this file needed?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants