Skip to content

Optimize sliding window late data insertion with skip list #245

@yunfengzhou-hub

Description

@yunfengzhou-hub

SlidingWindowKeyedProcessFunction.java:

                if (timestamps.isEmpty() || timestamps.get(timestamps.size() - 1) < timestamp) {
                    timestampList.add(timestamp);
                } else {
                    // TODO: Use skip list to optimize the performance of inserting late data.
                    for (int insertionIdx = 0; insertionIdx < timestamps.size(); insertionIdx++) {
                        if (timestamps.get(insertionIdx) < timestamp) {
                            continue;
                        }
                        timestamps.add(insertionIdx, timestamp);
                        break;
                    }
                    timestampList.update(timestamps);
                }

Metadata

Metadata

Assignees

No one assigned

    Labels

    type:improvementimprovement to existing functionality

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions