Commit aeb6577
authored
[Step 2/4] Queue Time Histgram (#6467)
# Description
Add two main class for queue time histogram lambda:
- TimeIntervalGenerator: generate list of time intervals based on the
latest timestamp from source table and target table
- generate a single data from source table if no timestamp found in
target table
- throw error if the interval is greater than maxi-interval threshold
- WorkerPoolProcessor:
- input the interval list from TimeIntervalGenerator, and process
histogram data in parrallel
- It process each interval by calling the QueueTimeProcessor.process(),
and in the end write all result to destination
# Others
- added unit tests for all utilization methods
- clickhouse_client is not thread-safe, make sure each thread has its
own client in thread pool
# Next steps
step 3
- Add HistogramGenerator
- Add runner_label logic in HistogramGenerator
- Fix the issue with GithubConfigRetriever
- Change the target db to misc. instead of fortesting
step 4
- setup AWS resources
- setup deployment1 parent 4e8d1bd commit aeb6577
File tree
2 files changed
+989
-221
lines changed- aws/lambda
- oss_ci_job_queue_time
- tests
2 files changed
+989
-221
lines changed
0 commit comments