@@ -7,47 +7,47 @@ Codeflash reports benchmarking results that look like this.
77To measure the runtime of code, Codeflash runs the function multiple times, on several inputs
88and sums the minimum time of each input to get the total runtime.
99
10- A simplified pseudocode of the benchmarking that Codeflash implements looks like this -
10+ A simplified pseudocode of Codeflash benchmarking looks like this -
1111
1212``` python
1313loops = 0
14- start_time = time.time()
1514min_input_runtime = [float (' inf' )] * len (test_inputs)
15+ start_time = time.time()
1616while loops <= 5 or time.time() - start_time < 10 :
1717 loops += 1
1818 for input_index, input in enumerate (test_inputs):
19- t = time(function (input ))
19+ t = time(function_to_optimize (input ))
2020 if t < min_input_runtime[input_index]:
2121 min_input_runtime[input_index] = t
2222total_runtime = sum (min_input_runtime)
2323number_of_runs = loops
2424```
2525
26- The above code runs the function multiple times on different inputs and takes the minimum time for each input.
26+ The above code runs the function multiple times on different inputs and uses the minimum time for each input.
2727
2828In this document we explain -
29- - how we measure the runtime of code
30- - how we determine if an optimization is actually faster
31- - why we measure the timing as best of N runs
32- - how we measure the runtime when we run on a wide variety of test cases.
29+ - How we measure the runtime of code
30+ - How we determine if an optimization is actually faster
31+ - Why we measure the timing as best of N runs
32+ - How we measure the runtime when we run on a wide variety of test cases.
3333
3434## Goals of Codeflash auto-benchmarking
3535
36- A core design of Codeflash is that it does not make assumptions
37- on the types of optimizations that might be faster. It generates multiple possible optimizations with LLMs and then automatically benchmarks the code
36+ A core principle of Codeflash is that it does not make assumptions
37+ on what types of optimizations that might be faster. It generates multiple possible optimizations with LLMs and then automatically benchmarks the code
3838on a variety of inputs to verify empirically if the optimization is actually faster.
3939
4040The goals of Codeflash auto-benchmarking are:
4141- Accurately measure the runtime of code.
42- - Measure runtime of a wide variety of codes .
42+ - Measure runtime for a wide variety of code .
4343- Measure runtime on a variety of inputs.
44- - Do all the above on real machine, where there might be other processes running, causing timing measurement noise.
45- - Make a binary decision on whether an optimization is faster or not.
44+ - Do all the above on a real machine, where there might be other processes running, causing timing measurement noise.
45+ - Finally make a binary decision whether an optimization is faster or not.
4646
47- ## A useful train analogy -
47+ ## Racing Trains as an analogy -
4848
4949Imagine that you are a boss at a train company who wants to purchase a train to run between the two cities of San Francisco and Los Angeles.
50- You are deciding between two trains, Train A and Train B, and want to run the train that is the fastest between the two cities.
50+ You want to decide between two trains, Train A and Train B, and want to run the train that is the fastest between the two cities.
5151
5252You can measure the speed of the trains by timing how long it takes to go from San Francisco to Los Angeles.
5353
0 commit comments