|
1 | 1 | --- |
2 | 2 | sidebar_position: 4 |
3 | 3 | --- |
4 | | -# Optimize Workflows End-to-End by Tracing them. |
| 4 | +# Optimize Workflows End-to-End. |
5 | 5 |
|
6 | | -Codeflash supports optimizing an entire Python script end to end by tracing the execution of the script and generating Replay Tests. Tracing means following the execution of a script and capturing the inputs to all the functions called, so they can be replayed again while optimizing. These Replay Tests can be used to optimize all the functions called in the script. |
| 6 | +Codeflash supports optimizing an entire Python script end-to-end by tracing the script's execution and generating Replay Tests. Tracing follows the execution of a script, profiles it and captures inputs to all called functions, allowing them to be replayed during optimization. Codeflash uses these Replay Tests to optimize all functions called in the script, starting from the most important ones. |
7 | 7 |
|
8 | | -## Motivation for Tracing a workflow |
| 8 | +To optimize a script, `python myscript.py`, replace `python` with `codeflash optimize` and run the following command: |
9 | 9 |
|
10 | | -One of the hard problems with optimizing new code is verifying correctness and performance gains. |
11 | | -The way Codeflash verifies correctness and performance gains is by running the function under optimization against a set of test cases. |
12 | | -These test cases can be part of your existing unit test suite or generated by Codeflash. However, it's tedious to write test cases for every function you want to optimize, plus it's hard to come up with test cases for complex inputs and inputs that cover all edge cases. |
13 | | -Additionally, running the function with these test cases is not a great way to verify performance gains as the test cases might not be representative of the real-world usage of the function. |
| 10 | +```bash |
| 11 | +codeflash optimize myscript.py |
| 12 | +``` |
| 13 | + |
| 14 | +To optimize code within pytest tests that you normally run like `python -m pytest tests/`, use this command: |
14 | 15 |
|
15 | | -Codeflash Tracer solves these issues. |
| 16 | +```bash |
| 17 | +codeflash optimize -m pytest tests/ |
| 18 | +``` |
16 | 19 |
|
17 | | -## What is Codeflash Tracer? |
18 | | -Codeflash Tracer is a tool that traces the execution of your workflow and generates a set of test cases that are derived from how your code is actually run. |
| 20 | +This powerful command creates high-quality optimizations, making it ideal when you need to optimize a workflow or script. The initial tracing process is slow, so try to limit your script's runtime to under 1 minute for best results. If your workflow is longer, consider breaking it into smaller sections and optimizing them separately with limited but representative data. |
| 21 | + |
| 22 | +## What is the codeflash optimize command? |
| 23 | + |
| 24 | +`codeflash optimize` does everything that an expert engineer would do while optimizing a workflow. It profiles your code, traces the execution of your workflow and generates a set of test cases that are derived from how your code is actually run. |
19 | 25 | Codeflash Tracer works by recording the inputs of your functions as they are called in your codebase. These inputs are then used to generate test cases that are representative of the real-world usage of your functions. |
20 | 26 | We call these generated test cases "Replay Tests" because they replay the inputs that were recorded during the tracing phase. |
21 | 27 |
|
22 | 28 | Then, Codeflash Optimizer can use these replay tests to verify correctness and calculate accurate performance gains for the optimized functions. |
23 | | -Using Replay Tests, Codeflash can verify that the optimized function produces the same output as the original function and also measure the performance gains of the optimized function on the real-world inputs. |
24 | | -This way you can be _sure_ that the optimized function causes no changes of behavior for the traced workflow and also, that it is faster than the original function. |
| 29 | +Using Replay Tests, Codeflash can verify that the optimized functions produce the same output as the original function and also measure the performance gains of the optimized function on the real-world inputs. |
| 30 | +This way you can be *sure* that the optimized function causes no changes of behavior for the traced workflow and also, that it is faster than the original function. |
25 | 31 |
|
26 | | -## Using Codeflash Tracer |
| 32 | +## Using codeflash optimize |
27 | 33 |
|
28 | | -Codeflash Tracer can be used in two ways: |
| 34 | +Codeflash Tracer can be used in three ways: |
29 | 35 |
|
30 | | -1. **As a command line module -** |
| 36 | +1. **As an integrated command** |
31 | 37 |
|
32 | | - You can use Codeflash Tracer as a module when you run Python. |
33 | | - If you run a Python script as follows - |
| 38 | + If you run a Python script as follows |
| 39 | + |
34 | 40 | ```bash |
35 | 41 | python path/to/your/file.py --your_options |
36 | 42 | ``` |
37 | | - You can trace the execution of the script by running - |
| 43 | + |
| 44 | + You can start tracing and optimizing your code with the following command |
| 45 | + |
38 | 46 | ```bash |
39 | | - python -m codeflash.tracer -o codeflash.trace path/to/your/file.py --your_options |
| 47 | + codeflash optimize path/to/your/file.py --your_options |
40 | 48 | ``` |
41 | 49 |
|
42 | | - So adding a `-m codeflash.tracer -o codeflash.trace` before your script will trace the execution of the script and save the trace to a file called `codeflash.trace`. |
43 | | - If your script itself runs as a module, you can run it as follows - |
| 50 | + The above command should suffice in most situations. You can add a argument like `codeflash optimize -o trace_file_path.trace` if you want to customize the trace file location. Otherwise, it defaults to `codeflash.trace` in the cwd. |
| 51 | + |
| 52 | +2. **Trace and optimize as two separate steps** |
| 53 | + |
| 54 | + If you want more control over the tracing and optimization process. You can trace first and then optimize with the replay tests later. Each replay test is associated with a trace file. |
| 55 | + |
| 56 | + To first create just the trace file, run |
| 57 | + |
| 58 | + ```python |
| 59 | + codeflash optimize -o trace_file.trace --trace-only path/to/your/file.py --your_options |
| 60 | + ``` |
| 61 | + |
| 62 | + This will create a replay test file. To optimize with the replay test, run the |
| 63 | + |
44 | 64 | ```bash |
45 | | - python -m codeflash.tracer -o codeflash.trace -m path.to.your.module --your_options |
| 65 | + codeflash --replay-test /path/to/test_replay_test_0.py |
46 | 66 | ``` |
47 | | - More Options: |
48 | | - - `--max-function-count`: The maximum number of times to trace a single function. More calls to a function will not be traced. Default is 100. |
| 67 | + |
| 68 | + More Options: |
49 | 69 | - `--tracer-timeout`: The maximum time in seconds to trace the entire workflow. Default is indefinite. This is useful while tracing really long workflows. |
50 | | -2. **As a Context Manager -** |
51 | | - |
52 | | - You can also use Codeflash Tracer as a context manager in your codebase. |
53 | | - You can wrap the code you want to trace in a `with` statement as follows - |
54 | | - ```python |
55 | | - from codeflash.tracer import Tracer |
56 | | - |
57 | | - with Tracer(output="codeflash.trace"): |
58 | | - # Your code here |
59 | | - ``` |
60 | | - This is useful to only trace and optimize a part of your executable, not the entire script. |
61 | | - Sometimes, if using the tracer as a module fails, then the Context Manager can also be used to trace the code sections. |
62 | | - |
63 | | - More Options: |
| 70 | +3. **As a Context Manager -** |
| 71 | + |
| 72 | + To trace only very specific sections of your codeflash, You can also use the Codeflash Tracer as a context manager. |
| 73 | + You can wrap the code you want to trace in a `with` statement as follows - |
| 74 | + |
| 75 | + ```python |
| 76 | + from codeflash.tracer import Tracer |
| 77 | + |
| 78 | + with Tracer(output="codeflash.trace"): |
| 79 | + # Your code here |
| 80 | + ``` |
| 81 | + |
| 82 | + Sometimes, if using the tracer as a module fails, then the Context Manager can also be used to trace the code sections. This also is much faster than tracing the whole script. |
| 83 | + |
| 84 | + After this finishes, you can optimize using the replay tests |
| 85 | + |
| 86 | + ```bash |
| 87 | + codeflash --replay-test /path/to/test_replay_test_0.py |
| 88 | + ``` |
| 89 | + |
| 90 | + More Options to the Tracer: |
| 91 | + |
64 | 92 | - `disable`: If set to `True`, the tracer will not trace the code. Default is `False`. |
65 | 93 | - `max_function_count`: The maximum number of times to trace a single function. More calls to a function will not be traced. Default is 100. |
66 | 94 | - `timeout`: The maximum time in seconds to trace the entire workflow. Default is indefinite. This is useful while tracing really long workflows, to not wait indefinitely. |
67 | 95 | - `output`: The file to save the trace to. Default is `codeflash.trace`. |
68 | 96 | - `config_file_path`: The path to the `pyproject.toml` file which stores the Codeflash config. This is auto-discovered by default. |
69 | | - You can also disable the tracer in the code by setting the `disable=True` option in the `Tracer` constructor. |
70 | | - |
71 | | -## Optimizing with Replay Tests |
72 | | -After the tracing phase is complete, the tracer will generate a trace file as well as a Replay Test file. |
73 | | -The path of the generated replay test is printed on the console after the tracing is complete. It will be located in your tests directory and have a name like `test_file_getting_traced__replay_test_0.py`. |
74 | | -The Replay Test file is a Python test file that when run will call the traced functions with the recorded inputs, i.e. replay them. |
75 | | -Now Codeflash Optimizer can use these Replay Tests to verify correctness and calculate performance gains of the optimized functions. |
76 | | - |
77 | | -To optimize all the functions traced, you can run the following command - |
78 | | -```bash |
79 | | -codeflash --replay-test tests/test_file_getting_traced__replay_test_0.py |
80 | | -``` |
81 | | -Codeflash will auto-discover all the functions that were traced, and use the replay tests, plus will discover existing unit tests and generate more tests to get the best optimizations. |
82 | | -Codeflash will open pull requests with the optimized functions as it finds them, which should speed up your end to end workflow! |
83 | | - |
84 | | - |
| 97 | + You can also disable the tracer in the code by setting the `disable=True` option in the `Tracer` constructor. |
0 commit comments