You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: README.md
+1-38Lines changed: 1 addition & 38 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -21,44 +21,7 @@ This part of the library is under development. There are three kinds of agents:
21
21
The design is simple in that each agent is responding to state of error vs. success. In the case of a step agent, the return code determines to continue or try again. In the case of a helper, the input is typically an erroneous response (or something that needs changing) with respect to a goal.
22
22
For a manager, we are making a choice based on a previous erroneous step.
23
23
24
-
See [examples/agent](examples/agent) for an example.
25
-
26
-
#### To do items
27
-
28
-
- refactor manager to not handle prompt, just get step when retries come back.
29
-
- then need to decide how to handle kubernetes job creating additional structures.
30
-
- Get basic runner working
31
-
- Add in ability to get log and optimize - the manager will need to use goal
32
-
- We likely want the manager to be able to edit the prompt.
33
-
- should be provided with entire prompt?
34
-
- When pod pending, it can be due to resource issues (and will never start). Right now we will time out, but we should be able to catch that earlier.
35
-
36
-
#### Research Questions
37
-
38
-
**And experiment ideas**
39
-
40
-
- How do we define stability?
41
-
- What are the increments of change (e.g., "adding a library")? We should be able to keep track of times for each stage and what changed, and an analyzer LLM can look at result and understand (categorize) most salient contributions to change.
42
-
- We also can time the time it takes to do subsequent changes, when relevant. For example, if we are building, we should be able to use cached layers (and the build times speed up) if the LLM is changing content later in the Dockerfile.
43
-
- We can also save the successful results (Dockerfile builds, for example) and compare for similarity. How consistent is the LLM?
44
-
- How does specificity of the prompt influence the result?
45
-
- For an experiment, we would want to do a build -> deploy and successful run for a series of apps and get distributions of attempts, reasons for failure, and a general sense of similarity / differences.
46
-
- For the optimization experiment, we'd want to do the same, but understand gradients of change that led to improvement.
47
-
48
-
#### Observations
49
-
50
-
- Specifying cpu seems important - if you don't it wants to do GPU
51
-
- If you ask for a specific example, it sometimes tries to download data (tell it where data is)
52
-
- Always include common issues in the initial prompt
53
-
- If you are too specific about instance types, it adds node selectors/affinity, and that often doesn't work.
54
-
55
-
#### Ideas
56
-
57
-
- The manager agent is currently generated an updated prompt AND choosing the step.
58
-
- Arguably we should have a separation of responsibility so a step can ask to fix an error without a manager.
59
-
- I think we need one more level of agent - a step agent should have helper agents that can:
60
-
- take an error message and analyze to get a fix.
61
-
24
+
See [examples/agent](examples/agent) for an example, along with observations, research questions, ideas, and experiment brainstorming!
fractale agent build lammps --environment "google cloud CPU" --outfile Dockerfile --details "Ensure all globbed files from examples/reaxff/HNS from the root of the lammps codebase are in the WORKDIR. Clone the latest branch of LAMMPS."
14
14
```
15
15
16
+
Note that we are specific about the data and using CPU, which is something the builder agent would have to guess.
16
17
That might generate the [Dockerfile](Dockerfile) here, and a container that defaults to the application name "lammps"
17
18
18
19
### Kubernetes Job
@@ -27,9 +28,20 @@ kind load docker-image lammps
27
28
To start, we will assume a kind cluster running and tell the agent the image is loaded into it (and so the pull policy will be never).
fractale agent kubernetes-job lammps --environment "google cloud CPU" --context-file ./Dockerfile --no-pull --details "Run in.reaxff.hns in the pwd with lmp" --outfile ./job.yaml
31
32
```
32
33
34
+
## With Cache
35
+
36
+
The same steps can be run using a cache. This will save to a deterministic path in the present working directory, and means that you can run steps a la carte, and run a workflow later to re-use the context (and not wait again).
37
+
Note that when you save a cache, you often don't need to save the output file, because it will be the result in the context.
38
+
39
+
```bash
40
+
fractale agent build lammps --environment "google cloud CPU" --details "Ensure all globbed files from examples/reaxff/HNS from the root of the lammps codebase are in the WORKDIR. Clone the latest branch of LAMMPS." --use-cache
41
+
```
42
+
43
+
And then try running with the manager (below) with the cache to see it being used.
44
+
33
45
## Manager
34
46
35
47
Let's run with a manager. Using a manager means we provide a plan along with a goal. The manager itself takes on a similar structure to a step agent, but it has a high level goal. The manager will follow the high level structure of the plan, and step
For this first design, we are taking an approach where we only re-assess the state and go back to a previous step given a last step failure. The assumption is that if a previous step fails, we keep trying until it succeeds. We only need to backtrack if the last step in a sequence is not successful, and it is due to failure at some stage in the process. But I do think we have a few options:
62
+
We haven't hit the case yet where the manager needs to take over - that needs further development, along with being goal oriented (e.g., parsing a log and getting an output).
63
+
64
+
## Notes
65
+
66
+
#### To do items
67
+
68
+
- Figure out optimization agent (with some goal)
69
+
70
+
#### Research Questions
71
+
72
+
**And experiment ideas**
73
+
74
+
- How do we define stability?
75
+
- What are the increments of change (e.g., "adding a library")? We should be able to keep track of times for each stage and what changed, and an analyzer LLM can look at result and understand (categorize) most salient contributions to change.
76
+
- We also can time the time it takes to do subsequent changes, when relevant. For example, if we are building, we should be able to use cached layers (and the build times speed up) if the LLM is changing content later in the Dockerfile.
77
+
- We can also save the successful results (Dockerfile builds, for example) and compare for similarity. How consistent is the LLM?
78
+
- How does specificity of the prompt influence the result?
79
+
- For an experiment, we would want to do a build -> deploy and successful run for a series of apps and get distributions of attempts, reasons for failure, and a general sense of similarity / differences.
80
+
- For the optimization experiment, we'd want to do the same, but understand gradients of change that led to improvement.
81
+
82
+
#### Observations
48
83
49
-
1. Allow the manager to decide what to do on _every_ step (likely not ideal)
50
-
2. Allow step managers to execute until success, always (too much issue if a step is failing because of dependency)
51
-
3. Allow step managers to execute until success unless a limit is set, and then let the manager take over (in other words, too many failures means we hand it back to the manager to look.)
84
+
- Specifying cpu seems important - if you don't it wants to do GPU
85
+
- If you ask for a specific example, it sometimes tries to download data (tell it where data is)
86
+
- There are issues that result from not enough information. E.g., if you don't tell it what to run / data, it can only guess. It will loop forever.
87
+
- As an example, we know where in a git clone is the data of interest. The LLM can only guess. It's easier to tell it exactly.
88
+
- An LLM has no sense of time with respect to versions. For example, the reax data changed from reaxc to reaxff in the same path, and which you get depends on the clone. Depending on when the LLM was trained with how to build lammps, it might select an older (or latest) branch. Instead of a juggling or guessing game (that again) would result in an infinite loop, we need to tell it the branch and data file explicitly.
89
+
- Always include common issues in the initial prompt
90
+
- If you are too specific about instance types, it adds node selectors/affinity, and that often doesn't work.
0 commit comments