You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: README.md
+18-8Lines changed: 18 additions & 8 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -9,13 +9,29 @@ This library is primarily being used for development for the descriptive thrust
9
9
10
10
## Design
11
11
12
-
### Simple
12
+
### Agents
13
+
14
+
The `fractale agent` command provides means to run build, job generation, and deployment agents.
15
+
This part of the library is under development. There are three kinds of agents:
16
+
17
+
-`step` agents are experts on doing specific tasks (do hold state)
18
+
-`manager` agents know how to orchestrate step agents and choose between them (don't hold state, but could)
19
+
-`helper` agents are used by step agents to do small tasks (e.g., suggest a fix for an error)
20
+
21
+
The design is simple in that each agent is responding to state of error vs. success. In the case of a step agent, the return code determines to continue or try again. In the case of a helper, the input is typically an erroneous response (or something that needs changing) with respect to a goal.
22
+
For a manager, we are making a choice based on a previous erroneous step.
23
+
24
+
See [examples/agent](examples/agent) for an example, along with observations, research questions, ideas, and experiment brainstorming!
25
+
26
+
### Job Specifications
27
+
28
+
#### Simple
13
29
14
30
We provide a simple translation layer between job specifications. We take the assumption that although each manager has many options, the actual options a user would use is a much smaller set, and it's relatively straight forward to translate (and have better accuracy).
15
31
16
32
See [examples/transform](examples/transform) for an example.
17
33
18
-
### Complex
34
+
####Complex
19
35
20
36
We want to:
21
37
@@ -32,12 +48,6 @@ For graph tool:
32
48
conda install -c conda-forge graph-tool
33
49
```
34
50
35
-
## Questions
36
-
37
-
- Should other subsystem types have edges? How used?
38
-
- Should we try to map them to nodes in the graph or use another means (or assume global across cluster nodes as we do now)?
39
-
- Can we simplify spack subsystem graph (it's really big...)
Let's use fractale to run build, execute, and deploy agents. Right now we will run these a-la-carte, and eventually we will group them together to loop back and repeat steps if needed.
3
+
Let's use fractale to run build, execute, and deploy agents. First now we will run these a-la-carte, and then we will group them together to be run by an agent to request steps when needed.
4
4
5
-
## Build
5
+
## A-la-carte
6
+
7
+
### Build
6
8
7
9
The build agent will use the Gemini API to generate a Dockerfile and then build until it succeeds. We would need subsequent agents to test it.
8
10
Here is how to first ask the build agent to generate a lammps container for Google cloud.
fractale agent build lammps --environment "google cloud CPU" --outfile Dockerfile --details "Ensure all globbed files from examples/reaxff/HNS from the root of the lammps codebase are in the WORKDIR. Clone the latest branch of LAMMPS."
12
14
```
13
15
16
+
Note that we are specific about the data and using CPU, which is something the builder agent would have to guess.
14
17
That might generate the [Dockerfile](Dockerfile) here, and a container that defaults to the application name "lammps"
15
18
16
-
## Kubernetes Job
19
+
###Kubernetes Job
17
20
18
21
The kubernetes job agent agent will be asked to run a command, and will be provided the Dockerfile and name of the container. We assume that another agent (or you) have built and either pushed the image to a registry, or loaded it. Let's create our cluster and load the image:
19
22
@@ -25,6 +28,63 @@ kind load docker-image lammps
25
28
To start, we will assume a kind cluster running and tell the agent the image is loaded into it (and so the pull policy will be never).
fractale agent kubernetes-job lammps --environment "google cloud CPU" --context-file ./Dockerfile --no-pull --details "Run in.reaxff.hns in the pwd with lmp" --outfile ./job.yaml
32
+
```
33
+
34
+
## With Cache
35
+
36
+
The same steps can be run using a cache. This will save to a deterministic path in the present working directory, and means that you can run steps a la carte, and run a workflow later to re-use the context (and not wait again).
37
+
Note that when you save a cache, you often don't need to save the output file, because it will be the result in the context.
38
+
39
+
```bash
40
+
fractale agent build lammps --environment "google cloud CPU" --details "Ensure all globbed files from examples/reaxff/HNS from the root of the lammps codebase are in the WORKDIR. Clone the latest branch of LAMMPS." --use-cache
29
41
```
30
42
43
+
And then try running with the manager (below) with the cache to see it being used.
44
+
45
+
## Manager
46
+
47
+
Let's run with a manager. Using a manager means we provide a plan along with a goal. The manager itself takes on a similar structure to a step agent, but it has a high level goal. The manager will follow the high level structure of the plan, and step
48
+
managers can often run independently for some number of attempts. If a step manager
49
+
returns after these attempts still with a failure, or if the last step is a failure,
50
+
the manager can decide how to act. For example, if a Kubernetes job deploys but fails,
51
+
it could be due to the Dockerfile build (the build manager) or the manifest for the Job.
52
+
The Job manager needs to prepare the updated context to return to this step, and then
We haven't hit the case yet where the manager needs to take over - that needs further development, along with being goal oriented (e.g., parsing a log and getting an output).
63
+
64
+
## Notes
65
+
66
+
#### To do items
67
+
68
+
- Figure out optimization agent (with some goal)
69
+
70
+
#### Research Questions
71
+
72
+
**And experiment ideas**
73
+
74
+
- How do we define stability?
75
+
- What are the increments of change (e.g., "adding a library")? We should be able to keep track of times for each stage and what changed, and an analyzer LLM can look at result and understand (categorize) most salient contributions to change.
76
+
- We also can time the time it takes to do subsequent changes, when relevant. For example, if we are building, we should be able to use cached layers (and the build times speed up) if the LLM is changing content later in the Dockerfile.
77
+
- We can also save the successful results (Dockerfile builds, for example) and compare for similarity. How consistent is the LLM?
78
+
- How does specificity of the prompt influence the result?
79
+
- For an experiment, we would want to do a build -> deploy and successful run for a series of apps and get distributions of attempts, reasons for failure, and a general sense of similarity / differences.
80
+
- For the optimization experiment, we'd want to do the same, but understand gradients of change that led to improvement.
81
+
82
+
#### Observations
83
+
84
+
- Specifying cpu seems important - if you don't it wants to do GPU
85
+
- If you ask for a specific example, it sometimes tries to download data (tell it where data is)
86
+
- There are issues that result from not enough information. E.g., if you don't tell it what to run / data, it can only guess. It will loop forever.
87
+
- As an example, we know where in a git clone is the data of interest. The LLM can only guess. It's easier to tell it exactly.
88
+
- An LLM has no sense of time with respect to versions. For example, the reax data changed from reaxc to reaxff in the same path, and which you get depends on the clone. Depending on when the LLM was trained with how to build lammps, it might select an older (or latest) branch. Instead of a juggling or guessing game (that again) would result in an infinite loop, we need to tell it the branch and data file explicitly.
89
+
- Always include common issues in the initial prompt
90
+
- If you are too specific about instance types, it adds node selectors/affinity, and that often doesn't work.
0 commit comments