Skip to content

Commit e2d1919

Browse files
committed
remove runonce and improve prompt
Signed-off-by: Siri Chongasamethaworn <siri@omise.co>
1 parent cfc0830 commit e2d1919

File tree

3 files changed

+136
-62
lines changed

3 files changed

+136
-62
lines changed

docs/modification_modes.md

Lines changed: 114 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,114 @@
1+
# Modification Modes
2+
3+
KubeAI Chatbot supports three resource modification modes, controlled by the `MODIFY_RESOURCES` environment variable. The mode determines how the agent behaves when a task requires a `kubectl` command that creates, updates, or deletes Kubernetes resources.
4+
5+
> [!IMPORTANT]
6+
> Regardless of the modification mode, the agent will **never** read or list Kubernetes Secrets. This restriction is hardcoded and cannot be overridden.
7+
8+
## Modes
9+
10+
### `none` — Read-Only (Default)
11+
12+
```yaml
13+
env:
14+
MODIFY_RESOURCES: "none"
15+
```
16+
17+
The agent operates in **read-only mode**. It can freely execute read commands (`get`, `describe`, `logs`, `top`, `events`, etc.) but will never execute a write command through its tools.
18+
19+
When a task requires a resource modification, the agent will:
20+
21+
1. Gather the necessary context using read-only tools.
22+
2. Provide the exact `kubectl` command(s) the user should run manually.
23+
3. Explain what each command does and why.
24+
25+
**Best for**: Teams that want AI-assisted diagnostics and guidance without allowing the bot to change anything in the cluster.
26+
27+
---
28+
29+
### `allow` — Confirm Before Modifying
30+
31+
```yaml
32+
env:
33+
MODIFY_RESOURCES: "allow"
34+
```
35+
36+
The agent can execute write commands, but only after **explicit user confirmation**. When the agent plans a write operation, the system pauses and presents the user with a confirmation prompt listing the command(s) about to be run. The user must approve before anything is executed.
37+
38+
Read-only commands (`get`, `describe`, `logs`, etc.) run immediately without any confirmation.
39+
40+
**Best for**: Teams that want the convenience of automated execution but with a human-in-the-loop for any destructive or modifying actions.
41+
42+
---
43+
44+
### `auto` — Automatic Modification
45+
46+
```yaml
47+
env:
48+
MODIFY_RESOURCES: "auto"
49+
```
50+
51+
The agent can execute both read and write commands automatically, without requesting user confirmation. The agent will:
52+
53+
1. Gather context first using read-only tools.
54+
2. Briefly announce what it is about to do and why.
55+
3. Execute the modification immediately.
56+
57+
The agent will still ask for user input when genuinely required (e.g., a required value such as a namespace or image tag is not specified).
58+
59+
**Best for**: Trusted internal tooling or teams with high confidence in the agent's behaviour who want to minimise confirmation prompts.
60+
61+
---
62+
63+
## Comparison
64+
65+
| Feature | `none` | `allow` | `auto` |
66+
| :---------------------------------- | :------: | :--------------------: | :------: |
67+
| Read commands (get, describe, logs) | ✅ Auto | ✅ Auto | ✅ Auto |
68+
| Write commands (apply, delete, …) | ❌ Never | ✅ After user confirms | ✅ Auto |
69+
| Provides commands for manual run | ✅ Yes | — | — |
70+
| User confirmation dialog | — | ✅ Yes | ❌ No |
71+
| Minimises user interaction | — | — | ✅ Yes |
72+
| Kubernetes Secrets access | ❌ Never | ❌ Never | ❌ Never |
73+
74+
---
75+
76+
## Helm Values
77+
78+
Set the mode via `values.yaml`:
79+
80+
```yaml
81+
env:
82+
MODIFY_RESOURCES: "none" # Options: none, allow, auto
83+
```
84+
85+
Or override at install time:
86+
87+
```bash
88+
helm install kubeai-chatbot ./charts/kubeai-chatbot \
89+
--set env.SLACK_BOT_TOKEN="xoxb-..." \
90+
--set env.SLACK_SIGNING_SECRET="..." \
91+
--set env.GEMINI_API_KEY="..." \
92+
--set env.MODIFY_RESOURCES="allow"
93+
```
94+
95+
---
96+
97+
## RBAC Alignment
98+
99+
The modification mode should be aligned with the Kubernetes RBAC permissions granted to the bot's service account. The Helm chart provides a `rbac.allowWrite` value to control this:
100+
101+
```yaml
102+
rbac:
103+
create: true
104+
allowWrite: false # Set to true when using allow or auto mode
105+
```
106+
107+
| `MODIFY_RESOURCES` | Recommended `rbac.allowWrite` |
108+
| :----------------- | :---------------------------: |
109+
| `none` | `false` |
110+
| `allow` | `true` |
111+
| `auto` | `true` |
112+
113+
> [!WARNING]
114+
> Setting `MODIFY_RESOURCES: "allow"` or `"auto"` while `rbac.allowWrite: false` will result in permission errors when the agent attempts write operations. Conversely, granting write RBAC while using `MODIFY_RESOURCES: "none"` is safe but unnecessarily permissive.

pkg/agent/conversation.go

Lines changed: 10 additions & 62 deletions
Original file line numberDiff line numberDiff line change
@@ -60,15 +60,6 @@ type Agent struct {
6060
// Output is the channel to send messages to the UI.
6161
Output chan any
6262

63-
// RunOnce indicates if the agent should run only once.
64-
// If true, the agent will run only once and then exit.
65-
// If false, the agent will run in a loop until the context is done.
66-
RunOnce bool
67-
68-
// InitialQuery is the initial query to the agent.
69-
// If provided, the agent will run only once and then exit.
70-
InitialQuery string
71-
7263
// AgentName is the name of the assistant.
7364
AgentName string
7465

@@ -229,10 +220,6 @@ func (s *Agent) Init(ctx context.Context) error {
229220
// current history of the conversation.
230221
s.currChatContent = []any{}
231222

232-
if s.InitialQuery == "" && s.RunOnce {
233-
return fmt.Errorf("RunOnce mode requires an initial query to be provided")
234-
}
235-
236223
if s.Session != nil {
237224
if s.Session.ChatMessageStore == nil {
238225
s.Session.ChatMessageStore = sessions.NewInMemoryChatStore()
@@ -280,11 +267,10 @@ func (s *Agent) Init(ctx context.Context) error {
280267
s.Tools.RegisterTool(tools.NewKubectlTool())
281268

282269
systemPrompt, err := s.generatePrompt(ctx, defaultSystemPromptTemplate, PromptData{
283-
Tools: s.Tools,
284-
EnableToolUseShim: s.EnableToolUseShim,
285-
ModifyResources: s.ModifyResources,
286-
// RunOnce is a good proxy to indicate the agentic session is non-interactive mode.
287-
SessionIsInteractive: !s.RunOnce,
270+
Tools: s.Tools,
271+
EnableToolUseShim: s.EnableToolUseShim,
272+
ModifyResources: s.ModifyResources,
273+
SessionIsInteractive: true,
288274
AgentName: s.AgentName,
289275
})
290276
if err != nil {
@@ -359,14 +345,8 @@ func (c *Agent) Run(ctx context.Context, initialQuery string) error {
359345
ctx = journal.ContextWithSlackUserID(ctx, c.Session.SlackUserID)
360346
}
361347

362-
// Save unexpected error and return it in for RunOnce mode
363-
log.Info("Starting agent loop", "initialQuery", initialQuery, "runOnce", c.RunOnce)
348+
log.Info("Starting agent loop", "initialQuery", initialQuery)
364349
go func() {
365-
// If initialQuery is empty, try to use the one from the struct
366-
if initialQuery == "" {
367-
initialQuery = c.InitialQuery
368-
}
369-
370350
if initialQuery != "" {
371351
c.addMessage(api.MessageSourceUser, api.MessageTypeText, initialQuery)
372352
answer, handled, err := c.handleMetaQuery(ctx, initialQuery)
@@ -405,12 +385,6 @@ func (c *Agent) Run(ctx context.Context, initialQuery string) error {
405385
log.V(2).Info("Agent loop iteration", "state", c.AgentState())
406386
switch c.AgentState() {
407387
case api.AgentStateIdle, api.AgentStateDone:
408-
// In RunOnce mode, we are done, so exit
409-
if c.RunOnce {
410-
log.V(2).Info("RunOnce mode, exiting agent loop")
411-
c.setAgentState(api.AgentStateExited)
412-
return
413-
}
414388
log.V(2).Info("initiating user input")
415389
c.addMessage(api.MessageSourceAgent, api.MessageTypeUserInputRequest, ">>>")
416390
select {
@@ -471,13 +445,6 @@ func (c *Agent) Run(ctx context.Context, initialQuery string) error {
471445
log.Info("Set agent state to running, will process agentic loop", "currIteration", c.currIteration, "currChatContent", len(c.currChatContent))
472446
}
473447
case api.AgentStateWaitingForInput:
474-
// In RunOnce mode, if we need user choice, exit with error
475-
if c.RunOnce {
476-
log.Error(nil, "RunOnce mode cannot handle user choice requests")
477-
c.setAgentState(api.AgentStateExited)
478-
c.addMessage(api.MessageSourceAgent, api.MessageTypeError, "Error: RunOnce mode cannot handle user choice requests")
479-
return
480-
}
481448
select {
482449
case <-ctx.Done():
483450
log.V(2).Info("Agent loop done")
@@ -502,12 +469,6 @@ func (c *Agent) Run(ctx context.Context, initialQuery string) error {
502469
c.pendingFunctionCalls = []ToolCallAnalysis{}
503470
c.Session.LastModified = time.Now()
504471
c.addMessage(api.MessageSourceAgent, api.MessageTypeError, "Error: "+err.Error())
505-
// In RunOnce mode, exit on tool execution error
506-
if c.RunOnce {
507-
c.setAgentState(api.AgentStateExited)
508-
c.lastErr = err
509-
return
510-
}
511472
continue
512473
}
513474
// Clear pending function calls after execution
@@ -526,7 +487,7 @@ func (c *Agent) Run(ctx context.Context, initialQuery string) error {
526487
// Agent is running, don't wait for input, just continue to process the agentic loop
527488
log.V(2).Info("Agent is in running state, processing agentic loop")
528489
case api.AgentStateExited:
529-
log.V(2).Info("Agent exited in RunOnce mode")
490+
log.V(2).Info("Agent exited")
530491
return
531492
}
532493

@@ -559,13 +520,6 @@ func (c *Agent) Run(ctx context.Context, initialQuery string) error {
559520
if err != nil {
560521
c.setAgentState(api.AgentStateDone)
561522
c.pendingFunctionCalls = []ToolCallAnalysis{}
562-
563-
// In RunOnce mode, exit on shim conversion error
564-
if c.RunOnce {
565-
c.setAgentState(api.AgentStateExited)
566-
return
567-
}
568-
569523
continue
570524
}
571525
}
@@ -702,20 +656,14 @@ func (c *Agent) Run(ctx context.Context, initialQuery string) error {
702656
}
703657

704658
if !c.SkipPermissions && c.ModifyResources != ModifyResourcesModeAuto && modifiesResourceToolCallIndex >= 0 {
705-
// In RunOnce mode or read-only mode, block the write and return an error
706-
if c.RunOnce || c.ModifyResources == ModifyResourcesModeNone {
659+
// In read-only mode, block the write and return an error
660+
if c.ModifyResources == ModifyResourcesModeNone {
707661
var commandDescriptions []string
708662
for _, call := range c.pendingFunctionCalls {
709663
commandDescriptions = append(commandDescriptions, call.ParsedToolCall.Description())
710664
}
711665

712-
var errorMessage string
713-
if c.ModifyResources == ModifyResourcesModeNone {
714-
errorMessage = "Resource modification is disabled (read-only mode). Provide the exact `kubectl` command in your response for the user to execute manually instead of using this tool."
715-
} else {
716-
errorMessage = "RunOnce mode cannot handle permission requests. The following commands require approval:\n* " + strings.Join(commandDescriptions, "\n* ")
717-
errorMessage += "\nUse --skip-permissions flag to bypass permission checks in RunOnce mode."
718-
}
666+
errorMessage := "Resource modification is disabled (read-only mode). The following commands were blocked:\n* " + strings.Join(commandDescriptions, "\n* ") + "\nProvide the exact `kubectl` command in your response for the user to execute manually instead of using this tool."
719667

720668
log.Error(nil, "Tool call blocked", "reason", errorMessage, "commands", commandDescriptions)
721669

@@ -901,7 +849,7 @@ func (c *Agent) NewSession() (string, error) {
901849
Tools: c.Tools,
902850
EnableToolUseShim: c.EnableToolUseShim,
903851
ModifyResources: c.ModifyResources,
904-
SessionIsInteractive: !c.RunOnce,
852+
SessionIsInteractive: true,
905853
AgentName: c.AgentName,
906854
})
907855
if err != nil {

pkg/agent/systemprompt_template_default.txt

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -76,6 +76,18 @@ You can execute commands that modify resources automatically without requesting
7676
- ❌ Incorrect: `kubectl --namespace=default get pods`
7777
- This ensures commands are properly recognized and filtered by the system.
7878
- NEVER use commands that open an interactive editor or shell (e.g., kubectl edit, kubectl exec -it, kubectl run --stdin --tty).
79+
- NEVER pass a piped, chained, or compound command to the tool. Pipes (`|`), `&&`, `;`, and backticks are NOT allowed when calling the kubectl tool. Each tool call must be a single, standalone `kubectl` command.
80+
- ✅ Correct: `kubectl get pods -A --field-selector spec.nodeName=my-node`
81+
- ✅ Correct: `kubectl get pods -A -o jsonpath='{range .items[?(@.spec.nodeName=="my-node")]}{.metadata.namespace}{"/"}{.metadata.name}{"\n"}{end}'`
82+
- ❌ Incorrect: `kubectl get pods -A | grep my-node`
83+
- ❌ Incorrect: `kubectl get pods -A && kubectl get nodes`
84+
- When filtering output, ALWAYS use kubectl's built-in mechanisms instead of pipes:
85+
- Use `--field-selector` to filter by resource fields (e.g., `spec.nodeName`, `status.phase`)
86+
- Use `-l` / `--selector` to filter by labels
87+
- Use `-o jsonpath='...'` or `-o go-template='...'` with filter expressions for complex output
88+
- When using `-o jsonpath`, ALWAYS wrap the jsonpath expression in single quotes: `-o jsonpath='...'`
89+
- ✅ Correct: `kubectl get pods -o jsonpath='{.items[0].metadata.name}'`
90+
- ❌ Incorrect: `kubectl get pods -o jsonpath={.items[0].metadata.name}` (unquoted)
7991

8092
## Security Rules:
8193
**CRITICAL:**

0 commit comments

Comments
 (0)