Skip to content

Commit 8cb9ad5

Browse files
joelteplyjoelteply
authored
Candle token limiting: prevent NaN in quantized models (#268)
* Observability Phase 1: Non-blocking Rust logging + Sentinel log commands Rust logging improvements: - Add clog_info!, clog_warn!, clog_error!, clog_debug! macros for easy logging - Auto-route logs by module_path!() to appropriate directories - Wire to LoggerModule's concurrent writer (sync_channel + try_send = NEVER blocks) - Convert println!/eprintln! to clog_* in orm/sqlite, voice/orchestrator, voice/tts/phonemizer, ai/adapter, concurrent/message_processor New commands: - logging/enable, logging/disable, logging/status - control Rust logging - sentinel/logs/list - list log streams for a sentinel with size/mtime - sentinel/logs/read - read log content with offset/limit - sentinel/logs/tail - get last N lines (like Unix tail) SentinelLogWriter: - Non-blocking async TypeScript log writer for per-sentinel logs - Event streaming via sentinel:{handle}:log for real-time UI - Directory structure: .sentinel-workspaces/{handle}/logs/ Documentation: - docs/OBSERVABILITY-ARCHITECTURE.md - overall vision - docs/SENTINEL-LOGGING-PLAN.md - sentinel logging design * Candle token limiting: prevent NaN in quantized models - Add SAFE_INPUT_TOKENS (800) limit for quantized model input - Token-based truncation using tokenizer (not character-based) - Preserves 30% start (system prompt) + 60% end (recent messages) - Logs TOKEN LIMIT warning when truncation occurs Also: - Fix ts-rs binding generator to ignore harmless serde warnings - Add threshold test for future NaN threshold validation - Add INCIDENT_CAPTURE logging for reproducibility Verified: Helper AI and Teacher AI now produce coherent responses instead of garbage output from NaN/Inf in logits.
1 parent 541a275 commit 8cb9ad5

File tree

54 files changed

+5538
-39
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

54 files changed

+5538
-39
lines changed

‎src/debug/jtag/browser/generated.ts‎

Lines changed: 19 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
/**
22
* Browser Structure Registry - Auto-generated
33
*
4-
* Contains 11 daemons and 201 commands and 2 adapters and 28 widgets.
4+
* Contains 11 daemons and 204 commands and 2 adapters and 28 widgets.
55
* Generated by scripts/generate-structure.ts - DO NOT EDIT MANUALLY
66
*/
77

@@ -146,6 +146,9 @@ import { WebFetchBrowserCommand } from './../commands/interface/web/fetch/browse
146146
import { InterfaceWebmcpCallBrowserCommand } from './../commands/interface/webmcp/call/browser/InterfaceWebmcpCallBrowserCommand';
147147
import { InterfaceWebmcpDiscoverBrowserCommand } from './../commands/interface/webmcp/discover/browser/InterfaceWebmcpDiscoverBrowserCommand';
148148
import { ListBrowserCommand } from './../commands/list/browser/ListBrowserCommand';
149+
import { LoggingDisableBrowserCommand } from './../commands/logging/disable/browser/LoggingDisableBrowserCommand';
150+
import { LoggingEnableBrowserCommand } from './../commands/logging/enable/browser/LoggingEnableBrowserCommand';
151+
import { LoggingStatusBrowserCommand } from './../commands/logging/status/browser/LoggingStatusBrowserCommand';
149152
import { LogsConfigBrowserCommand } from './../commands/logs/config/browser/LogsConfigBrowserCommand';
150153
import { LogsListBrowserCommand } from './../commands/logs/list/browser/LogsListBrowserCommand';
151154
import { LogsReadBrowserCommand } from './../commands/logs/read/browser/LogsReadBrowserCommand';
@@ -958,6 +961,21 @@ export const BROWSER_COMMANDS: CommandEntry[] = [
958961
className: 'ListBrowserCommand',
959962
commandClass: ListBrowserCommand
960963
},
964+
{
965+
name: 'logging/disable',
966+
className: 'LoggingDisableBrowserCommand',
967+
commandClass: LoggingDisableBrowserCommand
968+
},
969+
{
970+
name: 'logging/enable',
971+
className: 'LoggingEnableBrowserCommand',
972+
commandClass: LoggingEnableBrowserCommand
973+
},
974+
{
975+
name: 'logging/status',
976+
className: 'LoggingStatusBrowserCommand',
977+
commandClass: LoggingStatusBrowserCommand
978+
},
961979
{
962980
name: 'logs/config',
963981
className: 'LogsConfigBrowserCommand',

‎src/debug/jtag/commands/logging/disable/.npmignore‎

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,20 @@
1+
# Development files
2+
.eslintrc*
3+
tsconfig*.json
4+
vitest.config.ts
5+
6+
# Build artifacts
7+
*.js.map
8+
*.d.ts.map
9+
10+
# IDE
11+
.vscode/
12+
.idea/
13+
14+
# Logs
15+
*.log
16+
npm-debug.log*
17+
18+
# OS files
19+
.DS_Store
20+
Thumbs.db

‎src/debug/jtag/commands/logging/disable/README.md‎

Lines changed: 166 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,166 @@
1+
# Logging Disable Command
2+
3+
Disable logging for a persona. Persists to .continuum/logging.json
4+
5+
## Table of Contents
6+
7+
- [Usage](#usage)
8+
- [CLI Usage](#cli-usage)
9+
- [Tool Usage](#tool-usage)
10+
- [Parameters](#parameters)
11+
- [Result](#result)
12+
- [Examples](#examples)
13+
- [Testing](#testing)
14+
- [Unit Tests](#unit-tests)
15+
- [Integration Tests](#integration-tests)
16+
- [Getting Help](#getting-help)
17+
- [Access Level](#access-level)
18+
- [Implementation Notes](#implementation-notes)
19+
20+
## Usage
21+
22+
### CLI Usage
23+
24+
From the command line using the jtag CLI:
25+
26+
```bash
27+
./jtag logging/disable --persona=<value>
28+
```
29+
30+
### Tool Usage
31+
32+
From Persona tools or programmatic access using `Commands.execute()`:
33+
34+
```typescript
35+
import { Commands } from '@system/core/shared/Commands';
36+
37+
const result = await Commands.execute('logging/disable', {
38+
// your parameters here
39+
});
40+
```
41+
42+
## Parameters
43+
44+
- **persona** (required): `string` - Persona uniqueId to disable logging for (e.g., 'helper', 'codereview')
45+
- **category** (optional): `string` - Specific category to disable. If not specified, disables all logging for the persona
46+
47+
## Result
48+
49+
Returns `LoggingDisableResult` with:
50+
51+
Returns CommandResult with:
52+
- **persona**: `string` - The persona that was disabled
53+
- **enabled**: `boolean` - Whether any logging remains enabled for this persona
54+
- **categories**: `string[]` - Categories still enabled (empty if all disabled)
55+
- **message**: `string` - Human-readable status message
56+
57+
## Examples
58+
59+
### Disable all logging for helper persona
60+
61+
```bash
62+
./jtag logging/disable --persona=helper
63+
```
64+
65+
**Expected result:**
66+
{ persona: 'helper', enabled: false, categories: [], message: 'Disabled all logging for helper' }
67+
68+
### Disable only training logs
69+
70+
```bash
71+
./jtag logging/disable --persona=helper --category=training
72+
```
73+
74+
**Expected result:**
75+
{ persona: 'helper', enabled: true, categories: ['cognition'], message: 'Disabled training logging for helper' }
76+
77+
## Getting Help
78+
79+
### Using the Help Tool
80+
81+
Get detailed usage information for this command:
82+
83+
**CLI:**
84+
```bash
85+
./jtag help logging/disable
86+
```
87+
88+
**Tool:**
89+
```typescript
90+
// Use your help tool with command name 'logging/disable'
91+
```
92+
93+
### Using the README Tool
94+
95+
Access this README programmatically:
96+
97+
**CLI:**
98+
```bash
99+
./jtag readme logging/disable
100+
```
101+
102+
**Tool:**
103+
```typescript
104+
// Use your readme tool with command name 'logging/disable'
105+
```
106+
107+
## Testing
108+
109+
### Unit Tests
110+
111+
Test command logic in isolation using mock dependencies:
112+
113+
```bash
114+
# Run unit tests (no server required)
115+
npx tsx commands/Logging Disable/test/unit/LoggingDisableCommand.test.ts
116+
```
117+
118+
**What's tested:**
119+
- Command structure and parameter validation
120+
- Mock command execution patterns
121+
- Required parameter validation (throws ValidationError)
122+
- Optional parameter handling (sensible defaults)
123+
- Performance requirements
124+
- Assertion utility helpers
125+
126+
**TDD Workflow:**
127+
1. Write/modify unit test first (test-driven development)
128+
2. Run test, see it fail
129+
3. Implement feature
130+
4. Run test, see it pass
131+
5. Refactor if needed
132+
133+
### Integration Tests
134+
135+
Test command with real client connections and system integration:
136+
137+
```bash
138+
# Prerequisites: Server must be running
139+
npm start # Wait 90+ seconds for deployment
140+
141+
# Run integration tests
142+
npx tsx commands/Logging Disable/test/integration/LoggingDisableIntegration.test.ts
143+
```
144+
145+
**What's tested:**
146+
- Client connection to live system
147+
- Real command execution via WebSocket
148+
- ValidationError handling for missing params
149+
- Optional parameter defaults
150+
- Performance under load
151+
- Various parameter combinations
152+
153+
**Best Practice:**
154+
Run unit tests frequently during development (fast feedback). Run integration tests before committing (verify system integration).
155+
156+
## Access Level
157+
158+
**ai-safe** - Safe for AI personas to call autonomously
159+
160+
## Implementation Notes
161+
162+
- **Shared Logic**: Core business logic in `shared/LoggingDisableTypes.ts`
163+
- **Browser**: Browser-specific implementation in `browser/LoggingDisableBrowserCommand.ts`
164+
- **Server**: Server-specific implementation in `server/LoggingDisableServerCommand.ts`
165+
- **Unit Tests**: Isolated testing in `test/unit/LoggingDisableCommand.test.ts`
166+
- **Integration Tests**: System testing in `test/integration/LoggingDisableIntegration.test.ts`

‎src/debug/jtag/commands/logging/disable/browser/LoggingDisableBrowserCommand.ts‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
/**
2+
* Logging Disable Command - Browser Implementation
3+
*
4+
* Disable logging for a persona. Persists to .continuum/logging.json
5+
*/
6+
7+
import { CommandBase, type ICommandDaemon } from '@daemons/command-daemon/shared/CommandBase';
8+
import type { JTAGContext } from '@system/core/types/JTAGTypes';
9+
import type { LoggingDisableParams, LoggingDisableResult } from '../shared/LoggingDisableTypes';
10+
11+
export class LoggingDisableBrowserCommand extends CommandBase<LoggingDisableParams, LoggingDisableResult> {
12+
13+
constructor(context: JTAGContext, subpath: string, commander: ICommandDaemon) {
14+
super('logging/disable', context, subpath, commander);
15+
}
16+
17+
async execute(params: LoggingDisableParams): Promise<LoggingDisableResult> {
18+
console.log('🌐 BROWSER: Delegating Logging Disable to server');
19+
return await this.remoteExecute(params);
20+
}
21+
}

‎src/debug/jtag/commands/logging/disable/package.json‎

Lines changed: 35 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,35 @@
1+
{
2+
"name": "@jtag-commands/logging/disable",
3+
"version": "1.0.0",
4+
"description": "Disable logging for a persona. Persists to .continuum/logging.json",
5+
"main": "server/LoggingDisableServerCommand.ts",
6+
"types": "shared/LoggingDisableTypes.ts",
7+
"scripts": {
8+
"test": "npm run test:unit && npm run test:integration",
9+
"test:unit": "npx vitest run test/unit/*.test.ts",
10+
"test:integration": "npx tsx test/integration/LoggingDisableIntegration.test.ts",
11+
"lint": "npx eslint **/*.ts",
12+
"typecheck": "npx tsc --noEmit"
13+
},
14+
"peerDependencies": {
15+
"@jtag/core": "*"
16+
},
17+
"files": [
18+
"shared/**/*.ts",
19+
"browser/**/*.ts",
20+
"server/**/*.ts",
21+
"test/**/*.ts",
22+
"README.md"
23+
],
24+
"keywords": [
25+
"jtag",
26+
"command",
27+
"logging/disable"
28+
],
29+
"license": "MIT",
30+
"author": "",
31+
"repository": {
32+
"type": "git",
33+
"url": ""
34+
}
35+
}

‎src/debug/jtag/commands/logging/disable/server/LoggingDisableServerCommand.ts‎

Lines changed: 74 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,74 @@
1+
/**
2+
* Logging Disable Command - Server Implementation
3+
*
4+
* Disable logging for a persona. Persists to .continuum/logging.json
5+
*/
6+
7+
import { CommandBase, type ICommandDaemon } from '@daemons/command-daemon/shared/CommandBase';
8+
import type { JTAGContext } from '@system/core/types/JTAGTypes';
9+
import { ValidationError } from '@system/core/types/ErrorTypes';
10+
import type { LoggingDisableParams, LoggingDisableResult } from '../shared/LoggingDisableTypes';
11+
import { createLoggingDisableResultFromParams } from '../shared/LoggingDisableTypes';
12+
import { LoggingConfig, LOGGING_CATEGORIES } from '@system/core/logging/LoggingConfig';
13+
14+
export class LoggingDisableServerCommand extends CommandBase<LoggingDisableParams, LoggingDisableResult> {
15+
16+
constructor(context: JTAGContext, subpath: string, commander: ICommandDaemon) {
17+
super('logging/disable', context, subpath, commander);
18+
}
19+
20+
async execute(params: LoggingDisableParams): Promise<LoggingDisableResult> {
21+
// Validate persona parameter
22+
if (!params.persona || params.persona.trim() === '') {
23+
throw new ValidationError(
24+
'persona',
25+
`Missing required parameter 'persona'. ` +
26+
`Use: ./jtag logging/disable --persona=helper [--category=cognition]`
27+
);
28+
}
29+
30+
const persona = params.persona.trim();
31+
const category = params.category?.trim();
32+
33+
// Disable logging
34+
if (category) {
35+
// Disable specific category
36+
LoggingConfig.setEnabled(persona, category, false);
37+
} else {
38+
// Disable all logging for persona
39+
LoggingConfig.setPersonaEnabled(persona, false);
40+
}
41+
42+
// Get current state after update
43+
const config = LoggingConfig.getConfig();
44+
const normalizedId = persona.toLowerCase().replace(/\s+ai$/i, '').replace(/\s+/g, '-').replace(/[^a-z0-9-]/g, '');
45+
const personaConfig = config.personas[normalizedId];
46+
47+
// Determine remaining enabled categories
48+
let categories: string[] = [];
49+
let enabled = false;
50+
51+
if (personaConfig && personaConfig.enabled) {
52+
enabled = true;
53+
if (!personaConfig.categories || personaConfig.categories.length === 0) {
54+
categories = Object.values(LOGGING_CATEGORIES);
55+
} else if (personaConfig.categories.includes('*')) {
56+
categories = Object.values(LOGGING_CATEGORIES);
57+
} else {
58+
categories = personaConfig.categories;
59+
}
60+
}
61+
62+
const message = category
63+
? `Disabled ${category} logging for ${persona}`
64+
: `Disabled all logging for ${persona}`;
65+
66+
return createLoggingDisableResultFromParams(params, {
67+
success: true,
68+
persona,
69+
enabled,
70+
categories,
71+
message,
72+
});
73+
}
74+
}

0 commit comments

Comments
 (0)