Skip to content

Commit bd35dba

Browse files
authored
feat: implement SQLite FTS, improve config system, rename CLI to 'put', fix analyzer warnings (#1097)
## Summary This PR implements SQLite Full-Text Search and includes several related improvements to config system, CLI, and code quality: ### 1. SQLite FTS Implementation (~850 lines new code) - **Core**: `SqliteFtsIndex` implementing FTS5 with stemming support - **Interfaces**: `ISearchIndex`, `IFtsIndex`, `FtsMatch` for search abstraction - **Factory**: `SearchIndexFactory` to instantiate search indexes from config - **Integration**: Wired into CLI commands - content is automatically indexed on upsert - **Tests**: Comprehensive unit tests (310 lines) + integration tests (264 lines) **Usage:** ```bash $ km put "Machine learning is a subset of AI" --id ml-001 $ km put "Natural language processing uses transformers" --id nlp-001 # Content is automatically indexed in SQLite FTS database ``` ### 2. Config System Improvements - **Specific discriminators**: `sqliteFTS`, `sqliteVector` (not generic `"fts"`, `"vector"`) - **Auto-creation**: Config file created automatically on first command use - **Config paths relative**: All paths in auto-created config are relative to config file location - **New command**: `km config --create` to persist default config to disk - **ConfigPathService**: DI-based config path access for commands ### 3. CLI Improvements - **Rename command**: `upsert` → `put` (HTTP-style naming) - **Remove premature features**: `embeddingsCache` removed from default config (not yet implemented) - **Help text updated**: All examples and documentation updated ### 4. Code Quality (Zero-Tolerance Build) - Fixed CA1859: `SearchIndexFactory` return type optimization - Fixed CA1031: Specific exception types in test cleanup - Fixed CA1711: Test collection naming convention - **All code passes `/warnaserror` build** ## Test Plan - [x] All 301 tests passing (192 Main.Tests + 109 Core.Tests) - [x] FTS indexing: Content indexed and searchable - [x] Config auto-creation: Works with all CLI commands - [x] Config discriminators: Validated in tests - [x] "put" command: Works with all options - [x] Zero-tolerance build: `build.sh` succeeds ## Files Changed (52 files, +2150/-265 lines) **New FTS Implementation:** - `src/Core/Search/SqliteFtsIndex.cs` (194 lines) - `src/Core/Search/IFtsIndex.cs`, `ISearchIndex.cs`, `FtsMatch.cs` - `src/Main/Services/SearchIndexFactory.cs` (60 lines) - `tests/Core.Tests/Search/SqliteFtsIndexTests.cs` (310 lines) - `tests/Core.Tests/Search/FtsIntegrationTests.cs` (264 lines) **Config System:** - `src/Core/Config/ConfigParser.cs` - Auto-creation logic - `src/Core/Config/AppConfig.cs` - Remove embeddingsCache - `src/Main/CLI/Infrastructure/ConfigPathService.cs` - DI service - `src/Main/CLI/Commands/ConfigCommand.cs` - Add --create flag - `tests/Core.Tests/Config/ConfigParserAutoCreateTests.cs` (203 lines) **CLI Changes:** - `src/Main/CLI/CliApplicationBuilder.cs` - Rename upsert → put - `src/Main/CLI/Commands/GetCommand.cs`, `ListCommand.cs` - Help text - 14 test files updated for "put" command ## Breaking Changes - CLI command renamed from `upsert` to `put` - Default config no longer includes `embeddingsCache` - Config discriminators changed: `fts` → `sqliteFTS`, `vector` → `sqliteVector`
1 parent 4e30479 commit bd35dba

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

52 files changed

+2150
-265
lines changed

AGENTS.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,6 @@
11
- Follow @docs/AGENTS.md instructions without exceptions
22
- Ignore the "archived" directory
3+
- During development don't EVER write to my personal ~/.km dir
34
- For any task with multiple steps, create a todo list FIRST, then execute.
45
- Avoid destructive operations like "get reset", "git clean", etc.
56

docs

Submodule docs updated from 6c3214d to 5565295

src/Core/Config/AppConfig.cs

Lines changed: 14 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -57,24 +57,31 @@ public void Validate(string path = "")
5757

5858
/// <summary>
5959
/// Creates a default configuration with a single "personal" node
60-
/// using local SQLite storage
60+
/// using local SQLite storage in the user's home directory
6161
/// </summary>
6262
public static AppConfig CreateDefault()
6363
{
6464
var homeDir = Environment.GetFolderPath(Environment.SpecialFolder.UserProfile);
6565
var kmDir = Path.Combine(homeDir, ".km");
66-
var personalNodeDir = Path.Combine(kmDir, "nodes", "personal");
66+
return CreateDefault(kmDir);
67+
}
68+
69+
/// <summary>
70+
/// Creates a default configuration with a single "personal" node
71+
/// using local SQLite storage in the specified base directory
72+
/// </summary>
73+
/// <param name="baseDir">Base directory for data storage</param>
74+
public static AppConfig CreateDefault(string baseDir)
75+
{
76+
var personalNodeDir = Path.Combine(baseDir, "nodes", "personal");
6777

6878
return new AppConfig
6979
{
7080
Nodes = new Dictionary<string, NodeConfig>
7181
{
7282
["personal"] = NodeConfig.CreateDefaultPersonalNode(personalNodeDir)
73-
},
74-
EmbeddingsCache = CacheConfig.CreateDefaultSqliteCache(
75-
Path.Combine(kmDir, "embeddings-cache.db")
76-
),
77-
LLMCache = null
83+
}
84+
// EmbeddingsCache and LLMCache intentionally omitted - add when features are implemented
7885
};
7986
}
8087
}

src/Core/Config/ConfigParser.cs

Lines changed: 70 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,6 @@
11
// Copyright (c) Microsoft. All rights reserved.
22
using System.Text.Json;
3+
using System.Text.Json.Serialization.Metadata;
34
using KernelMemory.Core.Config.Cache;
45
using KernelMemory.Core.Config.ContentIndex;
56
using KernelMemory.Core.Config.SearchIndex;
@@ -19,28 +20,55 @@ public static class ConfigParser
1920
/// - Case insensitive property names
2021
/// - Comments allowed
2122
/// - Trailing commas allowed
23+
/// - Supports polymorphic deserialization
2224
/// </summary>
2325
private static readonly JsonSerializerOptions s_jsonOptions = new()
2426
{
2527
PropertyNameCaseInsensitive = true,
2628
ReadCommentHandling = JsonCommentHandling.Skip,
2729
AllowTrailingCommas = true,
28-
PropertyNamingPolicy = JsonNamingPolicy.CamelCase
30+
PropertyNamingPolicy = JsonNamingPolicy.CamelCase,
31+
TypeInfoResolver = new DefaultJsonTypeInfoResolver()
2932
};
3033

3134
/// <summary>
32-
/// Loads configuration from a file, or returns default config if file doesn't exist
35+
/// JSON serializer options for writing config files
36+
/// - Indented formatting
37+
/// - Camel case property names
38+
/// - Omit null values
39+
/// </summary>
40+
private static readonly JsonSerializerOptions s_writeJsonOptions = new()
41+
{
42+
WriteIndented = true,
43+
PropertyNamingPolicy = JsonNamingPolicy.CamelCase,
44+
DefaultIgnoreCondition = System.Text.Json.Serialization.JsonIgnoreCondition.WhenWritingNull
45+
};
46+
47+
/// <summary>
48+
/// Loads configuration from a file, or creates default config if file doesn't exist.
49+
/// The config file is always ensured to exist on disk after loading.
3350
/// Performs tilde expansion on paths (~/ → home directory)
3451
/// </summary>
3552
/// <param name="filePath">Path to configuration file</param>
3653
/// <returns>Validated AppConfig instance</returns>
3754
/// <exception cref="ConfigException">Thrown when file exists but parsing or validation fails</exception>
3855
public static AppConfig LoadFromFile(string filePath)
3956
{
40-
// If file doesn't exist, return default configuration
57+
AppConfig config;
58+
59+
// If file doesn't exist, create default configuration relative to config file location
4160
if (!File.Exists(filePath))
4261
{
43-
return AppConfig.CreateDefault();
62+
var configDir = Path.GetDirectoryName(filePath);
63+
var baseDir = string.IsNullOrEmpty(configDir) ? "." : configDir;
64+
65+
// Create default config relative to config file location
66+
config = AppConfig.CreateDefault(baseDir);
67+
68+
// Write the config file
69+
WriteConfigFile(filePath, config);
70+
71+
return config;
4472
}
4573

4674
try
@@ -49,11 +77,14 @@ public static AppConfig LoadFromFile(string filePath)
4977
var json = File.ReadAllText(filePath);
5078

5179
// Parse and validate
52-
var config = ParseFromString(json);
80+
config = ParseFromString(json);
5381

5482
// Expand tilde paths
5583
ExpandTildePaths(config);
5684

85+
// Always ensure the config file exists (recreate if deleted between load and save)
86+
WriteConfigFileIfMissing(filePath, config);
87+
5788
return config;
5889
}
5990
catch (ConfigException)
@@ -251,4 +282,38 @@ private static string ExpandTilde(string path, string homeDir)
251282

252283
return path;
253284
}
285+
/// <summary>
286+
/// Writes the config file to disk if it doesn't exist.
287+
/// Used to ensure config file is always present after any operation.
288+
/// </summary>
289+
/// <param name="filePath">Path to configuration file</param>
290+
/// <param name="config">Configuration to write</param>
291+
private static void WriteConfigFileIfMissing(string filePath, AppConfig config)
292+
{
293+
if (File.Exists(filePath))
294+
{
295+
return;
296+
}
297+
298+
WriteConfigFile(filePath, config);
299+
}
300+
301+
/// <summary>
302+
/// Writes the config file to disk, creating directories if needed.
303+
/// </summary>
304+
/// <param name="filePath">Path to configuration file</param>
305+
/// <param name="config">Configuration to write</param>
306+
private static void WriteConfigFile(string filePath, AppConfig config)
307+
{
308+
// Create the directory if needed
309+
var configDir = Path.GetDirectoryName(filePath);
310+
if (!string.IsNullOrEmpty(configDir) && !Directory.Exists(configDir))
311+
{
312+
Directory.CreateDirectory(configDir);
313+
}
314+
315+
// Write the config file
316+
var json = System.Text.Json.JsonSerializer.Serialize(config, s_writeJsonOptions);
317+
File.WriteAllText(filePath, json);
318+
}
254319
}

src/Core/Config/ContentIndex/ContentIndexConfig.cs

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -9,7 +9,7 @@ namespace KernelMemory.Core.Config.ContentIndex;
99
/// Base class for content index configurations
1010
/// Content index is the source of truth, backed by Entity Framework
1111
/// </summary>
12-
[JsonPolymorphic(TypeDiscriminatorPropertyName = "$type")]
12+
[JsonPolymorphic(TypeDiscriminatorPropertyName = "type")]
1313
[JsonDerivedType(typeof(SqliteContentIndexConfig), typeDiscriminator: "sqlite")]
1414
[JsonDerivedType(typeof(PostgresContentIndexConfig), typeDiscriminator: "postgres")]
1515
public abstract class ContentIndexConfig : IValidatable

src/Core/Config/Embeddings/EmbeddingsConfig.cs

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -8,7 +8,7 @@ namespace KernelMemory.Core.Config.Embeddings;
88
/// <summary>
99
/// Base class for embeddings provider configurations
1010
/// </summary>
11-
[JsonPolymorphic(TypeDiscriminatorPropertyName = "$type")]
11+
[JsonPolymorphic(TypeDiscriminatorPropertyName = "type")]
1212
[JsonDerivedType(typeof(OllamaEmbeddingsConfig), typeDiscriminator: "ollama")]
1313
[JsonDerivedType(typeof(OpenAIEmbeddingsConfig), typeDiscriminator: "openai")]
1414
[JsonDerivedType(typeof(AzureOpenAIEmbeddingsConfig), typeDiscriminator: "azureOpenAI")]

src/Core/Config/NodeConfig.cs

Lines changed: 15 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -71,10 +71,24 @@ public void Validate(string path)
7171
this.FileStorage?.Validate($"{path}.FileStorage");
7272
this.RepoStorage?.Validate($"{path}.RepoStorage");
7373

74+
// Validate search indexes
7475
for (int i = 0; i < this.SearchIndexes.Count; i++)
7576
{
7677
this.SearchIndexes[i].Validate($"{path}.SearchIndexes[{i}]");
7778
}
79+
80+
// Ensure all search index IDs are unique within this node
81+
var duplicateIds = this.SearchIndexes
82+
.GroupBy(idx => idx.Id)
83+
.Where(g => g.Count() > 1)
84+
.Select(g => g.Key)
85+
.ToList();
86+
87+
if (duplicateIds.Count > 0)
88+
{
89+
throw new ConfigException($"{path}.SearchIndexes",
90+
$"Duplicate search index IDs found: {string.Join(", ", duplicateIds)}. Each search index must have a unique ID within a node.");
91+
}
7892
}
7993

8094
/// <summary>
@@ -97,16 +111,10 @@ internal static NodeConfig CreateDefaultPersonalNode(string nodeDir)
97111
{
98112
new FtsSearchIndexConfig
99113
{
114+
Id = "sqlite-fts",
100115
Type = SearchIndexTypes.SqliteFTS,
101116
Path = Path.Combine(nodeDir, "fts.db"),
102117
EnableStemming = true
103-
},
104-
new VectorSearchIndexConfig
105-
{
106-
Type = SearchIndexTypes.SqliteVector,
107-
Path = Path.Combine(nodeDir, "vectors.db"),
108-
Dimensions = 768,
109-
Metric = VectorMetrics.Cosine
110118
}
111119
}
112120
};

src/Core/Config/SearchIndex/FtsSearchIndexConfig.cs

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -33,6 +33,12 @@ public sealed class FtsSearchIndexConfig : SearchIndexConfig
3333
/// <inheritdoc />
3434
public override void Validate(string path)
3535
{
36+
// Validate ID is provided
37+
if (string.IsNullOrWhiteSpace(this.Id))
38+
{
39+
throw new ConfigException($"{path}.Id", "Search index ID is required");
40+
}
41+
3642
this.Embeddings?.Validate($"{path}.Embeddings");
3743

3844
var isSqlite = this.Type == SearchIndexTypes.SqliteFTS;

src/Core/Config/SearchIndex/GraphSearchIndexConfig.cs

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -27,6 +27,12 @@ public sealed class GraphSearchIndexConfig : SearchIndexConfig
2727
/// <inheritdoc />
2828
public override void Validate(string path)
2929
{
30+
// Validate ID is provided
31+
if (string.IsNullOrWhiteSpace(this.Id))
32+
{
33+
throw new ConfigException($"{path}.Id", "Search index ID is required");
34+
}
35+
3036
this.Embeddings?.Validate($"{path}.Embeddings");
3137

3238
var hasPath = !string.IsNullOrWhiteSpace(this.Path);

src/Core/Config/SearchIndex/SearchIndexConfig.cs

Lines changed: 11 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -9,12 +9,20 @@ namespace KernelMemory.Core.Config.SearchIndex;
99
/// <summary>
1010
/// Base class for search index configurations
1111
/// </summary>
12-
[JsonPolymorphic(TypeDiscriminatorPropertyName = "$type")]
13-
[JsonDerivedType(typeof(FtsSearchIndexConfig), typeDiscriminator: "fts")]
14-
[JsonDerivedType(typeof(VectorSearchIndexConfig), typeDiscriminator: "vector")]
12+
[JsonPolymorphic(TypeDiscriminatorPropertyName = "type")]
13+
[JsonDerivedType(typeof(FtsSearchIndexConfig), typeDiscriminator: "sqliteFTS")]
14+
[JsonDerivedType(typeof(VectorSearchIndexConfig), typeDiscriminator: "sqliteVector")]
1515
[JsonDerivedType(typeof(GraphSearchIndexConfig), typeDiscriminator: "graph")]
1616
public abstract class SearchIndexConfig : IValidatable
1717
{
18+
/// <summary>
19+
/// Unique identifier for this search index instance.
20+
/// Must be unique within a node's SearchIndexes array.
21+
/// Used to identify index instances in operations pipeline (stable across config changes).
22+
/// </summary>
23+
[JsonPropertyName("id")]
24+
public string Id { get; set; } = string.Empty;
25+
1826
/// <summary>
1927
/// Type of search index
2028
/// </summary>

0 commit comments

Comments
 (0)