Skip to content

Commit de0a86b

Browse files
jongallowayjongalloway
authored
Merge pull request #47 from jongalloway/copilot/fix-36
⚙️ Implement YAML and XML Configuration Format Support
2 parents 9f36a1c + 9e1b069 commit de0a86b

File tree

12 files changed

+1445
-0
lines changed

12 files changed

+1445
-0
lines changed

‎README.md‎

Lines changed: 55 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -191,6 +191,61 @@ app.UseNLWebNet(); // Add NLWebNet middleware (optional)
191191
app.MapNLWebNet(); // Map NLWebNet minimal API endpoints
192192
```
193193

194+
### Configuration Format Support
195+
196+
NLWebNet supports multiple configuration formats for enhanced flexibility:
197+
198+
#### YAML Configuration
199+
200+
```csharp
201+
// Enable YAML configuration support
202+
builder.Configuration.AddNLWebConfigurationFormats(builder.Environment);
203+
builder.Services.AddNLWebConfigurationFormats(builder.Configuration);
204+
```
205+
206+
Example YAML configuration (`config_retrieval.yaml`):
207+
208+
```yaml
209+
# Multi-backend configuration
210+
write_endpoint: primary_backend
211+
endpoints:
212+
primary_backend:
213+
enabled: true
214+
db_type: azure_ai_search
215+
priority: 1
216+
217+
# NLWeb settings
218+
nlweb:
219+
default_mode: List
220+
enable_streaming: true
221+
tool_selection_enabled: true
222+
```
223+
224+
#### XML Tool Definitions
225+
226+
Define tools for the tool selection framework:
227+
228+
```xml
229+
<?xml version="1.0" encoding="utf-8"?>
230+
<ToolDefinitions>
231+
<Tool id="search-tool" name="Enhanced Search" type="search" enabled="true">
232+
<Description>Advanced search with semantic understanding</Description>
233+
<Parameters>
234+
<MaxResults>50</MaxResults>
235+
<TimeoutSeconds>30</TimeoutSeconds>
236+
</Parameters>
237+
<TriggerPatterns>
238+
<Pattern>search for*</Pattern>
239+
<Pattern>find*</Pattern>
240+
</TriggerPatterns>
241+
</Tool>
242+
</ToolDefinitions>
243+
```
244+
245+
#### Backward Compatibility
246+
247+
All existing JSON configuration continues to work unchanged. See the [Configuration Format Guide](doc/configuration-format-updates.md) for detailed documentation and migration examples.
248+
194249
### Prerequisites
195250

196251
- .NET 9.0 SDK

‎doc/configuration-format-updates.md‎

Lines changed: 223 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,223 @@
1+
# Configuration Format Updates
2+
3+
This document describes the new configuration format support added to NLWebNet in response to the NLWeb June 2025 release requirements.
4+
5+
## Overview
6+
7+
NLWebNet now supports multiple configuration formats while maintaining full backward compatibility:
8+
9+
- **YAML Configuration**: Enhanced multi-backend configuration with `enabled` flags
10+
- **XML Tool Definitions**: Structured tool definitions for the tool selection framework
11+
- **JSON Configuration**: Existing JSON configuration format continues to work unchanged
12+
13+
## YAML Configuration Support
14+
15+
### Basic Usage
16+
17+
To enable YAML configuration support in your application:
18+
19+
```csharp
20+
var builder = WebApplication.CreateBuilder(args);
21+
22+
// Add YAML configuration format support
23+
builder.Configuration.AddNLWebConfigurationFormats(builder.Environment);
24+
25+
// Register configuration format services
26+
builder.Services.AddNLWebConfigurationFormats(builder.Configuration);
27+
```
28+
29+
### YAML Configuration Format
30+
31+
The new YAML format supports the multi-backend configuration structure introduced in June 2025:
32+
33+
```yaml
34+
# config_retrieval.yaml
35+
write_endpoint: primary_backend
36+
endpoints:
37+
primary_backend:
38+
enabled: true
39+
db_type: azure_ai_search
40+
priority: 1
41+
properties:
42+
service_name: "nlweb-search"
43+
index_name: "nlweb-index"
44+
api_version: "2023-11-01"
45+
46+
secondary_backend:
47+
enabled: true
48+
db_type: mock
49+
priority: 0
50+
properties:
51+
data_source: "sample_data"
52+
enable_fuzzy_matching: true
53+
54+
backup_backend:
55+
enabled: false
56+
db_type: azure_ai_search
57+
priority: -1
58+
properties:
59+
service_name: "nlweb-backup"
60+
index_name: "nlweb-backup-index"
61+
62+
# Multi-backend settings
63+
enable_parallel_querying: true
64+
enable_result_deduplication: true
65+
max_concurrent_queries: 3
66+
backend_timeout_seconds: 30
67+
68+
# General NLWeb settings
69+
nlweb:
70+
default_mode: "List"
71+
enable_streaming: true
72+
default_timeout_seconds: 30
73+
max_results_per_query: 50
74+
enable_caching: true
75+
cache_expiration_minutes: 60
76+
tool_selection_enabled: true
77+
```
78+
79+
### Auto-Detection
80+
81+
The configuration system automatically looks for these YAML files:
82+
83+
- `config_retrieval.yaml` / `config_retrieval.yml`
84+
- `nlweb.yaml` / `nlweb.yml`
85+
- Environment-specific versions (e.g., `config_retrieval.Development.yaml`)
86+
87+
### Manual YAML Loading
88+
89+
You can also manually load YAML configuration:
90+
91+
```csharp
92+
builder.Configuration.AddYamlFile("my-config.yaml", optional: true, reloadOnChange: true);
93+
94+
// Or from a stream
95+
using var stream = File.OpenRead("config.yaml");
96+
builder.Configuration.AddYamlStream(stream);
97+
```
98+
99+
## XML Tool Definitions
100+
101+
### Tool Definition Loader Service
102+
103+
The `IToolDefinitionLoader` service provides XML-based tool definition support:
104+
105+
```csharp
106+
// Register the service (included in AddNLWebConfigurationFormats)
107+
builder.Services.AddSingleton<IToolDefinitionLoader, ToolDefinitionLoader>();
108+
109+
// Use the service
110+
var toolLoader = serviceProvider.GetRequiredService<IToolDefinitionLoader>();
111+
var toolDefinitions = await toolLoader.LoadFromFileAsync("tool_definitions.xml");
112+
```
113+
114+
### XML Tool Definition Format
115+
116+
```xml
117+
<?xml version="1.0" encoding="utf-8"?>
118+
<ToolDefinitions>
119+
<Tool id="enhanced-search" name="Enhanced Search Tool" type="search" enabled="true" priority="10">
120+
<Description>Advanced search capability with semantic understanding</Description>
121+
<Parameters>
122+
<MaxResults>50</MaxResults>
123+
<TimeoutSeconds>30</TimeoutSeconds>
124+
<EnableCaching>true</EnableCaching>
125+
<CustomProperties>
126+
<Property name="semantic_search" value="true" type="boolean" />
127+
<Property name="relevance_threshold" value="0.7" type="float" />
128+
</CustomProperties>
129+
</Parameters>
130+
<TriggerPatterns>
131+
<Pattern>search for*</Pattern>
132+
<Pattern>find*</Pattern>
133+
<Pattern>what is*</Pattern>
134+
</TriggerPatterns>
135+
<SupportedBackends>
136+
<Backend>azure_ai_search</Backend>
137+
<Backend>mock</Backend>
138+
</SupportedBackends>
139+
</Tool>
140+
</ToolDefinitions>
141+
```
142+
143+
### Tool Types
144+
145+
Supported tool types:
146+
- `search` - Enhanced search capabilities
147+
- `details` - Retrieve specific information about items
148+
- `compare` - Side-by-side comparison of items
149+
- `ensemble` - Create cohesive sets of related items
150+
151+
## Backward Compatibility
152+
153+
### Existing JSON Configuration
154+
155+
All existing JSON configuration continues to work unchanged:
156+
157+
```json
158+
{
159+
"NLWebNet": {
160+
"DefaultMode": "List",
161+
"EnableStreaming": true,
162+
"MultiBackend": {
163+
"Enabled": true,
164+
"WriteEndpoint": "primary",
165+
"Endpoints": {
166+
"primary": {
167+
"Enabled": true,
168+
"BackendType": "azure_ai_search"
169+
}
170+
}
171+
}
172+
}
173+
}
174+
```
175+
176+
### Migration Path
177+
178+
1. **Phase 1**: Add YAML support alongside existing JSON
179+
2. **Phase 2**: Gradually migrate configuration to YAML format
180+
3. **Phase 3**: Optionally remove JSON configuration files (JSON support remains)
181+
182+
No code changes are required to existing applications - the new formats are additive.
183+
184+
## Configuration Options
185+
186+
### ConfigurationFormatOptions
187+
188+
Control configuration format behavior:
189+
190+
```csharp
191+
builder.Services.Configure<NLWebOptions>(options =>
192+
{
193+
options.ConfigurationFormat.EnableYamlSupport = true;
194+
options.ConfigurationFormat.EnableXmlToolDefinitions = true;
195+
options.ConfigurationFormat.AutoDetectConfigurationFiles = true;
196+
options.ConfigurationFormat.YamlConfigPath = "custom-config.yaml";
197+
options.ConfigurationFormat.XmlToolDefinitionsPath = "tools.xml";
198+
});
199+
```
200+
201+
## Examples
202+
203+
See the demo application for complete examples:
204+
- `/samples/Demo/config_retrieval.yaml` - Multi-backend YAML configuration
205+
- `/samples/Demo/tool_definitions.xml` - XML tool definitions
206+
- `/samples/Demo/Program.cs` - Integration example
207+
208+
## Dependencies
209+
210+
The new configuration format support adds these dependencies:
211+
- `YamlDotNet` (16.2.1) - YAML parsing
212+
- `Microsoft.Extensions.Configuration.Xml` (9.0.6) - XML configuration support
213+
214+
## Testing
215+
216+
Comprehensive tests cover:
217+
- YAML parsing and configuration binding
218+
- XML tool definition loading and validation
219+
- Service registration and dependency injection
220+
- Backward compatibility with existing JSON configuration
221+
- Error handling and validation
222+
223+
Run tests with: `dotnet test --filter "ConfigurationFormatSupportTests"`

‎samples/Demo/Program.cs‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,9 @@
66

77
var builder = WebApplication.CreateBuilder(args);
88

9+
// Configure configuration with NLWeb format support
10+
builder.Configuration.AddNLWebConfigurationFormats(builder.Environment);
11+
912
// Detect if running in Aspire and configure accordingly
1013
var isAspireEnabled = !string.IsNullOrEmpty(Environment.GetEnvironmentVariable("DOTNET_ASPIRE_URLS")) ||
1114
!string.IsNullOrEmpty(Environment.GetEnvironmentVariable("OTEL_EXPORTER_OTLP_ENDPOINT"));
@@ -82,6 +85,9 @@
8285
});
8386
}
8487

88+
// Add NLWeb configuration format support (YAML, XML tool definitions)
89+
builder.Services.AddNLWebConfigurationFormats(builder.Configuration);
90+
8591
// IMPORTANT: Override the default MockDataBackend with our enhanced version AFTER AddNLWebNet
8692
// Use enhanced data backend with RSS feed integration and better sample data
8793
builder.Services.AddScoped<NLWebNet.Services.IDataBackend, EnhancedMockDataBackend>();

‎samples/Demo/config_retrieval.yaml‎

Lines changed: 55 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,55 @@
1+
# NLWeb Multi-Backend Configuration Example
2+
# This file demonstrates the YAML-style configuration format introduced in June 2025
3+
4+
# Primary backend for write operations
5+
write_endpoint: primary_backend
6+
7+
# Backend endpoint configurations
8+
endpoints:
9+
primary_backend:
10+
enabled: true
11+
db_type: azure_ai_search
12+
priority: 1
13+
properties:
14+
service_name: "nlweb-search"
15+
index_name: "nlweb-index"
16+
api_version: "2023-11-01"
17+
# connection_string: "..." # Set via environment variables
18+
19+
secondary_backend:
20+
enabled: true
21+
db_type: mock
22+
priority: 0
23+
properties:
24+
data_source: "sample_data"
25+
enable_fuzzy_matching: true
26+
27+
backup_backend:
28+
enabled: false
29+
db_type: azure_ai_search
30+
priority: -1
31+
properties:
32+
service_name: "nlweb-backup"
33+
index_name: "nlweb-backup-index"
34+
35+
# Multi-backend settings
36+
enable_parallel_querying: true
37+
enable_result_deduplication: true
38+
max_concurrent_queries: 3
39+
backend_timeout_seconds: 30
40+
41+
# General NLWeb settings that can also be configured via YAML
42+
nlweb:
43+
default_mode: "List"
44+
enable_streaming: true
45+
default_timeout_seconds: 30
46+
max_results_per_query: 50
47+
enable_caching: true
48+
cache_expiration_minutes: 60
49+
tool_selection_enabled: true
50+
51+
# Tool selection configuration
52+
tool_selection:
53+
default_tool: "search"
54+
enable_auto_detection: true
55+
fallback_to_search: true

0 commit comments

Comments
 (0)