Skip to content

Commit 3faa8c3

Browse files
ghost1facekrlosflipdevjohannschopplichCopilot
authored
Toon Spec 3.0 (#23)
* Implemented Specs v3.0 * - Removed `IsIdentifierSegment` and `IdentifierPattern` from `PathExpansion.cs` - Replaced dot (.) with `Constants.DOT` in `PathExpansion.cs` - Added `ToonPathExpansionException` - Updated spec generator aligned to v3.0.0 * Update README.md Co-authored-by: Johann Schopplich <johann@schopplich.com> * Fixes - Make Encode windows-friendly - Make SpecGenerator make a clean of tests folder and re-generate the files - Updated `specgen.sh` aligned to .ps1 file * Revert "Fixes" This reverts commit f3c8e11. * Fixes - Updated `specgen.sh` as same to ps1 file - Added exception type validation in `FixtureWriter.cs` * Fixed tests * Updated tests * Fixed tests on windows * feat: Ensure numeric values are fully represented * feat: Remove LengthMarker references * chore: Remove unused converters * chore: Test cleanup * chore: Change test project structure, manual tests/specs are no longer copied from the SpecGen project * feat: Test to validate issue #7 is resolved * chore: Default namespaces to Toon.Format * chore: Regenerate test file for appropriate line endings * Update README.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update README.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * chore: Code cleanup based on code review --------- Co-authored-by: Carlos Felipe <develop.krlosflip@gmail.com> Co-authored-by: Johann Schopplich <johann@schopplich.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
1 parent f7b338c commit 3faa8c3

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

72 files changed

+1600
-313
lines changed

‎README.md‎

Lines changed: 236 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -4,15 +4,37 @@
44
[![.NET version](https://img.shields.io/badge/.NET-8.0%20%7C%209.0-512BD4)](https://dotnet.microsoft.com/)
55
[![License: MIT](https://img.shields.io/badge/license-MIT-blue.svg)](./LICENSE)
66

7-
**Token-Oriented Object Notation** is a compact, human-readable format designed for passing structured data to Large Language Models with significantly reduced token usage.
7+
**Token-Oriented Object Notation** is a compact, human-readable encoding of the JSON data model that minimizes tokens and makes structure easy for models to follow. Combines YAML-like indentation with CSV-like tabular arrays. Fully compatible with the [official TOON specification v3.0](https://github.com/toon-format/spec).
88

9-
## Status
9+
**Key Features:** Minimal syntax • TOON Encoding and Decoding • Tabular arrays for uniform data • Path expansion • Strict mode validation • .NET 8.0, 9.0 and 10.0 • 520+ tests with 99.7% spec coverage.
1010

11-
🚧 **This package is currently a namespace reservation.** Full implementation coming soon!
11+
## Quick Start
1212

13-
### Example
13+
```csharp
14+
using Toon.Format;
15+
16+
var data = new
17+
{
18+
users = new[]
19+
{
20+
new { id = 1, name = "Alice", role = "admin" },
21+
new { id = 2, name = "Bob", role = "user" }
22+
}
23+
};
24+
25+
Console.WriteLine(ToonEncoder.Encode(data));
26+
```
27+
28+
**Output:**
29+
30+
```
31+
users[2]{id,name,role}:
32+
1,Alice,admin
33+
2,Bob,user
34+
```
35+
36+
**Compared to JSON (30-60% token reduction):**
1437

15-
**JSON** (verbose):
1638
```json
1739
{
1840
"users": [
@@ -22,35 +44,232 @@
2244
}
2345
```
2446

25-
**TOON** (compact):
47+
## Installation
48+
49+
```bash
50+
dotnet add package Toon.Format
51+
```
52+
53+
## Type Conversions
54+
55+
.NET-specific types are automatically normalized for LLM-safe output:
56+
57+
| Input Type | Output |
58+
| --- | --- |
59+
| Number (finite) | Decimal form; `-0` → `0`; no scientific notation |
60+
| Number (`NaN`, `±Infinity`) | `null` |
61+
| `decimal`, `double`, `float` | Decimal number |
62+
| `DateTime`, `DateTimeOffset` | ISO 8601 string in quotes |
63+
| `Guid` | String in quotes |
64+
| `IDictionary<,>`, `Dictionary<,>` | Object with string keys |
65+
| `IEnumerable<>`, arrays | Arrays |
66+
| Nullable types | Unwrapped value or `null` |
67+
68+
## API
69+
70+
### `ToonEncoder.Encode(object value): string`
71+
72+
### `ToonEncoder.Encode(object value, ToonEncodeOptions options): string`
73+
74+
Converts any .NET object to TOON format.
75+
76+
**Parameters:**
77+
78+
- `value` – Any .NET object (class, record, dictionary, list, or primitive). Non-serializable values are converted to `null`. DateTime types are converted to ISO strings.
79+
- `options` – Optional encoding options:
80+
- `Indent` – Number of spaces per indentation level (default: `2`)
81+
- `Delimiter` – Delimiter for array values: `ToonDelimiter.COMMA` (default), `TAB`, or `PIPE`
82+
- `KeyFolding` – Collapse nested single-key objects: `ToonKeyFolding.Off` or `Safe` (default: `Off`)
83+
84+
**Returns:**
85+
86+
A TOON-formatted string with no trailing newline or spaces.
87+
88+
**Example:**
89+
90+
```csharp
91+
using Toon.Format;
92+
93+
record Item(string Sku, int Qty, double Price);
94+
record Data(List<Item> Items);
95+
96+
var item1 = new Item("A1", 2, 9.99);
97+
var item2 = new Item("B2", 1, 14.5);
98+
var data = new Data(new List<Item> { item1, item2 });
99+
100+
Console.WriteLine(ToonEncoder.Encode(data));
101+
```
102+
103+
**Output:**
104+
105+
```
106+
Items[2]{Sku,Qty,Price}:
107+
A1,2,9.99
108+
B2,1,14.5
109+
```
110+
111+
#### Delimiter Options
112+
113+
Alternative delimiters can provide additional token savings:
114+
115+
**Tab Delimiter:**
116+
117+
```csharp
118+
var options = new ToonEncodeOptions
119+
{
120+
Delimiter = ToonDelimiter.TAB
121+
};
122+
Console.WriteLine(ToonEncoder.Encode(data, options));
123+
```
124+
125+
**Output:**
126+
127+
```
128+
Items[2 ]{Sku Qty Price}:
129+
A1 2 9.99
130+
B2 1 14.5
26131
```
132+
133+
**Pipe Delimiter:**
134+
135+
```csharp
136+
var options = new ToonEncodeOptions
137+
{
138+
Delimiter = ToonDelimiter.PIPE
139+
};
140+
Console.WriteLine(ToonEncoder.Encode(data, options));
141+
```
142+
143+
**Output:**
144+
145+
```
146+
Items[2|]{Sku|Qty|Price}:
147+
A1|2|9.99
148+
B2|1|14.5
149+
```
150+
151+
#### Key Folding
152+
153+
Collapse nested single-key objects for more compact output:
154+
155+
```csharp
156+
var data = new { user = new { profile = new { name = "Alice" } } };
157+
158+
var options = new ToonEncodeOptions
159+
{
160+
KeyFolding = ToonKeyFolding.Safe
161+
};
162+
Console.WriteLine(ToonEncoder.Encode(data, options));
163+
// Output: user.profile.name: Alice
164+
```
165+
166+
### `ToonDecoder.Decode(string toon): JsonNode`
167+
168+
### `ToonDecoder.Decode(string toon, ToonDecodeOptions options): JsonNode`
169+
170+
### `ToonDecoder.Decode<T>(string toon): T`
171+
172+
### `ToonDecoder.Decode<T>(string toon, ToonDecodeOptions options): T`
173+
174+
Converts TOON-formatted strings back to .NET objects.
175+
176+
**Parameters:**
177+
178+
- `toon` – TOON-formatted input string
179+
- `options` – Optional decoding options:
180+
- `Indent` – Number of spaces per indentation level (default: `2`)
181+
- `Strict` – Enable validation mode (default: `true`). When `true`, throws `ToonFormatException` on invalid input.
182+
- `ExpandPaths` – Expand dotted keys: `ToonPathExpansion.Off` (default) or `ToonPathExpansion.Safe`
183+
184+
**Returns:**
185+
186+
For generic overloads: Returns a `JsonNode` (JsonObject, JsonArray, or JsonValue) or deserialized type `T`.
187+
188+
**Example:**
189+
190+
```csharp
191+
using Toon.Format;
192+
193+
string toon = """
27194
users[2]{id,name,role}:
28195
1,Alice,admin
29196
2,Bob,user
197+
""";
198+
199+
// Decode to JsonNode
200+
var result = ToonDecoder.Decode(toon);
201+
202+
// Decode to specific type
203+
var users = ToonDecoder.Decode<List<User>>(toon);
30204
```
31205

32-
## Resources
206+
#### Path Expansion
207+
208+
Expand dotted keys into nested objects:
33209

34-
- [TOON Specification](https://github.com/toon-format/spec/blob/main/SPEC.md)
35-
- [Main Repository](https://github.com/toon-format/toon)
36-
- [Benchmarks & Performance](https://github.com/toon-format/toon#benchmarks)
37-
- [Other Language Implementations](https://github.com/toon-format/toon#other-implementations)
210+
```csharp
211+
string toon = "a.b.c: 1";
212+
213+
var options = new ToonDecodeOptions
214+
{
215+
ExpandPaths = ToonPathExpansion.Safe
216+
};
38217

39-
## Future Usage
218+
var result = ToonDecoder.Decode(toon, options);
219+
// Result: { "a": { "b": { "c": 1 } } }
220+
```
40221

41-
Once implemented, the package will provide:
222+
#### Round-Trip Conversion
42223

43224
```csharp
44225
using Toon.Format;
45226

46-
var data = // your data structure
47-
var toonString = ToonEncoder.Encode(data);
48-
var decoded = ToonDecoder.Decode(toonString);
227+
// Original data
228+
var data = new
229+
{
230+
id = 123,
231+
name = "Ada",
232+
tags = new[] { "dev", "admin" }
233+
};
234+
235+
// Encode to TOON
236+
string toon = ToonEncoder.Encode(data);
237+
238+
// Decode back to objects
239+
var decoded = ToonDecoder.Decode(toon);
240+
241+
// Or decode to specific type
242+
var typed = ToonDecoder.Decode<MyType>(toon);
49243
```
50244

245+
For more examples and options, see the [tests](./tests/ToonFormat.Tests/).
246+
247+
## Project Status
248+
249+
**This project is 100% compliant with TOON Specification v3.0**
250+
251+
This implementation:
252+
- Passes 370+ specification tests (100% coverage)
253+
- Supports all TOON v3.0 features
254+
- Handles all edge cases and strict mode validations
255+
- Fully documented with XML comments
256+
- Production-ready for .NET 8.0, .NET 9.0 and .NET 10.0
257+
258+
See [CONTRIBUTING.md](CONTRIBUTING.md) for detailed guidelines.
259+
260+
## Documentation
261+
262+
- [📘 TOON Specification v3.0](https://github.com/toon-format/spec/blob/main/SPEC.md) - Official specification
263+
- [🔧 API Tests](./tests/ToonFormat.Tests/) - Comprehensive test suite with examples
264+
- [📋 Project Plan](SPEC_V3_PROJECT_PLAN.md) - Implementation details and compliance checklist
265+
- [🤝 Contributing](CONTRIBUTING.md) - Contribution guidelines
266+
- [🏠 Main Repository](https://github.com/toon-format/toon) - TOON format home
267+
- [📊 Benchmarks](https://github.com/toon-format/toon#benchmarks) - Performance comparisons
268+
- [🌐 Other Implementations](https://github.com/toon-format/toon#other-implementations) - TypeScript, Java, Python, etc.
269+
51270
## Contributing
52271

53-
Interested in implementing TOON for .NET? Check out the [specification](https://github.com/toon-format/spec/blob/main/SPEC.md) and feel free to contribute!
272+
Interested in contributing? Check out the [specification](https://github.com/toon-format/spec/blob/main/SPEC.md) and [contribution guidelines](CONTRIBUTING.md)!
54273

55274
## License
56275

‎specgen.ps1‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,8 @@
11
# generate spec
22
$GH_REPO = "https://github.com/toon-format/spec.git"
3-
$OUT_DIR = "./tests/ToonFormat.Tests"
3+
$OUT_DIR = "./tests/ToonFormat.Tests/GeneratedTests"
44

55
# build and execute spec generator
66
dotnet build tests/ToonFormat.SpecGenerator
77

8-
dotnet run --project tests/ToonFormat.SpecGenerator -- --url="$GH_REPO" --output="$OUT_DIR" --branch="v2.0.0" --loglevel="Information"
8+
dotnet run --project tests/ToonFormat.SpecGenerator -- --url="$GH_REPO" --output="$OUT_DIR" --branch="v3.0.0" --loglevel="Information"

‎specgen.sh‎

100644100755
Lines changed: 8 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,8 @@
1-
# generate spec
2-
GH_REPO="https://github.com/toon-format/spec.git"
3-
OUT_DIR="./tests/ToonFormat.Tests"
4-
5-
# build and execute spec generator
6-
dotnet build tests/ToonFormat.SpecGenerator
7-
8-
dotnet run --project tests/ToonFormat.SpecGenerator -- --url="$GH_REPO" --output="$OUT_DIR" --branch="v2.0.0" --loglevel="Information"
1+
# generate spec
2+
GH_REPO="https://github.com/toon-format/spec.git"
3+
OUT_DIR="./tests/ToonFormat.Tests/GeneratedTests"
4+
5+
# build and execute spec generator
6+
dotnet build tests/ToonFormat.SpecGenerator
7+
8+
dotnet run --project tests/ToonFormat.SpecGenerator -- --url="$GH_REPO" --output="$OUT_DIR" --branch="v3.0.0" --loglevel="Information"

0 commit comments

Comments
 (0)