Merge pull request mischasigtermans#12 from mischasigtermans/feat/spec-test-fixtures

TOON v3.0 Spec Compliance

Author: mischasigtermans

.github/workflows/tests.yml

@@ -52,5 +52,8 @@ jobs:
           composer require "laravel/framework:${{ matrix.laravel }}" "orchestra/testbench:${{ matrix.testbench }}" --no-interaction --no-update
           composer update --prefer-stable --prefer-dist --no-interaction
 
+      - name: Install spec fixtures
+        run: npm install
+
       - name: Execute tests
         run: vendor/bin/pest

CHANGELOG.md

@@ -7,13 +7,13 @@ All notable changes to this project will be documented in this file.
 ### Added
 - Full [TOON v3.0 specification](https://github.com/toon-format/spec/blob/main/SPEC.md) compliance
 - Global helper functions: `toon_encode()` and `toon_decode()`
-- Performance optimizations for encoding and decoding
-- Config injection support for encoder/decoder constructors (improved testability)
 - Spec-compliant string quoting (strings with special characters are now quoted with `"..."`)
 - Proper escape sequences within quoted strings (`\n`, `\r`, `\t`, `\"`, `\\`)
 - Delimiter support: comma (default), tab (`\t`), and pipe (`|`) via `delimiter` config option
 - Strict mode for decoding with validation errors via `strict` config option
-- Official specification test suite (65 compliance tests)
+- Key folding: collapse single-key nested objects into dot notation via `key_folding` config
+- Path expansion: expand dotted keys back to nested objects via `expand_paths` config
+- Official specification test fixtures from [toon-format/spec](https://github.com/toon-format/spec)
 - Inline primitive array format (`key[N]: a,b,c`)
 
 ### Changed
@@ -24,11 +24,12 @@ All notable changes to this project will be documented in this file.
 - Float encoding now preserves full IEEE 754 double precision (16 significant digits)
 
 ### Migration Guide
-The decoder maintains backward compatibility and will correctly parse both the old backslash-escaped format and the new quoted string format. However, if you have code that expects the old output format, be aware that:
-1. Encoded output will now use quoted strings for special characters
-2. The `escape_style` config option has been removed
-3. Republish config to get new options: `php artisan vendor:publish --tag=toon-config --force`
-4. Set `strict => false` in config if parsing legacy TOON that may have formatting issues
+The decoder maintains backward compatibility and parses both old backslash-escaped format and new quoted strings.
+
+If you have code that expects the old output format:
+1. Encoded output now uses quoted strings for special characters
+2. Republish config: `php artisan vendor:publish --tag=toon-config --force`
+3. Set `strict => false` if parsing legacy TOON with formatting issues
 
 ## [0.2.2] - 2025-12-28
 
@@ -63,11 +64,10 @@ The decoder maintains backward compatibility and will correctly parse both the o
 
 ### Added
 - Initial release
-- TOON encoding with automatic nested object flattening using dot notation
+- TOON encoding with tabular array format
 - TOON decoding with nested object reconstruction
-- Tabular array format for compact representation
 - Type preservation (int, float, bool, null)
 - Special character escaping (comma, colon, newline)
-- Configurable flatten depth and table thresholds
+- Configurable table thresholds
 - Laravel 9, 10, 11, and 12 support
 - PHP 8.2+ support

README.md

@@ -3,9 +3,9 @@
 [![Latest Version on Packagist](https://img.shields.io/packagist/v/mischasigtermans/laravel-toon.svg?style=flat-square)](https://packagist.org/packages/mischasigtermans/laravel-toon)
 [![Total Downloads](https://img.shields.io/packagist/dt/mischasigtermans/laravel-toon.svg?style=flat-square)](https://packagist.org/packages/mischasigtermans/laravel-toon)
 
-A spec-compliant TOON (Token-Optimized Object Notation) encoder/decoder for Laravel with intelligent nested object handling.
+The most complete [TOON](https://toonformat.dev/) implementation for Laravel, and the only one with full [TOON v3.0 specification](https://github.com/toon-format/spec/blob/main/SPEC.md) compliance.
 
-TOON is a compact, YAML-like format designed to reduce token usage when sending data to LLMs. This package implements the [TOON v3.0 specification](https://github.com/toon-format/spec/blob/main/SPEC.md) and achieves **40-60% token reduction** compared to JSON while maintaining full round-trip fidelity.
+TOON (Token-Optimized Object Notation) is a compact, YAML-like format designed to reduce token usage when sending data to LLMs. This package achieves **~50% token reduction** compared to JSON while maintaining full round-trip fidelity, backed by 470 tests.
 
 ## Installation
 
@@ -20,8 +20,8 @@ use MischaSigtermans\Toon\Facades\Toon;
 
 $data = [
     'users' => [
-        ['id' => 1, 'name' => 'Alice', 'role' => ['id' => 'admin', 'level' => 10]],
-        ['id' => 2, 'name' => 'Bob', 'role' => ['id' => 'user', 'level' => 1]],
+        ['id' => 1, 'name' => 'Alice', 'active' => true],
+        ['id' => 2, 'name' => 'Bob', 'active' => false],
     ],
 ];
 
@@ -34,10 +34,9 @@ $original = Toon::decode($toon);
 
 **Output:**
 ```
-users:
-  items[2]{id,name,role.id,role.level}:
-    1,Alice,admin,10
-    2,Bob,user,1
+users[2]{id,name,active}:
+  1,Alice,true
+  2,Bob,false
 ```
 
 ### Global Helper Functions
@@ -84,78 +83,76 @@ $toon = $user->toToon();
 
 When building MCP servers or LLM-powered applications, every token counts. JSON's verbosity wastes context window space with repeated keys and structural characters.
 
-**JSON (398 bytes):**
+**JSON (201 bytes):**
 ```json
-{"orders":[{"id":"ord_1","status":"shipped","customer":{"id":"cust_1","name":"Alice"},"total":99.99},{"id":"ord_2","status":"pending","customer":{"id":"cust_2","name":"Bob"},"total":149.50}]}
+{"users":[{"id":1,"name":"Alice","role":"admin"},{"id":2,"name":"Bob","role":"user"},{"id":3,"name":"Carol","role":"user"}]}
 ```
 
-**TOON (186 bytes) - 53% smaller:**
+**TOON (62 bytes) - 69% smaller:**
 ```
-orders:
-  items[2]{id,status,customer.id,customer.name,total}:
-    ord_1,shipped,cust_1,Alice,99.99
-    ord_2,pending,cust_2,Bob,149.5
+users[3]{id,name,role}:
+  1,Alice,admin
+  2,Bob,user
+  3,Carol,user
 ```
 
 ## Benchmarks
 
+For a typical paginated API response (50 records):
+- **JSON**: ~7,597 tokens
+- **TOON**: ~3,586 tokens
+- **Saved**: ~4,000 tokens per request
+
 Real-world benchmarks from a production application with 17,000+ records:
 
 | Data Type | JSON | TOON | Savings |
 |-----------|------|------|---------|
-| 50 records (nested objects) | 13,055 bytes | 5,080 bytes | **61%** |
-| 100 records (nested objects) | 26,156 bytes | 10,185 bytes | **61%** |
-| 500 records (nested objects) | 129,662 bytes | 49,561 bytes | **62%** |
-| 1,000 records (nested objects) | 258,965 bytes | 98,629 bytes | **62%** |
-| 100 records (mixed nesting) | 43,842 bytes | 26,267 bytes | **40%** |
-| Single object | 169 bytes | 124 bytes | **27%** |
-
-### Token Impact
-
-For a typical paginated API response (50 records):
-- **JSON**: ~3,274 tokens
-- **TOON**: ~1,279 tokens
-- **Saved**: ~2,000 tokens per request
+| 50 records | 30,389 bytes | 14,343 bytes | **53%** |
+| 100 records | 60,856 bytes | 28,498 bytes | **53%** |
+| 500 records | 303,549 bytes | 140,154 bytes | **54%** |
+| 1,000 records | 604,408 bytes | 277,614 bytes | **54%** |
 
 ## Features
 
-### Nested Object Flattening
+### Tabular Format
 
-The key differentiator. Arrays containing objects with nested properties are automatically flattened using dot notation:
+Arrays of uniform objects with primitive values are encoded as compact tables:
 
 ```php
 $data = [
-    ['id' => 1, 'author' => ['name' => 'Jane', 'email' => 'jane@example.com']],
-    ['id' => 2, 'author' => ['name' => 'John', 'email' => 'john@example.com']],
+    ['id' => 1, 'name' => 'Alice', 'role' => 'admin'],
+    ['id' => 2, 'name' => 'Bob', 'role' => 'user'],
 ];
 
 $toon = Toon::encode($data);
-// items[2]{id,author.name,author.email}:
-//   1,Jane,jane@example.com
-//   2,John,john@example.com
-
-$decoded = Toon::decode($toon);
-// Returns original nested structure
+// [2]{id,name,role}:
+//   1,Alice,admin
+//   2,Bob,user
 ```
 
-### Multi-Level Nesting
+### List Format for Nested Objects
 
-Handles deeply nested structures:
+Arrays containing objects with nested properties use list format for clarity:
 
 ```php
 $data = [
-    [
-        'id' => 1,
-        'product' => [
-            'name' => 'Widget',
-            'category' => ['id' => 'cat_1', 'name' => 'Electronics'],
-        ],
-    ],
+    ['id' => 1, 'author' => ['name' => 'Jane', 'email' => 'jane@example.com']],
+    ['id' => 2, 'author' => ['name' => 'John', 'email' => 'john@example.com']],
 ];
 
 $toon = Toon::encode($data);
-// items[1]{id,product.name,product.category.id,product.category.name}:
-//   1,Widget,cat_1,Electronics
+// [2]:
+//   - id: 1
+//     author:
+//       name: Jane
+//       email: jane@example.com
+//   - id: 2
+//     author:
+//       name: John
+//       email: john@example.com
+
+$decoded = Toon::decode($toon);
+// Returns original nested structure
 ```
 
 ### Type Preservation
@@ -207,9 +204,6 @@ return [
     // Arrays with fewer items use regular object format instead of tables
     'min_rows_for_table' => 2,
 
-    // How deep to flatten nested objects (deeper = JSON string)
-    'max_flatten_depth' => 3,
-
     // Delimiter for array values: ',' (default), '\t' (tab), or '|' (pipe)
     'delimiter' => ',',
 
@@ -335,18 +329,19 @@ This package implements the [TOON v3.0 specification](https://github.com/toon-fo
 
 - **String quoting**: Safe strings unquoted, special characters properly escaped (`\n`, `\r`, `\t`, `\"`, `\\`)
 - **Delimiter support**: Comma (default), tab, and pipe delimiters
-- **Array formats**: Inline primitives (`[N]: a,b,c`) and tabular objects (`[N]{fields}:`)
-- **Nested object flattening**: Dot notation for nested properties in tabular format
+- **Tabular format**: Compact tables for arrays of primitive-only objects (`[N]{fields}:`)
+- **List format**: Readable structure for arrays with nested objects (`[N]:` with `- field:` items)
+- **Inline arrays**: Primitive arrays on single line (`key[N]: a,b,c`)
 - **Strict mode**: Optional validation during decoding
-- **Backward compatibility**: Decoder accepts legacy backslash escaping
+- **Backward compatibility**: Decoder accepts legacy formats (backslash escaping, dot-notation columns)
 
 ## Testing
 
 ```bash
 composer test
 ```
 
-The test suite includes 118 tests covering encoding, decoding, nested object handling, and official spec compliance fixtures.
+The test suite includes 470 tests covering encoding, decoding, nested object handling, and official spec compliance fixtures.
 
 ## Requirements
 

config/toon.php

@@ -67,6 +67,42 @@
     */
     'strict' => true,
 
+    /*
+    |--------------------------------------------------------------------------
+    | Key Folding
+    |--------------------------------------------------------------------------
+    |
+    | When set to 'safe', single-key nested objects are folded into dot notation.
+    | Example: {user: {name: "Alice"}} becomes user.name: Alice
+    |
+    | Supported values: 'off' (default), 'safe'
+    |
+    */
+    'key_folding' => 'off',
+
+    /*
+    |--------------------------------------------------------------------------
+    | Key Folding Depth
+    |--------------------------------------------------------------------------
+    |
+    | Maximum depth for key folding. Only applies when key_folding is 'safe'.
+    |
+    */
+    'key_folding_depth' => null,
+
+    /*
+    |--------------------------------------------------------------------------
+    | Expand Paths
+    |--------------------------------------------------------------------------
+    |
+    | When set to 'safe', dotted keys are expanded back to nested objects
+    | during decoding. Example: user.name: Alice becomes {user: {name: "Alice"}}
+    |
+    | Supported values: 'off' (default), 'safe'
+    |
+    */
+    'expand_paths' => 'off',
+
     /*
     |--------------------------------------------------------------------------
     | Omit Values

package-lock.json

@@ -0,0 +1,5 @@
+{
+  "devDependencies": {
+    "@toon-format/spec": "^3.0.1"
+  }
+}

package.json

@@ -6,5 +6,8 @@
         <testsuite name="Feature">
             <directory suffix="Test.php">./tests/Feature</directory>
         </testsuite>
+        <testsuite name="Specs">
+            <directory suffix="Test.php">./tests/Specs</directory>
+        </testsuite>
     </testsuites>
 </phpunit>