Skip to content

Latest commit

 

History

History
300 lines (250 loc) · 6.88 KB

README.MD

File metadata and controls

300 lines (250 loc) · 6.88 KB

DToon - Delphi Support Lib for Toon (Token-Oriented Object Notation)

DToon is a lightweight Delphi library for parsing TOON (Token-Oriented Object Notation), a modern data serialization format designed specifically for the AI era.

It is based on the official TOON format overview TOON and models data in a JSON-like way, but with a more human-friendly, indentation-based syntax.

What is TOON?

While JSON is the standard for web APIs, it is "token-heavy" due to excessive punctuation (braces, quotes, commas). TOON solves this by using a clean, indentation-based syntax similar to YAML, but with the strict structural model of JSON.

Key Benefits:

  • Token Efficiency: Reduces token usage by removing syntactic noise, saving costs and context window space in LLM interactions.
  • High Density: Supports "Table Arrays" (e.g., users[N]{fields}), allowing lists of objects to be represented as dense CSV-like rows.
  • Human Readable: Easier to read and edit manually than minified JSON.

DToon brings this efficiency to the Delphi ecosystem, allowing you to easily integrate token-optimized data streams into your applications.

TOON Data Model

TOON models data the same way as JSON:

  • Primitives

    • strings
    • numbers
    • booleans
    • null

  • Objects

    • mappings from string keys to values

  • Arrays

    • ordered sequences of values

Standard TOON (like YAML) excels at readability but suffers with arrays of objects. Repeating keys and indentation for every item wastes tokens. Uniform arrays are converted into a tabular format supported TToonParser. This separates schema from data, drastically reducing overhead.

Json:

{
  "meta": {
    "version": "1.0",
    "generated": "2023-10-27"
  },
  "users": [
    {
      "id": 101,
      "name": "Alice Johnson",
      "role": "Admin",
      "active": true,
      "login_count": 42
    },
    {
      "id": 102,
      "name": "Bob Smith",
      "role": "Editor",
      "active": true,
      "login_count": 15
    },
    {
      "id": 103,
      "name": "Charlie Brown",
      "role": "Viewer",
      "active": false,
      "login_count": 3
    },
    {
      "id": 104,
      "name": "Diana Prince",
      "role": "Admin",
      "active": true,
      "login_count": 128
    },
    {
      "id": 105,
      "name": "Evan Wright",
      "role": "Viewer",
      "active": true,
      "login_count": 0
    }
  ],
  "inventory": [
    {
      "sku": "WID-001",
      "desc": "Widget A",
      "stock": 500,
      "price": 19.99
    },
    {
      "sku": "WID-002",
      "desc": "Widget B",
      "stock": 0,
      "price": 25.5
    },
    {
      "sku": "GAD-001",
      "desc": "Gadget X",
      "stock": 120,
      "price": 99
    }
  ]
}

Standard TOON (Nested)

meta:
  version: 1.0
  generated: 2023-10-27
users:
  item_0:
    id: 101
    name: Alice Johnson
    role: Admin
    active: true
    login_count: 42
  item_1:
    id: 102
    name: Bob Smith
    role: Editor
    active: true
    login_count: 15
  item_2:
    id: 103
    name: Charlie Brown
    role: Viewer
    active: false
    login_count: 3
  item_3:
    id: 104
    name: Diana Prince
    role: Admin
    active: true
    login_count: 128
  item_4:
    id: 105
    name: Evan Wright
    role: Viewer
    active: true
    login_count: 0
inventory:
  item_0:
    sku: WID-001
    desc: Widget A
    stock: 500
    price: 19.99
  item_1:
    sku: WID-002
    desc: Widget B
    stock: 0
    price: 25.5
  item_2:
    sku: GAD-001
    desc: Gadget X
    stock: 120
    price: 99

Optimized TOON (Flat)

meta:
  version: 1.0
  generated: 2023-10-27
users[5]{id,name,role,active,login_count}:
  101,Alice Johnson,Admin,true,42
  102,Bob Smith,Editor,true,15
  103,Charlie Brown,Viewer,false,3
  104,Diana Prince,Admin,true,128
  105,Evan Wright,Viewer,true,0
inventory[3]{sku,desc,stock,price}:
  WID-001,Widget A,500,19.99
  WID-002,Widget B,0,25.5
  GAD-001,Gadget X,120,99
Token Usage Analysis Usage
JSON 100% ██████████████
Standard TOON (Nested) 74% ██████████░░░░
Optimized TOON (Flat) 35% █████░░░░░░░░░
  • Based on character count approximation (1 token ≈ 4 chars). Actual LLM tokenization may vary slightly.

Root Forms

A TOON document can represent different root forms:

  • Root object (most common)
    Fields appear at depth 0 with no parent key.

  • Root array
    Begins with:

    • [N]:
    • or [N]{fields}:

    at depth 0, where N is the array length.

  • Root primitive
    A single primitive value (string, number, boolean, or null).

Main Types

The unit exposes two core classes:

TToonNode

Represents a node in the TOON tree:

  • NodeType: TToonNodeType – kind of node:
    • tntObject, tntArray, tntString, tntNumber, tntBoolean, tntNull
  • Key: string – field name (for object children)
  • Value: string – raw string value for primitive nodes
  • Items[Index: Integer]: TToonNode – indexed access to children
  • Pair[const KeyName: string]: TToonNode – access by key (for objects)
  • Count: Integer – number of children

Helper methods:

  • AddChild(AType: TToonNodeType; AKey: string = ''): TToonNode
  • AsString: string
  • AsInteger: Integer
  • AsBoolean: Boolean
  • ToString: string – debug: prints the subtree as text

TToonParser

Parses a TOON document from a string into a TToonNode tree:

  • function Parse(const AToonContent: string): TToonNode;

Internally it:

  • splits content into lines,
  • calculates indentation depth,
  • builds a stack of parent nodes,
  • creates object/array/primitive nodes according to TOON rules.

Basic Usage

uses
  DToon;

procedure DemoToon;
const
  CToonSample =
    'name: "Example"' + sLineBreak +
    'version: 1'       + sLineBreak +
    'enabled: true'    + sLineBreak +
    'items[2]:'        + sLineBreak +
    '  0: "foo"'       + sLineBreak +
    '  1: "bar"';
var
  Parser: TToonParser;
  Root: TToonNode;
begin
  Parser := TToonParser.Create;
  try
    Root := Parser.Parse(CToonSample);

    // Object fields
    Writeln('Name: ',    Root.Pair['name'].AsString);
    Writeln('Version: ', Root.Pair['version'].AsInteger);
    Writeln('Enabled: ', BoolToStr(Root.Pair['enabled'].AsBoolean, True));

    // Array items
    Writeln('First item: ', Root.Pair['items'][0].AsString);

    // Debug print
    Writeln(Root.ToString);
  finally
    Parser.Free;
  end;
end;

Requirements

  • Delphi (any modern version with:
    • System.SysUtils
    • System.Classes
    • System.Generics.Collections
    • System.RegularExpressions
    • System.StrUtils
  • No external dependencies beyond the RTL.

License

This project is licensed under the MIT License - see the LICENSE. file for details.