v2.0.0-preview

[2.0.0-preview] - 2025-01-14

🚀 Major Release - ASON 2.0

Complete rewrite of ASON with new syntax, architecture, and optimizations.

Added

• New ASON 2.0 Syntax - Cleaner, more intuitive format

  • Sections with @section syntax (replaces object references)
  • Semantic references $var instead of numeric #0
  • Tabular arrays [N]{fields} with pipe | delimiter
  • Dot notation for nested objects config.debug:true
  • Support for inline objects {key:value} and inline arrays [a,b,c]

• Lexer → Parser → AST → Compiler Architecture

  • Complete modular parsing pipeline
  • Token-based lexer (src/lexer/Lexer.js, TokenType.js, Token.js)
  • Recursive descent parser (src/parser/Parser.js)
  • AST nodes (src/parser/nodes/)
  • Optimized compiler/serializer (src/compiler/Serializer.js)

• Advanced Optimizations

  • Inline Compact Objects: Small objects serialized as {key:value} without spaces
  • Dot Notation in Schemas: Nested objects flattened in tabular arrays [N]{id,price.amount,price.currency}
  • Array Fields in Schemas: Arrays marked with [] suffix [N]{id,tags[]}
  • minRowsForTabular = 2: Tabular optimization from just 2 rows

• Smart Analyzers

  • ReferenceAnalyzer: Detects repeated strings for $def: section
  • SectionAnalyzer: Identifies large objects for @section markers
  • TabularAnalyzer: Finds uniform arrays with support for nested objects and arrays
  • DefinitionBuilder: Manages reference definitions

Changed

• Breaking: Complete syntax change from ASON 1.x to ASON 2.0

  • &obj0 → @section (sections instead of object references)
  • #0 → $var (semantic variable names)
  • @field1,field2 → [N]{field1,field2} (explicit count + schema)
  • , → | (pipe delimiter in tabular arrays)

• Improved Compression

  • Better detection of optimization opportunities
  • Combined optimizations (dot notation + arrays + inline objects)
  • Smarter decisions on when to use each format

• Enhanced Round-Trip Fidelity

  • 100% lossless compression/decompression
  • Proper handling of edge cases (negative numbers, nested structures)
  • Fixed YAML-style list parsing at root level

Fixed

• Parser Bugs

  • Fixed YAML-style lists (-) not working at root level
  • Fixed nested object parsing in arrays
  • Fixed negative number tokenization
  • Fixed empty object/array handling

• Serializer Issues

  • Fixed duplicate properties in sections
  • Fixed illogical ordering (sections now appear after primitives)
  • Fixed proper newline handling

Documentation

• Updated /docs/index.html - ASON 2.0 Playground
• Updated /docs/docs.html - ASON 2.0 Documentation
• Updated /docs/benchmarks.html - ASON 2.0 Benchmarks
• Updated /docs/tokenizer.html - Token Comparison Tool
• All examples updated to ASON 2.0 syntax

Performance

• 20-60% token reduction vs JSON (maintained from 1.x)
• Improved human readability with semantic references
• Better LLM compatibility with cleaner syntax
• Scalable format suitable for large datasets

Migration Guide

ASON 1.x is not compatible with ASON 2.0. To migrate:

1. Recompress your data with ASON 2.0
2. Update any manual ASON strings to new syntax
3. See /docs/docs.html for complete syntax reference

Internal Changes

• Complete rewrite of codebase (Lexer → Parser → AST → Compiler)
• Improved test coverage (30 passing tests)
• Better separation of concerns
• Modular analyzer system
• Cleaner code organization

v1.1.4

[1.1.4] - 2025-11-13

Added

• Interactive Token Comparison Tool - /docs/tokenizer.html
  • Real GPT tokenization using gpt-tokenizer library
  • Compare token counts across 6 formats: Pretty JSON, JSON, YAML, TOON, ASON, CSV
  • Visual token highlighting - each token displayed with unique color
  • 16 pastel color palette for token visualization
  • Preset datasets: small-simple, small-nested, medium-users, large-complex (Stripe)
  • Custom JSON input support with live analysis
  • Dynamic baseline comparison (compare any format against any other)
  • All permutations comparison table showing all datasets
  • Compact Vercel-style UI matching docs design
  • Local TOON library integration (/docs/js/toon.js + wrapper)
  • Interactive hover effects on tokens (scale + shadow)
  • Responsive grid layout for format cards
  • Key findings section explaining token efficiency

Changed

• CI/CD - Automated Publishing with Provenance

  • Updated /.github/workflows/npm-publish.yml with npm provenance support
  • Added --provenance flag to npm publish command
  • Added permissions.id-token: write for cryptographic signing
  • Package now displays verified checkmark ✓ on npm
  • Automated GitHub release creation with changelogithub
  • Supports both manual releases and git tag triggers (v*)
  • Node.js 20.x (updated from 18.x)
  • Full transparency log and build attestation

• Documentation Navigation

  • Added "Token Comparison" link to main docs navigation
  • Blue highlight for new tokenizer tool link
  • Consistent navigation across all doc pages

Fixed

• Tokenizer Library Loading

  • Replaced tiktoken (ES module issues) with gpt-tokenizer
  • Added local TOON library at /docs/js/toon.js
  • Created /docs/js/toon-wrapper.js to expose TOON globally
  • Removed broken CDN imports
  • All libraries now load correctly in browser
  • Console logs verify library availability

• CSV Conversion

  • Improved CSV generator to handle all unique keys
  • Added automatic quoting for values containing commas
  • Better detection of nested objects (shows "N/A (nested objects)")
  • Handles null/undefined values gracefully

• TOON Conversion

  • Enhanced manual TOON fallback implementation
  • Proper handling of non-uniform arrays
  • String quoting for values with commas/spaces
  • Clear error messages: "N/A (non-uniform)", "N/A (nested objects)"

v1.1.3

[1.1.3] - 2025-01-12

Fixed

• Critical Bug Fix: Object keys with reserved symbols now properly escaped during compression
  • Keys starting with @, $, #, &, -, [ are now wrapped in quotes
  • Empty keys "" are now properly escaped
  • Keys matching reserved values (null, true, false) are now escaped
  • Keys that look like numbers are now escaped
  • Fixes issue where "@ason-format/ason" would be interpreted as uniform array marker
  • Ensures perfect round-trip fidelity for all valid JSON object keys

v1.1.2

[1.1.2] - 2024-11-11

Fixed

• TypeScript Declarations: Fixed TypeScript declaration files to only expose public API
  • Reduced .d.ts file size from ~1434 lines to 191 lines (87% reduction)
  • Removed internal/private methods from public type definitions
  • Private methods (prefixed with _) are no longer visible in TypeScript autocomplete
  • Added proper TypeScript interfaces: SmartCompressorOptions and TokenComparisonStats
  • Improved IDE experience for TypeScript users

Changed

• Modified tsup.config.js to use custom TypeScript declarations instead of auto-generation
• Added custom src/index.d.ts with clean, public-only API definitions

v1.1.1

[1.1.1] - 2025-11-11

Changed

• Documentation Improvements - Enhanced README visual presentation
  • Updated npm version badge to use shields.io style (img.shields.io) for consistency
  • Added overview image (preview.png) to nodejs-compressor README
  • Improved visual consistency across all documentation files

v1.1.0

Changed

• Package Rename - Migrated from @ason-format/ason

  • Updated package name to reflect organization scope
  • Updated all documentation and examples with new package name
  • Updated repository URLs to https://github.com/ason-format/ason

• Token Estimation - Removed external dependency for browser compatibility

  • Removed gpt-tokenizer dependency from package
  • Implemented lightweight approximation: Math.ceil(text.length / 4)
  • Reduced package size and improved browser compatibility
  • Updated all documentation to reflect token approximation method

• Documentation Improvements

  • Fixed indentation documentation (valid values: 1, 2, or 4 spaces - not 0)
  • Updated all code examples to use indent: 1 instead of indent: 0
  • Corrected performance metrics throughout docs (up to 23% vs previous claims)
  • Improved clarity in all README files

• Benchmarks Page Redesign

  • Redesigned /docs/benchmarks.html with Vercel/GitHub-style color palette
  • Changed from bright colors to subtle emerald-600/rose-600 scheme
  • Added clean table layout showing Dataset | JSON | ASON | Toon | Winner | Comparisons
  • Improved readability with font-mono for numbers and better visual hierarchy
  • Added proper badges and SVG icons for key findings

Added

• Automated Documentation Build - npm run build now auto-generates docs
  • Added build:docs script that copies dist/index.js to docs/js/ason.js
  • Eliminated code duplication between source and documentation
  • Ensures docs always use latest built version

Fixed

• CI/CD Pipeline - GitHub Actions now properly build before testing

  • Added npm run build step in test.yml workflow
  • Added npm run build step in npm-publish.yml workflow
  • Prevents test failures due to missing dist files

• Browser Compatibility - Fixed web visualizer errors

  • Removed Node.js-specific dependencies from browser builds
  • Fixed "Cannot read properties of undefined" errors
  • Fixed "Failed to resolve module specifier" errors in console

v1.0.0

[1.0.0] - 2025-01-10

Added

• Initial Release - ASON (Aliased Serialization Object Notation)

  • Complete Node.js implementation with SmartCompressor
  • Automatic pattern detection (zero configuration required)
  • Object references for repeated structures (&obj0, &obj1, etc.)
  • Inline-first value dictionary optimized for LLM readability (#0, #1, etc.)
  • Uniform array compression with @keys notation
  • Path flattening for nested single-property objects
  • Configurable indentation (0, 1, or 2 spaces)
  • Lossless compression with perfect round-trip fidelity

• CLI Tool - Command-line interface for JSON ↔ ASON conversion

  • Auto-detection of format from file extension or content
  • --stats flag for visual token count comparison with ASCII table
  • --delimiter option for comma/tab/pipe separators
  • Stdin/stdout support for piping workflows (e.g., cat data.json | npx ason)
  • Comprehensive help documentation with -h/--help
  • Support for --no-references and --no-dictionary flags

• TypeScript Support - Full type definitions included

  • .d.ts files for ESM (48.2 KB)
  • .d.cts files for CommonJS (48.2 KB)
  • Full IntelliSense and autocomplete support in VSCode and other IDEs
  • Proper type exports for both import and require

• Build System - Integrated tsup for optimized bundling

  • Minified ESM build (14.5 KB) - 70% smaller than source
  • Minified CJS build (14.5 KB) for Node.js legacy compatibility
  • Tree-shaking enabled for smaller bundle sizes
  • Dual package exports for seamless ESM/CJS interop

• Testing & Quality

  • Complete test suite with Jest
  • 33 passing tests across 3 test suites
  • Full coverage of compression/decompression cycles

• Documentation & Community

  • Interactive web visualizer with compress/decompress modes at /docs/index.html
  • Complete API documentation at /docs/docs.html
  • Interactive benchmarks at /docs/benchmarks.html
  • Premium README with head-to-head ASON vs Toon comparison
  • ASCII bar charts for visual benchmark comparison
  • Complete API reference with TypeScript examples
  • CLI usage guide with real-world examples
  • Professional badges (npm, TypeScript, License, Node.js)
  • Detailed table of contents with deep anchor links
  • "When to use" guide comparing ASON, Toon, JSON, and CSV
  • Examples in /nodejs-compressor/examples/
  • Community files: CODE_OF_CONDUCT, CONTRIBUTING, SECURITY
  • MIT License

Performance

• Achieves 33.26% token reduction on real-world data (28KB JSON)
• Benchmarks show ASON achieves +4.94% average token reduction
• Beats Toon format by 8 tokens (1,808 vs 1,816)
• Toon shows -6.75% average (worse than JSON in some cases)
• ASON wins on 3 out of 4 tested real-world datasets
• Processes ~35ms for compression + decompression cycle
• Memory efficient: ~10MB for large datasets
• CLI --stats flag provides instant compression metrics

Package Distribution

• Tarball size: 37.8 KB (includes types, CLI, and docs)
• Unpacked size: 148 KB
• Only 7 files published (optimized for production)
• Package ships with built/minified code instead of source files
• package.json configured with proper exports field for ESM/CJS resolution
• Updated files field to include only dist/, src/cli.js, and README.md
• Repository URL format corrected to use git+ prefix (eliminates npm publish warning)