feat(baml_language): Implement BAML V2 lexer with complete operator support (#2686)

Implemented a lossless, incremental lexer for BAML V2 that produces
structural
tokens only, leaving keyword detection to the parser.

Changes:
- Replace keyword tokens (Function, Class, Enum, etc.) with single Word
token
- Add raw string support with multiple delimiter levels (#"..."#,
##"..."##, etc.)
- Add comprehensive operator support:
  * Arithmetic: +, -, *, /, %, ++, --
  * Assignment: =, +=, -=, *=, /=, %=, &=, |=, ^=, <<=, >>=
  * Comparison: ==, !=, <, >, <=, >=
  * Logical: &&, ||, !
  * Bitwise: &, |, ^, ~, <<, >>
  * Other: ::, ;, .
- Add documentation for unquoted string handling (parser responsibility)
- Update SyntaxKind enum with all new token types
- Update parser to map all new tokens correctly
- Add comprehensive test coverage (20 lexer tests)

Design decisions:
- Lexer is "dumb" - only identifies token boundaries, not semantic
meaning
- Parser checks Word token text to determine if it's a keyword
- Raw strings are opaque - Jinja/template parsing happens in parser
layer
- Lossless tokenization preserved for perfect source reconstruction
- Operator ordering ensures longer tokens match first (e.g., <<= before
<< before <)

All tests passing:
- 20 lexer unit tests
- 20 integration tests
- Updated snapshots to reflect new token structure

<!-- CURSOR_SUMMARY -->
---

> [!NOTE]
> Introduces a lossless structural lexer with raw-string and
comprehensive operator support, updates parser/syntax mappings,
refreshes tests/snapshots, and wires project/workspace APIs.
> 
> - **Lexer (baml_lexer)**:
> - Replace keyword tokens with `Word`; add `Quote`/`Hash` and
raw-string handling strategy.
> - Add comprehensive operators/punctuation (assignment, comparison,
logical, bitwise, shift, arithmetic) and preserve whitespace/comments.
>   - Document unquoted/raw string behavior; add extensive unit tests.
> - **Parser & Syntax (baml_parser, baml_syntax)**:
>   - Map all new `TokenKind` variants to `SyntaxKind`.
> - Extend `SyntaxKind` with `WORD`, `QUOTE`, `HASH`, new operators, and
punctuation.
> - **DB/Workspace (baml_db, baml_workspace)**:
> - `RootDatabase::add_file` now creates `SourceFile` with `FileId`; add
`set_project_root`; re-export workspace.
>   - Add file discovery (`discover_baml_files`) with tests.
> - **Tests/Infra (baml_tests)**:
>   - Update generated tests and snapshots to new tokenization.
> - Add benchmark generation scaffolding for incremental/scale benches.
> 
> <sup>Written by [Cursor
Bugbot](https://cursor.com/dashboard?tab=bugbot) for commit
64982ce. This will update automatically
on new commits. Configure
[here](https://cursor.com/dashboard?tab=bugbot).</sup>
<!-- /CURSOR_SUMMARY -->

Author: hellovai

baml_language/crates/baml_base/src/core_types.rs

@@ -1,7 +1,8 @@
 //! Core types used throughout the compiler.
 
-use smol_str::SmolStr;
 use std::fmt;
+
+use smol_str::SmolStr;
 use text_size::{TextRange, TextSize};
 
 /// Unique identifier for a source file

baml_language/crates/baml_base/src/files.rs

@@ -2,9 +2,10 @@
 //!
 //! Defines the core structures for accessing file contents and paths.
 
-use crate::FileId;
 use std::path::PathBuf;
 
+use crate::FileId;
+
 /// Input structure representing a source file in the compilation.
 ///
 /// This is a salsa input, which means it's the primary way to provide

baml_language/crates/baml_db/src/lib.rs

@@ -3,9 +3,10 @@
 //! This crate purely combines all the compiler traits into a single database.
 //! All testing happens in the separate `baml_tests` crate.
 
-use std::path::PathBuf;
-use std::sync::Arc;
-use std::sync::atomic::AtomicU32;
+use std::{
+    path::PathBuf,
+    sync::{Arc, atomic::AtomicU32},
+};
 
 // Re-export all public APIs
 pub use baml_base::*;

baml_language/crates/baml_diagnostics/src/lib.rs

@@ -3,9 +3,10 @@
 //! This crate converts compiler diagnostics into beautiful error messages.
 //! It doesn't define error types - those live in each compiler phase.
 
+use std::collections::HashMap;
+
 use ariadne::{Color, Label, Report, ReportKind, Source};
 use baml_base::{Diagnostic, FileId, Severity, Span};
-use std::collections::HashMap;
 
 /// Convert a compiler diagnostic to Ariadne span format.
 fn span_to_ariadne(span: Span) -> (usize, std::ops::Range<usize>) {
@@ -72,10 +73,11 @@ pub fn render_diagnostics(
 
 #[cfg(test)]
 mod tests {
-    use super::*;
     use baml_base::{FileId, Span};
     use text_size::{TextRange, TextSize};
 
+    use super::*;
+
     // A simple test diagnostic
     #[derive(Debug)]
     struct TestError {