init

Author: siriuslee69

.github/workflows/ci.yml

@@ -0,0 +1,21 @@
+name: ci
+
+on:
+  push:
+  pull_request:
+
+jobs:
+  test:
+    runs-on: windows-latest
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v4
+        with:
+          submodules: recursive
+      - name: Setup Nim
+        uses: nim-lang/setup-nim-action@v2
+        with:
+          nim-version: stable
+      - name: Run tests
+        working-directory: nimAutoWrapper
+        run: nimble test_all

.gitignore

@@ -0,0 +1,10 @@
+nimcache/
+**/nimcache_*/
+testCRepos/builds/
+*.exe
+*.dll
+*.obj
+*.pdb
+nimble.develop
+nimble.paths
+nimbledeps

README.md

@@ -0,0 +1,235 @@
+# nimAutoWrapper
+
+Modular C header wrapper generator for Nim, with small real-world validation harnesses.
+
+This README explains how the parser works, module order, naming rules, and debug output.
+
+-------------------------------------------------------------------------------
+Quick start
+-------------------------------------------------------------------------------
+
+```sh
+nim c -r nimAutoWrapper.nim <input.h> <output.nim>
+```
+
+This writes:
+- `output.nim` (bindings)
+- `output.debug.json` (debug markers and collisions)
+
+-------------------------------------------------------------------------------
+High-level pipeline
+-------------------------------------------------------------------------------
+
+```
+    C header text
+          |
+          v
+      tokenizer
+          |
+          v
+    parser registry (ordered)
+          |
+          v
+     output lines + debug log
+          |
+          v
+  output.nim + output.debug.json
+```
+
+-------------------------------------------------------------------------------
+Module map (what each file does)
+-------------------------------------------------------------------------------
+
+Core:
+- `nimAutoWrapper.nim`: CLI + file I/O; writes debug JSON.
+- `src/tokenizer.nim`: turns C text into flat tokens with line/col.
+- `src/parser_core.nim`: runs parsers in order; logs unparsed tokens.
+- `src/default_parsers.nim`: parser registry order.
+- `src/types.nim`: token types, parser state, debug entry type.
+- `src/utils.nim`: token helpers and output helpers.
+
+Parsers (each handles one C shape):
+- `src/preprocessor_parser.nim`: consumes non-define/include directives.
+- `src/extern_parser.nim`: consumes `extern "C" { ... }` blocks.
+- `src/define_parser.nim`: `#define` to `const` or `template`.
+- `src/static_const_parser.nim`: `static const` variables with simple init.
+- `src/enum_parser.nim`: `enum` to Nim enum.
+- `src/macro_struct_parser.nim`: `MACRO(struct ...)` wrappers.
+- `src/struct_parser.nim`: `struct` to Nim object.
+- `src/typedef_parser.nim`: `typedef` to Nim alias.
+- `src/function_parser.nim`: function prototypes to `proc`.
+
+Naming + debug helpers:
+- `src/name_mangle.nim`: sanitize identifiers, importc pragmas.
+- `src/name_registry.nim`: collision resolution + tracking.
+- `src/debugger.nim`: debug entry collection + JSON writer.
+- `src/cast_utils.nim`: strips leading C casts like `((long)0)` in init/defines.
+
+-------------------------------------------------------------------------------
+Parser order (exact)
+-------------------------------------------------------------------------------
+
+The registry order is important. The default order is:
+
+```
+1) preprocessor_parser  (non-define/include directives)
+2) extern_parser        (extern "C" { ... } blocks)
+3) define_parser        (#define -> const/template)
+4) include_parser       (#include -> comment)
+5) static_const_parser  (static const vars)
+6) enum_parser          (enum -> Nim enum)
+7) macro_struct_parser  (MACRO(struct ...))
+8) struct_parser        (struct -> object)
+9) typedef_parser       (typedef -> distinct pointer)
+10) function_parser     (prototype -> proc)
+```
+
+If no parser matches, `parser_core` consumes one token and logs it as `unparsed`.
+
+-------------------------------------------------------------------------------
+Name mangling and collisions
+-------------------------------------------------------------------------------
+
+We sanitize every emitted name, then reserve a unique Nim identifier.
+
+Sanitization rules (in `src/name_mangle.nim`):
+- Strip leading and trailing underscores: `_my_func_` -> `my_func`
+- If invalid or Nim keyword, prefix with `c_`: `type` -> `c_type`
+- Parameters use `p_` prefix or `p{index}` fallback
+
+Collision rules (in `src/name_registry.nim`):
+1) Try base name (sanitized)
+2) If taken, try kind-specific suffix:
+   - `struct` -> `_str` (ex: `foo_str`)
+   - `typedef` -> `_tyd` (ex: `foo_tyd`)
+3) If still taken, append numeric suffixes (`_1`, `_2`, ...)
+
+All renamed symbols preserve the original C name via `importc`.
+
+Example:
+```
+// C
+struct blake2s_param__ { ... };
+typedef struct blake2s_param__ blake2s_param;
+
+// Nim (collision resolution)
+type
+  blake2s_param_str* {.importc: "blake2s_param__".} = object ...
+  blake2s_param_tyd* {.importc: "blake2s_param".} = distinct pointer
+```
+
+-------------------------------------------------------------------------------
+Debug output (output.debug.json)
+-------------------------------------------------------------------------------
+
+Every wrapper run writes a debug JSON file next to the output.
+
+What is logged:
+- `unparsed`: a token was not handled by any parser
+- `skipped`: a directive was intentionally consumed
+  - `preprocessor`: #if/#endif/#pragma/etc (non-define/include)
+  - `extern_block`: `extern "C" {`
+  - `extern_block_end`: closing brace for the extern block
+- `collision`: a name was renamed to avoid duplicate Nim symbols
+- `static_const_*`: static const values we skipped (missing/complex init)
+
+Example entry:
+```json
+{
+  "kind": "collision",
+  "reason": "name_collision",
+  "line": 0,
+  "col": 0,
+  "text": "blake2s_param",
+  "context": "blake2s_param -> blake2s_param_tyd"
+}
+```
+
+Tip: if you want to audit parser coverage, search for `unparsed` entries.
+
+-------------------------------------------------------------------------------
+Examples (what gets generated)
+-------------------------------------------------------------------------------
+
+Function prototype:
+```
+// C
+int foo(const void* in, size_t n);
+
+// Nim
+proc foo*(p_in: pointer, n: csize_t): cint {.importc.}
+```
+
+Define -> const:
+```
+// C
+#define AES_BLOCKLEN 16
+
+// Nim
+const AES_BLOCKLEN* = 16
+```
+
+Define -> template:
+```
+// C
+#define MAX(a, b) ((a) > (b) ? (a) : (b))
+
+// Nim
+template MAX*(a: untyped, b: untyped): untyped =
+  ## C macro: ((a) > (b) ? (a) : (b))
+  discard
+```
+
+Static const:
+```
+// C
+static const WGPUTextureUsage WGPUTextureUsage_None = 0x0;
+
+// Nim
+const WGPUTextureUsage_None* = 0x0
+```
+
+Macro-wrapped struct:
+```
+// C
+BLAKE2_PACKED(struct blake2s_param__ { ... });
+
+// Nim
+type blake2s_param_str* {.importc: "blake2s_param__".} = object
+  ...
+```
+
+Extern "C":
+```
+// C
+#if defined(__cplusplus)
+extern "C" {
+#endif
+...
+#if defined(__cplusplus)
+}
+#endif
+```
+The extern block is skipped and logged in `output.debug.json`.
+
+-------------------------------------------------------------------------------
+Layout
+-------------------------------------------------------------------------------
+
+- `nimAutoWrapper.nim`: CLI entry point.
+- `src/`: tokenizer, parser registry, parser modules.
+- `tests/functionality/`: unit tests for tokenizer and helpers.
+- `tests/realworld/`: wrapper + validation runners for real C libraries.
+- `testCRepos/repos/`: C repos (submodules).
+- `testCRepos/builds/`: generated wrappers and build artifacts.
+
+-------------------------------------------------------------------------------
+Nimble tasks
+-------------------------------------------------------------------------------
+
+- `nimble build_repos`
+- `nimble test_functionality`
+- `nimble test_realworld`
+- `nimble test_all`
+- `nimble setup`
+- `nimble start`

_tmp_refs/c2nim

@@ -0,0 +1 @@
+Subproject commit 835ba0e49242ce086d8cb7a8d943d92f179de352

config.nims

@@ -0,0 +1,4 @@
+# begin Nimble config (version 2)
+when withDir(thisDir(), system.fileExists("nimble.paths")):
+  include "nimble.paths"
+# end Nimble config

nimAutoWrapper.nim

@@ -0,0 +1,69 @@
+import os
+import strutils
+import src/debugger
+import src/types
+import src/tokenizer
+import src/utils
+import src/parser_core
+import src/default_parsers
+
+proc defaultConfig*(): WrapperConfig =
+  ## returns the default wrapper config
+  ## Enables template and comment emission by default.
+  var
+    cfg: WrapperConfig = WrapperConfig(emitTemplates: true, emitComments: true)
+  result = cfg
+
+proc wrapTextState*(a: string, b: WrapperConfig): ParserState =
+  ## a: input C source text
+  ## b: wrapper config
+  ## Tokenizes and parses input C text into a parser state.
+  ## Example: `wrapTextState("int foo(void);", defaultConfig())` yields output lines.
+  var
+    tokens: seq[Token] = tokenizeC(a)
+    state: ParserState = initState(tokens, b)
+    registry: ParserRegistry = buildDefaultRegistry()
+  parseAll(state, registry)
+  result = state
+
+proc wrapText*(a: string, b: WrapperConfig): seq[string] =
+  ## a: input C source text
+  ## b: wrapper config
+  ## Tokenizes and parses input C text into Nim output lines.
+  ## Example: `wrapText("int foo(void);", defaultConfig())` yields a proc line.
+  var
+    state: ParserState = wrapTextState(a, b)
+  result = state.output
+
+proc wrapFile*(a: string, b: string, c: WrapperConfig) =
+  ## a: input file path
+  ## b: output file path
+  ## c: wrapper config
+  ## Reads a C header file and writes a Nim wrapper file.
+  var
+    inputText: string = readFile(a)
+    state: ParserState = wrapTextState(inputText, c)
+    outputText: string = state.output.join("\n")
+  writeFile(b, outputText)
+  writeDebugJson(b, state.debugEntries)
+
+proc main*() =
+  ## entry point for CLI usage
+  ## Parses CLI args and runs a file-to-file wrapper pass.
+  ## Example: `nimAutoWrapper input.h output.nim`.
+  var
+    args: seq[string] = commandLineParams()
+    inputPath: string = ""
+    outputPath: string = ""
+    config: WrapperConfig
+  if args.len < 2:
+    echo "usage: nimAutoWrapper <input.h> <output.nim>"
+    return
+  inputPath = args[0]
+  outputPath = args[1]
+  config = defaultConfig()
+  wrapFile(inputPath, outputPath, config)
+  echo "wrote: " & outputPath
+
+when isMainModule:
+  main()

nimAutoWrapper.nimble

@@ -0,0 +1,43 @@
+version       = "0.1.0"
+author        = "n1ght"
+description   = "Modular C header wrapper generator for Nim."
+license       = "UNLICENSED"
+
+task build_aes, "Generate wrapper for tiny-AES-c":
+  exec "nim r tools/ensure_env.nim -- --submodules --builddirs"
+  exec "nim c -r nimAutoWrapper.nim testCRepos/repos/tiny-AES-c/aes.h testCRepos/builds/tiny-AES-c/aes_wrapper.nim"
+
+task build_blake2, "Generate wrapper for BLAKE2 reference code":
+  exec "nim r tools/ensure_env.nim -- --submodules --builddirs"
+  exec "nim c -r nimAutoWrapper.nim testCRepos/repos/BLAKE2/ref/blake2.h testCRepos/builds/BLAKE2/blake2_wrapper.nim"
+
+task build_repos, "Generate wrappers for all test repos":
+  exec "nim r tools/ensure_env.nim -- --submodules --builddirs"
+  exec "nim c -r nimAutoWrapper.nim testCRepos/repos/tiny-AES-c/aes.h testCRepos/builds/tiny-AES-c/aes_wrapper.nim"
+  exec "nim c -r nimAutoWrapper.nim testCRepos/repos/BLAKE2/ref/blake2.h testCRepos/builds/BLAKE2/blake2_wrapper.nim"
+
+task setup, "Fetch submodules for test repos":
+  exec "nim r tools/ensure_env.nim -- --submodules"
+
+task start, "Fetch submodules and build test wrappers":
+  exec "nim r tools/ensure_env.nim -- --submodules --builddirs"
+  exec "nimble build_repos"
+
+task test_functionality, "Run tokenizer and utils tests":
+  exec "nim r tools/ensure_env.nim -- --submodules --builddirs"
+  exec "nim c -r --path:. tests/functionality/test_tokenizer.nim"
+  exec "nim c -r --path:. tests/functionality/test_utils.nim"
+
+task test_realworld, "Run real-world wrapper tests":
+  exec "nim r tools/ensure_env.nim -- --submodules --builddirs"
+  exec "nimble build_repos"
+  exec "nim c -r tests/realworld/tiny_aes_c_runner.nim"
+  exec "nim c -r tests/realworld/blake2_ref_runner.nim"
+
+task test_all, "Run all tests":
+  exec "nim r tools/ensure_env.nim -- --submodules --builddirs"
+  exec "nimble build_repos"
+  exec "nim c -r --path:. tests/functionality/test_tokenizer.nim"
+  exec "nim c -r --path:. tests/functionality/test_utils.nim"
+  exec "nim c -r tests/realworld/tiny_aes_c_runner.nim"
+  exec "nim c -r tests/realworld/blake2_ref_runner.nim"