manifest: Tezos-inspired manifest system for dune generation#18425
Draft
dannywillems wants to merge 28 commits intocompatiblefrom
Draft
manifest: Tezos-inspired manifest system for dune generation#18425dannywillems wants to merge 28 commits intocompatiblefrom
dannywillems wants to merge 28 commits intocompatiblefrom
Conversation
Add a manifest system in manifest/ that centralizes build metadata in OCaml and generates dune files programmatically. Includes: - Dune s-expression parser, pretty-printer, and structural comparator - DSL for declaring libraries and executables with PPX presets - Check mode (--check) for CI verification - Opam file generation support - 22 Alcotest tests covering parsing, comparison, and DSL - 10 proof-of-concept libraries + 3 executables declared The manifest only depends on str and unix from OCaml stdlib. Run with: dune exec manifest/main.exe
Wave 0: Add manifest features needed for full migration: - private_library/private_executable for targets without public_name - kind field (ppx_deriver) - ppx_runtime_libraries - flags for libraries - link_flags for executables - enabled_if for libraries/executables - js_of_ocaml field - file_stanzas for top-level stanzas (vendored_dirs, include_subdirs) - Fix structural comparison for atom vs single-element list Wave 1: Migrate 36 leaf libraries + PPX ecosystem: allocation_functor, codable, comptime, error_json, integers_stubs_js, key_value_database, linked_tree, logproc_lib, interpolator_lib, mina_wire_types, one_or_two, otp_lib, participating_state, perf_histograms, proof_cache_tag, rosetta_coding, rosetta_models, sgn_type, structured_log_events, sync_status, unsigned_extended, visualization, webkit_trace_event, webkit_trace_event.binary, graphql_basic_scalars, graphql_wrapper, mina_compile_config, ppx_annot, ppx_register_event, ppx_version, ppx_version.runtime, ppx_mina, ppx_to_enum, ppx_representatives, ppx_representatives.runtime, storage + 7 ppx_mina test libraries
Migrate 36 libraries from Wave 2: - mina_stdlib, blake2, bignum_bigint, string_sign - snark_keys_header, plonkish_prelude, random_oracle_input - outside_hash_image, hash_prefixes, sgn, timeout_lib - quickcheck_lib, test_util, disk_cache.intf - run_in_thread (virtual + native/fake), interruptible - promise (virtual + native/js/js_helpers) - internal_tracing + context_call - mina_metrics (virtual + none/prometheus) - cache_dir (virtual + native/fake) - node_config (intf + main + for_unit_tests + profiled + unconfigurable_constants + version) Also fix flags type from string list to Dune_s_expr.t list to support nested s-expression flags like (:standard -w a -warn-error +a). 87 dune files checked, 0 differences. 22 tests pass.
Migrate 22 libraries from Wave 3: - logger (virtual + context_logger/fake/file_system/native) - o1trace, o1trace_webkit_event - mina_stdlib_unix, mina_numbers - cache_lib, trust_system, parallel - mina_version.dummy, mina_version.runtime - mina_signature_kind (virtual + type/config/testnet/mainnet) - multi_key_file_storage 104 dune files checked, 0 differences. 22 tests pass.
Add test_integration.exe that registers all product_mina declarations and verifies each generated dune file is structurally identical to the original using s-expr comparison. - Add Manifest.check() returning structured results per file - Add Manifest.reset() to clear registered targets - Extract target_path before check for proper ordering - Split product_mina into manifest_product library so it can be shared between main.exe and test_integration.exe
Adds declarations for the crypto layer: kimchi bindings (js/node/web backends, pasta poseidon), kimchi backend (common, pasta, basic, constraint system, gadgets), crypto params, pickles (base, types, composition types, plonk checks, pseudo, limb vector, backend, main), snark params/bits, random oracle (+ permutation virtual/impls), non_zero_curve_point, signature_lib, secrets, key_gen, bowe_gabizon_hash. 143 dune files pass structural comparison, 0 differences.
Adds test stanza support and forbidden_libraries to the manifest DSL. Migrates: hash_prefix_states (+ create virtual/native/js), data_hash_lib, sparse_ledger_lib, block_time, proof_carrying_data, protocol_version, genesis_constants, network_peer, node_addrs_and_ports, user_command_input, fields_derivers (+ json, graphql, zkapps), rocksdb, key_cache (+ sync, async, native), parallel_scan, merkle_address, merkle_list_prover/verifier, lmdb_storage, disk_cache (virtual + filesystem, identity, lmdb, test_lib, test, utils), dummy_values (+ gen_values exe), mina_base.test_helpers. 180 dune files pass structural comparison, 0 differences.
Adds test stanza support (with package), inline_tests_deps, inline_tests_bare, forbidden_libraries for executables, and name-implied-by-public_name equivalence in the comparator. Migrates: staged_ledger_diff, mina_transaction, mina_transaction_logic (+tests), transaction_witness, merkle_ledger (+tests), merkle_mask, mina_ledger (+test_helpers), transaction_protocol_state, transaction_snark (+20 test subdirs), transaction_snark_work, transaction_snark_scan_state, zkapp_command_builder, zkapp_vk_cache_tag. 218 dune files pass structural comparison, 0 differences.
Add declarations for blockchain_snark, mina_state, mina_block, consensus (with modules_exclude for shared-dir executables), staged_ledger, snark_work_lib, snark_worker, precomputed_values, genesis_ledger, genesis_proof, genesis_ledger_helper, vrf_lib, vrf_evaluator, snark_profiler_lib, rosetta_lib, generated_graphql_queries, ledger_proof, prover, verifier, mina_incremental, and mina_plugins. New manifest features: - preprocessor_deps for executables - modules_exclude for (:standard \ ...) syntax - file_deps for test stanzas (file dependencies) 247 files pass structural comparison, 0 differences.
Split the monolithic product_mina.ml into domain-based layer files (ppx, base, concurrency, test, tooling, node, snarky, crypto, storage, infra, domain, ledger, transaction, protocol, network) and individual product files (one per src/app directory). Add manifest DSL features: bisect_sigterm, modes, enabled_if for tests, and symlink .ocamlformat for consistent formatting. 324 dune files managed, 0 structural differences, 22 tests passing.
Auto-generated dune files from the manifest system. Changes are formatting-only (vertical layout, sorted deps, header comment). No semantic changes to build targets.
Member
|
@dannywillems what's the benefit of using this approach vs. plain dune files? To me they seem more or less the same thing, with this approach being less familiar to an Ocaml developer. So there would be some benefits in doing the transition, to justify the work? |
Organize manifest files into a cleaner structure: - manifest/layers/ — 15 domain layer declarations - manifest/products/ — 27 product (executable) declarations - manifest/test/ — unit and integration tests Uses (include_subdirs unqualified) so dune finds modules in subdirs.
Define each opam dependency once in externals.ml (114 bindings) and reference them by name in all layer/product files, replacing 1,977 raw opam string calls with typed values. Gives compile-time typo detection and single point of truth for external dep names.
Add manifest-build, manifest-generate, manifest-check, and manifest-test Makefile targets. Add GitHub Actions workflow that runs on PRs touching manifest/ or src/**/dune to verify generated dune files stay in sync with manifest declarations.
Member
|
Have you talked with someone from the ocaml team? If so could you provide a thead so I can look at?
|
Create layer_consensus, layer_snark_worker, layer_service, and layer_base_types to break circular module dependencies between the 15 existing layers. Move ~30 library definitions to their proper layers: - consensus/state/vrf libs from network to layer_consensus - snark work/scan state libs from network+transaction to layer_snark_worker - verifier/prover from network to layer_service Replace backward cross-layer references with local "xxx" calls to break OCaml compilation cycles while preserving identical dep values in generated dune files. Fix forward references caused by converting from function-body style (let register () = ...) to top-level let bindings. Remove dead register() dispatch pattern from product_mina.ml. Verified: builds cleanly, --check shows 0 differences, 22/22 tests pass.
Move proof_cache_tag from layer_base to layer_crypto where it belongs since it depends on pickles. Remove 361 decorative section comments (tier headers, dashed separators) across all manifest files.
Move libraries out of layer_infra, layer_base, and layer_domain into the layers that match their domain: - pipe_lib, parallel → layer_concurrency - network_peer, trust_system → layer_network - mina_stdlib_unix, mina_numbers, mina_version_dummy, mina_version_runtime → layer_base - cache_lib, multi_key_file_storage → layer_storage - mina_signature_kind (+ type, config, testnet, mainnet) → layer_protocol
The infra layer now only contains logging-related libraries (logger, logger_file_system, logger_context_logger, logger_fake, o1trace, o1trace_webkit_event). Rename to reflect its actual scope.
Extract kimchi/plonk bindings and backends into layer_kimchi (17 libs) and pickles proof composition into layer_pickles (10 libs). Also move snarky_taylor into layer_snarky. layer_crypto retains general crypto: blake2, bignum_bigint, snark_params, random_oracle, signature_lib, secrets, key_gen, and related utilities. Compilation order: layer_kimchi -> layer_crypto -> layer_pickles.
Split the single manifest_product library into tiered libraries: manifest_foundation -> layer_kimchi -> layer_crypto -> layer_pickles -> manifest_upper -> manifest_products. The compiler now rejects cross-layer violations, e.g. layer_crypto cannot reference Layer_pickles, and layer_kimchi cannot reference Layer_crypto.
Reorder definitions in layer_ppx.ml so that ppx_version_runtime, ppx_to_enum, ppx_representatives, and ppx_representatives_runtime are defined before their use, replacing local "..." with direct OCaml value references.
Member
|
I would prefer to wait on this until after the hard fork, and to discuss why it's necessary to do this. I'd also rather keep the ocaml build system fairly vanilla, unless we can't avoid extending it for whatever reason. |
Add a third dep_kind (Submodule) to distinguish git submodule dependencies from both opam and local manifest-managed libraries. Define all 12 snarky submodule libraries as typed constants in a Snarky_lib module and replace 148 local "..." string references across 16 layer/product files.
Move externals into manifest_lib and give layer_snarky its own dune library depending only on manifest_lib, with no dependency on any other layer.
Add snarky_blake2, snarky_curves, snarky_field_extensions, snarky_group_map, snarky_log, and snark_keys_header to layer_snarky. Move user_command_input from layer_domain to layer_transaction.
Move libraries to their semantically correct layers: - structured_log_events: layer_base -> layer_logging - perf_histograms: layer_base -> layer_tooling - child_processes: layer_base -> layer_concurrency - rosetta_coding, rosetta_models: layer_base -> new layer_rosetta - rosetta_lib: layer_network -> new layer_rosetta Create layer_rosetta for Rosetta API support code. Update cross-layer references across 12 files.
Organize all ~deps: lists in 21 layer files and 23 product files into consistent groups: opam deps first, cross-layer refs second, local refs third — each group sorted alphabetically.
dannywillems
commented
Feb 9, 2026
| (** Dune S-expression AST, pretty-printing, parsing, | ||
| and structural comparison. *) | ||
|
|
||
| type t = Atom of string | List of t list | Comment of string |
Member
Author
There was a problem hiding this comment.
There is already a sexpr library. Claude doesn't know about it...
| @@ -0,0 +1,9 @@ | |||
| (** Mina base types layer: core types that sit above crypto primitives. | |||
Member
Author
|
OPAM files generation to support. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
manifest/) that centrally declares all library and executable targets, then generates dune files from those declarationssrc/appdirectory)Test plan
dune build manifest/main.exesucceedsdune exec manifest/main.exe -- --checkshows 0 differencesdune exec manifest/test/test_manifest.exepasses 22 testsdune buildsucceeds (no build regressions)