tools/publish-crates.sh is more flexible for crate paths (#3185)

# Description of Changes

I updated `tools/publish-crates.sh` and `tools/find-publish-list.py` to
be more flexible to where our crates can be located (rather than
hardcoding `crates/foo`). This is in preparation for
#3181.

Now, `find-publish-list.py` loads the output of `cargo metadata` to
dynamically find the path of the crate in question. This also allows us
to "properly" determine which crates are ours, instead of the
`spacetimedb-` string prefix check that we were doing before.

It also now supports `--directories` to output directory paths, rather
than just crate names. `publish-crates.sh` now uses those output paths
rather than assuming that it knows where to find crates.

As a bonus, this approach is also much faster (~0.3s to find the crate
list, vs ~8 before to load and process all the tomls).

# API and ABI breaking changes

None

# Expected complexity level and risk

2

# Testing

- [x] `find-publish-list.py` lists the same crates before and after:
```bash
$ ( git checkout master && python3 tools/find-publish-list.py --recursive --quiet bindings sdk cli standalone > before.txt )

$ ( git checkout bfops/flexible-publish-scripts && python3 tools/find-publish-list.py --recursive --quiet spacetimedb spacetimedb-sdk spacetimedb-cli spacetimedb-standalone > after.txt )

# the new script prints out crate names rather than directory names, so we need to tweak a bit
$ diff -U2 before.txt <(cat after.txt | sed 's/^spacetimedb-//' | sed 's/^spacetimedb$/bindings/')
--- before.txt	2025-08-20 10:18:07.323217870 -0700
+++ /dev/fd/63	2025-08-20 10:35:38.344074842 -0700
@@ -8,17 +8,17 @@
 data-structures
 schema
-table
-expr
-physical-plan
 paths
 fs-utils
 commitlog
+table
 durability
-execution
+expr
+physical-plan
 snapshot
+execution
 client-api-messages
 query
-subscription
 vm
+subscription
 datastore
 auth
```
Some lines are reordered because we find dependencies via metadata
instead of toml now - I spot-checked some of the moved lines to see
whether they were in an invalid place, and I did not find any evidence
of that.

- [x] `--directories` flag prints correct directories instead of names
```bash
$ python3 tools/find-publish-list.py --directories --quiet --recursive spacetimedb-sdk
/home/lead/work/clockwork-localhd/SpacetimeDBPrivate/public/crates/memory-usage/Cargo.toml
/home/lead/work/clockwork-localhd/SpacetimeDBPrivate/public/crates/primitives/Cargo.toml
/home/lead/work/clockwork-localhd/SpacetimeDBPrivate/public/crates/metrics/Cargo.toml
/home/lead/work/clockwork-localhd/SpacetimeDBPrivate/public/crates/bindings-macro/Cargo.toml
/home/lead/work/clockwork-localhd/SpacetimeDBPrivate/public/crates/sats/Cargo.toml
/home/lead/work/clockwork-localhd/SpacetimeDBPrivate/public/crates/lib/Cargo.toml
/home/lead/work/clockwork-localhd/SpacetimeDBPrivate/public/crates/client-api-messages/Cargo.toml
/home/lead/work/clockwork-localhd/SpacetimeDBPrivate/public/crates/data-structures/Cargo.toml
/home/lead/work/clockwork-localhd/SpacetimeDBPrivate/public/crates/sdk/Cargo.toml
```

---------

Co-authored-by: Zeke Foppa <[email protected]>

Author: bfops

tools/find-publish-list.py

@@ -1,29 +1,35 @@
 import argparse
-import toml
+import json
+import subprocess
 import sys
 from pathlib import Path
-
-def find_spacetimedb_dependencies(cargo_toml_path):
-    with open(cargo_toml_path, 'r') as file:
-        cargo_data = toml.load(file)
-
-    deps = cargo_data.get('dependencies', {})
-    return [dep for dep in deps if dep.startswith("spacetimedb-")]
-
-def dep_to_crate_dir(dep_name):
-    return dep_name.replace("spacetimedb-", "", 1)
-
-def process_crate(crate_name, crates_dir, recursive=False, debug=False):
-    cargo_toml_path = crates_dir / crate_name / "Cargo.toml"
-
-    if not cargo_toml_path.is_file():
-        print(f"Warning: Cargo.toml not found for crate '{crate_name}' at {cargo_toml_path}", file=sys.stderr)
-        return []
-
+from typing import Dict
+
+def get_all_crate_metadata() -> Dict[str, dict]:
+    result = subprocess.run(
+        ['cargo', 'metadata', '--format-version=1', '--no-deps'],
+        capture_output=True,
+        text=True,
+        check=True
+    )
+    metadata = json.loads(result.stdout)
+    return {pkg['name']: pkg for pkg in metadata.get('packages', [])}
+
+def find_spacetimedb_dependencies(crate_metadata, crate):
+    deps = crate_metadata[crate].get('dependencies', [])
+    # filter out dev-dependencies. otherwise, we get dependency cycles.
+    deps = [ dep for dep in deps if dep['kind'] != 'dev' ]
+    dep_names = [ dep['name'] for dep in deps ]
+    # We use --no-deps to generate the metadata, so a dep will be in crate_metadata only if it is
+    # one we create in this workspace.
+    dep_names = [ dep for dep in dep_names if dep in crate_metadata ]
+    return dep_names
+
+def process_crate(crate_name, crate_metadata, recursive=False, debug=False):
     if debug:
         print(f"\nChecking crate '{crate_name}'...")
 
-    deps = find_spacetimedb_dependencies(cargo_toml_path)
+    deps = find_spacetimedb_dependencies(crate_metadata, crate_name)
 
     if debug:
         if deps:
@@ -36,8 +42,7 @@ def process_crate(crate_name, crates_dir, recursive=False, debug=False):
 
     if recursive:
         for dep_name in deps:
-            sub_crate = dep_to_crate_dir(dep_name)
-            sub_deps = process_crate(sub_crate, crates_dir, recursive=True, debug=debug)
+            sub_deps = process_crate(dep_name, crate_metadata, recursive=True, debug=debug)
             all_deps.extend(sub_deps)
 
     return all_deps
@@ -56,14 +61,18 @@ def ordered_dedup(items):
     parser.add_argument("root", nargs="+", help="One or more crate names to start with")
     parser.add_argument("--recursive", action="store_true", help="Recursively resolve dependencies")
     parser.add_argument("--quiet", action="store_true", help="Only print the final output")
+    parser.add_argument("--directories", action="store_true", help="Print crate paths instead of names")
     args = parser.parse_args()
 
-    crates_dir = Path("crates")
+    if not args.quiet:
+        print("Loading crate metadata...", file=sys.stderr)
+    crate_metadata = get_all_crate_metadata()
+
     all_crates = list(args.root)
 
     for crate in args.root:
-        deps = process_crate(crate, crates_dir, recursive=args.recursive, debug=not args.quiet)
-        all_crates.extend(dep_to_crate_dir(dep) for dep in deps)
+        deps = process_crate(crate, crate_metadata, recursive=args.recursive, debug=not args.quiet)
+        all_crates.extend(deps)
 
     # It takes a bit of reasoning to conclude that this is, in fact, going to be a legitimate
     # dependency-order of all of these crates. Because of how the list is constructed, once it's reversed,
@@ -76,4 +85,7 @@ def ordered_dedup(items):
     if not args.quiet:
         print("\nAll crates to publish, in order:")
     for crate in publish_order:
-        print(crate)
+        if args.directories:
+            print(crate_metadata[crate]['manifest_path'])
+        else:
+            print(crate)

tools/publish-crates.sh

@@ -39,34 +39,34 @@ if [ $DRY_RUN -ne 1 ]; then
 fi
 
 BASEDIR=$(pwd)
-declare -a ROOTS=(bindings sdk)
-declare -a CRATES=($(python3 tools/find-publish-list.py --recursive --quiet "${ROOTS[@]}"))
+declare -a ROOTS=(spacetimedb spacetimedb-sdk)
+declare -a CRATES=($(python3 tools/find-publish-list.py --recursive --quiet --directories "${ROOTS[@]}"))
 
 echo Crates to publish: "${CRATES[@]}"
 echo
 
-for crate in "${CRATES[@]}"; do
-    if [ ! -d "${BASEDIR}/crates/${crate}" ]; then
-        echo "This crate does not exist: ${crate}"
+for path in "${CRATES[@]}"; do
+    if [ ! -d "${path}" ]; then
+        echo "This crate does not exist: ${path}"
         exit 1
     fi
 done
 
 i=0
-for crate in "${CRATES[@]}"; do
+for path in "${CRATES[@]}"; do
     i=$(($i+1))
-    cd "${BASEDIR}/crates/${crate}"
+    cd "${path}"
 
     PUBLISH_CMD="cargo publish"
     [[ $DRY_RUN -eq 1 ]] && PUBLISH_CMD+=" --dry-run"
     [[ $ALLOW_DIRTY -eq 1 ]] && PUBLISH_CMD+=" --allow-dirty"
 
-    echo "[$i/${#CRATES[@]}] Publishing crate: $crate with command: $PUBLISH_CMD"
+    echo "[$i/${#CRATES[@]}] Publishing crate at $path with command: $PUBLISH_CMD"
     if ! OUTPUT=$($PUBLISH_CMD 2>&1); then
         if [ $SKIP_ALREADY_PUBLISHED -eq 1 ] && echo "$OUTPUT" | grep -q "already exists"; then
-            echo "WARNING: Crate $crate version is already published. Skipping..."
+            echo "WARNING: Crate at $path is already published at this version. Skipping..."
         else
-            echo "ERROR: Failed to publish $crate. Check logs:"
+            echo "ERROR: Failed to publish crate at $path. Check logs:"
             echo "$OUTPUT"
             exit 1
         fi