One-to-many asset processing #21889

andriyDev · 2025-11-20T03:05:42Z

Objective

Allow processors to produce multiple files instead of just one.

Solution

Instead of passing a &mut Writer to Process, instead we pass a WriterContext. Users then just call write_single or write_multiple to decide whether they want a single file or multiple files. Once users are done writing, they call SingleWriter::finish or MultipleWriter::finish.
I also added a new asset action - Decomposed. This makes it more obvious to callers that they are querying for the wrong asset.

This means if you have an asset path like path/to/my_asset.json, and the Process produces files "part1.thing", "part2.thing", and "dir/mesh.gltf", these assets will be accessible at path/to/my_asset.json/part1.thing, path/to/my_asset.json/part2.thing, and path/to/my_asset.json/dir/mesh.gltf. In essence, the asset in the unprocessed directory becomes a folder in the processed directory.

Testing

The asset_processing example works and has been updated to include a "multiple" processor.
Updated unit tests to account for "multiple" files.
Created new tests too.

JMS55

Very exciting! Left some feedback

I really don't like the single vs multi naming. I think we need better naming.
As someone not very familiar with the asset system, I don't quite get how the multi files get mapped to asset server concepts. Sub assets? Or?
Do we support asset savers besides filesystems? How does this work with those?

JMS55 · 2025-11-20T16:35:40Z

release-content/release-notes/one_to_many_processing.md

+        }
+        for (i, line) in bytes.lines().map(Result::unwrap).enumerate() {
+            let mut writer = writer_context
+                .write_multiple(Path::new(&format!("Line{i}.line")))


write_multiple() in the loop, for writing a single item, feels kind of weird to me. Can we call it something else?

write_part, write_subasset, write_new_asset, ?

Fair point! I tried renaming write_single and write_multiple to write_full and write_partial respectively. The naming here really sucks. We shouldn't use subasset because that's already taken and it has a completely different meaning here.

I want to make sure that the "full" vs "partial" are like antonyms (like single vs multiple). Just to give the intuition that these can't be mixed.

JMS55 · 2025-11-20T16:36:37Z

release-content/release-notes/one_to_many_processing.md

+}
+```
+
+Then if you have an asset like `shakespeare.txt`, you can load these separate files as


Can we show some code for that?

How are these represented in terms of bevy's asset system? Are they subassets? I don't remember how things like meshes within gltf files are represented and how to load subparts of it and the different asset_server.load() syntax, and how that all maps together.

I gave one example.

They are effectively completely separate assets. The Process trait has no idea about subassets. Writing "multiple"/"partial" files just writes a new file that you can point the asset server at.

JMS55 · 2025-11-20T16:36:52Z

release-content/release-notes/one_to_many_processing.md

+from scratch within the asset processor!
+
+We plan to use this to break apart large glTF files into smaller, easier-to-load pieces -
+particularly for producing meshlets.


Suggested change

particularly for producing meshlets.

particularly for producing virtual geometry meshes.

examples/asset/processing/assets/multi.lines

JMS55 · 2025-11-20T16:38:44Z

crates/bevy_asset/src/meta.rs

    Process {
        processor: String,
        settings: ProcessSettings,
    },


This is essentially saying "once an asset is processed into multiple new assets, you can't load the original asset anymore"? Can we state it more like that if so?

That's a much more obvious way to put it! I slightly reworded it, but overall the same.

JMS55 · 2025-11-20T16:40:36Z

crates/bevy_asset/src/processor/process.rs

+
+    /// Start writing a single output file, which can be loaded with the `load_settings`.
+    ///
+    /// Returns an error if you have previously started writing.


started writing <with either of the two apis>

I actually changed this to specifically mention "partial" writes - since it's now impossible to call single twice (since it takes an owned self).

andriyDev

To respond to your questions:

Good point! Discussed in a comment.
Multi-files are just treated as individual assets. There is no requirement that subassets map to some multi-file. If the process wants to throw out the bytes of the original asset and write 100 copies of the Bee Movie script, that's totally up to it!
The Process trait has no inherent relationship to AssetSaver, both before and after this PR. The AssetSaver is used by the particular Process implementation LoadTransformAndSave, which makes it slightly more convenient to write Process implementations. We could figure out a similar convenience for one-to-many, but I think we should leave that for another PR, since that seems a little tricky to get right, and I don't think we need it for a glTF processor.

andriyDev · 2025-11-21T03:29:38Z

crates/bevy_asset/src/meta.rs

    Process {
        processor: String,
        settings: ProcessSettings,
    },


That's a much more obvious way to put it! I slightly reworded it, but overall the same.

andriyDev · 2025-11-21T03:39:39Z

release-content/release-notes/one_to_many_processing.md

+}
+```
+
+Then if you have an asset like `shakespeare.txt`, you can load these separate files as


I gave one example.

They are effectively completely separate assets. The Process trait has no idea about subassets. Writing "multiple"/"partial" files just writes a new file that you can point the asset server at.

andriyDev · 2025-11-21T03:46:14Z

crates/bevy_asset/src/processor/process.rs

+
+    /// Start writing a single output file, which can be loaded with the `load_settings`.
+    ///
+    /// Returns an error if you have previously started writing.


I actually changed this to specifically mention "partial" writes - since it's now impossible to call single twice (since it takes an owned self).

andriyDev · 2025-11-21T03:47:07Z

release-content/release-notes/one_to_many_processing.md

+from scratch within the asset processor!
+
+We plan to use this to break apart large glTF files into smaller, easier-to-load pieces -
+particularly for producing meshlets.


andriyDev · 2025-11-21T03:50:28Z

release-content/release-notes/one_to_many_processing.md

+        }
+        for (i, line) in bytes.lines().map(Result::unwrap).enumerate() {
+            let mut writer = writer_context
+                .write_multiple(Path::new(&format!("Line{i}.line")))


Fair point! I tried renaming write_single and write_multiple to write_full and write_partial respectively. The naming here really sucks. We shouldn't use subasset because that's already taken and it has a completely different meaning here.

I want to make sure that the "full" vs "partial" are like antonyms (like single vs multiple). Just to give the intuition that these can't be mixed.

…cessing doesn't rerun if nothing has changed.

…i" asset.

…nd "ghost meta files".

…ssets.

… an asset to prevent failing to process.

…d-assets.

…y asset processing.

…d processors.

…l respectively.

andriyDev added this to the 0.18 milestone Nov 20, 2025

andriyDev requested a review from JMS55 November 20, 2025 03:06

andriyDev force-pushed the one-to-many branch 3 times, most recently from 08313c5 to 77091f1 Compare November 20, 2025 03:51

JMS55 suggested changes Nov 20, 2025

View reviewed changes

andriyDev commented Nov 21, 2025

View reviewed changes

andriyDev requested a review from JMS55 November 21, 2025 04:00

andriyDev added 16 commits November 21, 2025 00:01

Create a test to ensure that we actually gate on file transactions.

3621295

Implement one-to-many asset processing.

4ded24c

Add a test for one-to-many asset processing.

0abe60a

Update the "wrong_hash" test to also check that one-to-many asset pro…

f65170f

…cessing doesn't rerun if nothing has changed.

Update the gates_asset_path_on_process test to also include a "mult…

db67b4b

…i" asset.

Use let-else for initializing the asset processor.

c4dbf69

Delete the meta file of empty folders to make sure we don't keep arou…

77a66f8

…nd "ghost meta files".

Treat any directory with a meta file as an asset path for processed a…

c6e5a92

…ssets.

Delete processed assets (or multi-processed-assets) before processing…

809ecfc

… an asset to prevent failing to process.

Recursively look for processor asset infos to block on multi-processe…

7694049

…d-assets.

Create a migration guide for Process trait changes.

60fc4bb

Create release notes for one-to-many asset processing.

2c4be0f

Update the asset processing example to show that we can do one-to-man…

8181a26

…y asset processing.

Add some checks to verify the behavior of some error cases for invali…

c50d0f9

…d processors.

Update some doc comment wording and release note wording.

9ccaaf9

Rename write_single and write_multiple to write_full and write_partia…

c8f2933

…l respectively.

andriyDev force-pushed the one-to-many branch from 7cf4977 to c8f2933 Compare November 21, 2025 08:01

	particularly for producing meshlets.
	particularly for producing virtual geometry meshes.

Uh oh!

One-to-many asset processing #21889

Are you sure you want to change the base?

One-to-many asset processing #21889

Conversation

andriyDev commented Nov 20, 2025

Objective

Solution

Testing

Uh oh!

JMS55 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

andriyDev left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants