perf(codegen): preload assets in parallel using rayon#252
Draft
perf(codegen): preload assets in parallel using rayon#252
Conversation
Collect all unique asset paths (costumes + sounds) upfront before zip writing begins and hash them concurrently via rayon::par_iter(). Subsequent load() calls hit the pre-populated cache with no I/O or MD5 cost, reducing compile time on asset-heavy projects.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Collect all unique asset paths (costumes + sounds) from the entire project upfront, before zip writing begins, and read + MD5-hash them concurrently via
rayon::par_iter().Subsequent
load()calls during codegen just hit the pre-populated cache with zero I/O or hashing cost.Changes
Cargo.toml: addrayonas a native-only dependency (excluded from WASM targets)src/codegen/assets.rs: addAssetObjectStore::preload()method (cfg non-wasm32) that parallelises file reads and MD5 hashingsrc/codegen/sb3.rs: callpreload()at the start ofSb3::project()with all deduplicated asset pathsEffect
On projects with many assets the wall-clock compile time is reduced proportionally to the number of CPU cores available, since file I/O and hashing previously ran serially.