Experimenting with replacing the haste-map in Metro

[afoxman]

Expo: Evan Bacon
Meta: Moti Silberman
Microsoft: Adam Foxman, Andrew Coates

Recording: Link

Metro uses jest-haste-map to crawl the monorepo file-system. It gathers metadata on each file and watches for changes so it can keep the metadata up-to-date. The metadata includes things like a file hash and info about haste annotations within the file (e.g. @providesModule). The haste annotations allow a module to depend on another without using an explicit import or require statement.

Metro crawls the file-system synchronously when it is launched, before it bundles or starts a bundle server. In large monorepos, crawling takes a long time and uses a lot of CPU, memory and disk resources. jest-haste-map uses caching to alleviate this. Even still, the cached data can take several gigabytes of memory. At this size, the haste map is too large and causes watchman to become unresponsive or crash.

Haste, as a module loading system, is not widely used by the react-native developer community. It is mostly used by Meta (Facebook). Yet, every developer pays the cost to run it on their machines and in CI loops.

Metro recently landed a PR which lets developers replace the default haste map implementation with their own code. We can use this to create a "lazy" crawler which doesn't eagerly collect file metadata. Instead, it reads metadata on demand.

The working theory is that the file graph needed for bundling will be much smaller than the monorepo file graph, and will therefore use fewer resources.

To validate this, we need a prototype implementation and a set of measurable experiments which look at CPU time, memory usage, disk reads/writes, time to create a bundle, and time to start a bundle server and deliver a bundle to a waiting app.

If this approach proves to be worthwhile, it should land in jest-haste-map as a third crawler implementation named lazy, next to node and watchman. In metro, lazy should be used by default, and only revert to node or watchman when expliclty configured to do so.

The metadata found by lazy should be cached, just as watchman and node data is cached.

lazy must continue to support file-watching for those files which are queried by Metro.

Metro assumes it will have synchronous access to an in-memory map of all files and their metadata. This will need to be replaced with synchronous calls to the lazy crawler to fill in missing pieces of the map.

[afoxman]

I wrote up a feature proposal in jest-haste-map, outlining the new lazy crawler: jestjs/jest#12231

[ecreeth]

FYI facebook/metro#812 - "Open source our jest-haste-map fork as metro-file-map"