Skip to content

Commit 8226d83

Browse files
committed
Resolver: Collect file system observations per resolution under unstable_incrementalResolution
Stacked on #1948. ## Summary The first consumer of the `Observations` API from #1944. When `resolver.unstable_incrementalResolution` is enabled, `ModuleResolver.resolveDependency` creates an observations record at the start of each resolution, and everything that resolution looks up in the file map is recorded into it. The record is returned on the `BundlerResolution` as `unstable_observations`. Nothing reads it yet - invalidating from it is the next step - so there's no behaviour change, and with the flag off nothing is allocated. The resolution context given to `metro-resolver` doesn't change. `ModuleResolver` already builds a context per resolution, so when the flag is on it binds the file system capabilities to that resolution's record (`fileSystemLookup: p => fileSystemLookup(p, observations)`), and when it's off it passes the same functions as before. That means a custom `resolveRequest` records whatever it looks up through the context, without knowing about any of this. The observations parameter only appears on Metro's internal option types. What's recorded: - `fileSystemLookup`, and `resolveAsset` through it. - `getPackageForModule`, down to `hierarchicalLookup`. There's no memo in the way since #1948. - `doesFileExist`. `FileSystem.exists` can't record, so when recording this is answered by `lookup`, which agrees with it - true for exactly the paths found to be a file. Putting the record on the `BundlerResolution` means `DependencyGraph`'s whole-resolution cache replays it for free - every origin given a cached resolution gets its observations too. The one other memo is the empty module, which is resolved once and cached, so a resolution that lands on it returns a copy with the union of what led there and what resolving the empty module observed. Not included, to follow: - The contents of `package.json`. `getPackage` reads with `fs` outside the file map, and Metro has no way to turn an absolute path into a canonical one. The file is recorded in `existence` by `hierarchicalLookup`, so adding or removing one is covered, but editing one isn't. - Haste lookups, which have no bucket yet. - Failed resolutions. They throw, so there's nowhere to put what they observed. ## Benchmark Resolution time in isolation, replaying every `DependencyGraph.resolveDependency` call made by one iOS dev build of the benchmark app (described in the test plan), through the whole-resolution cache. Nothing is transformed or serialised, so these are percentages of resolution, not of a build. 7 interleaved rounds. Resolutions are identical with the flag on and off. | scenario | #1948 | this diff, flag off | this diff, flag on | |---|---|---|---| | first build (ms) | 95.9 | 95.8 (−0.1%) | 111.6 (+16.5%) | | rebuild, nothing changed (ms) | 7.61 | 7.64 (+0.4%) | 7.80 (+2.1%) | | after a source edit, re-resolving the edited module's dependencies (µs) | 153 | 159 (+4.1%) | 180 (+13.2%) | | after a `package.json` change, re-resolving everything (ms) | 79.8 | 80.1 (+0.4%) | 93.1 (+16.2%) | | heap retained after resolving (MB) | | 5.3 | 18.8 (3.5x) | Flag off is within noise of #1948. (The source edit row is 8 resolutions, so that's under a microsecond each, and the runs overlap.) Flag on costs about 16% of resolution time wherever everything is resolved, which is ~2µs per resolution, and 3.5x the retained heap. On the benchmark app that's 100k entries across 6.9k distinct resolutions - a median of 14 each, 90th percentile 27 - naming 28k distinct paths. Nearly all of it is `existence`, and most of that is extension and platform candidates that missed (`a.ios.js`, `a.native.js`, ...), which can't collapse because the directory they're in exists. That's the memory to go after before this is worth turning on, e.g. by recording the directory once instead. ## Test plan ``` yarn jest packages/metro-file-map packages/metro-resolver packages/metro/src/node-haste packages/metro/src/DeltaBundler packages/metro/src/integration_tests yarn flow check yarn typecheck-ts yarn verify-api-snapshots ``` New tests in `resolver-test.js` assert the exact sets recorded for a relative import's candidates, each level probed for a package, a symlink (recorded as `content`, with lookups recorded by real path), asset variants, and a resolution that lands on the empty module. They also check that there's no `unstable_observations` by default, and that origins sharing a cached resolution share its observations. **Benchmark app** - a synthetic mid-size RN 0.87 app: 23 common dependencies (Reanimated, React Navigation, TanStack Query, lodash, etc.) and ~1,000 generated first-party modules, 60% of them in a workspace package outside `projectRoot` with its own `node_modules`. The iOS dev bundle is 2.8k modules, from a file map of 34k files. Timings replay the `resolveDependency` calls a real build of it makes against a fresh `DependencyGraph` per process, in interleaved rounds, with source and `package.json` edits made on disk and picked up by the watcher. A real build of this app takes around 4s even with a warm transform cache, so resolution is about 2% of it. Changelog: Internal
1 parent 378980b commit 8226d83

6 files changed

Lines changed: 331 additions & 26 deletions

File tree

‎packages/metro/src/DeltaBundler/__tests__/resolver-test.js‎

Lines changed: 184 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2897,6 +2897,190 @@ function dep(name: string): TransformResultDependency {
28972897
});
28982898
});
28992899

2900+
describe('unstable_incrementalResolution', () => {
2901+
const incremental: InputConfigT = {
2902+
resolver: {unstable_incrementalResolution: true},
2903+
};
2904+
// Observed paths are canonical: relative to the root, with system
2905+
// separators.
2906+
const canonical = (...posixPaths: Array<string>) =>
2907+
new Set(posixPaths.map(posixPath => joinPath(...posixPath.split('/'))));
2908+
2909+
test('resolutions carry no observations by default', async () => {
2910+
setMockFileSystem({'index.js': '', 'a.js': ''});
2911+
resolver = await createResolver();
2912+
expect(
2913+
resolver.resolve(p('/root/index.js'), dep('./a')),
2914+
).not.toHaveProperty('unstable_observations');
2915+
});
2916+
2917+
test('records every candidate probed for a relative import', async () => {
2918+
setMockFileSystem({'index.js': '', 'a.js': ''});
2919+
resolver = await createResolver(incremental, 'ios');
2920+
expect(resolver.resolve(p('/root/index.js'), dep('./a'))).toEqual({
2921+
type: 'sourceFile',
2922+
filePath: p('/root/a.js'),
2923+
unstable_observations: {
2924+
// A more specific candidate appearing changes the result
2925+
existence: canonical(
2926+
'a',
2927+
'a.ios.js',
2928+
'a.native.js',
2929+
'a.js',
2930+
'package.json',
2931+
// The stand-in for the files in the directory, which would
2932+
// change their package if it appeared as a package root
2933+
'.metro-package-scope',
2934+
),
2935+
content: new Set(),
2936+
},
2937+
});
2938+
});
2939+
2940+
test('records each level probed for a package', async () => {
2941+
setMockFileSystem({
2942+
deep: {dir: {'index.js': ''}},
2943+
node_modules: {
2944+
pkg: {
2945+
'package.json': JSON.stringify({name: 'pkg', main: 'main.js'}),
2946+
'main.js': '',
2947+
},
2948+
},
2949+
});
2950+
resolver = await createResolver(incremental, 'ios');
2951+
expect(
2952+
resolver.resolve(p('/root/deep/dir/index.js'), dep('pkg')),
2953+
).toEqual({
2954+
type: 'sourceFile',
2955+
filePath: p('/root/node_modules/pkg/main.js'),
2956+
unstable_observations: {
2957+
existence: canonical(
2958+
// The closest package to the origin
2959+
'deep/dir/index.js',
2960+
'deep/dir/package.json',
2961+
'deep/package.json',
2962+
'package.json',
2963+
// Each node_modules on the way up. A missing one is one entry,
2964+
// however many packages are looked up beneath it.
2965+
'deep/dir/node_modules',
2966+
'deep/node_modules',
2967+
'node_modules',
2968+
// A file is preferred to a package of the same name
2969+
'node_modules/.metro-package-scope',
2970+
'node_modules/pkg.ios.js',
2971+
'node_modules/pkg.native.js',
2972+
'node_modules/pkg.js',
2973+
'node_modules/pkg.ios.json',
2974+
'node_modules/pkg.native.json',
2975+
'node_modules/pkg.json',
2976+
'node_modules/pkg',
2977+
'node_modules/pkg/package.json',
2978+
'node_modules/pkg/.metro-package-scope',
2979+
'node_modules/pkg/main.js',
2980+
),
2981+
content: new Set(),
2982+
},
2983+
});
2984+
});
2985+
2986+
test('records a traversed symlink as content, and lookups by real path', async () => {
2987+
setMockFileSystem({'index.js': '', real: {'x.js': ''}});
2988+
fs.symlinkSync(p('/root/real'), p('/root/link'));
2989+
resolver = await createResolver(incremental, 'ios');
2990+
expect(resolver.resolve(p('/root/index.js'), dep('./link/x'))).toEqual({
2991+
type: 'sourceFile',
2992+
filePath: p('/root/real/x.js'),
2993+
unstable_observations: {
2994+
existence: canonical(
2995+
'real/x',
2996+
'real/x.ios.js',
2997+
'real/x.native.js',
2998+
'real/x.js',
2999+
'real/.metro-package-scope',
3000+
'real/package.json',
3001+
'package.json',
3002+
),
3003+
// Retargeting the link changes what every path through it means
3004+
content: canonical('link'),
3005+
},
3006+
});
3007+
});
3008+
3009+
test('records every asset variant probed', async () => {
3010+
setMockFileSystem({
3011+
'index.js': '',
3012+
'asset.png': '',
3013+
'asset@2x.png': '',
3014+
});
3015+
resolver = await createResolver(incremental, 'ios');
3016+
expect(
3017+
resolver.resolve(p('/root/index.js'), dep('./asset.png'))
3018+
.unstable_observations,
3019+
).toEqual({
3020+
existence: canonical(
3021+
'asset.png',
3022+
'asset@1x.png',
3023+
'asset@1.5x.png',
3024+
'asset@2x.png',
3025+
'asset@3x.png',
3026+
'asset@4x.png',
3027+
'package.json',
3028+
),
3029+
content: new Set(),
3030+
});
3031+
});
3032+
3033+
test('a resolution landing on the empty module includes what resolving the empty module observed', async () => {
3034+
setMockFileSystem({
3035+
'empty.js': '',
3036+
withBrowser: {
3037+
'package.json': JSON.stringify({
3038+
name: 'with-browser',
3039+
browser: {'./gone.js': false},
3040+
}),
3041+
'index.js': '',
3042+
'gone.js': '',
3043+
},
3044+
});
3045+
resolver = await createResolver(
3046+
{
3047+
resolver: {
3048+
emptyModulePath: p('/root/empty.js'),
3049+
unstable_incrementalResolution: true,
3050+
},
3051+
},
3052+
'ios',
3053+
);
3054+
expect(
3055+
resolver.resolve(p('/root/withBrowser/index.js'), dep('./gone')),
3056+
).toEqual({
3057+
type: 'sourceFile',
3058+
filePath: p('/root/empty.js'),
3059+
unstable_observations: {
3060+
existence: canonical(
3061+
// What led to the empty module
3062+
'withBrowser/gone',
3063+
'withBrowser/package.json',
3064+
// What resolving the empty module observed
3065+
'empty.js',
3066+
'.metro-package-scope',
3067+
'package.json',
3068+
),
3069+
content: new Set(),
3070+
},
3071+
});
3072+
});
3073+
3074+
test('origins sharing a cached resolution share its observations', async () => {
3075+
setMockFileSystem({'index.js': '', 'other.js': '', 'a.js': ''});
3076+
resolver = await createResolver(incremental, 'ios');
3077+
const first = resolver.resolve(p('/root/index.js'), dep('./a'));
3078+
const second = resolver.resolve(p('/root/other.js'), dep('./a'));
3079+
expect(second).toBe(first);
3080+
expect(second.unstable_observations?.existence.size).toBeGreaterThan(0);
3081+
});
3082+
});
3083+
29003084
describe('schemeResolvers', () => {
29013085
test('config schemeResolvers are applied to scheme-prefixed specifiers', async () => {
29023086
setMockFileSystem({'index.js': '', 'a.js': ''});

‎packages/metro/src/DeltaBundler/types.js‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -151,9 +151,21 @@ export type AllowOptionalDependenciesWithOptions = {
151151
export type AllowOptionalDependencies =
152152
boolean | AllowOptionalDependenciesWithOptions;
153153

154+
// What a resolution observed of the file system: canonical paths whose
155+
// addition or removal can change its result, and those whose modification
156+
// can too. Satisfies `metro-file-map`'s `Observations`, which populates it.
157+
export type ResolutionObservations = Readonly<{
158+
existence: Set<string>,
159+
content: Set<string>,
160+
}>;
161+
154162
export type BundlerResolution = Readonly<{
155163
type: 'sourceFile',
156164
filePath: string,
165+
// What this resolution observed of the file system, if
166+
// `resolver.unstable_incrementalResolution` is enabled. It is shared by
167+
// every caller given this resolution, so must not be mutated.
168+
unstable_observations?: ResolutionObservations,
157169
}>;
158170

159171
export type Options<T = MixedOutput> = Readonly<{

‎packages/metro/src/node-haste/DependencyGraph.js‎

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -11,6 +11,7 @@
1111

1212
import type {
1313
BundlerResolution,
14+
ResolutionObservations,
1415
TransformResultDependency,
1516
} from '../DeltaBundler/types';
1617
import type {DynamicRoot} from '../lib/dynamicRoots';
@@ -135,8 +136,8 @@ export default class DependencyGraph extends EventEmitter {
135136
);
136137
this._resolutionCache = new Map();
137138
this.#packageCache = new PackageCache({
138-
getClosestPackage: absoluteModulePath =>
139-
this._getClosestPackage(absoluteModulePath),
139+
getClosestPackage: (absoluteModulePath, observations) =>
140+
this._getClosestPackage(absoluteModulePath, observations),
140141
});
141142
this._createModuleResolver();
142143
});
@@ -193,6 +194,7 @@ export default class DependencyGraph extends EventEmitter {
193194

194195
_getClosestPackage(
195196
absoluteModulePath: string,
197+
observations?: ?ResolutionObservations,
196198
): ?{packageJsonPath: string, packageRelativePath: string} {
197199
const result = this._fileSystem.hierarchicalLookup(
198200
absoluteModulePath,
@@ -201,6 +203,7 @@ export default class DependencyGraph extends EventEmitter {
201203
breakOnSegment: 'node_modules',
202204
subpathType: 'f',
203205
},
206+
observations,
204207
);
205208
return result
206209
? {

0 commit comments

Comments
 (0)