Move stable ECMAScript plugins to @babel/core

[nicolo-ribaudo]

View Rendered Text

I wrote this RFC as an attempt to solve the "how do we make sure that plugins are compatible with older @babel/core versions" problem.
This change should be transparent to most of our users, but I decided to open an RFC since it changes our monorepo structure a lot (it would remove about half of the packages).

This is another step towards #3, even if it was not my goal when writing this RFC.

[nicolo-ribaudo]

Maybe it should be builtin:[feature]?

[JLHwung]

To minimize the adoption friction, I propose we implement its opposite: "external:", which means always check node modules instead of builtin plugins in @babel/core.

So, if users upgrade from @babel/core from 8.x to 8.y, they don't have to modify their config to opt in the builtin plugins. And for some users, their Babel config is inside node_modules which make it harder to edit, and they have to wait for the whole ecosystem to migrate to builtin plugins.

This happens whenever a stage 3 feature gets advanced. Therefore I suggest we implement "external:[feature]" instead, users can opt out to the builtin features if they strongly prefer the unmaintained archived versions over the built-in version in @babel/core, but then they have to modify the config or lock the @babel/core versions.

[nicolo-ribaudo]

So I have thought for a while about this, but I still prefer internal: for a few reasons:

• Completely changing the resolution process for a specific plugin in version can cause confusion: we already have multiple ways of specifying plugins, that are then normalized/resolved, but at least they are deterministic. I would prefer avoiding one that depends on which specific @babel/core version.
• When a proposal becomes standard, we want to rename it from -proposal- to -transform-. We have not done this in Babel 8 to avoid renaming the package, and it has caused confusion.
• I don't think that "being stuck on an older version" is too bad: usually changes to plugins are minimal once they become standard.
• If we want the "automatic migration" to happen, whenever we move a plugin to @babel/core we can release a new version of the -proposal- plugin that starts like this:

  if (api.version >= 7.x.y) return { inherits: "internal:name-of-the-internal-plugin" };

  and we can thus still easily provide the "automatically use the internal plugin" behavior.

[JLHwung]

Here is my comments on this RFC:

Also rename plugins to transform-* when it is stage 4

When we copy stage 4 plugins to @babel/core, we can also rename it to internal:transform-*. In that case, we also need to inform users that proposal-* is deprecated either via npm deprecate or a runtime warnings, since then no further bug fix will be applied to that plugin. Users will have to migrate to internal plugins and here comes the chances for us to actually rename the plugins.

Bugfix-*

Are we going to move bugfix-* to @babel/core?

Alternative API design

I think we can also export only availableInternalPlugins: Set<string>. Downstream users can check whether a plugin is internal, or provide end users a range of valid internal plugins.

Comments on alternative proposal

Drop SemVer for the @babel/core peerDependency of plugins, so that we can require an higher @babel/core version when needed.

I don't think it is exclusive to this RFC. Even if this RFC is accepted, we can still consider whether we should bump core version on proposal plugins when it relies on new traverse methods, or new helper in @babel/helpers.

[nicolo-ribaudo]

I made different updates to the RFC, mostly integrating @JLHwung's feedback and with some additional thoughts I've had during the past year. I force-pushed, but you can see the diff in the first file of a9c8ab3e0841979b530703d5203df6c0f128074d..ec11d33878c40f3a3c52a18e8b3f88c84a8c16c2#diff-8712fd2cf7.

I am interested on restarting to work on this RFC.

[nicolo-ribaudo]

I'm starting to work on this, to see how it looks in practice when implemented. Please review this RFC and give your opinion!

[liuxingbaoyu]

All the plugins for stable ECMAScript features should be moved from the ./packages/babel-plugin-* folders into ./packages/babel-core/src/internal-plugins/*.

I prefer to keep it as it is now.

Drop SemVer for the @babel/core peerDependency of plugins, so that we can require an higher @babel/core version when needed. Plugins would then be able to require an higher @bebel/core version even without waiting for the next major release.

👍Looking forward to this very much!

As for the others, to be honest, I am not familiar with them, so I cannot comment.

[nicolo-ribaudo]

From @liuxingbaoyu

I'm a little concerned that this PR is blurring the distinction between @babel/core and @babel/preset-env.
Previously @babel/core was a framework, while @babel/preset-env resembled a convenient collection of plugins.
Now a large number of plugins are included in @babel/core, and only a few remain in @babel/preset-env, which is not very ideal.

At the same time, the advantages of purpose also have some limitations.
Mainly it is the proposals that reach the fourth stage to benefit, but they are rarely affected by @babel/core, @babel/traverse, because bug fixes and standard alignment also happen more in <=3 proposals and ts. (now the ts plugin is widely used, and ts fixes and feature support often want newer @babel/core)

The main way I understand in rfc to encourage users to upgrade is to deprecate the proposal that becomes the fourth phase, and use the package manager to remind users to upgrade @babel/core.
This appears to be a "soft limit" only, we still need compatibility for older versions of @babel/core. And once we implement "hard limits", the soft limits will no longer be necessary and we will be able to reap most of these benefits.

But I don't block this PR, I just want to have more detailed discussions. 😃

[nicolo-ribaudo]

I'm a little concerned that this PR is blurring the distinction between @babel/core and @babel/preset-env.
Previously @babel/core was a framework, while @babel/preset-env resembled a convenient collection of plugins.
Now a large number of plugins are included in @babel/core, and only a few remain in @babel/preset-env, which is not very ideal.

The end goal is to eventually move to a model where, if a user only wants to use standard JavaScript features, they have to install a single package (#3). This may seem like a step back towards what Babel 5 was, except that we still internally maintain the current plugin-based flexibility and we still have all the advantages that the Babel 6 model (when we first split the single big package into many plugins and presets) has. Ideally, users with "standard" setups would only have to install Babel and create a .browserslist file, and @babel/core would automatically behave as if they specified targets and presets: ["@babel/preset-env"]. No Babel config file needed, unless you want to use experimental features, third-party polyfills injected using Babel, custom plugins, or advanced compiler assumptions.

At the same time, the advantages of purpose also have some limitations.
Mainly it is the proposals that reach the fourth stage to benefit, but they are rarely affected by @babel/core, @babel/traverse, because bug fixes and standard alignment also happen more in <=3 proposals and ts. (now the ts plugin is widely used, and ts fixes and feature support often want newer @babel/core)

While it's true that a big part of changes happen in the "it's a proposal" phase, we still actively maintain many plugins for stable features. They are riskier when it comes to backward compat with @babel/core, since any change we make will be enabled by default and not under a special "version": "..." option (like we do for decorators, or for pipeline).

It's true that TS fixed&features often want newer version of @babel/core (specifically, of @babel/types and @babel/parser), but except for a few cases where we need to visit new AST node types we don't have to worry about backward compat: simply, the new feature will not work unless the user also updates @babel/core.

The main way I understand in rfc to encourage users to upgrade is to deprecate the proposal that becomes the fourth phase, and use the package manager to remind users to upgrade @babel/core.
This appears to be a "soft limit" only, we still need compatibility for older versions of @babel/core. And once we implement "hard limits", the soft limits will no longer be necessary and we will be able to reap most of these benefits.

When a package reaches Stage 4 (and starting from Babel 8), we would simply copy-paste its code into a sub-folder in @babel/core and completely stop maintaining the old package. In this new copy we can now make any change we want relying on new @babel/core (and its deps) features, because that code will always be used with the correct Babel version. The old plugin will not receive those "risky" changes, because it wouldn't be released anymore.
So yes, the limit is "soft" as in "you don't get an error if you don't stop using the plugin", but also you don't need any more bug fixes and new features. The only way you have to upgrade is by migrating to the internal plugin (or better, by simply deleting the plugin from your config since it would be now enabled by @babel/preset-env).

Hopefully I have clarified some things, I can remove the PR from the 7.21.0 milestone if you want more time to discuss it :)

[liuxingbaoyu]

The end goal is to eventually move to a model where, if a user only wants to use standard JavaScript features, they have to install a single package (#3). This may seem like a step back towards what Babel 5 was, except that we still internally maintain the current plugin-based flexibility and we still have all the advantages that the Babel 6 model (when we first split the single big package into many plugins and presets) has. Ideally, users with "standard" setups would only have to install Babel and create a .browserslist file, and @babel/core would automatically behave as if they specified targets and presets: ["@babel/preset-env"]. No Babel config file needed, unless you want to use experimental features, third-party polyfills injected using Babel, custom plugins, or advanced compiler assumptions.

This definitely makes sense (maybe it's worth releasing a new package? Not core or preset-env)

So yes, the limit is "soft" as in "you don't get an error if you don't stop using the plugin", but also you don't need any more bug fixes and new features. The only way you have to upgrade is by migrating to the internal plugin (or better, by simply deleting the plugin from your config since it would be now enabled by @babel/preset-env).

I am wondering if there is any way to make ts and other plugins also benefit.
I'm not sure how people feel about the alternatives in the rfc.
Or without modifying peerDependency, perhaps allow plugins to carry parser or traverse and have core use the latest automatically?
But these are still ideas and may not be worth it compared to the PR that has been done now.

Hopefully I have clarified some things, I can remove the PR from the 7.21.0 milestone if you want more time to discuss it :)

I don't want to block it, because it's an improvement anyway, and it seems reasonable to accept it in the absence of a better solution, in my opinion.

By the way, I want to keep the layout of the monorepo even if we don't publish them anymore. The current layout is more convenient for development in my opinion. : )

[JLHwung]

I want to keep the layout of the monorepo even if we don't publish them anymore.

I agree. We can mark mature plugin as private and bundle @babel/core so that any plugins can be inlined by bundlers.

[conartist6]

It think this proposal risks damaging the stability integrity, and independence of the babel core. Before your incentives were to build a stable independent core that could work equally well for everyone, because that would be the only way for your to ship your own functionality in a stable, sustainable way.

Babel would be removing the requirement that it serve everyone else's needs in order to serve its own. Instead of being incentivized to maintain stable core APIs, the project will instead be incentivized to maintain stable core transformation semantics because those are the semantics you'd be (more) committed to not breaking as opposed to the API semantics.

I actually already think that Babel's monorepo already exposes it to this problem, as it's trivial to change a definition in core APIs and (seemingly) change all usages as well in a single PR. This leads people not to think about the real relationships between the independent parts as you can always just change everything at once, without even needing to consider the state where the core is ahead of (and incompatible with) the current plugins!