chore: add --changedSince base SHA to Jest in CI (#110568)

JoshuaKGoldberg · web-flow · commit d9cf05df4751 · 2026-03-30T10:17:08.000-04:00
_"Why run many unit test when few do trick?"_ – Kevin Malone, Senior Frontend Developer Changes the Frontend > Jest jobs to use Jest's [`--changedSince` option](https://jestjs.io/docs/cli#--changedsince) when possible. That has them only run test suites that are impacted by the changes in a PR based on the module dependency graph. Jest requires the commit history since the base commit, so this adds a second _"run only necessary Jest tests"_ flow in `frontend-optional.yml`. Specifically, that: 1. Increases the frontend-optional fetch-depth in PRs to `100`, which should get most PRs' commits pretty quickly 2. Tries to compute a `MERGE_BASE` using `git merge-base` * Note that this also clears `MERGE_BASE` if the PR touches any non-`static/` files. 3. If `MERGE_BASE` could be computed, passes it as a process env var to test scripts for use as Jest's `changedSince` * If `MERGE_BASE` couldn't be computed, the new _"run only necessary Jest tests"_ flow is skipped My plan (thanks to reviewer feedback) is to keep this running as a secondary, optional test flow on `master` for a couple weeks. If all seems well, we can remove the original _"run all Jest tests"_ logic and only run the _"run only necessary Jest tests"_ logic. The following PRs exercise different scales of changes: | PR | Source Files | Test Files | Interconnectivity | Ran Suites | / 1,881 suites | |---|---|---|---|---|---| | [#110624](#110624) | 0 | 1 | ⬜️ None | 1 | 0.05% | | [#111073](#111073) | 0 | 6 | ⬜️ None | 6 | 0.32% | | [#111122](#111122) | 0 | 100 | ⬜️ None | 100 | 5.3% | | [#111074](#111074) | 1 | 0 | 🟦 Low | 5 | 0.27% | | [#111075](#111075) | 4 | 0 | 🟦 Low | 7 | 0.37% | | [#111076](#111076) | 3 | 3 | 🟦 Low | 13 | 0.69% | | [#111110](#111110) | 20 | 20 | 🟦 Low | 74 | 3.9% | | [#111111](#111111) | 100 | 100 | 🟦 Low | 196 | 10.4% | | [#111119](#111119) | 1 | 1 | 🟨 Medium | 30 | 1.6% | | [#111628](#111628) | 3 | 3 | 🟨 Medium | 78 | 4.1% | | [#111629](#111629) | 10 | 10 | 🟨 Medium | 167 | 8.9% | | [#111633](#111633) | 25 | 25 | 🟨 Medium | 227 | 12.1% | | [#111652](#111652) | 100 | 100 | 🟨 Medium | 549 | 29.2% | | [#111115](#111115) | 3 | 3 | 🟧 High | 1,652 | 87.8% | | [#111083](#111083) | 20 | 20 | 🟧 High | 1,660 | 88.3% | | [#111086](#111086) | 100 | 100 | 🟧 High | 1,721 | 91.5% | | [#110568](#110568) | 0 | 0 | 🟥 Config (Full) | 1,881 | 100% | For low-connectivity changes (isolated components, feature-specific views), `--changedSince` provides pretty great savings: even 100 low-connectivity files only trigger ~10% of suites. Medium-connectivity files still scale pretty well, with 100 files hitting ~29%. High-connectivity files (widely-imported utils, core components) sadly trigger 85-90% of suites, which is inevitable given Sentry's highly connected module graph. Fixes ENG-7103
diff --git a/.github/workflows/frontend-optional.yml b/.github/workflows/frontend-optional.yml
@@ -25,8 +25,11 @@ jobs:
       testable_rules_changed: ${{ steps.changes.outputs.testable_rules_changed }}
       typecheckable_rules_changed: ${{ steps.changes.outputs.typecheckable_rules_changed }}
       frontend_all: ${{ steps.changes.outputs.frontend_all }}
+      merge_base: ${{ steps.merge_base.outputs.merge_base }}
     steps:
       - uses: actions/checkout@692973e3d937129bcbf40652eb9f2f61becf3332 # v4.1.7
+        with:
+          fetch-depth: 100
 
       - name: Check for frontend file changes
         uses: dorny/paths-filter@0bc4621a3135347011ad047f9ecf449bf72ce2bd # v3.0.0
@@ -36,6 +39,88 @@ jobs:
           filters: .github/file-filters.yml
           list-files: shell
 
+      # On PRs, HEAD is the merge commit; its parents (HEAD^1, HEAD^2) are base and head.
+      # Merge base of those two is what Jest --changedSince needs.
+      # If merge base can't be computed or non-frontend files changed, output is empty
+      # and the optional Jest job will be skipped entirely.
+      - name: Get merge base for changedSince
+        id: merge_base
+        run: |
+          MERGE_BASE=$(git merge-base HEAD^1 HEAD^2 2>/dev/null) || true
+          if [ -n "$MERGE_BASE" ]; then
+            CHANGED=$(git diff --name-only "$MERGE_BASE" HEAD^2)
+            if echo "$CHANGED" | grep -qvE '^static/'; then
+              echo "Non-frontend file changed — skipping optional Jest"
+              MERGE_BASE=""
+            else
+              echo "Merge base: $MERGE_BASE (Jest will use --changedSince)"
+            fi
+          else
+            echo "Could not compute merge base — skipping optional Jest"
+          fi
+          echo "merge_base=${MERGE_BASE:-}" >> "$GITHUB_OUTPUT"
+
+  # This job intentionally mirrors `frontend-jest-tests` in frontend.yml.
+  # Our intent is to try it out for a few weeks and see if it's stable.
+  frontend-jest-tests-changed-only:
+    if: >-
+      needs.files-changed.outputs.merge_base != '' &&
+      (needs.files-changed.outputs.testable_rules_changed == 'true' || needs.files-changed.outputs.testable_modified == 'true')
+    needs: [files-changed]
+    name: Jest
+    # If you change the runs-on image, you must also change the runner in jest-balance.yml
+    # so that the balancer runs in the same environment as the tests.
+    runs-on: ubuntu-24.04
+    timeout-minutes: 30
+    strategy:
+      # This helps not having to run multiple jobs because one fails, thus, reducing resource usage
+      # and reducing the risk that one of many runs would turn red again (read: intermittent tests)
+      fail-fast: false
+      matrix:
+        # XXX: When updating this, make sure you also update CI_NODE_TOTAL.
+        instance: [0, 1, 2, 3]
+
+    steps:
+      - uses: actions/checkout@692973e3d937129bcbf40652eb9f2f61becf3332 # v4.1.7
+        name: Checkout sentry
+        with:
+          # PRs need history so we can compute merge base for Jest --changedSince.
+          # 100 is an arbitrary depth that will get most reasonable PRs' commits.
+          fetch-depth: ${{ github.event_name == 'pull_request' && '100' || '1' }}
+
+      - uses: ./.github/actions/setup-node-pnpm
+
+      - name: Download jest-balance.json
+        id: download-artifact
+        uses: dawidd6/action-download-artifact@ac66b43f0e6a346234dd65d4d0c8fbb31cb316e5 # v11
+        with:
+          workflow: 38531594 # jest-balancer.yml
+          workflow_conclusion: success # The conclusion of the workflow we're looking for
+          branch: master # The branch we're looking for
+          name: jest-balance.json # Artifact name
+          name_is_regexp: false
+          path: tests/js/test-balancer/ # Directory where to extract artifact(s), defaults to the current directory
+          search_artifacts: true # Search for the last workflow run whose stored the artifact we're looking for
+          if_no_artifact_found: warn # Can be one of: "fail", "warn", "ignore"
+
+      - name: jest
+        env:
+          GITHUB_PR_SHA: ${{ github.event.pull_request.head.sha || github.sha }}
+          GITHUB_PR_REF: ${{ github.event.pull_request.head.ref || github.ref }}
+          # XXX: CI_NODE_TOTAL must be hardcoded to the length of strategy.matrix.instance.
+          #      Otherwise, if there are other things in the matrix, using strategy.job-total
+          #      wouldn't be correct.
+          CI_NODE_TOTAL: 4
+          CI_NODE_INDEX: ${{ matrix.instance }}
+          # Disable testing-library from printing out any of of the DOM to
+          # stdout. No one actually looks through this in CI, they're just
+          # going to run it locally.
+          #
+          # This quiets up the logs quite a bit.
+          DEBUG_PRINT_LIMIT: 0
+          MERGE_BASE: ${{ needs.files-changed.outputs.merge_base }}
+        run: pnpm run test-ci --forceExit
+
   typescript-native:
     if: needs.files-changed.outputs.frontend_all == 'true'
     needs: files-changed
diff --git a/eslint.config.ts b/eslint.config.ts
@@ -928,6 +928,7 @@ export default typescript.config([
     name: 'files/jest related',
     files: [
       'tests/js/jest-pegjs-transform.js',
+      'tests/js/sentry-test/jest-environment.js',
       'tests/js/sentry-test/mocks/*',
       'tests/js/sentry-test/loadFixtures.ts',
       'tests/js/setup.ts',
diff --git a/jest.config.ts b/jest.config.ts
@@ -69,7 +69,18 @@ let JEST_TESTS: string[] | undefined;
 // to reexec itself here
 if (CI && !process.env.JEST_LIST_TESTS_INNER) {
   try {
-    const stdout = execFileSync('pnpm', ['exec', 'jest', '--listTests', '--json'], {
+    const listTestArguments = ['exec', 'jest', '--listTests', '--json'];
+
+    if (process.env.MERGE_BASE) {
+      console.log('MERGE_BASE detected:', process.env.MERGE_BASE);
+      listTestArguments.push(
+        '--changedSince',
+        process.env.MERGE_BASE,
+        '--passWithNoTests'
+      );
+    }
+
+    const stdout = execFileSync('pnpm', listTestArguments, {
       stdio: 'pipe',
       encoding: 'utf-8',
       env: {...process.env, JEST_LIST_TESTS_INNER: '1'},
@@ -108,6 +119,10 @@ function getTestsForGroup(
   allTests: ReadonlyArray<string>,
   testStats: Record<string, number>
 ): string[] {
+  if (allTests.length === 0) {
+    return [];
+  }
+
   const speculatedSuiteDuration = Object.values(testStats).reduce((a, b) => a + b, 0);
   const targetDuration = speculatedSuiteDuration / nodeTotal;
 
@@ -122,8 +137,13 @@ function getTestsForGroup(
   const tests = new Map<string, number>();
   const SUITE_P50_DURATION_MS = 1500;
 
+  const allTestsSet = new Set(allTests);
+
   // First, iterate over all of the tests we have stats for.
   Object.entries(testStats).forEach(([test, duration]) => {
+    if (!allTestsSet.has(test)) {
+      return;
+    }
     if (duration <= 0) {
       throw new Error(`Test duration is <= 0 for ${test}`);
     }
@@ -199,8 +219,8 @@ function getTestsForGroup(
     }
   }
 
-  if (!groups[nodeIndex]) {
-    throw new Error(`No tests found for node ${nodeIndex}`);
+  if (!groups[nodeIndex]?.length) {
+    return ['<rootDir>/__no_tests_for_this_shard__'];
   }
   return groups[nodeIndex].map(test => `<rootDir>/${test}`);
 }
@@ -285,6 +305,7 @@ const config: Config.InitialOptions = {
     // window/cookies state.
     '@sentry/toolbar': '<rootDir>/tests/js/sentry-test/mocks/sentryToolbarMock.js',
   },
+  passWithNoTests: !!process.env.MERGE_BASE,
   setupFiles: [
     '<rootDir>/static/app/utils/silence-react-unsafe-warnings.ts',
     'jest-canvas-mock',
@@ -333,8 +354,7 @@ const config: Config.InitialOptions = {
    */
   clearMocks: true,
 
-  // To disable the sentry jest integration, set this to 'jsdom'
-  testEnvironment: '@sentry/jest-environment/jsdom',
+  testEnvironment: '<rootDir>/tests/js/sentry-test/jest-environment.js',
   testEnvironmentOptions: {
     globalsCleanup: 'on',
     sentryConfig: {
diff --git a/tests/js/sentry-test/jest-environment.js b/tests/js/sentry-test/jest-environment.js
@@ -0,0 +1,28 @@
+const SentryEnvironment = require('@sentry/jest-environment/jsdom');
+
+// @sentry/jest-environment mutates config.projectConfig.testEnvironmentOptions
+// .sentryConfig.init in-place (pushing integrations and calling Sentry.init).
+// When Jest runs in-band (≤1 test, e.g. via --changedSince), those mutations
+// create circular references that crash ScriptTransformer's config serialisation.
+// Deep-cloning sentryConfig isolates the mutation from the original config object.
+class SafeSentryEnvironment extends SentryEnvironment {
+  /** @param {import('@jest/environment').JestEnvironmentConfig} config @param {import('@jest/environment').EnvironmentContext} context */
+  constructor(config, context) {
+    const sentryConfig = config.projectConfig.testEnvironmentOptions?.sentryConfig;
+    if (sentryConfig) {
+      config = {
+        ...config,
+        projectConfig: {
+          ...config.projectConfig,
+          testEnvironmentOptions: {
+            ...config.projectConfig.testEnvironmentOptions,
+            sentryConfig: structuredClone(sentryConfig),
+          },
+        },
+      };
+    }
+    super(config, context);
+  }
+}
+
+module.exports = SafeSentryEnvironment;
diff --git a/tests/js/sentry-test/sentry-jest-environment.d.ts b/tests/js/sentry-test/sentry-jest-environment.d.ts
@@ -0,0 +1,7 @@
+declare module '@sentry/jest-environment/jsdom' {
+  // eslint-disable-next-line import/no-extraneous-dependencies -- transitive dep of jest
+  import type {JestEnvironment} from '@jest/environment';
+
+  const SentryEnvironment: typeof JestEnvironment;
+  export = SentryEnvironment;
+}
