Enable Cypress Testing in devcontainer - #1649
Conversation
…root level package.json for standing up the preview server for Cypress tests.
There was a problem hiding this comment.
Pull request overview
This PR updates the repository’s devcontainer so the existing Cypress e2e workflows (which run against Chrome) can run successfully in a fresh container environment.
Changes:
- Adds a root
previewscript that starts thepackages/test-cypresspreview server. - Updates the devcontainer to install required Cypress/Chrome runtime dependencies (
libgtk-3-0,xvfb) and adds a local devcontainer feature to installgoogle-chrome-stable. - Configures the devcontainer remote environment to clear the inherited
DISPLAYvariable to avoid Cypress/Electron startup failures.
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
| package.json | Adds a root-level preview script delegating to the packages/test-cypress workspace preview. |
| .devcontainer/devcontainer.json | Clears DISPLAY in remoteEnv, installs libgtk-3-0 + xvfb, and enables the new local Chrome feature. |
| .devcontainer/features/chrome/install.sh | Installs Google Chrome Stable by adding Google’s apt repo and key, then installing google-chrome-stable. |
| .devcontainer/features/chrome/devcontainer-feature.json | Declares the local devcontainer “chrome” feature metadata. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| #!/usr/bin/env bash | ||
| set -e | ||
|
|
||
| apt-get update | ||
|
|
||
| apt-get install -y \ | ||
| wget \ | ||
| gnupg \ | ||
| ca-certificates | ||
|
|
||
| mkdir -p /etc/apt/keyrings | ||
|
|
||
| wget -qO- https://dl.google.com/linux/linux_signing_key.pub \ | ||
| | gpg --dearmor \ | ||
| > /etc/apt/keyrings/google-chrome.gpg | ||
|
|
||
| echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/google-chrome.gpg] https://dl.google.com/linux/chrome/deb/ stable main" \ | ||
| > /etc/apt/sources.list.d/google-chrome.list | ||
|
|
||
| apt-get update | ||
|
|
||
| apt-get install -y google-chrome-stable | ||
|
|
||
| google-chrome --version No newline at end of file |
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
siefkenj
left a comment
siefkenj
left a comment
There was a problem hiding this comment.
We still don't run any cypress tests in the dev container. We need to modify the devcontainer job in the github yml files to include at least one of the cypress tests.
|
|
||
| apt-get update | ||
|
|
||
| apt-get install -y google-chrome-stable |
There was a problem hiding this comment.
Let's change this to a chromium install. It's best to stay away from any "features" that chrome adds.
There was a problem hiding this comment.
@siefkenj I tried the Ubuntu chromium-browser package, but on Ubuntu 24.04 ARM64 it resolves to a Snap-based stub rather than installing a working browser. As a result, Cypress is unable to detect any Chromium browser after installation and only sees Electron.
The previous Google Chrome feature worked correctly and allowed the existing Cypress workflows to run after addressing the DISPLAY issue. It also aligns with what the GitHub Actions CI jobs are currently using, since the Cypress diagnostics reported google-chrome as the browser executable for the CI test runs.
If Chromium is preferred, we'll likely need a different installation mechanism, such as a Playwright-managed Chromium install or another non-Snap Chromium distribution. Which direction would you like me to take?
| "./features/python": {} | ||
| "./features/python": {}, | ||
| // Install files needed for Chrome Cypress testing. | ||
| "./features/chrome": {} |
| "lint:i18n": "npm run lint:i18n -w @doenet/i18n", | ||
| "prettier:check": "prettier --check .", | ||
| "prettier:format": "prettier --write .", | ||
| "preview": "npm run preview --workspace packages/test-cypress", |
There was a problem hiding this comment.
This doesn't seem like the right way to run cypress tests. Remove this line entirely.
There was a problem hiding this comment.
@siefkenj This script doesn't run Cypress tests itself. It simply provides a top-level shortcut for starting the preview server from the packages/test-cypress workspace:
npm run preview
instead of:
npm run preview --workspace packages/test-cypress
My thinking was that we already expose a number of top-level convenience scripts that forward into specific workspaces:
npm run test:e2e-group1
npm run test:e2e-group2
npm run test:e2e-group3
npm run test:e2e-group4
npm run test:e2e-group5
and this was intended to serve a similar purpose for the preview server that those tests depend on.
That said, I agree that preview may not be the best name if it implies that it is part of the normal application workflow. If we want to keep the convenience script, I'd be happy to rename it to something more explicit such as:
npm run test:preview
which would better convey that it exists to support the Cypress testing workflow rather than being a general-purpose preview command.
There was a problem hiding this comment.
@siefkenj I've also now explored using the Chrome for Testing binaries -- but they don't provide a Linux-ARM64 version. So that isn't an option either.
One note on the Chrome installation approach: the implementation in this PR adds Google's apt repository and installs google-chrome-stable through apt rather than downloading a .deb directly.
The repository-based approach is preferred because it integrates Chrome into the normal apt package management workflow. Once the repository is configured, Chrome can be installed, upgraded, and dependency-resolved like any other package. It also remains architecture-aware through apt metadata and more closely matches how Chrome is typically installed on Linux systems. In practice, this was also the approach that successfully reproduced the browser environment used by the GitHub Actions Cypress runs.
An alternative would be to download and install the browser package directly. For example:
curl -L -o google-chrome-stable_arm64.deb \
https://dl.google.com/linux/direct/google-chrome-stable_current_arm64.deb
apt-get update
apt-get install -y ./google-chrome-stable_arm64.deb
rm google-chrome-stable_arm64.debThis approach avoids configuring a long-lived external apt repository, but it introduces a few tradeoffs:
- The installation becomes architecture-specific (
arm64vsamd64). - Future updates require downloading and installing a new package rather than using normal apt upgrade paths.
- The setup script becomes responsible for managing download URLs directly.
- Any future packaging or architecture changes would need to be handled manually within the installer.
If maintaining the Google apt repository is undesirable, supporting a direct-package installation path in the feature would be a reasonable alternative and could be added to the setup script. However, given that the repository-based installation is simpler to maintain long-term, more flexible across architectures, and was already verified to work with the existing Cypress workflows, it seemed like a safe default implementation.
|
@copilot Can you address the review comments? |
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 4 out of 4 changed files in this pull request and generated no new comments.
Suppressed comments (2)
.devcontainer/features/chrome/install.sh:5
- This feature install script doesn’t follow the pattern used by the other devcontainer feature installers in this repo (e.g.
set -euo pipefail,DEBIAN_FRONTEND=noninteractive,--no-install-recommends, and cleaning/var/lib/apt/lists/*). Withoutpipefail, thewget | gpgpipeline can fail in a way that doesn’t reliably stop the script, and leaving apt lists increases image size.
#!/usr/bin/env bash
set -e
apt-get update
package.json:52
- The new root
previewscript shells out to anothernpm run, which makes it hard to forward preview-server flags (e.g.--host/--port/--strictPort) fromnpm run preview -- .... Any extra args will currently be interpreted as npm CLI options on the inner command rather than being passed to the workspace preview script.
"preview": "npm run preview --workspace packages/test-cypress",
…re script patterns. Renamed the base path convenience macro prefix from `preview` to `test:preview` for consistency. Alternative implementation scripts have been left in place for now but should be removed once we settle on a final approach. At this point, the only alternative I see worth investigating is installing Playwright Chromium directly. That said, I recommend sticking with the current working setup unless there's a compelling reason to revisit it.
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 7 out of 7 changed files in this pull request and generated no new comments.
Suppressed comments (5)
.devcontainer/features/chrome/install.sh:26
- This feature doesn’t clear /var/lib/apt/lists, which leaves APT cache layers behind and makes the devcontainer image larger. Also consider --no-install-recommends for the Chrome install step for consistency with other feature scripts.
apt-get install -y google-chrome-stable
google-chrome --version
.devcontainer/features/chrome_for_testing/install.sh:21
- Trailing whitespace in the PLATFORM assignment can cause noisy diffs and style/lint issues; remove the extra space.
PLATFORM="linux-arm64"
.devcontainer/features/chrome_for_testing/install.sh:3
- This feature directory contains install.sh but no devcontainer-feature.json, so it can’t be referenced/used as a devcontainer feature (unlike the other feature folders). Either add the missing devcontainer-feature.json or remove this unused/unfinished feature folder.
#!/usr/bin/env bash
set -euo pipefail
.devcontainer/features/chromium/install.sh:8
- This script documents that Ubuntu’s chromium-browser APT package is typically just a Snap transitional package, but it still installs chromium-browser and then symlinks /usr/bin/chromium-browser. In containers without Snap support this commonly fails or yields a non-functional wrapper, so this feature is likely broken as written; additionally it isn’t referenced from .devcontainer/devcontainer.json. Consider removing it, or switching it to a non-Snap installation approach (e.g., Chrome for Testing download, or the Google Chrome APT repo like the chrome feature).
apt-get update
apt-get install -y --no-install-recommends \
chromium-browser
.devcontainer/features/chrome/install.sh:11
- Use --no-install-recommends for the prerequisite packages to match other devcontainer features and avoid pulling in unnecessary packages during image build.
apt-get install -y \
wget \
gnupg \
ca-certificates
…es changes suggested by copilot to install script for chrome to reduce container size.
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 7 out of 7 changed files in this pull request and generated no new comments.
Suppressed comments (5)
.devcontainer/features/chromium/devcontainer-feature.json:9
- This entire
chromiumfeature (and the relatedchrome_for_testingfiles) is stated to be "not used in the devcontainer build process" and is currently unreferenced. Keeping unused devcontainer features in-tree adds maintenance burden and can confuse future contributors; consider removing these exploratory features from the PR (or moving the investigation notes to documentation instead).
"id": "chromium",
"version": "1.0.0",
"name": "Chromium",
"description": "Installs Chromium and supporting dependencies for Cypress browser testing.",
"installsAfter": ["ghcr.io/devcontainers/features/common-utils"]
}
.devcontainer/features/chromium/install.sh:6
- The shebang is not on the first line of this script. If this feature is ever executed, the OS may ignore the intended interpreter and try to run it under
/bin/sh(or treat it as a plain text file), which can breakset -euo pipefailand other bash-specific syntax.
## Included for review purposes only. This file is not used in the devcontainer build process.
## But it was a path explored for resolving the chrome-cypress issue.
#!/usr/bin/env bash
set -euo pipefail
package.json:54
- The new root script
test:previewstarts the Cypress preview server without ensuringpackages/test-cypresshas been built first. In CI, the workflow runsnpm run build --workspace packages/test-cypressbeforevite preview, and a fresh checkout/devcontainer may not havedist/yet, causingvite previewto fail.
"test:preview": "npm run preview --workspace packages/test-cypress",
.devcontainer/features/chrome_for_testing/install.sh:5
- The shebang is not on the first line of this script. If this feature is ever executed, the OS may ignore the intended interpreter and try to run it under
/bin/sh(or treat it as a plain text file), which can break bash-specific syntax.
## Included for review purposes only. This file is not used in the devcontainer build process.
## But it was a path explored for resolving the chrome-cypress issue.
#!/usr/bin/env bash
set -euo pipefail
.devcontainer/features/chromium/devcontainer-feature.json:3
devcontainer-feature.jsonmust be valid JSON; the leading//comment lines make this file invalid and will break any tooling that tries to consume the feature definition.
This issue also appears on line 4 of the same file.
// Included for review purposes only. This file is not used in the devcontainer build process.
// But it was a path explored for resolving the chrome-cypress issue.
{
|
@siefkenj I spent some time exploring alternatives to install Chromium instead of Chrome. However, our CI environment explicitly uses The Chrome approach is also currently the cleanest and most reliable option. It's working end-to-end, requires less special handling than the Chromium alternatives I investigated, and more closely matches what we're running in CI. My proposed next steps are:
This should leave us with a simpler implementation that mirrors CI closely while providing confidence that the browser setup works as expected. The only remaining alternative I see worth exploring is installing Chromium through Playwright. My initial testing in that direction was not promising, so it would likely require additional investigation. If there's a strong preference for Chromium over Chrome, I'm happy to take a deeper look. Otherwise, I think my time would be better spent on other issues, since the current solution already resolves the core problem of getting Cypress testing running successfully in the devcontainer. |
|
I still don't like the chrome approach. It needs to download pgp keys from a website to install. The package manager should have chromium available. Does it not? |
If you look above I’ve already documented that it does not. |
|
|
||
| mkdir -p /etc/apt/keyrings | ||
|
|
||
| wget -qO- https://dl.google.com/linux/linux_signing_key.pub \ |
There was a problem hiding this comment.
If we don't need keys, can we remove these lines?
There was a problem hiding this comment.
After digging into this further, I noticed we're already installing a Playwright-managed Chromium browser as part of the Python feature:
PLAYWRIGHT_BROWSERS_PATH=/opt/playwright-browsers playwright install chromiumand I was able to verify that Cypress can successfully run against that Chromium binary directly.
Given that, I'm going to spend some time tomorrow exploring a Chromium-based implementation that reuses the existing Playwright installation instead of adding a separate system Chrome install.
There are still some challenges to work through:
- Playwright installs Chromium in a versioned location (e.g.
chromium-1234), so we'd need a stable way for Cypress to discover and launch it. - We'd need to validate the full Cypress workflow and ensure the solution is reliable across devcontainer rebuilds.
- Browser lifecycle and versioning would become tied to Playwright's browser management rather than OS package management.
- We'd effectively be wiring together two separate ecosystems that were not designed to manage each other's browser lifecycle.
For the current implementation, however, the PGP key setup is still required. The key is necessary for APT to trust Google's repository and perform the initial installation:
apt-get update
apt-get install -y --no-install-recommends google-chrome-stableWithout the key, that installation path would fail signature verification and Chrome would not install successfully. So if we keep the current APT-based approach, the keyring configuration needs to stay.
If we move entirely to the Playwright-managed Chromium approach, we could remove the Google repository and associated keys altogether. However, I expect that solution to be more fragile in the long term because it will require bridging Cypress' browser discovery and execution model with Playwright's browser installation and version-management model, introducing additional coordination, maintenance, and potential failure modes compared to a traditional system-installed browser.
That said, I still don't fully understand the objections to the system-browser approach. I would be perfectly happy to make that installation Chromium instead of Chrome if there were a reliable way to do so in this environment. Unfortunately, on Linux ARM64, the options we've explored so far have either been unavailable, Snap-based, unmaintained, or otherwise problematic.
Given those constraints, I still lean toward a stable system-installed browser. It keeps browser management at the operating-system level, aligns with how Cypress expects to discover and launch browsers, uses the same browser as GitHub Actions CI testing, and avoids introducing an implicit dependency between Cypress and Playwright-managed browser assets. I'll evaluate the Playwright Chromium approach tomorrow since it's already present and appears functional, but at the moment I view it as a workaround worth investigating in response to the concerns raised about the system-level install rather than a clearly cleaner or more maintainable solution than a conventional system-installed browser.
There was a problem hiding this comment.
Are you running on a mac? I thought the dev container was set up as an x86 container.
There was a problem hiding this comment.
@siefkenj Yes, I am running on Mac. I could force the x86 build through emulation, but it would be much slower.
…r chrome for testing. The two viable paths appear to be the system-level chrome install or the Playwright managed Chromium install, already being done as part of the Python feature.
Summary of current state of PR to enable seamless Cypress Testing into DoenetML devcontainer@siefkenj @dqnykamp @cqnykamp (this is basically the tl;dr version of all my comments above) The core issue is not that ARM64 Chromium doesn't exist. In fact, we already have two viable browser options:
The difficulty is specifically finding a distribution-packaged Chromium solution for Ubuntu ARM64 that works cleanly inside a devcontainer. Some of the challenges we've run into:
As a result, the current PR installs Chrome through Google's repository, which gives us a conventional system-level browser installation that Cypress can discover and launch normally. The obvious alternative is reusing the Playwright Chromium binary that's already being installed. I confirmed that Cypress can launch it successfully when pointed at the Playwright-managed executable directly. However, I'm not yet convinced that replacing the system-level browser with the Playwright-managed browser is actually a net improvement. With the current system installation, Cypress interacts with a browser exactly as expected: cypress run -b chromeand the browser exists as a system-managed installation. With the Playwright approach, we'd be depending on browser assets that are managed by a completely different toolchain. That means we'd need to maintain some mechanism that continuously maps: instead of simply having a browser installed on the system. That may work today, but it creates additional maintenance considerations:
In contrast, the current PR provides a straightforward system-level browser that behaves the same way developers and CI systems generally expect browsers to behave. So while I plan to investigate the Playwright Chromium option further, I currently view it as a potentially useful workaround that avoids the repository and key-management concerns, not as an obviously cleaner solution than the existing system-browser approach. In fact, given the additional coupling between Cypress and Playwright that it introduces, my current preference remains the system-level installation already implemented in my existing PR. |
|
Submitting this PR for review. I believe this is in its final state after evaluating a number of alternative approaches. This implementation provides a robust, low-maintenance solution that works natively on both x86 and Apple Silicon development environments, enabling Cypress testing without requiring architecture emulation and the associated performance penalties. Several alternatives were explored, particularly around using Chromium instead of a system-installed Chrome. The most viable option identified was leveraging the Chromium binary installed by Playwright. However, that approach introduces ongoing maintenance concerns due to Playwright's browser versioning and monthly update cadence. It also creates an awkward separation of responsibilities where browser installation and lifecycle management are handled by Playwright while test execution is handled by Cypress, increasing the potential for future compatibility and troubleshooting issues. The chosen approach of installing Chrome at the system level was ultimately deemed the most robust and sustainable option. In addition to avoiding cross-toolchain browser management, it offers several practical benefits:
Given those tradeoffs, a system-level Chrome installation provides the best balance of reliability, developer experience, and long-term maintainability within our current devcontainer and CI environments, while avoiding significant changes to established workflows. Alternative approaches remain possible, but are likely better suited for a future effort (if deemed worthwhile) than for adding additional complexity to this PR. At this point, the implementation is more robust than what I strictly needed for my own development workflow to locally run Cypress tests. If others find value in having a cross-architecture solution that works consistently across local development and CI, then I believe this PR should be merged. Otherwise, future contributors, whether carbon- or silicon-based, are welcome to revisit the tradeoffs. The original problem has been solved in my development environment, and I consider this work complete. I'm happy to leave the final disposition of the PR to the reviewers; any further evolution of the approach will need to be driven by someone with a stronger opinion on the matter than I currently have. |
Summary
This change updates the development container to support the existing Cypress test workflows used by the repository.
The Cypress test scripts in
packages/test-cypressare currently configured to run using Chrome:However, a fresh devcontainer did not provide the dependencies necessary for those scripts to run successfully.
Changes
libgtk-3-0xvfbgoogle-chrome-stableDISPLAYenvironment variable:Chrome Feature
google-chrome-stableis not available from the default Ubuntu repositories used by the devcontainer. The feature adds Google's signing key and apt repository before installing Chrome Stable. Unlikevim,xvfb, andlibgtk-3-0, Chrome cannot be installed through the existing apt-packages feature alone.Before Changes
A fresh devcontainer could not successfully run the repository's existing Cypress workflows.
Missing dependencies included:
libgtk-3-0google-chrome-stableAdditionally, the devcontainer inherited:
which Cypress interpreted as a valid X display. However, no usable X server was available within the container. As a result, Cypress failed during startup with errors similar to:
The standard Cypress commands failed:
A temporary workaround was:
which demonstrated that the issue was related to the inherited display environment rather than the tests themselves.
After Changes
The devcontainer now:
DISPLAY=:0environment variable for VS Code remote sessionsAs a result, Cypress can be run from a fresh devcontainer using the same commands already used throughout the repository and CI, without requiring additional wrappers or environment variable modifications.
To run Cypress tests from the devcontainer:
Start the preview server in one terminal:
Run the desired Cypress test group in another terminal:
or any other existing Cypress test script defined by the repository.
No additional
DISPLAY=prefix or container-specific command wrappers are required.Investigation
Several approaches were explored before settling on the current solution.
Missing dependencies
Initial Cypress runs failed due to missing browser runtime dependencies. Installing the following packages resolved those issues:
Chrome was also not available in the devcontainer despite CI executing Cypress tests using Chrome.
CI browser verification
A temporary diagnostic step was added to the GitHub Actions workflow to determine which browser the Cypress jobs actually use.
The workflow reported:
This confirmed that the repository's Cypress jobs run against Google Chrome in CI.
DISPLAY investigation
Despite Chrome and the required libraries being installed, Cypress still failed in the devcontainer with:
The container environment exposed:
However:
and
were both unable to connect to the display.
Additional investigation showed:
/tmp/.X11-unix/X0existed.Xauthorityfile was present$XAUTHORITYwas unsetChrome testing
Chrome itself was tested independently and successfully launched in headless mode:
This worked both with
DISPLAYunset and withDISPLAY=:0.This suggested that Chrome was not the primary source of the failure.
Cypress testing
Cypress behaved differently.
The following succeeded:
The following failed:
as did:
with failures occurring before any Chrome test execution began.
This indicates that Cypress/Electron startup is affected by the inherited invalid display environment.
Attempted Cypress configuration changes
The following approaches were explored but did not resolve the issue:
before:browser:launch--headlesswith--headless=newInvestigation showed that the failure occurs before the browser launch hook is reached, making those configuration changes ineffective.
Alternative Considered
An alternative approach would be to leave the devcontainer environment unchanged and add container-specific wrapper scripts to the Cypress package configuration, e.g.:
and similarly for each existing Cypress script.
While functional, that approach introduces container-specific behavior into the test scripts themselves and diverges from the commands executed in CI.
The implementation in this PR keeps the existing Cypress commands unchanged and instead adjusts the devcontainer environment so the existing workflow functions as expected. Specifically, it unsets the inherited
DISPLAY=:0environment variable, which Cypress interprets as a valid X display even though no usable X server is available within the container.