feature: engine extension #13718
Merged
feature: engine extension #13718
+13,638
−1,650
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Collaborator
✅ Snyk checks have passed. No issues have been found so far.
💻 Catch issues earlier using the plugins for VS Code, JetBrains IDEs, Visual Studio, and Eclipse. |
gordonwoodhull
force-pushed
the
feature/engine-extension-3
branch
from
ecb4468 to
9aedc82
Compare
This comment was marked as outdated.
This comment was marked as outdated.
gordonwoodhull
force-pushed
the
feature/engine-extension-3
branch
2 times, most recently
from
bf9a428 to
8f530a4
Compare
Contributor
Author
|
I’ve resolved the circular dependencies and other build issues with reasonable elegance. Since I’m out for the next week, I propose to merge on Dec 11. Still need to clean up the commit history but I think this is solid. We are hitting what appears to be an esbuild bug, necessitating marking some minified symbols as external for eslint. Seems to be harmless. evanw/esbuild#4348 |
gordonwoodhull
force-pushed
the
feature/engine-extension-3
branch
from
7e7ddc7 to
0ec129b
Compare
Contributor
Author
|
The only really ugly thing in here is the commands that need the list of engines: Often we need the list really early, when parsing command-line arguments. Previously this was known statically, actually initialized at module level. But we can't do that anymore: we need a project context and we need to load extensions. So there's a new utility I tried to isolate the badness... we can revisit the implementation later, or feel free to review here! |
Begin major architectural refactoring to enable external execution engines: - Create @quarto/types package with QuartoAPI interface and type definitions - Introduce ExecutionEngineDiscovery/Instance pattern replacing legacy engine system - Port Julia engine to new pattern and expand API with jupyter, path, format, and system namespaces - Create EngineProjectContext for engine-specific project access Co-Authored-By: Claude <[email protected]>
Complete migration of Jupyter, Knitr, and Julia engines to ExecutionEngineDiscovery pattern: - Port Jupyter and Knitr engines - Move Julia to bundled extension - Remove legacy ExecutionEngine adapter pattern and _discovery flags - Add git subtree infrastructure for managing bundled extensions in separate repositories - Make quarto check discover engines - Quarto API: system.checkRender(), system.onCleanup(), console namespace - Improve Julia tests with proper Deno test steps and cleanup Co-Authored-By: Claude <[email protected]>
Completes the external engine ecosystem with user-facing tools and organizational improvements. Git Subtree Organization: - Separate git subtree extensions into extension-subtrees/ directory to distinguish from other bundled extensions - Update subtree extension directory handling and tests - Remove bundled subtree extensions from input to pandoc to avoid duplicate processing Engine Template: - Add 'engine' extension template for quarto create - Include Discovery/Instance pattern boilerplate - Fix hardcoded "example" references in engine template - Improve engine template and error handling - Add cell language prompt for engine configuration - Add engine extension type to basic creation tests Metadata Display: - Filter bundled engines from metadata display output - Suppress expected/noisy engine metadata while preserving user's custom engines Co-Authored-By: Claude <[email protected]>
Single-file renders now initialize a ProjectContext with config, enabling
both engine and metadata extensions to work without requiring _quarto.yml.
Key changes:
- Single-file ProjectContext now has config = { project: {} } instead of
undefined, making it consistent with multi-file projects
- Pass full RenderOptions to singleFileProjectContext for extension services
- Export mergeExtensionMetadata() and resolveEngineExtensions() from
project-context.ts for use in single-file path
- Call both functions in singleFileProjectContext() following same order
as regular projects
- Use context.dir consistently for extension discovery
Extension output-dir handling:
- When extensions contribute output-dir metadata, set forceClean to match
the behavior of --output-dir flag, ensuring proper cleanup of the
temporary .quarto directory (addresses #9745, #13625)
- Warn users when extension contributes output-dir, showing the path and
whether cleanup will occur (respects --no-clean flag)
Co-Authored-By: Claude <[email protected]>
that reached out of quarto-cli and caused quarto-latexmk not to build correctly
yet another instance of #9737
this was causing dependencies to blow up where quarto-latexmk was trying to load all execution engines
Co-Authored-By: Claude <[email protected]>
…ests Implement `quarto call build-ts-extension` to type-check and bundle TypeScript engine extensions using deno bundle with import map support. - flexible entry point resolution (CLI arg, config, or auto-detect) - smart output path inference - initialize deno.json with --init-config flag LLM documentation for Quarto API and error message style, and smoke tests. Co-Authored-By: Claude <[email protected]>
…e context Fixes the issue where `quarto check julia` was failing with "Invalid value 'julia'" even though Julia is available as a bundled engine extension. The problem: When running commands like `quarto check julia` outside a project directory, no project context exists, so bundled engine extensions were never loaded. The solution: Create a zeroFileProjectContext() that provides just enough structure to discover and register bundled engine extensions when no actual project or file exists. The function initializeProjectContextAndEngines() now falls back to this zero-file context when no project is found. Also changed initializeProjectContextAndEngines() to return Promise<void> instead of Promise<ProjectContext | undefined>, since the name suggests initialization rather than retrieval and both callers ignore the return value. Now `quarto check julia` works both inside and outside project directories. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]>
also, load builtin engine extensions
Add a --dev flag to `quarto run` that uses the main development import map (src/import_map.json) instead of the user-facing run_import_map.json. This allows internal development tools (like import-report scripts) to access the full quarto development environment including deno_ral and other internal modules. Not visible for end users; description is for documentation purposes only. Usage: quarto run --dev script.ts [args...] The --dev flag should be used for internal quarto development tools that import from src/deno_ral or src/core. User-facing scripts should continue to use the standard `quarto run` without --dev. Co-Authored-By: Claude <[email protected]>
…ules If you have a dependency cycle where all of the modules are async (transitively because of their dependencies) then it would probably deadlock if you ran it. esbuild gives up and emits invalid JS. So you must either break these cycles, or prevent the async from propagating to the cycles. This tool uses ILP to show - Minimal places to break remove static imports to stop transitive dependencies before they hit cycles (minimal hitting set) - Minimal places to remove static imports to break these cycles (minimum feedback arc set) Co-Authored-By: Claude <[email protected]>
Eliminate circular dependencies by restructuring quarto-api into modular src/core/api/ architecture with dependency inversion. Move shared utilities from execute/ to core/, refactor into 13 modules with registry pattern for eager validation and lazy initialization, and replace Proxy-based API with explicit getQuartoAPI() function. Also implement dynamic engine registration to resolve module initialization order issues. Co-Authored-By: Claude <[email protected]>
…tils.ts Refactored to break the 2-cycle between src/execute/engine.ts ↔ src/command/command-utils.ts by moving CLI command logic out of the core engine module. ## Changes **Created:** src/command/call/engine-cmd.ts - Moved engineCommand definition from engine.ts - Imports core functions: executionEngine(), executionEngines() - Imports initializeProjectContextAndEngines from command-utils.ts - Follows existing command pattern (check/cmd.ts, render/cmd.ts) **Modified:** src/execute/engine.ts - Removed engineCommand export (40 lines) - Removed imports: Command from cliffy, initializeProjectContextAndEngines - Now contains only core engine logic **Modified:** src/command/call/cmd.ts - Updated import to use new ./engine-cmd.ts **Modified:** src/command/command-utils.ts - Made ProjectContext import type-only ## Impact The 2-cycle is eliminated: - engine.ts no longer imports from command-utils.ts - command-utils.ts cleanly imports from engine.ts - engineCommand now lives in proper command location - Better architecture: CLI separated from core engine logic - Net reduction: 38 lines of code 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]>
…ation - Fix timing issues where both create commands attempted to use engines before they were registered. - Move engine initialization to start of action handlers and fix create-project to use current directory (not non-existent project directory) when initializing engine registry. Co-Authored-By: Claude <[email protected]>
gordonwoodhull
force-pushed
the
feature/engine-extension-3
branch
from
516c41f to
a1348c8
Compare
gordonwoodhull
force-pushed
the
feature/engine-extension-3
branch
from
a1348c8 to
fec8673
Compare
…m commit 1de70acbd git-subtree-dir: src/resources/extension-subtrees/julia-engine git-subtree-split: 1de70acbdfb95b49a0115876453cfc1238c91f8b
…rces/extension-subtrees/julia-engine'
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
replaces #13701
Working branch for CI while I work out dependency and build issues.