-
-
Notifications
You must be signed in to change notification settings - Fork 0
Release v1.0.0: Focus on RST/MyST to Jupyter Notebook Conversion #62
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Conversation
- Removed 220 lines from full.py (1504→1284 lines, 15% reduction) - Simplified depart_reference from 143 to 45 lines - Removed all tojupyter_target_pdf/html conditionals - Removed book index construction logic - Removed PDF content_depth skip logic - Always use Markdown-style links and formatting - All tests passing (base, full, glue, proof) This is part of the v2.0 refactoring to focus solely on notebook generation.
- ✅ All tests pass after v2.0 refactoring (no output changes) - Added latex_macros to tests-full session - Created dedicated test-macros session - Updated tests/README.md with comprehensive documentation - Created TEST-REVIEW.md, TEST-VALIDATION-PLAN.md, TEST-RESULTS.md Test Matrix Results: - Python 3.11, 3.12, 3.13 ✅ - Sphinx 7.4, 8.2 ✅ - Base (26 RST files) ✅ - MyST-NB glue ✅ - Sphinx-proof ✅ - LaTeX macros ✅ (newly added to suite) No reference notebooks need updating - output identical despite 220-line code reduction.
Captures all accomplishments from Phase 1-2.5: - Code refactoring (220 lines removed) - Test validation (all pass with zero changes) - Test suite improvements (latex_macros added) - Documentation and progress tracking Ready for Phase 3: Documentation updates.
- Added MIGRATION.md with comprehensive migration guide from v1.x to v2.0 - Updated CHANGELOG.md with detailed v2.0.0 breaking changes - Updated README.md with v2.0 focus statement - Updated docs/index.md with v2.0 notice and migration link - Simplified docs/config-extension.md to notebooks-only - Updated docs/config-extension-notebooks.md: - Added v2.0 notice - Removed jupyter_target_html section - Completely rewrote docs/builders.md for v2.0: - Documented jupyter builder only - Added removal notice for jupyterpdf - Provided Jupyter Book alternatives - Updated docs/installation.md: - Added Python >= 3.11, Sphinx >= 7.0 requirements - Added v2.0 notice about Jupyter Book - Added migration guide link - Removed conda-forge todo - Deleted obsolete documentation: - docs/config-extension-pdf.md - docs/config-extension-html.md - docs/config-extension-execution.md - docs/config-extension-coverage.md - docs/config-extension-exercise.md Total changes: - 5 files deleted - 10 files updated - 1 new file (MIGRATION.md) Phase 3 complete: Documentation now reflects v2.0's focused mission of notebook generation only, with clear migration path for v1.x users.
Added pytest-based tests for v2.0 configuration handling: New test file: - tests/test_config_validation.py - test_deprecated_config_options_dont_crash: Ensures deprecated v1.x options (jupyter_target_pdf, jupyter_target_html, jupyter_execute_notebooks, etc.) don't cause crashes - test_core_config_options_work: Validates core v2.0 options work correctly - test_minimal_config: Tests minimal configuration succeeds New nox session: - test-config: Runs configuration validation tests across Python/Sphinx matrix - Added to noxfile.py with parametrization for Python 3.11/3.12/3.13 and Sphinx 7.4/8.2 Updated documentation: - tests/README.md: Added configuration validation test suite section - Added test coverage items for config validation and deprecated options Test results: ✅ All 3 tests pass ✅ Validates backward compatibility with deprecated options ✅ Confirms core v2.0 functionality Phase 4 complete: Test suite enhanced to ensure v2.0 configuration robustness and backward compatibility.
RESTRUCTURING (Major):
- Flattened package structure for better organization
- Moved writers/utils.py → utils.py (used across multiple components)
- Moved directive/jupyter.py → directive.py (single file module)
- Removed empty directive/ and writers/ directories
- Removed duplicate builders/ directory (used root builder.py)
- Updated all imports to reflect new structure
CODE QUALITY IMPROVEMENTS:
- Removed Python 2 compatibility code (we require Python >= 3.11)
- Added comprehensive module docstrings to utils.py and directive.py
- Added docstring to python27_glob() function (kept for test compatibility)
- Removed backup file translators/full.py.backup
- Imported fnmatch at module level (was conditionally imported for Python 2)
DOCUMENTATION:
- Added CODE-QUALITY-REVIEW.md with comprehensive audit results
- Documented all identified issues with severity levels
- Provided recommendations for future improvements
New package structure:
sphinx_tojupyter/
├── __init__.py
├── builder.py # Jupyter builder (was using duplicate in builders/)
├── writer.py # Jupyter writer
├── directive.py # Jupyter directives (moved from directive/jupyter.py)
├── utils.py # Shared utilities (moved from writers/utils.py)
└── translators/
├── __init__.py
├── code.py # Code-only translator
└── full.py # Full document translator
Benefits:
✅ Clearer package structure
✅ Easier navigation (less directory nesting)
✅ Removed dead/duplicate code
✅ Better module organization
✅ All tests pass
Validation:
- ✅ All unit tests pass
- ✅ Import paths updated correctly
- ✅ No functional changes (pure refactoring)
This refactor will be released as v1.0.0 (not v2.0.0) because: - v0.6.0 serves as the pre-release baseline (users can pin to it) - v1.0.0 better represents 'first stable, focused release' - Saves v2.x for future major architecture changes - More room for incremental MyST improvements (v1.1, v1.2, etc.) Changes: - Updated VERSION in setup.py: v2.0.0 → v1.0.0 - Updated all documentation references - Updated CHANGELOG.md to reflect v1.0.0 as first stable release - Updated MIGRATION.md title: v0.6.0 → v1.0.0 - Deleted the incorrectly created v1.0.0 tag/release
Consolidation: - Created FUTURE-DEVELOPMENT-PLAN.md with v1.x roadmap - Moved MyST test results to docs/myst-feature-testing.md - Moved MyST coverage analysis to docs/myst-feature-coverage.md - Updated CHANGELOG.md with MyST limitations and future plans - Updated docs/index.md to include MyST documentation Removed temporary files: - SESSION-SUMMARY.md (info integrated into FUTURE-DEVELOPMENT-PLAN.md) - TEST-VALIDATION-PLAN.md (completed, no longer needed) - TEST-RESULTS.md (completed, no longer needed) - TEST-REVIEW.md (completed, no longer needed) - CODE-QUALITY-REVIEW.md (issues resolved, no longer needed) - CLEANUP-PROGRESS.md (completed, no longer needed) - VERSIONING-DECISION.md (decision recorded in git history) Root directory now contains only essential files: - REFACTOR-PLANNING.md (historical planning) - FUTURE-DEVELOPMENT-PLAN.md (v1.x roadmap) - CHANGELOG.md, MIGRATION.md, README.md (user-facing) - TESTING.md, TESTING_QUANTECON.md (development docs)
Removed obsolete documentation: - config-project.md (referenced removed features: jupyter_make_site, jupyter_generate_html, jupyter_execute_notebooks, jupyter_target_html) - directives.md (documented exercise/exerciselist directives removed in v1.0) Rewrote config-example.md: - Created modern minimal and complete examples - Only includes v1.0 supported options - Added MyST, MyST-NB, and sphinx-proof examples - Added migration notes from v0.6.0 - Included validation commands and resources Updated examples.md: - Removed obsolete jupyter_images_markdown reference - Updated coverage test reference - Modernized solution/test block descriptions Updated config-extension-notebooks.md: - Removed obsolete coverage test reference - Simplified jupyter_ignore_no_execute description Updated docs/index.md: - Removed deleted files from toctree - Reordered for better flow: installation → builders → config - Added config-extension-notebooks to toctree All documentation now reflects v1.0 scope and features.
Fixed malformed YAML frontmatter and content duplication. File now has: - Clean YAML frontmatter - Proper markdown formatting - Simplified but complete configuration examples - All v1.0 options only
Quality assurance fixes:
- Added YAML frontmatter to latex-macros.md
- Added YAML frontmatter to myst-feature-coverage.md
- Added YAML frontmatter to myst-feature-testing.md
- Fixed broken {doc} cross-reference in config-example.md
All documentation files now have:
✅ Properly formatted YAML frontmatter
✅ Valid MyST markdown syntax
✅ Even number of code fences (matched ```)
✅ Valid Python syntax in code blocks
✅ Proper MyST anchor definitions
✅ Working cross-references
- Add .github/workflows/docs.yml for automatic deployment to GitHub Pages - Add .github/workflows/README.md documenting the deployment workflow - Update docs/conf.py: change theme from sphinx_rtd_theme to alabaster - Fix docs/config-extension.md: remove references to deleted files - Update copilot-instructions.md with comprehensive v1.0 information: - Add project overview and package structure - Document all nox test sessions and CI/CD workflows - Add test structure and common development tasks - Include debugging tips and practical examples
- Modified sphinx_tojupyter/translators/full.py to prefix internal cross-document links with configured tojupyter_urlpath - In-page anchors (starting with #) remain as local references - External URLs (http://, https://) are not modified - Added comprehensive test suite in tests/urlpath/ - Added nox session 'test-urlpath' to validate behavior - Tests verify cross-document links are prefixed and in-page anchors are local Fixes issue where exercise solution links were using local anchors instead of full URLs when tojupyter_urlpath was configured.
- Added tojupyter_drop_raw_html config option (default: True) - Drops raw HTML blocks (Thebe config, scripts, etc.) from notebooks - Prevents web-specific content from cluttering notebook output - Modified sphinx_tojupyter/__init__.py to register config - Modified translators/code.py to read config with fallback - Modified translators/full.py visit_raw() to skip HTML when enabled - Added comprehensive test suite in tests/raw_html/ - Added nox session 'test-raw-html' to validate behavior Fixes issue where Thebe configuration appeared as extra markdown cells at the end of generated notebooks. Resolves: THEBE_ISSUE.md reported by user Default behavior: Drop raw HTML (clean notebooks) Override: Set tojupyter_drop_raw_html = False to preserve HTML
- Tests tojupyter_drop_raw_html configuration with real Thebe blocks - Includes comparison test (drop vs passthrough modes) - Documents behavior with sphinx-thebe extension - Provides regression testing for raw HTML dropping Tests verify: - Default (drop_raw_html=True) removes <script> tags - Passthrough (drop_raw_html=False) preserves HTML - Simulates sphinx-thebe configuration injection Addresses testing gap identified in PR #149 review
- Documents all 15 active configuration options - Includes type, default, status, and purpose for each - Notes options removed in v1.0 refactor - Highlights recent fixes (tojupyter_urlpath, tojupyter_drop_raw_html) - Provides migration guidance from v0.x Useful reference for reviewing v1.0 configuration alignment
|
@HumphreyYang I have uploaded the jupyter.zip from continuous_time_mcs When you have time would you mind to cast an eye over them and let me know if you spot any errors. This PR is a massive re-write of this extension to solely focus on |
- Replace builder() and builder.warn() calls with builder.logger.warning() to fix TypeError and AttributeError in writer.py and translators/full.py - Update test_config_validation.py to use tojupyter_* config options and add assertions to verify config values affect notebook output - Fix documentation to reference correct tojupyter_* option names in config-example.md and config-extension-notebooks.md - Add explicit config option rename table to MIGRATION.md - Change default fallback from 'code' to 'all' to match config default Addresses code review feedback on PR #62
PR Description Updated ✅I've updated the PR description to ensure accuracy and completeness: Key Changes Made:
All updates ensure the PR description accurately reflects the current state of the v1.0 refactor branch. |
- Remove CONFIG_OPTIONS_SUMMARY.md (temporary working document) - Remove test-output.log (should be gitignored per .gitignore) - Keep documented planning files (REFACTOR-PLANNING.md, FUTURE-DEVELOPMENT-PLAN.md, TESTING.md) - Adheres to copilot-instructions.md policy: no summary artifacts
PR Description Corrected ✅Issue Found: The previous description incorrectly stated that config options were renamed from Correction: The What Changed:
Migration Impact: |
|
Many thanks @mmcky! I will take look at each of these notebooks. |

Summary
This PR represents a major refactoring of sphinx-tojupyter to focus exclusively on converting RST and MyST source files to Jupyter notebooks. This streamlines the project by removing execution, PDF generation, and website building features—users should now use dedicated tools like Jupyter Book, myst-nb, or nbconvert for those workflows.
Breaking Changes⚠️
Removed Features:
myst-nbor Jupyter Book instead)jupyter nbconvertor Jupyter Book instead)tojupyter_execute_notebooks,tojupyter_make_site,tojupyter_target_pdf, and 30+ other obsolete optionsRemoved Files:
builders/jupyterpdf.py(legacy PDF builder)writers/execute_nb.py(execution logic)writers/make_pdf.py(PDF generation)writers/make_site.py(site generation)Configuration Simplified:
tojupyter_*prefix (unchanged from v0.6)What's New ✨
Core Functionality
Package Restructuring
translators/package for conversion logicDocumentation Overhaul
Testing Infrastructure
Migration Guide
If you only used notebook generation:
Update your dependencies:
Then remove any deleted configuration options from your
conf.py(see below).If you used execution/PDF/site features:
Switch to recommended tools:
For notebook execution:
For PDF generation:
# Use nbconvert jupyter nbconvert --to pdf notebook.ipynbFor website building:
Configuration Changes⚠️
Configuration option names are unchanged - v0.6 already used the
tojupyter_*prefix, and v1.0 continues using it.What changed: Many options have been removed to focus on notebook generation only.
Remove these deleted options from your
conf.py:tojupyter_execute_notebooks- usemyst-nbinsteadtojupyter_make_site- use Jupyter Book insteadtojupyter_target_pdf- usenbconvertinsteadtojupyter_execute_nb- usemyst-nbinsteadtojupyter_generate_html- use Jupyter Book insteadtojupyter_html_template- use Jupyter Book insteadtojupyter_theme- use Jupyter Book insteadtojupyter_theme_path- use Jupyter Book insteadtojupyter_template_path- use Jupyter Book insteadtojupyter_threads_per_worker- no longer neededtojupyter_number_workers- no longer neededRetained options in v1.0 (with
tojupyter_*prefix):tojupyter_kernels- Kernel metadatatojupyter_default_lang- Default language for code blockstojupyter_lang_synonyms- Language synonymstojupyter_conversion_mode- Conversion mode ("all" or "code")tojupyter_drop_solutions- Drop solution blockstojupyter_drop_tests- Drop test blockstojupyter_static_file_path- Static files to copytojupyter_images_markdown- Use markdown for imagestojupyter_urlpath- Base URL for cross-document linkstojupyter_image_urlpath- Base URL for imagestojupyter_glue_urlpath- Base URL for MyST-NB glue figurestojupyter_latex_macros- LaTeX macros file pathtojupyter_dependency_lists- File dependenciesNew option in v1.0:
tojupyter_drop_raw_html(default:True) - Drop raw HTML/script blocksSee
MIGRATION.mdfor detailed upgrade guide.Documentation
📚 Full documentation available at: https://quantecon.github.io/sphinx-tojupyter/
Key pages:
Testing
All tests pass across the compatibility matrix:
Test Results:
Compatibility
Python: 3.11, 3.12, 3.13
Sphinx: 7.4, 8.2+
Required:
sphinx>=7.0,nbformat,nbconvert,pyyamlOptional:
myst-parser,myst-nb>=0.14,sphinx-proofDevelopment
Updated for contributors:
copilot-instructions.mdwith v1.0 architectureCommits in This PR
This PR includes 28 commits organized in phases:
Recent Code Review Fixes (Commits 30ccab5, bbc5314)
Following comprehensive PR review, these issues were resolved:
Documentation Config References ✅
docs/config-example.mdto use correcttojupyter_*optionsdocs/config-extension-notebooks.mdsection headersMIGRATION.mdwith accurate migration informationConfig Validation Tests ✅
tests/test_config_validation.pyto use propertojupyter_*optionsSphinx API Compatibility ✅
builder.warn()withbuilder.logger.warning()sphinx_tojupyter/writer.pyandsphinx_tojupyter/translators/full.pyRepository Cleanup ✅
copilot-instructions.mdpolicytest-output.log(should be gitignored)All tests passing (3/3 validation tests, full sphinx-build success).
Checklist
Related Issues
This addresses the need to modernize sphinx-tojupyter and focus on its core strength: converting documentation to notebooks. Users who need execution, PDF, or site generation should use the excellent tools specifically designed for those purposes.
Post-Merge Steps