Skip to content

Conversation

@mmcky
Copy link
Contributor

@mmcky mmcky commented Nov 19, 2025

Summary

This PR represents a major refactoring of sphinx-tojupyter to focus exclusively on converting RST and MyST source files to Jupyter notebooks. This streamlines the project by removing execution, PDF generation, and website building features—users should now use dedicated tools like Jupyter Book, myst-nb, or nbconvert for those workflows.

Breaking Changes ⚠️

Removed Features:

  • ❌ Notebook execution (use myst-nb or Jupyter Book instead)
  • ❌ PDF generation (use jupyter nbconvert or Jupyter Book instead)
  • ❌ Website/HTML site generation (use Jupyter Book instead)
  • ❌ Removed config options: tojupyter_execute_notebooks, tojupyter_make_site, tojupyter_target_pdf, and 30+ other obsolete options

Removed Files:

  • Deleted builders/jupyterpdf.py (legacy PDF builder)
  • Deleted writers/execute_nb.py (execution logic)
  • Deleted writers/make_pdf.py (PDF generation)
  • Deleted writers/make_site.py (site generation)

Configuration Simplified:

  • Reduced from 47 config options to 15 essential options (68% reduction)
  • All configs now focused on notebook generation only
  • Configuration options continue to use the tojupyter_* prefix (unchanged from v0.6)

What's New ✨

Core Functionality

  • Clean RST/MyST → Notebook conversion with improved translator
  • MyST-NB feature support (glue, code cells, MyST syntax)
  • sphinx-proof directives (theorems, proofs, algorithms)
  • LaTeX macro support (MathJax 2 & 3 compatibility)
  • Modern dependencies: Sphinx 7+, Python 3.11-3.13
  • Improved error handling with modern Sphinx logger API

Package Restructuring

  • Flattened package structure for better maintainability
  • Created translators/ package for conversion logic
  • Removed ~1,200 lines of obsolete code
  • Added comprehensive docstrings and type hints

Documentation Overhaul

  • Complete rewrite of documentation for v1.0
  • New MyST feature coverage and testing guides
  • Simplified configuration examples
  • Added migration guide from v0.6.0

Testing Infrastructure

  • Matrix testing: 6 combinations (Python 3.11/3.12/3.13 × Sphinx 7.4/8.2)
  • Feature tests: Dedicated test suites for glue, proof, macros
  • Nox automation: 12+ test sessions for different scenarios
  • CI/CD: Full GitHub Actions pipeline with multi-OS testing
  • GitHub Pages: Automatic documentation deployment

Migration Guide

If you only used notebook generation:

Update your dependencies:

pip install --upgrade sphinx-tojupyter

Then remove any deleted configuration options from your conf.py (see below).

If you used execution/PDF/site features:

Switch to recommended tools:

For notebook execution:

# conf.py - Use myst-nb instead
extensions = ["myst_nb"]
nb_execution_mode = "auto"  # or "cache"

For PDF generation:

# Use nbconvert
jupyter nbconvert --to pdf notebook.ipynb

For website building:

# Use Jupyter Book
pip install jupyter-book
jupyter-book build .

Configuration Changes ⚠️

Configuration option names are unchanged - v0.6 already used the tojupyter_* prefix, and v1.0 continues using it.

What changed: Many options have been removed to focus on notebook generation only.

Remove these deleted options from your conf.py:

  • tojupyter_execute_notebooks - use myst-nb instead
  • tojupyter_make_site - use Jupyter Book instead
  • tojupyter_target_pdf - use nbconvert instead
  • tojupyter_execute_nb - use myst-nb instead
  • tojupyter_generate_html - use Jupyter Book instead
  • tojupyter_html_template - use Jupyter Book instead
  • tojupyter_theme - use Jupyter Book instead
  • tojupyter_theme_path - use Jupyter Book instead
  • tojupyter_template_path - use Jupyter Book instead
  • tojupyter_threads_per_worker - no longer needed
  • tojupyter_number_workers - no longer needed
  • And 20+ other execution/HTML/PDF-related options

Retained options in v1.0 (with tojupyter_* prefix):

  • tojupyter_kernels - Kernel metadata
  • tojupyter_default_lang - Default language for code blocks
  • tojupyter_lang_synonyms - Language synonyms
  • tojupyter_conversion_mode - Conversion mode ("all" or "code")
  • tojupyter_drop_solutions - Drop solution blocks
  • tojupyter_drop_tests - Drop test blocks
  • tojupyter_static_file_path - Static files to copy
  • tojupyter_images_markdown - Use markdown for images
  • tojupyter_urlpath - Base URL for cross-document links
  • tojupyter_image_urlpath - Base URL for images
  • tojupyter_glue_urlpath - Base URL for MyST-NB glue figures
  • tojupyter_latex_macros - LaTeX macros file path
  • tojupyter_dependency_lists - File dependencies

New option in v1.0:

  • tojupyter_drop_raw_html (default: True) - Drop raw HTML/script blocks

See MIGRATION.md for detailed upgrade guide.

Documentation

📚 Full documentation available at: https://quantecon.github.io/sphinx-tojupyter/

Key pages:

Testing

All tests pass across the compatibility matrix:

# Run all tests
nox

# Test specific features
nox -s test-glue    # MyST-NB glue support
nox -s test-proof   # sphinx-proof directives
nox -s test-macros  # LaTeX macros

# Build documentation
nox -s docs

Test Results:

  • ✅ 6/6 core test combinations passing
  • ✅ All optional feature tests passing
  • ✅ Documentation builds with 0 errors, 0 warnings
  • ✅ Multi-OS testing (Ubuntu, macOS, Windows)

Compatibility

Python: 3.11, 3.12, 3.13
Sphinx: 7.4, 8.2+
Required: sphinx>=7.0, nbformat, nbconvert, pyyaml
Optional: myst-parser, myst-nb>=0.14, sphinx-proof

Development

Updated for contributors:

  • Comprehensive copilot-instructions.md with v1.0 architecture
  • Documented all 12 nox test sessions
  • GitHub Actions CI/CD pipeline
  • Automatic GitHub Pages deployment

Commits in This PR

This PR includes 28 commits organized in phases:

  • Phase 0: Project initialization and baseline tagging
  • Phase 1: Core cleanup (~1,200 lines removed, dependency updates)
  • Phase 2: Package restructuring (flattened structure)
  • Phase 3-4: Documentation overhaul and testing infrastructure
  • Final: CI/CD setup, documentation polish, and cleanup

Recent Code Review Fixes (Commits 30ccab5, bbc5314)

Following comprehensive PR review, these issues were resolved:

  1. Documentation Config References

    • Fixed docs/config-example.md to use correct tojupyter_* options
    • Updated docs/config-extension-notebooks.md section headers
    • Enhanced MIGRATION.md with accurate migration information
  2. Config Validation Tests

    • Corrected tests/test_config_validation.py to use proper tojupyter_* options
    • Added kernel metadata and cell type validation assertions
    • Fixed docstring version reference (v2.0 → v1.0)
  3. Sphinx API Compatibility

    • Replaced deprecated builder.warn() with builder.logger.warning()
    • Fixed in sphinx_tojupyter/writer.py and sphinx_tojupyter/translators/full.py
    • Ensures compatibility with Sphinx 7.4+ and 8.2+
  4. Repository Cleanup

    • Removed temporary summary files per copilot-instructions.md policy
    • Removed tracked test-output.log (should be gitignored)

All tests passing (3/3 validation tests, full sphinx-build success).

Checklist

  • All tests passing (6 Python/Sphinx combinations)
  • Documentation updated and building cleanly
  • Migration guide provided
  • CHANGELOG.md updated
  • Breaking changes clearly documented
  • GitHub Pages deployment configured
  • CI/CD pipeline verified
  • Code review findings addressed
  • Repository cleanup completed

Related Issues

This addresses the need to modernize sphinx-tojupyter and focus on its core strength: converting documentation to notebooks. Users who need execution, PDF, or site generation should use the excellent tools specifically designed for those purposes.

Post-Merge Steps

  1. Tag as v1.0.0
  2. Create GitHub release with notes
  3. Update PyPI package (optional, when ready)
  4. Update README badges if needed

mmcky added 23 commits November 19, 2025 09:00
- Removed 220 lines from full.py (1504→1284 lines, 15% reduction)
- Simplified depart_reference from 143 to 45 lines
- Removed all tojupyter_target_pdf/html conditionals
- Removed book index construction logic
- Removed PDF content_depth skip logic
- Always use Markdown-style links and formatting
- All tests passing (base, full, glue, proof)

This is part of the v2.0 refactoring to focus solely on notebook generation.
- ✅ All tests pass after v2.0 refactoring (no output changes)
- Added latex_macros to tests-full session
- Created dedicated test-macros session
- Updated tests/README.md with comprehensive documentation
- Created TEST-REVIEW.md, TEST-VALIDATION-PLAN.md, TEST-RESULTS.md

Test Matrix Results:
- Python 3.11, 3.12, 3.13 ✅
- Sphinx 7.4, 8.2 ✅
- Base (26 RST files) ✅
- MyST-NB glue ✅
- Sphinx-proof ✅
- LaTeX macros ✅ (newly added to suite)

No reference notebooks need updating - output identical despite 220-line code reduction.
Captures all accomplishments from Phase 1-2.5:
- Code refactoring (220 lines removed)
- Test validation (all pass with zero changes)
- Test suite improvements (latex_macros added)
- Documentation and progress tracking

Ready for Phase 3: Documentation updates.
- Added MIGRATION.md with comprehensive migration guide from v1.x to v2.0
- Updated CHANGELOG.md with detailed v2.0.0 breaking changes
- Updated README.md with v2.0 focus statement
- Updated docs/index.md with v2.0 notice and migration link
- Simplified docs/config-extension.md to notebooks-only
- Updated docs/config-extension-notebooks.md:
  - Added v2.0 notice
  - Removed jupyter_target_html section
- Completely rewrote docs/builders.md for v2.0:
  - Documented jupyter builder only
  - Added removal notice for jupyterpdf
  - Provided Jupyter Book alternatives
- Updated docs/installation.md:
  - Added Python >= 3.11, Sphinx >= 7.0 requirements
  - Added v2.0 notice about Jupyter Book
  - Added migration guide link
  - Removed conda-forge todo
- Deleted obsolete documentation:
  - docs/config-extension-pdf.md
  - docs/config-extension-html.md
  - docs/config-extension-execution.md
  - docs/config-extension-coverage.md
  - docs/config-extension-exercise.md

Total changes:
- 5 files deleted
- 10 files updated
- 1 new file (MIGRATION.md)

Phase 3 complete: Documentation now reflects v2.0's focused mission of
notebook generation only, with clear migration path for v1.x users.
Added pytest-based tests for v2.0 configuration handling:

New test file:
- tests/test_config_validation.py
  - test_deprecated_config_options_dont_crash: Ensures deprecated v1.x options (jupyter_target_pdf, jupyter_target_html, jupyter_execute_notebooks, etc.) don't cause crashes
  - test_core_config_options_work: Validates core v2.0 options work correctly
  - test_minimal_config: Tests minimal configuration succeeds

New nox session:
- test-config: Runs configuration validation tests across Python/Sphinx matrix
- Added to noxfile.py with parametrization for Python 3.11/3.12/3.13 and Sphinx 7.4/8.2

Updated documentation:
- tests/README.md: Added configuration validation test suite section
- Added test coverage items for config validation and deprecated options

Test results:
✅ All 3 tests pass
✅ Validates backward compatibility with deprecated options
✅ Confirms core v2.0 functionality

Phase 4 complete: Test suite enhanced to ensure v2.0 configuration
robustness and backward compatibility.
RESTRUCTURING (Major):
- Flattened package structure for better organization
- Moved writers/utils.py → utils.py (used across multiple components)
- Moved directive/jupyter.py → directive.py (single file module)
- Removed empty directive/ and writers/ directories
- Removed duplicate builders/ directory (used root builder.py)
- Updated all imports to reflect new structure

CODE QUALITY IMPROVEMENTS:
- Removed Python 2 compatibility code (we require Python >= 3.11)
- Added comprehensive module docstrings to utils.py and directive.py
- Added docstring to python27_glob() function (kept for test compatibility)
- Removed backup file translators/full.py.backup
- Imported fnmatch at module level (was conditionally imported for Python 2)

DOCUMENTATION:
- Added CODE-QUALITY-REVIEW.md with comprehensive audit results
- Documented all identified issues with severity levels
- Provided recommendations for future improvements

New package structure:
sphinx_tojupyter/
├── __init__.py
├── builder.py          # Jupyter builder (was using duplicate in builders/)
├── writer.py           # Jupyter writer
├── directive.py        # Jupyter directives (moved from directive/jupyter.py)
├── utils.py           # Shared utilities (moved from writers/utils.py)
└── translators/
    ├── __init__.py
    ├── code.py        # Code-only translator
    └── full.py        # Full document translator

Benefits:
✅ Clearer package structure
✅ Easier navigation (less directory nesting)
✅ Removed dead/duplicate code
✅ Better module organization
✅ All tests pass

Validation:
- ✅ All unit tests pass
- ✅ Import paths updated correctly
- ✅ No functional changes (pure refactoring)
This refactor will be released as v1.0.0 (not v2.0.0) because:
- v0.6.0 serves as the pre-release baseline (users can pin to it)
- v1.0.0 better represents 'first stable, focused release'
- Saves v2.x for future major architecture changes
- More room for incremental MyST improvements (v1.1, v1.2, etc.)

Changes:
- Updated VERSION in setup.py: v2.0.0 → v1.0.0
- Updated all documentation references
- Updated CHANGELOG.md to reflect v1.0.0 as first stable release
- Updated MIGRATION.md title: v0.6.0 → v1.0.0
- Deleted the incorrectly created v1.0.0 tag/release
Consolidation:
- Created FUTURE-DEVELOPMENT-PLAN.md with v1.x roadmap
- Moved MyST test results to docs/myst-feature-testing.md
- Moved MyST coverage analysis to docs/myst-feature-coverage.md
- Updated CHANGELOG.md with MyST limitations and future plans
- Updated docs/index.md to include MyST documentation

Removed temporary files:
- SESSION-SUMMARY.md (info integrated into FUTURE-DEVELOPMENT-PLAN.md)
- TEST-VALIDATION-PLAN.md (completed, no longer needed)
- TEST-RESULTS.md (completed, no longer needed)
- TEST-REVIEW.md (completed, no longer needed)
- CODE-QUALITY-REVIEW.md (issues resolved, no longer needed)
- CLEANUP-PROGRESS.md (completed, no longer needed)
- VERSIONING-DECISION.md (decision recorded in git history)

Root directory now contains only essential files:
- REFACTOR-PLANNING.md (historical planning)
- FUTURE-DEVELOPMENT-PLAN.md (v1.x roadmap)
- CHANGELOG.md, MIGRATION.md, README.md (user-facing)
- TESTING.md, TESTING_QUANTECON.md (development docs)
Removed obsolete documentation:
- config-project.md (referenced removed features: jupyter_make_site,
  jupyter_generate_html, jupyter_execute_notebooks, jupyter_target_html)
- directives.md (documented exercise/exerciselist directives removed in v1.0)

Rewrote config-example.md:
- Created modern minimal and complete examples
- Only includes v1.0 supported options
- Added MyST, MyST-NB, and sphinx-proof examples
- Added migration notes from v0.6.0
- Included validation commands and resources

Updated examples.md:
- Removed obsolete jupyter_images_markdown reference
- Updated coverage test reference
- Modernized solution/test block descriptions

Updated config-extension-notebooks.md:
- Removed obsolete coverage test reference
- Simplified jupyter_ignore_no_execute description

Updated docs/index.md:
- Removed deleted files from toctree
- Reordered for better flow: installation → builders → config
- Added config-extension-notebooks to toctree

All documentation now reflects v1.0 scope and features.
Fixed malformed YAML frontmatter and content duplication. File now has:
- Clean YAML frontmatter
- Proper markdown formatting
- Simplified but complete configuration examples
- All v1.0 options only
Quality assurance fixes:
- Added YAML frontmatter to latex-macros.md
- Added YAML frontmatter to myst-feature-coverage.md
- Added YAML frontmatter to myst-feature-testing.md
- Fixed broken {doc} cross-reference in config-example.md

All documentation files now have:
✅ Properly formatted YAML frontmatter
✅ Valid MyST markdown syntax
✅ Even number of code fences (matched ```)
✅ Valid Python syntax in code blocks
✅ Proper MyST anchor definitions
✅ Working cross-references
- Add .github/workflows/docs.yml for automatic deployment to GitHub Pages
- Add .github/workflows/README.md documenting the deployment workflow
- Update docs/conf.py: change theme from sphinx_rtd_theme to alabaster
- Fix docs/config-extension.md: remove references to deleted files
- Update copilot-instructions.md with comprehensive v1.0 information:
  - Add project overview and package structure
  - Document all nox test sessions and CI/CD workflows
  - Add test structure and common development tasks
  - Include debugging tips and practical examples
- Modified sphinx_tojupyter/translators/full.py to prefix internal
  cross-document links with configured tojupyter_urlpath
- In-page anchors (starting with #) remain as local references
- External URLs (http://, https://) are not modified
- Added comprehensive test suite in tests/urlpath/
- Added nox session 'test-urlpath' to validate behavior
- Tests verify cross-document links are prefixed and in-page anchors are local

Fixes issue where exercise solution links were using local anchors
instead of full URLs when tojupyter_urlpath was configured.
- Added tojupyter_drop_raw_html config option (default: True)
- Drops raw HTML blocks (Thebe config, scripts, etc.) from notebooks
- Prevents web-specific content from cluttering notebook output
- Modified sphinx_tojupyter/__init__.py to register config
- Modified translators/code.py to read config with fallback
- Modified translators/full.py visit_raw() to skip HTML when enabled
- Added comprehensive test suite in tests/raw_html/
- Added nox session 'test-raw-html' to validate behavior

Fixes issue where Thebe configuration appeared as extra markdown
cells at the end of generated notebooks.

Resolves: THEBE_ISSUE.md reported by user
Default behavior: Drop raw HTML (clean notebooks)
Override: Set tojupyter_drop_raw_html = False to preserve HTML
- Tests tojupyter_drop_raw_html configuration with real Thebe blocks
- Includes comparison test (drop vs passthrough modes)
- Documents behavior with sphinx-thebe extension
- Provides regression testing for raw HTML dropping

Tests verify:
- Default (drop_raw_html=True) removes <script> tags
- Passthrough (drop_raw_html=False) preserves HTML
- Simulates sphinx-thebe configuration injection

Addresses testing gap identified in PR #149 review
- Documents all 15 active configuration options
- Includes type, default, status, and purpose for each
- Notes options removed in v1.0 refactor
- Highlights recent fixes (tojupyter_urlpath, tojupyter_drop_raw_html)
- Provides migration guidance from v0.x

Useful reference for reviewing v1.0 configuration alignment
@mmcky
Copy link
Contributor Author

mmcky commented Nov 19, 2025

@HumphreyYang I have uploaded the ipynb build artifacts from QuantEcon/continuous_time_mcs#149

jupyter.zip from continuous_time_mcs

When you have time would you mind to cast an eye over them and let me know if you spot any errors.

This PR is a massive re-write of this extension to solely focus on Jupyter notebook generation for sphinx.

- Replace builder() and builder.warn() calls with builder.logger.warning()
  to fix TypeError and AttributeError in writer.py and translators/full.py
- Update test_config_validation.py to use tojupyter_* config options
  and add assertions to verify config values affect notebook output
- Fix documentation to reference correct tojupyter_* option names in
  config-example.md and config-extension-notebooks.md
- Add explicit config option rename table to MIGRATION.md
- Change default fallback from 'code' to 'all' to match config default

Addresses code review feedback on PR #62
@mmcky
Copy link
Contributor Author

mmcky commented Nov 19, 2025

PR Description Updated ✅

I've updated the PR description to ensure accuracy and completeness:

Key Changes Made:

  1. ✅ Fixed Configuration Count

    • Corrected from "14 options (70% reduction)" to "15 options (68% reduction)"
    • Accounts for the new tojupyter_drop_raw_html option added in this PR
  2. ✅ Added Configuration Rename Documentation

    • Added comprehensive "Configuration Option Renames" section
    • Includes complete mapping table (jupyter_*tojupyter_*)
    • Before/after code examples for easy migration
    • This breaking change now prominently documented
  3. ✅ Documented Recent Code Review Fixes

    • Added "Recent Code Review Fixes" section for commit 30ccab5
    • Documents resolution of 3 blocking issues:
      • Documentation config references fixed
      • Config validation tests corrected
      • Sphinx API compatibility ensured (builder.warn()builder.logger.warning())
  4. ✅ Enhanced "What's New" Section

    • Added: Improved error handling with modern Sphinx logger API
    • Added: Config options renamed to tojupyter_* prefix for better namespacing

All updates ensure the PR description accurately reflects the current state of the v1.0 refactor branch.

- Remove CONFIG_OPTIONS_SUMMARY.md (temporary working document)
- Remove test-output.log (should be gitignored per .gitignore)
- Keep documented planning files (REFACTOR-PLANNING.md, FUTURE-DEVELOPMENT-PLAN.md, TESTING.md)
- Adheres to copilot-instructions.md policy: no summary artifacts
@mmcky
Copy link
Contributor Author

mmcky commented Nov 19, 2025

PR Description Corrected ✅

Issue Found: The previous description incorrectly stated that config options were renamed from jupyter_* to tojupyter_*.

Correction: The tojupyter_* prefix has been used since at least v0.5.0. The actual breaking change in v1.0 is the removal of 32 config options (from 47 → 15), not a rename.

What Changed:

  • ❌ Removed the incorrect "Configuration Option Renames" section with the false jupyter_*tojupyter_* mapping table
  • ✅ Updated "Configuration Changes" section to accurately describe what was removed
  • ✅ Listed the 13 retained options (all keeping their tojupyter_* names from v0.6)
  • ✅ Added the 1 new option: tojupyter_drop_raw_html

Migration Impact:
Users upgrading from v0.6 need to remove deleted options from their conf.py, but they do not need to rename any options.

@HumphreyYang
Copy link
Member

Many thanks @mmcky! I will take look at each of these notebooks.

@HumphreyYang
Copy link
Member

HumphreyYang commented Nov 19, 2025

Hi Matt,

I read all the notebooks and here is what I found:

  • The cross-lecture referencing leads to a broken url.

For example:

  • In ergodicity.ipynb the link to other lectures are broken:

    image
    [again](https://continuous-time-mcs.quantecon.org/kolmogorov_fwd.ipynb#solvode) 
    
  • the links to equation lambdafromq, kfromqxx and algorithm ejc_algo outside of this lecture are rendered as broken links

To see this, let $ (\lambda, K) $ be the jump chain pair constructed from $ Q $ via
[(7.8)](https://continuous-time-mcs.quantecon.org/uc_mc_semigroups.ipynb#equation-lambdafromq), [(7.9)](https://continuous-time-mcs.quantecon.org/uc_mc_semigroups.ipynb#equation-kfromqxx) and [(7.10)](https://continuous-time-mcs.quantecon.org/uc_mc_semigroups.ipynb#equation-kfromqxy).
Let $ (Y_k) $ and $ (J_k) $ be the embedded jump chain and jump sequence generated
by [Algorithm 4.1](https://continuous-time-mcs.quantecon.org/kolmogorov_bwd.ipynb#ejc_algo), with $ Y_0 = u $.
  • the link to the reference list is also broken for those references:
See, for example, Proposition 3.1.2 of [[Lasota and Mackey, 1994](https://continuous-time-mcs.quantecon.org/zreferences.ipynb#id3)] or Lemma 8.2.3 of [[Stachurski, 2009](https://continuous-time-mcs.quantecon.org/zreferences.ipynb#id4)].

Potential Solution: perhaps we can direct those links to our lectures?

  • The destination tags to Theorem, Corollary, Lemma, and exercises within the lecture are not generated:

For example,

  • In ergodicity.ipynb this link is broken:

    [Corollary 8.1](#perimposs) shows that periodicity is not a concern for irreducible
    continuous time Markov chains.
    

    because there is no destination tag <a id='perimposs'></a> attached to Corollary 8.1.

  • ## Solution to[ Exercise 8.2](#ergodicity-ex-2) is also broken because there is no destination tag <a id='ergodicity-ex-2'></a> attached to Exercise 8.2

Potential Solution: I think adding the destination tags for those environments will fix all of them!

Other lectures have similar issues but ergodicity.ipynb will be the best testing ground as it contains all cases of cross-referencing!

Other than those, everything else looks perfect!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants