✨ NEW: Improve option parsing, add jupyter-book migration (#18)

Author: chrisjsewell

README.md

@@ -84,60 +84,100 @@ sections:
 - glob: subfolder/other*
 ```
 
-### Titles and Captions
+### File and URL titles
 
-By default, ToCs will use the initial header within a document as its title.
-
-With the `title` key you can set an alternative title for a document or URL in the ToC.
-Each part can also have a `caption`, e.g. for use in ToC side-bars:
+By default, the initial header within a `file` document will be used as its title in generated Table of Contents.
+With the `title` key you can set an alternative title for a document. and also for `url`:
 
 ```yaml
 root: intro
 parts:
-- caption: Part Caption
-  sections:
+- sections:
   - file: doc1
-    title: Document 1
+    title: Document 1 Title
   - url: https://example.com
-    title: Example Site
+    title: Example URL Title
 ```
 
-### Numbering
+### ToC tree (part) options
 
-You can automatically add numbers to all docs with a part by adding the `numbered: true` flag to it:
+Each part can be configured with a number of options (see also [sphinx `toctree` options](https://www.sphinx-doc.org/en/master/usage/restructuredtext/directives.html#directive-toctree)):
+
+- `caption` (string): A title for the whole the part, e.g. shown above the part in ToCs
+- `hidden` (boolean): Whether to show the ToC within (inline of) the document (default `False`).
+  By default it is appended to the end of the document, but see also the `tableofcontents` directive for positioning of the ToC.
+- `maxdepth` (integer): A maximum nesting depth to use when showing the ToC within the document.
+- `numbered` (boolean or integer): Automatically add numbers to all documents within a part (default `False`).
+  If set to `True`, all sub-parts will also be numbered based on nesting (e.g. with `1.1` or `1.1.1`),
+  or if set to an integer then the numbering will only be applied to that depth.
+- `reversed` (boolean): If `True` then the entries in the part will be listed in reverse order.
+  This can be useful when using `glob` sections.
+- `titlesonly` (boolean): If `True` then only the first heading in the document will be shown in the ToC, not other headings of the same level.
+
+These options can be set at the level of the part:
 
 ```yaml
 root: intro
 parts:
-- numbered: true
+- caption: Part Caption
+  hidden: False
+  maxdepth: 1
+  numbered: True
+  reversed: False
+  titlesonly: True
   sections:
   - file: doc1
-  - file: doc2
+    parts:
+    - titlesonly: True
+      sections:
+      - file: doc2
 ```
 
-You can also **limit the TOC numbering depth** by setting the `numbered` flag to an integer instead of `true`, e.g., `numbered: 3`.
-
-:::{note}
-By default, section numbering restarts for each `part`.
-If you want want this numbering to be continuous, check-out the [sphinx-multitoc-numbering extension](https://github.com/executablebooks/sphinx-multitoc-numbering).
-:::
+or, if you are using the shorthand for a single part, set options under an `options` key:
 
-### Defaults
+```yaml
+root: intro
+options:
+  caption: Part Caption
+  hidden: False
+  maxdepth: 1
+  numbered: True
+  reversed: False
+  titlesonly: True
+sections:
+- file: doc1
+  options:
+    titlesonly: True
+  sections:
+  - file: doc2
+```
 
-To have e.g. `numbered` added to all toctrees, set it under a `defaults` top-level key:
+You can also use the top-level `defaults` key, to set default options for all parts:
 
 ```yaml
-defaults:
-  numbered: true
 root: intro
+defaults:
+  titlesonly: True
+options:
+  caption: Part Caption
+  hidden: False
+  maxdepth: 1
+  numbered: True
+  reversed: False
 sections:
 - file: doc1
   sections:
   - file: doc2
-  - url: https://example.com
 ```
 
-Available keys: `numbered`, `titlesonly`, `reversed`
+:::{warning}
+`numbered` should not generally be used as a default, since numbering cannot be changed by nested parts, and sphinx will log a warning.
+:::
+
+:::{note}
+By default, section numbering restarts for each `part`.
+If you want want this numbering to be continuous, check-out the [sphinx-multitoc-numbering extension](https://github.com/executablebooks/sphinx-multitoc-numbering).
+:::
 
 ## Add a ToC to a page's content
 
@@ -159,6 +199,8 @@ MyST Markdown:
 
 Currently, only one `tableofcontents` should be used per page (all `toctree` will be added here), and only if it is a page with child/descendant documents.
 
+Note, this will override the `hidden` option set for a part.
+
 ## Excluding files not in ToC
 
 By default, Sphinx will build all document files, regardless of whether they are specified in the Table of Contents, if they:
@@ -313,9 +355,6 @@ intro:
 
 Questions / TODOs:
 
-- add migration CLI command for old jupyter-book format
-- Should `titlesonly` default to `True` (as in jupyter-book)?
-- nested numbered toctree not allowed (logs warning), so should be handled if `numbered: true` is in defaults
 - Add additional top-level keys, e.g. `appendices` (see https://github.com/sphinx-doc/sphinx/issues/2502) and `bibliography`
 - Using `external_toc_exclude_missing` to exclude a certain file suffix:
   currently if you had files `doc.md` and `doc.rst`, and put `doc.md` in your ToC,

codecov.yml

@@ -2,7 +2,7 @@ coverage:
   status:
     project:
       default:
-        target: 85%
+        target: 90%
         threshold: 0.2%
     patch:
       default:

docs/_toc.yml

@@ -1,9 +1,9 @@
+root: intro
 defaults:
-  numbered: 3
   titlesonly: false
-root: intro
 parts:
 - caption: Part 1
+  numbered: true
   sections:
   - file: doc1
   - file: doc2
@@ -12,5 +12,6 @@ parts:
     - url: https://example.com
       title: Example Link
 - caption: Part 2
+  numbered: true
   sections:
   - glob: subglobs/glob*

sphinx_external_toc/api.py

@@ -37,11 +37,16 @@ class TocItem:
             instance_of((GlobItem, FileItem, UrlItem)), instance_of(list)
         )
     )
-    caption: Optional[str] = attr.ib(None, validator=optional(instance_of(str)))
-    numbered: Union[bool, int] = attr.ib(False, validator=instance_of((bool, int)))
-    # TODO in jupyter-book titlesonly default is True, but why
-    titlesonly: bool = attr.ib(True, validator=instance_of(bool))
-    reversed: bool = attr.ib(False, validator=instance_of(bool))
+    caption: Optional[str] = attr.ib(
+        None, kw_only=True, validator=optional(instance_of(str))
+    )
+    hidden: bool = attr.ib(True, kw_only=True, validator=instance_of(bool))
+    maxdepth: int = attr.ib(-1, kw_only=True, validator=instance_of(int))
+    numbered: Union[bool, int] = attr.ib(
+        False, kw_only=True, validator=instance_of((bool, int))
+    )
+    reversed: bool = attr.ib(False, kw_only=True, validator=instance_of(bool))
+    titlesonly: bool = attr.ib(True, kw_only=True, validator=instance_of(bool))
 
     def files(self) -> List[str]:
         return [

sphinx_external_toc/cli.py

@@ -1,11 +1,15 @@
-from pathlib import PurePosixPath
+from pathlib import Path, PurePosixPath
 
 import click
 import yaml
 
 from sphinx_external_toc import __version__
 from sphinx_external_toc.parsing import create_toc_dict, parse_toc_yaml
-from sphinx_external_toc.tools import create_site_from_toc, create_site_map_from_path
+from sphinx_external_toc.tools import (
+    create_site_from_toc,
+    create_site_map_from_path,
+    migrate_jupyter_book,
+)
 
 
 @click.group(context_settings={"help_option_names": ["-h", "--help"]})
@@ -105,3 +109,17 @@ def create_toc(site_folder, extension, index, skip_match, guess_titles):
             site_map[docname].title = " ".join(words).capitalize()
     data = create_toc_dict(site_map)
     click.echo(yaml.dump(data, sort_keys=False, default_flow_style=False))
+
+
+@main.command("migrate")
+@click.argument("toc_file", type=click.Path(exists=True, file_okay=True))
+@click.option(
+    "-f",
+    "--format",
+    type=click.Choice(["jb-v0.10"]),
+    help="The format to migrate from.",
+)
+def migrate_toc(toc_file, format):
+    """Migrate a ToC from a previous revision."""
+    toc = migrate_jupyter_book(Path(toc_file))
+    click.echo(yaml.dump(toc, sort_keys=False, default_flow_style=False))

sphinx_external_toc/events.py

@@ -76,21 +76,26 @@ def parse_toc_to_env(app: Sphinx, config: Config) -> None:
             # we do not use `Path.glob` here, since it does not ignore hidden files:
             # https://stackoverflow.com/questions/49862648/why-do-glob-glob-and-pathlib-path-glob-treat-hidden-files-differently
             for path_str in glob.iglob(
-                str(Path(app.srcdir) / "**" / f"*[{suffix}]"), recursive=True
+                str(Path(app.srcdir) / "**" / f"*{suffix}"), recursive=True
             ):
                 path = Path(path_str)
                 if not path.is_file():
                     continue
                 posix = path.relative_to(app.srcdir).as_posix()
-                possix_no_suffix = posix[: -len(suffix)]
+                posix_no_suffix = posix[: -len(suffix)]
+                components = posix.split("/")
                 if not (
                     # files can be stored with or without suffixes
                     posix in site_map
-                    or possix_no_suffix in site_map
-                    # ignore anything already excluded
-                    or already_excluded(posix)
+                    or posix_no_suffix in site_map
+                    # ignore anything already excluded, we have to check against
+                    # the file path and all its sub-directory paths
+                    or any(
+                        already_excluded("/".join(components[: i + 1]))
+                        for i in range(len(components))
+                    )
                     # don't exclude docnames matching globs
-                    or any(patmatch(possix_no_suffix, pat) for pat in site_map.globs())
+                    or any(patmatch(posix_no_suffix, pat) for pat in site_map.globs())
                 ):
                     new_excluded.append(posix)
         if new_excluded:
@@ -201,13 +206,13 @@ def insert_toctrees(app: Sphinx, doctree: nodes.document) -> None:
         subnode.source = doctree.source
         subnode["entries"] = []
         subnode["includefiles"] = []
-        subnode["maxdepth"] = -1
+        subnode["maxdepth"] = toctree.maxdepth
         subnode["caption"] = toctree.caption
         # TODO this wasn't in the original code,
         # but alabaster theme intermittently raised `KeyError('rawcaption')`
         subnode["rawcaption"] = toctree.caption or ""
         subnode["glob"] = any(isinstance(entry, GlobItem) for entry in toctree.sections)
-        subnode["hidden"] = False if toc_placeholders else True
+        subnode["hidden"] = False if toc_placeholders else toctree.hidden
         subnode["includehidden"] = False
         subnode["numbered"] = toctree.numbered
         subnode["titlesonly"] = toctree.titlesonly

sphinx_external_toc/parsing.py

@@ -12,6 +12,15 @@
 GLOB_KEY = "glob"
 URL_KEY = "url"
 
+TOCTREE_OPTIONS = (
+    "caption",
+    "hidden",
+    "maxdepth",
+    "numbered",
+    "reversed",
+    "titlesonly",
+)
+
 
 class MalformedError(Exception):
     """Raised if toc file is malformed."""
@@ -50,7 +59,7 @@ def _parse_doc_item(
         # this is a shorthand for defining a single part
         if "parts" in data:
             raise MalformedError(f"Both 'sections' and 'parts' found: '{path}'")
-        parts_data = [{"sections": data["sections"]}]
+        parts_data = [{"sections": data["sections"], **data.get("options", {})}]
     elif "parts" in data:
         parts_data = data["parts"]
         if not (isinstance(parts_data, Sequence) and parts_data):
@@ -112,19 +121,11 @@ def _parse_doc_item(
                 sections.append(UrlItem(section[URL_KEY], section.get("title")))
 
         # generate toc key-word arguments
-        keywords = {}
-        for key in ("caption", "numbered", "titlesonly", "reversed"):
-            if key in part:
-                keywords[key] = part[key]
-            elif key in defaults:
+        keywords = {k: part[k] for k in TOCTREE_OPTIONS if k in part}
+        for key in defaults:
+            if key not in keywords:
                 keywords[key] = defaults[key]
 
-        # TODO this is a hacky fix for the fact that sphinx logs a warning
-        # for nested toctrees, see:
-        # sphinx/environment/collectors/toctree.py::TocTreeCollector::assign_section_numbers::_walk_toctree
-        if keywords.get("numbered") and path != "/":
-            keywords.pop("numbered")
-
         try:
             toc_item = TocItem(sections=sections, **keywords)
         except TypeError as exc: