Fix //torchx/github/docs:doctest on devservers and rid a few warnings. (#1017)

Summary:

Adds documentation and removes a few annoying warnings when building torchx (staticdocs) docs in a devserver.

NOTE: The developer still has to manually install `pandoc` by running `dnf install -y pandoc` (this is documented in `//torchx/github/docs/TARGETS`)

Reviewed By: andywag

Differential Revision: D71421188

Author: kiukchung

docs/Makefile

@@ -33,4 +33,4 @@ livehtml:
 	sphinx-autobuild --watch ../torchx --watch ../examples --re-ignore ".*(examples_.*|.new|source/.*(Dockerfile|.py))" "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
 
 papermill: html
-	./papermill.sh
+ 	./papermill.sh

docs/source/custom_components.md

@@ -14,15 +14,16 @@ jupyter:
 
 # Custom Components
 
-This is a guide on how to build a simple app and custom component spec
-and launch it via two different schedulers.
+This is a guide on how to build a simple app and custom component spec and
+launch it via two different schedulers.
 
 See the [Quickstart Guide](quickstart.md) for installation and basic usage.
 
-
 ## Builtins
 
-Before writing a custom component, check if any of the builtin components satisfy your needs. TorchX provides a number of builtin components with premade images. You can discover them via:
+Before writing a custom component, check if any of the builtin components
+satisfy your needs. TorchX provides a number of builtin components with premade
+images. You can discover them via:
 
 ```sh
 torchx builtins
@@ -35,7 +36,6 @@ you would any other component.
 torchx run utils.echo --msg "Hello :)"
 ```
 
-
 ## Hello World
 
 Lets start off with writing a simple "Hello World" python app. This is just a
@@ -71,8 +71,8 @@ if __name__ == "__main__":
     main(args.user)
 ```
 
-Now that we have an app we can write the component file for it. This
-function allows us to reuse and share our app in a user friendly way.
+Now that we have an app we can write the component file for it. This function
+allows us to reuse and share our app in a user friendly way.
 
 We can use this component from the `torchx` cli or programmatically as part of a
 pipeline.
@@ -99,8 +99,8 @@ def greet(user: str, image: str = "my_app:latest") -> specs.AppDef:
     )
 ```
 
-We can execute our component via `torchx run`. The
-`local_cwd` scheduler executes the component relative to the current directory.
+We can execute our component via `torchx run`. The `local_cwd` scheduler
+executes the component relative to the current directory.
 
 ```sh
 torchx run --scheduler local_cwd my_component.py:greet --user "your name"
@@ -137,13 +137,15 @@ We can then launch it on the local scheduler.
 torchx run --scheduler local_docker my_component.py:greet --image "my_app:latest" --user "your name"
 ```
 
-If you have a Kubernetes cluster you can use the [Kubernetes scheduler](schedulers/kubernetes.rst) to launch
-this on the cluster instead.
-
+If you have a Kubernetes cluster you can use the
+[Kubernetes scheduler](schedulers/kubernetes.rst) to launch this on the cluster
+instead.
 
 <!-- #md -->
+
 ```sh
 $ docker push my_app:latest
 $ torchx run --scheduler kubernetes my_component.py:greet --image "my_app:latest" --user "your name"
 ```
+
 <!-- #endmd -->

docs/source/ext/fbcode.py

@@ -9,21 +9,60 @@
 import os
 
 from docutils import nodes
+
+from docutils.parsers.rst import directives
 from sphinx.util.docutils import SphinxDirective
 from sphinx.util.nodes import nested_parse_with_titles
 
 
 class FbcodeDirective(SphinxDirective):
+    """
+    Includes the content of this directive if running in fbcode.
+
+    Used to add fb-specific (internal) documentation to display in
+    StaticDocs (https://www.internalfb.com/intern/staticdocs/torchx).
+
+    To exclude the contents in the directive (e.g. for oss-only documentation)
+    when building docs in fbcode, use the ``:exclude:`` option (see example below).
+
+    Usage:
+
+    ```
+    List of supported components:
+
+     * ``utils.python``
+
+     .. fbcode::
+        :exclude:
+
+        * ``utils.echo``
+
+     .. fbcode::
+
+        * ``fb.dist.hpc``
+    ```
+
+    In the example above, ``utils.echo`` will be listed only when building outside of fbcode.
+    Similarly ``fb.dist.hpc`` will be listed only when buildincg in fbcode.
+    """
+
     # this enables content in the directive
     has_content = True
+    option_spec = {
+        "exclude": directives.flag,  # if present, includes contents EXCEPT when in fbcode
+    }
 
     def run(self):
-        if "fbcode" not in os.getcwd():
+        exclude_in_fbcode = "exclude" in self.options
+        is_fbcode = "fbcode" in os.getcwd()
+
+        if is_fbcode ^ exclude_in_fbcode:
+            node = nodes.section()
+            node.document = self.state.document
+            nested_parse_with_titles(self.state, self.content, node)
+            return node.children
+        else:
             return []
-        node = nodes.section()
-        node.document = self.state.document
-        nested_parse_with_titles(self.state, self.content, node)
-        return node.children
 
 
 def setup(app):

docs/source/index.rst

@@ -111,17 +111,14 @@ Works With
 
 Examples
 ------------
-
 .. toctree::
    :maxdepth: 1
    :caption: Examples
 
-
    examples_apps/index
    examples_pipelines/index
 
 
-
 Components Library
 ---------------------
 .. _Components:

docs/source/schedulers/ray.rst

@@ -1,17 +1,20 @@
 Ray
 =================
 
-.. automodule:: torchx.schedulers.ray_scheduler
+.. fbcode::
+   :exclude:
 
-.. currentmodule:: torchx.schedulers.ray_scheduler
+   .. automodule:: torchx.schedulers.ray_scheduler
 
-.. autoclass:: RayScheduler
-   :members:
-   :show-inheritance:
+   .. currentmodule:: torchx.schedulers.ray_scheduler
 
-.. autofunction:: create_scheduler
-.. autofunction:: has_ray
-.. autofunction:: serialize
+   .. autoclass:: RayScheduler
+      :members:
+      :show-inheritance:
 
-.. autoclass:: RayJob
-   :members:
+   .. autofunction:: create_scheduler
+   .. autofunction:: has_ray
+   .. autofunction:: serialize
+
+   .. autoclass:: RayJob
+      :members: