Update to sphinx documentation (#3078)

This updates our website to use sphinx for its documentation site. This
includes some new content around the SQL API, as well as our API
Javadoc, which we will be publishing ourselves for the first time in a
while. Publishing happens at the end of the release process (which I've
been able to test by first applying these changes to a test repo of
mine).

---------

Co-authored-by: FDB Relational Layer Authors <[email protected]>

Author: alecgrieser

Diff for: .github/workflows/mixed_mode_test.yml

@@ -40,7 +40,7 @@ jobs:
       - name: Checkout Main
         run: git checkout main
       - name: Update release notes
-        run: python build/publish-mixed-mode-results.py ${{ inputs.tag }} --release-notes docs/ReleaseNotes.md --commit --run-link ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}
+        run: python build/publish-mixed-mode-results.py ${{ inputs.tag }} --release-notes docs/sphinx/source/ReleaseNotes.md --commit --run-link ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}
       - name: Push release notes update
         run: git push
 

Diff for: .github/workflows/release.yml

@@ -10,14 +10,14 @@ jobs:
       checks: write
       contents: write
       packages: write
-      pages: write
       pull-requests: write
     steps:
       - name: Checkout sources
         uses: actions/[email protected]
         with:
           ssh-key: ${{ secrets.DEPLOY_KEY }}
       - name: Setup Base Environment
+        id: setup-base
         uses: ./actions/setup-base-env
       - name: Setup FDB
         uses: ./actions/setup-fdb
@@ -44,4 +44,31 @@ jobs:
           MAVEN_USER: ${{ secrets.MAVEN_USER }}
           MAVEN_PASSWORD: ${{ secrets.MAVEN_PASSWORD }}
 
-    # TODO: Publish documentation updates
+      # Build documentation.
+      - name: Cache Python Environment
+        uses: actions/cache@v4
+        with:
+          path: docs/sphinx/.venv
+          key: ${{ runner.os }}-sphinx-python-${{ steps.setup-base.outputs.python-version }}-${{ hashFiles('docs/sphinx/requirements.txt') }}
+      - name: Build Documentation Site
+        uses: ./actions/run-gradle
+        with:
+          gradle_command: documentationSite -PreleaseBuild=true
+      - name: Upload Documentation
+        id: doc_upload
+        uses: actions/upload-pages-artifact@v3
+        with:
+          path: docs/sphinx/.out/html/
+
+  deploy_docs:
+    runs-on: ubuntu-latest
+    needs: gradle
+    permissions:
+      pages: write
+      id-token: write
+    environment:
+      name: github-pages
+      url: ${{ steps.doc_upload.outputs.page_url }}
+    steps:
+      - name: Deploy Documentation
+        uses: actions/deploy-pages@v4

Diff for: .gitignore

@@ -44,6 +44,11 @@ atlassian-ide-plugin.xml
 .idea/scala_settings.xml
 .idea/shelf
 
+# Built during documentation
+/docs/sphinx/.venv
+/docs/sphinx/source/api/**/index.md
+*.diagram.svg
+
 # VSCode specific files/directories
 .vscode
 **/bin

Diff for: CONTRIBUTING.md

@@ -83,7 +83,7 @@ it is addressing and, upon merging of the PR into the main code line, will
 automatically mark the issue as resolved.
 
 If your pull request results in a user-visible change to the Record Layer, you should
-also update the [release notes](docs/ReleaseNotes.md). For most changes, it
+also update the [release notes](docs/sphinx/source/ReleaseNotes.md). For most changes, it
 is sufficient to fill in one of the bullets in the "next release" section of that
 document. You should include a short description of the change as well as filling in
 the issue number. The "next release" section is commented out, so the change won't

Diff for: actions/release-build-publish/action.yml

@@ -51,8 +51,12 @@ runs:
       id: push_updates
       shell: bash
       run: git push origin
+      # Continue the build (including downstream steps). If the push fails, we'll create a PR
+      continue-on-error: true
     - name: Create Merge PR if conflict
-      if: failure() && steps.push_updates.conclusion == 'failure'
+      # Only create the PR if we've otherwise been successful, but the push failed. Note that
+      # we're checking the .outcome of the push step, which is applied before continue-on-error.
+      if: success() && steps.push_updates.outcome == 'failure'
       uses: peter-evans/create-pull-request@bb88e27d3f9cc69c8bc689eba126096c6fe3dded
       id: pr_on_conflict
       with:
@@ -62,3 +66,8 @@ runs:
         sign-commits: true
         body: |
           Updates from release for version ${{ steps.get_version.outputs.version }}. Conflicts during the build prevented automatic updating. Please resolve conflicts by checking out the current branch, merging, and then deleting this branch.
+
+    # Creating the PR can change the current branch. Explicitly check out the tag here for downstream builds
+    - name: Revert to tag
+      shell: bash
+      run: git checkout "${{ steps.get_version.outputs.version }}"

Diff for: actions/setup-base-env/action.yml

@@ -1,5 +1,9 @@
 name: Setup Base Environment
 
+outputs:
+  python-version:
+    value: ${{ steps.setup-python.outputs.python-version }}
+
 runs:
   using: "composite"
   steps:
@@ -11,6 +15,7 @@ runs:
     - name: Setup Gradle
       uses: gradle/actions/setup-gradle@0bdd871935719febd78681f197cd39af5b6e16a6
     - name: Setup Python
+      id: setup-python
       uses: actions/setup-python@v5
       with:
         python-version: '3.13'

Diff for: build.gradle

@@ -240,8 +240,8 @@ subprojects {
                     linksOffline "https://developers.google.com/protocol-buffers/docs/reference/java/", "${packageListDir}/protobuf/"
 
                     // This is for dependent sub-projects so that their links to things in the Record Layer work
-                    linksOffline "https://static.javadoc.io/org.foundationdb/fdb-extensions/${project.version}/", "${packageListDir}/fdb-extensions/"
-                    linksOffline "https://static.javadoc.io/org.foundationdb/fdb-record-layer-core/${project.version}/", "${packageListDir}/fdb-record-layer-core/"
+                    linksOffline "https://foundationdb.github.io/fdb-record-layer/api/fdb-extensions/", "${packageListDir}/fdb-extensions/"
+                    linksOffline "https://foundationdb.github.io/fdb-record-layer/api/fdb-record-layer-core/", "${packageListDir}/fdb-record-layer-core/"
                 }
             }
             compileJava {
@@ -333,6 +333,8 @@ clean.doLast {
     }
 }
 
+apply from: 'gradle/sphinx.gradle'
+
 if (!JavaVersion.current().isJava8Compatible()) {
     throw new Exception("Java 8 is required to build fdb-record-layer")
 }

Diff for: build/update_release_notes.bash

@@ -31,7 +31,7 @@
 script_dir="$( dirname $0 )"
 success=0
 
-release_notes_file="${script_dir}/../docs/ReleaseNotes.md"
+release_notes_file="${script_dir}/../docs/sphinx/source/ReleaseNotes.md"
 
 if [[ -n "${GIT_BRANCH:-}" ]] ; then
     branch="${GIT_BRANCH#*/}"

Diff for: docs/_config.yml

@@ -0,0 +1,41 @@
+#!/bin/python
+#
+# generate_railroad_svg.py
+#
+# This source file is part of the FoundationDB open source project
+#
+# Copyright 2015-2025 Apple Inc. and the FoundationDB project authors
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+from os import walk, path
+from railroad import Diagram, Choice, Terminal, NonTerminal, OneOrMore, Sequence, Optional, Stack
+import railroad
+
+# Vertical space
+railroad.VS = 15
+
+# Arc radius
+railroad.AR = 12
+
+# Find all diagram files and generate svgs
+base_dir = path.dirname(path.abspath(__file__))
+for root, dirs, files in walk('.'):
+    for file in files:
+        if file.endswith('.diagram'):
+            with open(path.join(root, file), 'r') as diagram_file:
+                diagram = eval(diagram_file.read())
+                with open(path.join(root, file + ".svg"), 'w') as svg_file:
+                    diagram.writeStandalone(svg_file.write)
+

Diff for: docs/_includes/footer.html

@@ -0,0 +1,27 @@
+alabaster==1.0.0
+babel==2.16.0
+beautifulsoup4==4.12.3
+certifi==2024.7.4
+charset-normalizer==3.3.2
+docutils==0.21.2
+furo==2024.8.6
+idna==3.7
+imagesize==1.4.1
+Jinja2==3.1.4
+MarkupSafe==2.1.5
+myst-parser==4.0.0
+packaging==24.1
+Pygments==2.18.0
+railroad-diagrams==3.0.1
+requests==2.32.3
+snowballstemmer==2.2.0
+soupsieve==2.5
+sphinx==8.1.3
+sphinx-basic-ng==1.0.0b2
+sphinxcontrib-devhelp==2.0.0
+sphinxcontrib-htmlhelp==2.1.0
+sphinxcontrib-jsmath==1.0.1
+sphinxcontrib-qthelp==2.0.0
+sphinxcontrib-serializinghtml==2.0.0
+tomli==2.0.1
+urllib3==2.2.2

Diff for: docs/_layouts/default.html

@@ -14,7 +14,7 @@ To keep the guide simple, we will not cover:
 * Modifying a schema
 * Using advanced [index](Overview.md#secondary-indexes) and [query](Overview.md#querying-for-records) functionality
 
-### Setup
+## Setup
 
 Install Java JDK for version 8 or above; for this demo we are using JDK `1.8.0_181`.
 
@@ -52,7 +52,7 @@ Now we should add the Record Layer as a dependency of our project. The Record La
 are published to Maven Central, so we can declare the dependency by adding the following to our
 project's `build.gradle` file:
 
-```gradle
+```groovy
 repositories {
     mavenCentral()
     maven {
@@ -64,11 +64,12 @@ dependencies {
     implementation 'org.foundationdb:fdb-record-layer-core-pb3:VERSION_NUMBER'
 }
 ```
+
 Replace `VERSION_NUMBER` with a recent version of the artifact from Maven Central (See
 [mvnrepository](https://mvnrepository.com/artifact/org.foundationdb/fdb-record-layer-core-pb3)
 for a listing of published versions).
 
-### ProtoBuf Configuration
+## Protobuf Configuration
 
 Our sample project will use [Protocol Buffers](https://developers.google.com/protocol-buffers/) (protobuf)
 to define our record meta-data. First, since we are using Gradle, let's include the
@@ -77,7 +78,7 @@ which will allow us to add protobuf compilation as a step in our build process.
 Add this to the top of your `build.gradle`, ahead of the above `repositories` and
 `dependencies` added above:
 
-```gradle
+```groovy
 plugins {
     id 'java'
     id 'com.google.protobuf' version "0.8.19"
@@ -88,7 +89,7 @@ include `id'java'`.
 
 Additionally, add the following:
 
-```gradle
+```groovy
 protobuf {
     generatedFilesBaseDir = "${projectDir}/src/main/generated"
     protoc {
@@ -119,7 +120,7 @@ One last step might be necessary to configure your IDE of choice to discover the
 offer auto-complete suggestions. The additional generated source directory can be added to the list of Java sources
 by adding the following to the project's `build.gradle` file:
 
-```gradle
+```groovy
 sourceSets {
     main {
         java {
@@ -180,7 +181,7 @@ Finally, the Record Layer requires we have a `UnionDescriptor` message which con
 record types to be stored in our record store (here only `Order`). We must either set the `usage = UNION`
 option for this message or we can omit the option and instead name the message `RecordTypeUnion`.
 
-### Creating an Application
+## Creating an Application
 
 Run `./gradlew generateProto` to see that the above configuration is correct.
 You should see the generated code put into `src/main/generated/main/java/RecordLayerDemoProto.java`.
@@ -411,7 +412,7 @@ flower {
 price: 34
 ```
 
-### Summary
+## Summary
 
 We've covered creating a very simple demo app from scratch using the FDB Record Layer.
 We downloaded and started up an FDB server, created a new Java project, brought in the Record Layer dependency,

Diff for: docs/_layouts/page.html

@@ -132,7 +132,7 @@ The Apache Commons library has been removed as a dependency. There were a few lo
 
 ### Breaking Changes
 
-Support for the Protobuf 2 runtime has been removed as of this version. All artifacts now use Protobuf version 3. Note that the choice of Protobuf runtime version is distinct from the choice of Protobuf message syntax, and that users wishing to retain Protobuf 2 behavior can still achieve the same semantics (including [optional field behavior]()) as long as they specify the syntax on their Protobuf file as `proto2`. Note that the Maven artifacts using Protobuf version 3 used to be suffixed with `-pb3`. Existing Protobuf 3 users must remove that suffix from their dependency declarations (e.g., `fdb-record-layer-core-pb3` should now be `fdb-record-layer-core`).
+Support for the Protobuf 2 runtime has been removed as of this version. All artifacts now use Protobuf version 3. Note that the choice of Protobuf runtime version is distinct from the choice of Protobuf message syntax, and that users wishing to retain Protobuf 2 behavior can still achieve the same semantics (including [optional field behavior](Overview.md#indexing-and-querying-of-missing--null-values)) as long as they specify the syntax on their Protobuf file as `proto2`. Note that the Maven artifacts using Protobuf version 3 used to be suffixed with `-pb3`. Existing Protobuf 3 users must remove that suffix from their dependency declarations (e.g., `fdb-record-layer-core-pb3` should now be `fdb-record-layer-core`).
 
 Starting with version [3.4.455.0](#344550), the semantics of `UnnestedRecordType` were changed in response to [Issue #2512](https://github.com/FoundationDB/fdb-record-layer/issues/2512). It was identified that extraneous synthetic records were being produced when one of the children was empty. This did not match the semantics of `FanOut` expressions, and so the unnesting calculation was changed. This means that any index on an existing `UnnestedRecordType` requires rebuilding to clear out any such entries from older indexes.
 
@@ -2168,7 +2168,7 @@ The `FDBDatabase::getReadVersion()` method has been replaced with the `FDBRecord
 
 ### Breaking Changes
 
-The Guava version has been updated to version 27. Users of the [shaded variants](Shaded.html#Variants) of the Record Layer dependencies should not be affected by this change. Users of the unshaded variant using an older version of Guava as part of their own project may find that their own Guava dependency is updated automatically by their dependency manager.
+The Guava version has been updated to version 27. Users of the [shaded variants](Versioning.md#variants) of the Record Layer dependencies should not be affected by this change. Users of the unshaded variant using an older version of Guava as part of their own project may find that their own Guava dependency is updated automatically by their dependency manager.
 
 ### 2.7.79.0
 

Diff for: docs/sphinx/generate_railroad_svg.py

@@ -0,0 +1,31 @@
+# SQL Reference
+
+The FDB relational subproject provides a SQL API for interacting with Record Layer databases. This SQL API is currently under active development and is expected
+to change frequently. This reference is currently also a work in progress to be updated as the SQL engine develops.
+
+
+```{toctree}
+:caption: SQL
+:maxdepth: 2
+
+reference/sql_types
+reference/sql_commands
+reference/Functions
+```
+
+```{toctree}
+:caption: Data Model
+:maxdepth: 2
+
+reference/Databases_Schemas_SchemaTemplates
+reference/Tables
+reference/Indexes
+```
+
+```{toctree}
+:caption: Miscellaneous
+:maxdepth: 2
+
+reference/Direct_Access_Api
+reference/understanding_bitmap
+```