SPICE-0018: pkldoc I/O Improvements

spices/SPICE-0018-pkldoc-io-improvements.adoc

@@ -0,0 +1,172 @@
+= `pkldoc` I/O Improvements
+
+* Proposal: link:./SPICE-0018-pkldoc-io-improvements.adoc[SPICE-0018]
+* Status: Accepted or Rejected
+* Implemented in: Pkl <version>
+* Category: Tooling
+
+== Introduction
+
+In order to re-generate pkldoc, too much metadata is read and written.
+This causes storage, memory, and I/O to balloon for large enough documentation sites.
+
+This SPICE describes a fix, reducing the amount of metadata changes needed when re-generating documentation.
+
+== Motivation
+
+The following problems currently exist when generating pkldoc:
+
+. In order to build the "known versions" and "known usages" metadata, the `package-data.json` file of every single package is required. This is an expensive operation, which grows linear to the number of package versions.
+. When generating runtime data, every single runtime data file is re-written.
+. When adding a new package version, the runtime data file for every historical version is changed. This results in too much I/O, and too much storage used (as the number of versions increases, the runtime data file of every single version increases).
+
+== Proposed Solution
+
+A series of changes will be made to reduce the amount of data written, as well as the amount of I/O needed.
+
+. Separate runtime data into package-level runtime data and package-version level runtime data.
+. Generate package-level runtime data by consuming the previously generated package-level runtime data.
+. Eliminate known subtype and known usage information at a cross-package level.
+. Generate the search index by consuming the previously generated search index.
+. Generate the main page by consuming only the "current" package-data.json files.
+
+[#_detailed_design]
+== Detailed design
+
+=== Runtime data changes
+
+Runtime data will be separated into package-verison level runtime data, and package level runtime data.
+
+The package level runtime data will be written to a directory called `_`, whereas package-version level runtime data will be written to a directory whose name matches the version.
+
+Here is an example folder structure, showing the `_` and `4.5.6` directory as the package level and package-version level runtime data files, respectively.
+
+[source]
+----
+data/my/package/
+├── 4.5.6
+│   ├── MyModule
+│   │   ├── MyClass.json
+│   │   └── index.json
+│   └── index.json
+└── _
+    ├── MyModule
+    │   ├── MyClass.json
+    │   └── index.json
+    └── index.json
+----
+
+There are three types of information captured:
+
+1. Known versions
+2. Known subtypes
+3. Known usages
+
+The known versions data will be recorded in the package level JSON runtime data.
+The other two kinds of data will be recorded in the package-version level runtime data.
+
+The known-subtypes and known-usages relationships will only be recorded inter-package.
+The known-usages at the package level will be preserved.
+
+When generating documentation for new package versions, any dependencies that are used which are also part of the site will have their known-usage information updated.
+
+==== Runtime data format changes
+
+The existing runtime data files will be used to generate new runtime data files.
+To improve machine readability of these files, they are changed from `.js` to `.json` files.
+
+Previous:
+
+.index.js
+[source,js]
+----
+runtimeData.links('known-versions', '[{ "text": "4.5.6", "href": "../4.5.6/index.html"}]');
+----
+
+New:
+
+.index.json
+[source.json]
+----
+{
+  "knownVersions": [
+    {
+      "text": "4.5.6",
+      "href": "../4.5.6/index.html"
+    }
+  ]
+}
+----
+
+==== HTML/JS changes
+
+The generated page HTML and `pkldoc.js` file will be changed to accommodate this.
+
+This extra script will be added to every HTML head:
+
+[source,html]
+----
+<script type="module">
+import perPackageData from "../../data/com.package1/_/index.json" with { type: "json" }
+import perPackageVersionData from "../../data/com.package1/1.2.3/index.json" with { type: "json" }
+
+runtimeData.knownVersions(perPackageData.knownVersions, "1.2.3");
+runtimeData.knownUsagesOrSubtypes("known-subtypes", perPackageVersionData.knownSubtypes);
+runtimeData.knownUsagesOrSubtypes("known-usages", perPackageVersionData.knownUsages);
+</script>
+----
+
+This script uses https://developer.mozilla.org/en-US/docs/Web/JavaScript/Guide/Modules#loading_non-javascript_resources[JavaScript modules] to load JSON as an import.
+This feature is supported in all major browsers.
+
+The pkldoc.js file will be changed accordingly.
+
+==== Generating new runtime data
+
+To generate new runtime data, the per-package information and per-package-version data need to be updated.
+
+To generate new package level runtime data, the existing package level runtime data is read and parsed.
+
+=== Search index generation changes
+
+Instead of parsing all existing `package-data.json` files, the existing search index is generated by consuming the previous search index.
+
+=== Main page generation changes
+
+The main page is generated by parsing the "current" `package-data.json` files.
+
+[#_data_migration]
+=== Data migration
+
+The structure of a pkldoc website changes.
+To record the data version, a schema version file is introduced at path `.pkldoc/VERSION`.
+
+If this file is missing, a pkldoc website is considered to be version 1.
+The changes reflected by this SPICE is captured as version 2.
+
+If running `pkldoc` against a version 1 website, an error is thrown.
+
+A new migration command is introduced, which migrates an existing pkldoc website from version 1 to version 2.
+
+This command is called using flag `--migrate`.
+
+[source,shell]
+----
+pkldoc --migrate
+----
+
+== Compatibility
+
+This is a breaking change, with a migration needed (see <<_data_migration>>).
+
+== Future directions
+
+N/A
+
+== Alternatives considered
+
+N/A
+
+== Acknowledgements
+
+N/A