diff --git a/spices/SPICE-0018-pkldoc-io-improvements.adoc b/spices/SPICE-0018-pkldoc-io-improvements.adoc new file mode 100644 index 0000000..bf557d8 --- /dev/null +++ b/spices/SPICE-0018-pkldoc-io-improvements.adoc @@ -0,0 +1,172 @@ += `pkldoc` I/O Improvements + +* Proposal: link:./SPICE-0018-pkldoc-io-improvements.adoc[SPICE-0018] +* Status: Accepted or Rejected +* Implemented in: Pkl +* Category: Tooling + +== Introduction + +In order to re-generate pkldoc, too much metadata is read and written. +This causes storage, memory, and I/O to balloon for large enough documentation sites. + +This SPICE describes a fix, reducing the amount of metadata changes needed when re-generating documentation. + +== Motivation + +The following problems currently exist when generating pkldoc: + +. In order to build the "known versions" and "known usages" metadata, the `package-data.json` file of every single package is required. This is an expensive operation, which grows linear to the number of package versions. +. When generating runtime data, every single runtime data file is re-written. +. When adding a new package version, the runtime data file for every historical version is changed. This results in too much I/O, and too much storage used (as the number of versions increases, the runtime data file of every single version increases). + +== Proposed Solution + +A series of changes will be made to reduce the amount of data written, as well as the amount of I/O needed. + +. Separate runtime data into package-level runtime data and package-version level runtime data. +. Generate package-level runtime data by consuming the previously generated package-level runtime data. +. Eliminate known subtype and known usage information at a cross-package level. +. Generate the search index by consuming the previously generated search index. +. Generate the main page by consuming only the "current" package-data.json files. + +[#_detailed_design] +== Detailed design + +=== Runtime data changes + +Runtime data will be separated into package-verison level runtime data, and package level runtime data. + +The package level runtime data will be written to a directory called `_`, whereas package-version level runtime data will be written to a directory whose name matches the version. + +Here is an example folder structure, showing the `_` and `4.5.6` directory as the package level and package-version level runtime data files, respectively. + +[source] +---- +data/my/package/ +├── 4.5.6 +│ ├── MyModule +│ │ ├── MyClass.json +│ │ └── index.json +│ └── index.json +└── _ + ├── MyModule + │ ├── MyClass.json + │ └── index.json + └── index.json +---- + +There are three types of information captured: + +1. Known versions +2. Known subtypes +3. Known usages + +The known versions data will be recorded in the package level JSON runtime data. +The other two kinds of data will be recorded in the package-version level runtime data. + +The known-subtypes and known-usages relationships will only be recorded inter-package. +The known-usages at the package level will be preserved. + +When generating documentation for new package versions, any dependencies that are used which are also part of the site will have their known-usage information updated. + +==== Runtime data format changes + +The existing runtime data files will be used to generate new runtime data files. +To improve machine readability of these files, they are changed from `.js` to `.json` files. + +Previous: + +.index.js +[source,js] +---- +runtimeData.links('known-versions', '[{ "text": "4.5.6", "href": "../4.5.6/index.html"}]'); +---- + +New: + +.index.json +[source.json] +---- +{ + "knownVersions": [ + { + "text": "4.5.6", + "href": "../4.5.6/index.html" + } + ] +} +---- + +==== HTML/JS changes + +The generated page HTML and `pkldoc.js` file will be changed to accommodate this. + +This extra script will be added to every HTML head: + +[source,html] +---- + +---- + +This script uses https://developer.mozilla.org/en-US/docs/Web/JavaScript/Guide/Modules#loading_non-javascript_resources[JavaScript modules] to load JSON as an import. +This feature is supported in all major browsers. + +The pkldoc.js file will be changed accordingly. + +==== Generating new runtime data + +To generate new runtime data, the per-package information and per-package-version data need to be updated. + +To generate new package level runtime data, the existing package level runtime data is read and parsed. + +=== Search index generation changes + +Instead of parsing all existing `package-data.json` files, the existing search index is generated by consuming the previous search index. + +=== Main page generation changes + +The main page is generated by parsing the "current" `package-data.json` files. + +[#_data_migration] +=== Data migration + +The structure of a pkldoc website changes. +To record the data version, a schema version file is introduced at path `.pkldoc/VERSION`. + +If this file is missing, a pkldoc website is considered to be version 1. +The changes reflected by this SPICE is captured as version 2. + +If running `pkldoc` against a version 1 website, an error is thrown. + +A new migration command is introduced, which migrates an existing pkldoc website from version 1 to version 2. + +This command is called using flag `--migrate`. + +[source,shell] +---- +pkldoc --migrate +---- + +== Compatibility + +This is a breaking change, with a migration needed (see <<_data_migration>>). + +== Future directions + +N/A + +== Alternatives considered + +N/A + +== Acknowledgements + +N/A \ No newline at end of file