Migrate docs from mdbook to docusaurus

[Keinberger]

Summary

This PR migrates the existing documentation from mdbook to Docusaurus. This includes syntax changes and structural updates.

---

🚨 DO NOT MERGE YET 🚨

Merging this PR before all other Docusaurus PRs are ready will break the docs deployment! Also, merging has to be done in correct order to ensure a smooth transition!

---

Note

1. This PR changes the markdown syntax from mdbook to Docusaurus. Please familiarize yourself with the new Docusaurus syntax.
2. The new markdown files have been moved to docs/src.
3. I separated the exported md files from non-exported md files that are intended as "internal docs". The exported files can be found inside of docs/src/external, internal docs inside of docs/src/internal.

Questions & Clarifications

I kept the md files inside of docs/src/internal to keep supporting mdbook, so that the internal docs can be hosted as an mdbook on 0xmiden.github.io/miden-node/. These files are files that are currently not exported to the Miden book.

That said, I believe it would be better to migrate these files to Docusaurus syntax as well. This would allow us to integrate the internal docs into the external documentation in the future, while also ensuring that developers only need to work with a single, consistent syntax (Docusaurus Markdown) instead of switching between mdBook and Docusaurus.

@bitwalker What do you think?

Changes

Syntax changes

As already mentioned before, this PR changes the markdown syntax of all docs markdown files.
Going forward all documentation will also be written using Docusaurus sytnax.

NOTE: Please review the updated Markdown files to understand the new format.

Local testing

To test docs locally before pushing changes:

1. Cd into docs:

cd docs/external

2. Install packages:

npm install

4. Start the local dev server:

npm run start:dev

Note: The index.md file might not appear in the correct sidebar order during local dev. In production deployments it will. To guarantee proper ordering, ensure index.md includes:

---
sidebar_position: 1
---

New Docs Architecture

The documentation system no longer relies on cron jobs or local scripts. Instead, changes are propagated through a CI-driven process. Here are the key changes:

1. Removal of cron task for initiating a bash script

Instead of relying on a cron task to update the docs, every repository now integrates a "trigger-docs-deploy.yml" workflow, which automatically triggers a rebuild and redeployment of the docs upon changes to next.

2. Versioning support

Upon Testnet deployments, a dedicated workflow inside of the miden-docs repository can be triggered to create a snapshot of the docs for that specific version. This workflow has to be triggered manually.

[bitwalker]

Thanks for doing this!

I've noted a couple of small changes I'd like to make. Could you also do the following in Makefile.toml:

1. Rename the docs task to internal-docs, and
   • Change it's category to Documentation
   • Update it's description to Builds the internal compiler documentation
2. Create a new task called docs just above internal-docs with the following definition:

[tasks.docs]
category = "Documentation"
description = "Opens the public compiler documentation"
dependencies = ["docusaurus"]
cwd = "./docs/external"
command = "npx"
args = ["start:dev"]

[tasks.docusaurus]
category = "Build"
cwd = "./docs/external"
command = "npx"
args = ["install"]

[bitwalker]

We probably want to only trigger this workflow if files in the docs directory change, no?

[Keinberger]

Absolutely agreed, will fix!

[bitwalker]

Let's pick a specific SHA revision of this action - depending on tags (which are mutable) is extremely risky when secrets are involved. We need to do this with our other uses, but I don't want to add any new GHA dependencies without selecting a revision that we've reviewed ourselves and vetted.

[Keinberger]

Absolutely agree with you on this one. I will pick a specific SHA ref and we can review it for safety.

[Keinberger]

How about we plan on taking this SHA ref?: a628c95fd17070f003ea24579a56e6bc89b25766

It's the latest version that was released yesterday.

[bitwalker]

@Keinberger You'll also have to make sure commit signing is turned on and sign your commits, let me know if you need any help getting that set up - I think all of our repos currently require it in order to merge, and while I can bypass it for this PR, you'll need to get that set up anyway, so we may as well do it now 🙂

[Keinberger]

Hey @bitwalker ,

I implemented the changes you asked for. Also implemented signed commits, my last commit is signed as you can see.

Furthermore, based on discussions I had with Philipp and Bobbin on the Docusaurus PR on the miden-base repository, I also included some more changes on this branch:

• _category_.json files are now in YAML format. That way we can add a comment in each file to explain the position parameter
• Added a small section in the README that explains the docs architecture (NPM dependencies and absorption into main docs page)
• Added a new build-docs.yml workflow to check md syntax for the exported docs

[bitwalker]

Shit, I just remembered I wasn't supposed to merge this 🤦 - @Keinberger will there be any issues once the other repos are migrated?

[Keinberger]

Yeah we have to revert the merge, otherwise the current Miden Books deployment will fail. Merging can only be done once all PRs are ready and everything is ready to be migrated to Docusaurus.

[Keinberger]

@bitwalker Next time when something like this happens, please go ahead and revert the merge immediately. Our Miden Book updated overnight and in that process the compiler docs were lost because it couldn't recognize the Docusaurus format (all other repos are still using mdbook).

I had to create #700 here and also manually revert the automated commit on the miden-docs repo to revert this change and make sure everything works again...