Exclude pre-release documentation from the sitemap and mark it noindex

[bgravenorst]

Summary

Pre-release documentation is being advertised to search engines and AI crawlers as though it were current. Exclude the /development/* pages from sitemap.xml and mark them noindex, while keeping them reachable through the version dropdown.

Current behaviour

The site publishes two versions: the stable release, served at the site root, and a pre-release development version served under /development/.

The sitemap lists both:

curl -s https://docs.web3signer.consensys.io/sitemap.xml | grep -c "<loc>"              # 82
curl -s https://docs.web3signer.consensys.io/sitemap.xml | grep -c "<loc>.*development" # 40

So roughly half of the 82 advertised URLs are pre-release pages, and they largely mirror the stable pages.

Those pages are also fully indexable. https://docs.web3signer.consensys.io/development/ returns 200, carries a self-referencing canonical tag, and contains no noindex directive.

Why this is needed

There are two problems, and the second matters more than the first.

The indexing problem is that a near-complete duplicate of the documentation competes with the stable pages for the same queries, and inbound signals get split across two copies of each page.

The safety problem is that development documents unreleased behaviour. A reader who arrives on a /development/ page from a search result, or an AI assistant that cites one, can end up following instructions for behaviour that does not exist in the current release. For a signing service that holds validator keys, acting on documentation for the wrong version is a materially worse outcome than a ranking issue.

What is needed

• Exclude /development/* URLs from sitemap.xml.
• Add a noindex robots directive to pre-release pages.
• Keep the development version reachable through the version dropdown, so anyone who wants it can still navigate to it.
• Leave the stable pages served at the root unchanged.