Disallow package docs except for latest

[oraNod]

The purpose of this change is to disallow crawlers from all Ansible package docs versions except for the latest version, currently 13. We want to do this so that search results return for latest versions.

[samccann]

lgtm but we could go all the way up to Ansible 10 (and soon Ansible 12).

[samccann]

lgtm

[oraNod]

@samccann I've been rethinking disallowing the supported package versions.

Consider the RTD docs about version states: https://docs.readthedocs.com/platform/latest/versions.html#version-states

The default behaviour is to disallow hidden versions in robots.txt. Only active versions are indexed.

We need to explicitly declare whatever rules are needed because I don't think subprojects get their own robots.txt and we're using a custom one. But I think we should consider allowing any of the versions that are in the flyout widget on RTD, which follows the same pattern as the supported branches in the source repo.

[oraNod]

Here are the versions on the flyout widget:

This corresponds to the branches in https://github.com/ansible/ansible-documentation/blob/2aada29638a9ad769e3a8c7d66868a4408e0d9b7/.github/workflows/pip-compile-dev.yml

@samccann What do you think about hiding the devel build on RTD? That build will still be available but do we want devel in the flyout if we're disallowing it in robots.txt?

[oraNod]

Maybe if we hide devel in the flyout menu we can also remove this from the banner:

You are reading the devel version of the Ansible documentation - this version is not guaranteed stable. Use the version selection to the left if you want the latest (stable) released version.

[gundalow]

Not to make this more complex, though I agree we should think about /devel/ vs /latest/.
I could extend my RTD Bulk Fix script to point to /latest/ on all links that exist. Once case I can think we might want /devel/ is links to roadmaps.

I've marked this PR as draft till Don had decided what he wants to do here.