Search engines pointing to draft docs

[c-clark]

Search engines (and potentially LLMs) are sometimes surfacing our draft server (draftdocs.acescentral.com) instead of the live docs.

For example, this AMF Use Cases draft appears in searches but is entirely missing the "Working Location" section found in the live documentation.

We should address this, potentially by updating the robots.txt on the draftdocs server to prevent indexing.

[scoopxyz]

Seems like a better approach would be to not deploy the dev/draft docs in this manner and instead utilize versioning of the docs via mike which our theme handles: https://squidfunk.github.io/mkdocs-material/setup/setting-up-versioning/

Then at least a user would see that they're looking at draft/dev docs and not latest version.

[scottdyer]

This was supposed to have been fixed in e692797

I let Google (Gemini) tell me about itself and how to remove previously indexed pages from search...

Waiting for Google to "forget" your staging site naturally can take anywhere from one week to three months, depending on how often Googlebot crawls your domain.

Since your staging site is already indexed, you have two main paths: the "Passive" way (waiting) and the "Active" way (forcing it out).

1. The Passive Way (Waiting)
   Now that you have the noindex meta tag in place, you’ve done the hard work. The next time Googlebot visits your staging site, it will see the noindex tag. It will immediately stop showing that page in search results.
   Timeline: Usually 7–30 days.
   Crucial Note: Do not block the staging site with a password yet. Google must be able to "see" the noindex tag to know it should delete the entry. If you block access entirely now, Google will just keep the old version in its index because it can't see the update.

2. The Active Way (Forcing Removal)
   If you want the staging site gone within 24 hours, you can use Google Search Console (GSC). This is the only way to "skip the line.

I'm no web expert, so if anyone knows of additional things I should be doing to prevent these from being indexed in the future, I welcome any help.

[scoopxyz]

Sounds like we still might be in the window for the normal timeline of cleanup, probably to be expected for low traffic sites like this.

I'd still advocate for versioned docs.