added x-robots-tag header

[r3dg33k]

You're A Rockstar

Thank you for submitting a Pull Request (PR) to the Cheat Sheet Series.

🚩 If your PR is related to grammar/typo mistakes, please double-check the file for other mistakes in order to fix all the issues in the current cheat sheet.

Please make sure that for your contribution:

• In case of a new Cheat Sheet, you have used the Cheat Sheet template.
• All the markdown files do not raise any validation policy violation, see the policy.
• All the markdown files follow these format rules.
• All your assets are stored in the assets folder.
• All the images used are in the PNG format.
• Any references to websites have been formatted as [TEXT](URL)
• You verified/tested the effectiveness of your contribution (e.g., the defensive code proposed is really an effective remediation? Please verify it works!).
• The CI build of your PR pass, see the build status here.

If your PR is related to an issue, please finish your PR text with the following line:

This PR fixes issue #<REPLACE WITH ISSUE NUMBER>.

Thank you again for your contribution 😃

[jmanico]

I'm not 100% sure on this one, @mackowski ?

[mackowski]

hey @righettod can you help with this PR? :)

[jmanico]

I tend to be thumbs-up on this PR. It seems reasonable to want to AI crawlers of the day to leave us alone. :)

[mackowski]

@jmanico the problem is that the recommendation is from different header

Under X-Robots-Tag we will have:
'Disable sending this header. To remove the X-AspNetMvc-Version header, add the below line in Global.asax file.'

[mackowski]

Update the 'Recommendation' section

[jmanico]

I'm with you now @mackowski

[Copilot]

Pull Request Overview

This PR adds documentation for the X-Robots-Tag HTTP header to the HTTP Headers Cheat Sheet. The X-Robots-Tag header is used to control how search engines and AI bots index and display non-webpage content.

• Added a new section documenting the X-Robots-Tag header with description and example usage
• Included a note about crawler behavior and compliance
• Added a recommendation section for the header

---

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

[mackowski]

@jmanico @szh I updated this PR (with copilot) to fix previous issue

[szh]

I'm still not 100% convinced we should outright recommend disallowing crawlers without caveats. There are downsides. For example, I would argue we definitely want AI to ingest the cheat sheets so they will be more likely to give good recommendations for people who will inevitably ask LLMs for security advice.

[mackowski]

Agree, we should add this header but think how to describe the recommendation.
By the way - on the pages where I want to disable crawler it is better to implement solid AuthN/Z ;)

[jmanico]

Overall we want this recommendation for any data we want to protect from crawlers. It's totally fair that public data does not need this.