From 4c807c35d9e8ec569d143c432b06b2360f42c80e Mon Sep 17 00:00:00 2001
From: r3dg33k <2913793+r3dg33k@users.noreply.github.com>
Date: Wed, 13 Aug 2025 14:37:53 +0300
Subject: [PATCH 1/5] added x-robots-tag header

---
 cheatsheets/HTTP_Headers_Cheat_Sheet.md | 14 ++++++++++++++
 1 file changed, 14 insertions(+)

diff --git a/cheatsheets/HTTP_Headers_Cheat_Sheet.md b/cheatsheets/HTTP_Headers_Cheat_Sheet.md
index bdd705a81c..998e9495b7 100644
--- a/cheatsheets/HTTP_Headers_Cheat_Sheet.md
+++ b/cheatsheets/HTTP_Headers_Cheat_Sheet.md
@@ -239,6 +239,20 @@ MvcHandler.DisableMvcResponseHeader = true;
 
 - *NOTE*: Remember that attackers have other means of fingerprinting your tech stack.
 
+### X-Robots-Tag
+
+The HTTP `X-Robots-Tag` response header helps control how search engines and AI bots index and show files like PDFs, images, and other non-webpage content. It works like the robots meta tag but is set in the server response, giving more control and flexibility.
+
+```lang-none
+X-Robots-Tag: noindex, nofollow
+```
+
+- *NOTE*: Only well-behaved crawlers follow these rules, and they must first access the resource to read its headers and meta tags.
+  
+#### Recommendation
+
+Disable sending this header. To remove the `X-AspNetMvc-Version` header, add the below line in `Global.asax` file.
+
 ### X-DNS-Prefetch-Control
 
 The `X-DNS-Prefetch-Control` HTTP response header controls DNS prefetching, a feature by which browsers proactively perform domain name resolution on both links that the user may choose to follow as well as URLs for items referenced by the document, including images, CSS, JavaScript, and so forth.

From fd43e2ca483db9b19e4c5e691d481de8ab32ea29 Mon Sep 17 00:00:00 2001
From: mackowski <35339942+mackowski@users.noreply.github.com>
Date: Fri, 3 Oct 2025 14:37:16 +0200
Subject: [PATCH 2/5] Update cheatsheets/HTTP_Headers_Cheat_Sheet.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
---
 cheatsheets/HTTP_Headers_Cheat_Sheet.md | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)

diff --git a/cheatsheets/HTTP_Headers_Cheat_Sheet.md b/cheatsheets/HTTP_Headers_Cheat_Sheet.md
index 998e9495b7..ba198dbbb2 100644
--- a/cheatsheets/HTTP_Headers_Cheat_Sheet.md
+++ b/cheatsheets/HTTP_Headers_Cheat_Sheet.md
@@ -251,8 +251,13 @@ X-Robots-Tag: noindex, nofollow
   
 #### Recommendation
 
-Disable sending this header. To remove the `X-AspNetMvc-Version` header, add the below line in `Global.asax` file.
+Set the `X-Robots-Tag` header to control how search engines and bots index your content. For example:
+
+> `X-Robots-Tag: noindex, nofollow`
+
+This will prevent search engines from indexing the resource and following links on it. Adjust the value as needed for your use case (e.g., `index, follow`, `noarchive`, etc.).
 
+You can also use this header to control indexing of specific file types (like PDFs or images) by configuring your web server to send the header only for those resources.
 ### X-DNS-Prefetch-Control
 
 The `X-DNS-Prefetch-Control` HTTP response header controls DNS prefetching, a feature by which browsers proactively perform domain name resolution on both links that the user may choose to follow as well as URLs for items referenced by the document, including images, CSS, JavaScript, and so forth.

From 0d27cf0c5caaa8957e8de87a1e3ff12176b50882 Mon Sep 17 00:00:00 2001
From: mackowski <35339942+mackowski@users.noreply.github.com>
Date: Fri, 3 Oct 2025 14:39:20 +0200
Subject: [PATCH 3/5] Enhance HTTP Headers Cheat Sheet with new header info

Added explanation for X-DNS-Prefetch-Control header.
---
 cheatsheets/HTTP_Headers_Cheat_Sheet.md | 1 +
 1 file changed, 1 insertion(+)

diff --git a/cheatsheets/HTTP_Headers_Cheat_Sheet.md b/cheatsheets/HTTP_Headers_Cheat_Sheet.md
index ba198dbbb2..77817fd5cd 100644
--- a/cheatsheets/HTTP_Headers_Cheat_Sheet.md
+++ b/cheatsheets/HTTP_Headers_Cheat_Sheet.md
@@ -258,6 +258,7 @@ Set the `X-Robots-Tag` header to control how search engines and bots index your
 This will prevent search engines from indexing the resource and following links on it. Adjust the value as needed for your use case (e.g., `index, follow`, `noarchive`, etc.).
 
 You can also use this header to control indexing of specific file types (like PDFs or images) by configuring your web server to send the header only for those resources.
+
 ### X-DNS-Prefetch-Control
 
 The `X-DNS-Prefetch-Control` HTTP response header controls DNS prefetching, a feature by which browsers proactively perform domain name resolution on both links that the user may choose to follow as well as URLs for items referenced by the document, including images, CSS, JavaScript, and so forth.

From 390d59f0a13e6a2e5a70931096b57f921bff2f51 Mon Sep 17 00:00:00 2001
From: mackowski <35339942+mackowski@users.noreply.github.com>
Date: Fri, 10 Oct 2025 11:48:42 +0200
Subject: [PATCH 4/5] Enhance X-Robots-Tag section for clarity and formatting

Clarified the explanation of the X-Robots-Tag header and its usage. Improved formatting and notes for better readability.
---
 cheatsheets/HTTP_Headers_Cheat_Sheet.md | 26 ++++++++++++++++---------
 1 file changed, 17 insertions(+), 9 deletions(-)

diff --git a/cheatsheets/HTTP_Headers_Cheat_Sheet.md b/cheatsheets/HTTP_Headers_Cheat_Sheet.md
index 77817fd5cd..2e956d9a8a 100644
--- a/cheatsheets/HTTP_Headers_Cheat_Sheet.md
+++ b/cheatsheets/HTTP_Headers_Cheat_Sheet.md
@@ -241,23 +241,31 @@ MvcHandler.DisableMvcResponseHeader = true;
 
 ### X-Robots-Tag
 
-The HTTP `X-Robots-Tag` response header helps control how search engines and AI bots index and show files like PDFs, images, and other non-webpage content. It works like the robots meta tag but is set in the server response, giving more control and flexibility.
+The HTTP `X-Robots-Tag` response header controls how search engines and other automated crawlers index and display resources such as PDFs, images, and other non-HTML content.  
+It functions similarly to the `<meta name="robots">` tag, but is applied via the HTTP response header, allowing greater flexibility (e.g., for non-HTML files or server-wide rules).
 
-```lang-none
+```none
 X-Robots-Tag: noindex, nofollow
-```
+````
+
+* **Note:** Only compliant crawlers respect these directives, and they must still make an HTTP request to read the headers before deciding how to handle the content.
 
-- *NOTE*: Only well-behaved crawlers follow these rules, and they must first access the resource to read its headers and meta tags.
-  
 #### Recommendation
 
-Set the `X-Robots-Tag` header to control how search engines and bots index your content. For example:
+Use the `X-Robots-Tag` header to control crawler behavior:
+
+* For **private or sensitive content** you don’t want indexed:
+
+  > `X-Robots-Tag: noindex, nofollow`
+  > This prevents compliant search engines from indexing the resource or following links on it.
 
-> `X-Robots-Tag: noindex, nofollow`
+* For **public content** you want indexed and discoverable (e.g., documentation, datasets):
 
-This will prevent search engines from indexing the resource and following links on it. Adjust the value as needed for your use case (e.g., `index, follow`, `noarchive`, etc.).
+  > `X-Robots-Tag: index, follow`
+  > This allows search engines to index the resource and follow its links.
 
-You can also use this header to control indexing of specific file types (like PDFs or images) by configuring your web server to send the header only for those resources.
+You can also use other directives such as `noarchive`, `nosnippet`, or `noimageindex` depending on your needs.
+Server configuration can apply this header selectively — for example, only on specific file types (like PDFs or images).
 
 ### X-DNS-Prefetch-Control
 

From 80c9f776d2d2bd717791a209c30b6f6db444e68d Mon Sep 17 00:00:00 2001
From: mackowski <35339942+mackowski@users.noreply.github.com>
Date: Fri, 10 Oct 2025 11:51:57 +0200
Subject: [PATCH 5/5] Fix formatting and redundancy in HTTP Headers Cheat Sheet

Corrected formatting and removed redundant text in the HTTP Headers Cheat Sheet.
---
 cheatsheets/HTTP_Headers_Cheat_Sheet.md | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/cheatsheets/HTTP_Headers_Cheat_Sheet.md b/cheatsheets/HTTP_Headers_Cheat_Sheet.md
index 2e956d9a8a..d520fc579d 100644
--- a/cheatsheets/HTTP_Headers_Cheat_Sheet.md
+++ b/cheatsheets/HTTP_Headers_Cheat_Sheet.md
@@ -248,18 +248,18 @@ It functions similarly to the `<meta name="robots">` tag, but is applied via the
 X-Robots-Tag: noindex, nofollow
 ````
 
-* **Note:** Only compliant crawlers respect these directives, and they must still make an HTTP request to read the headers before deciding how to handle the content.
+- **Note:** Only compliant crawlers respect these directives, and they must still make an HTTP request to read the headers before deciding how to handle the content.
 
 #### Recommendation
 
 Use the `X-Robots-Tag` header to control crawler behavior:
 
-* For **private or sensitive content** you don’t want indexed:
+- For **private or sensitive content** you don’t want indexed:
 
   > `X-Robots-Tag: noindex, nofollow`
   > This prevents compliant search engines from indexing the resource or following links on it.
 
-* For **public content** you want indexed and discoverable (e.g., documentation, datasets):
+- For **public content** you want indexed and discoverable (e.g., documentation, datasets):
 
   > `X-Robots-Tag: index, follow`
   > This allows search engines to index the resource and follow its links.