Prevent all robots from scanning the site

- Amend robots.txt to exclude all robots from the entire server - Add a comment making it clear to everyone that the directives from the robots.txt file disallow all content on the site from being crawled
Adroop · Dec 9, 2015 · 278f44e · 278f44e
1 parent ab345e0
commit 278f44e
Show file tree

Hide file tree

Showing 2 changed files with 6 additions and 2 deletions.
diff --git a/prod/robots.txt b/prod/robots.txt
@@ -1,3 +1,5 @@
 # http://www.robotstxt.org
+
+# Disallow crawling of all content
 User-agent: *
-Disallow:
+Disallow: /
diff --git a/public/robots.txt b/public/robots.txt
@@ -1,3 +1,5 @@
 # http://www.robotstxt.org
+
+# Disallow crawling of all content
 User-agent: *
-Disallow:
+Disallow: /