summaryrefslogtreecommitdiff
path: root/public/robots.txt
diff options
context:
space:
mode:
authorBen Bodenmiller <bbodenmiller@hotmail.com>2015-10-04 23:39:24 -0700
committerBen Bodenmiller <bbodenmiller@hotmail.com>2015-10-04 23:39:24 -0700
commitabf18abb52dbdb409d481854bde24ed880c76f56 (patch)
tree26b821bc702e64f48b19957211ed2d8a756b695e /public/robots.txt
parent0de7c83a78711601b40b5a739070da2e3af29b11 (diff)
downloadgitlab-ce-abf18abb52dbdb409d481854bde24ed880c76f56.tar.gz
allow crawling of commit page but not patch/diffs
Commit page has valuable information that search engines should be allowed to crawl however the .patch and .diff pages have no new information that is not on commit page
Diffstat (limited to 'public/robots.txt')
-rw-r--r--public/robots.txt3
1 files changed, 2 insertions, 1 deletions
diff --git a/public/robots.txt b/public/robots.txt
index 528f421083e..4f616c7f4c1 100644
--- a/public/robots.txt
+++ b/public/robots.txt
@@ -38,7 +38,8 @@ Disallow: /*/*/edit
Disallow: /*/*/raw
Disallow: /*/*/blame
Disallow: /*/*/commits/*/*
-Disallow: /*/*/commit
+Disallow: /*/*/commit/*.patch
+Disallow: /*/*/commit/*.diff
Disallow: /*/*/compare
Disallow: /*/*/branches/new
Disallow: /*/*/tags/new