diff options
| author | Ben Bodenmiller <bbodenmiller@hotmail.com> | 2015-10-04 23:39:24 -0700 |
|---|---|---|
| committer | Ben Bodenmiller <bbodenmiller@hotmail.com> | 2015-10-04 23:39:24 -0700 |
| commit | abf18abb52dbdb409d481854bde24ed880c76f56 (patch) | |
| tree | 26b821bc702e64f48b19957211ed2d8a756b695e | |
| parent | 0de7c83a78711601b40b5a739070da2e3af29b11 (diff) | |
| download | gitlab-ce-abf18abb52dbdb409d481854bde24ed880c76f56.tar.gz | |
allow crawling of commit page but not patch/diffs
Commit page has valuable information that search engines should be allowed to crawl however the .patch and .diff pages have no new information that is not on commit page
| -rw-r--r-- | public/robots.txt | 3 |
1 files changed, 2 insertions, 1 deletions
diff --git a/public/robots.txt b/public/robots.txt index 528f421083e..4f616c7f4c1 100644 --- a/public/robots.txt +++ b/public/robots.txt @@ -38,7 +38,8 @@ Disallow: /*/*/edit Disallow: /*/*/raw Disallow: /*/*/blame Disallow: /*/*/commits/*/* -Disallow: /*/*/commit +Disallow: /*/*/commit/*.patch +Disallow: /*/*/commit/*.diff Disallow: /*/*/compare Disallow: /*/*/branches/new Disallow: /*/*/tags/new |