diff options
| author | Ben Bodenmiller <bbodenmiller@hotmail.com> | 2015-10-04 23:39:24 -0700 |
|---|---|---|
| committer | Ben Bodenmiller <bbodenmiller@hotmail.com> | 2015-10-04 23:39:24 -0700 |
| commit | abf18abb52dbdb409d481854bde24ed880c76f56 (patch) | |
| tree | 26b821bc702e64f48b19957211ed2d8a756b695e /public/robots.txt | |
| parent | 0de7c83a78711601b40b5a739070da2e3af29b11 (diff) | |
| download | gitlab-ce-abf18abb52dbdb409d481854bde24ed880c76f56.tar.gz | |
allow crawling of commit page but not patch/diffs
Commit page has valuable information that search engines should be allowed to crawl however the .patch and .diff pages have no new information that is not on commit page
Diffstat (limited to 'public/robots.txt')
| -rw-r--r-- | public/robots.txt | 3 |
1 files changed, 2 insertions, 1 deletions
diff --git a/public/robots.txt b/public/robots.txt index 528f421083e..4f616c7f4c1 100644 --- a/public/robots.txt +++ b/public/robots.txt @@ -38,7 +38,8 @@ Disallow: /*/*/edit Disallow: /*/*/raw Disallow: /*/*/blame Disallow: /*/*/commits/*/* -Disallow: /*/*/commit +Disallow: /*/*/commit/*.patch +Disallow: /*/*/commit/*.diff Disallow: /*/*/compare Disallow: /*/*/branches/new Disallow: /*/*/tags/new |