diff options
| author | Rémy Coutable <remy@rymai.me> | 2016-09-26 10:45:30 +0000 |
|---|---|---|
| committer | Rémy Coutable <remy@rymai.me> | 2016-09-26 10:45:30 +0000 |
| commit | ebc4299d8c025601a0e6ca8066252b3132349291 (patch) | |
| tree | d29a9245d1ac6934231075c9432fd1f7074c32b0 | |
| parent | 477f0309868bb7adcbb7901a7d2f13913ca9b4ba (diff) | |
| parent | f1df7b1bc223a1faad266658169095c56a08b31a (diff) | |
| download | gitlab-ce-ebc4299d8c025601a0e6ca8066252b3132349291.tar.gz | |
Merge branch 'robots-snippets' into 'master'
Update robots.txt disallow to allow users/groups starting with "s"
## What does this MR do?
Update robots.txt "Disallow: /s" line to "Disallow: /s/"
## Why was this MR needed?
Currently any Project that is under a username or group name that starts with "s" cannot be indexed because of this disallow line.
For example on https://gitlab.com looking at the projects with the most stars there is this project: https://gitlab.com/somerobots/Trident
If you do a Google search for "site:gitlab.com Trident", the project comes up but with "A description for this result is not available because of this site's robots.txt"
Adding the trailing slash will still disallow all snippet short urls, but start allowing projects where the URI begins with "s"
See merge request !6481
| -rw-r--r-- | CHANGELOG | 1 | ||||
| -rw-r--r-- | public/robots.txt | 2 |
2 files changed, 2 insertions, 1 deletions
diff --git a/CHANGELOG b/CHANGELOG index 61e3ed748c9..463b07b635e 100644 --- a/CHANGELOG +++ b/CHANGELOG @@ -2,6 +2,7 @@ Please view this file on the master branch, on stable branches it's out of date. v 8.13.0 (unreleased) - Speed-up group milestones show page + - Fix robots.txt disallowing access to groups starting with "s" (Matt Harrison) v 8.12.2 (unreleased) - Fix Import/Export not recognising correctly the imported services. diff --git a/public/robots.txt b/public/robots.txt index 334f4c03533..7d69fad59d1 100644 --- a/public/robots.txt +++ b/public/robots.txt @@ -23,7 +23,7 @@ Disallow: /groups/*/edit Disallow: /users # Global snippets -Disallow: /s +Disallow: /s/ Disallow: /snippets/new Disallow: /snippets/*/edit Disallow: /snippets/*/raw |