Add latest changes from gitlab-org/gitlab@master

2 files changed, 27 insertions, 18 deletions

diff --git a/doc/development/testing_guide/ci.md b/doc/development/testing_guide/ci.md
index d9f66a827de..5bdd0a69d7f 100644
--- a/doc/development/testing_guide/ci.md
+++ b/doc/development/testing_guide/ci.md
@@ -4,27 +4,24 @@
 
 Our current CI parallelization setup is as follows:
 
-1. The `knapsack` job in the prepare stage that is supposed to ensure we have a
-   `knapsack/${CI_PROJECT_NAME}/rspec_report-master.json` file:
-   - The `knapsack/${CI_PROJECT_NAME}/rspec_report-master.json` file is fetched
-     from S3, if it's not here we initialize the file with `{}`.
-1. Each `rspec x y` job are run with `knapsack rspec` and should have an evenly
-   distributed share of tests:
-   - It works because the jobs have access to the
-     `knapsack/${CI_PROJECT_NAME}/rspec_report-master.json` since the "artifacts
-     from all previous stages are passed by default".
+1. The `retrieve-tests-metadata` job in the `prepare` stage ensures we have a
+   `knapsack/report-master.json` file:
+   - The `knapsack/report-master.json` file is fetched from S3, if it's not here
+     we initialize the file with `{}`.
+1. Each `[rspec|rspec-ee] [unit|integration|system|geo] n m` job are run with
+   `knapsack rspec` and should have an evenly distributed share of tests:
+   - It works because the jobs have access to the `knapsack/report-master.json`
+     since the "artifacts from all previous stages are passed by default".
    - the jobs set their own report path to
-     `KNAPSACK_REPORT_PATH=knapsack/${CI_PROJECT_NAME}/${JOB_NAME[0]}_node_${CI_NODE_INDEX}_${CI_NODE_TOTAL}_report.json`.
+     `"knapsack/${TEST_TOOL}_${TEST_LEVEL}_${DATABASE}_${CI_NODE_INDEX}_${CI_NODE_TOTAL}_report.json"`.
    - if knapsack is doing its job, test files that are run should be listed under
      `Report specs`, not under `Leftover specs`.
-1. The `update-knapsack` job takes all the
-   `knapsack/${CI_PROJECT_NAME}/${JOB_NAME[0]}_node_${CI_NODE_INDEX}_${CI_NODE_TOTAL}_report.json`
-   files from the `rspec x y` jobs and merge them all together into a single
-   `knapsack/${CI_PROJECT_NAME}/rspec_report-master.json` file that is then
-   uploaded to S3.
-
-After that, the next pipeline will use the up-to-date
-`knapsack/${CI_PROJECT_NAME}/rspec_report-master.json` file.
+1. The `update-tests-metadata` job (which only runs on scheduled pipelines for
+   [the canonical project](https://gitlab.com/gitlab-org/gitlab) takes all the
+   `knapsack/rspec*_pg_*.json` files and merge them all together into a single
+   `knapsack/report-master.json` file that is then uploaded to S3.
+
+After that, the next pipeline will use the up-to-date `knapsack/report-master.json` file.
 
 ## Monitoring
 
diff --git a/doc/development/testing_guide/end_to_end/best_practices.md b/doc/development/testing_guide/end_to_end/best_practices.md
index 2200069ecfd..042879b47aa 100644
--- a/doc/development/testing_guide/end_to_end/best_practices.md
+++ b/doc/development/testing_guide/end_to_end/best_practices.md
@@ -53,3 +53,15 @@ In summary:
 
 - **Do**: Split tests across separate files, unless the tests share expensive setup.
 - **Don't**: Put new tests in an existing file without considering the impact on parallelization.
+
+## Limit the use of `before(:all)` hook
+
+Limit the use of `before(:all)` to perform setup tasks with only API calls, non UI operations
+or basic UI operations such as login.
+
+We use [`capybara-screenshot`](https://github.com/mattheworiordan/capybara-screenshot) library to automatically save screenshots on failures.
+This library [saves the screenshots in the RSpec's `after` hook](https://github.com/mattheworiordan/capybara-screenshot/blob/master/lib/capybara-screenshot/rspec.rb#L97).
+[If there is a failure in `before(:all)`, the `after` hook is not called](https://github.com/rspec/rspec-core/pull/2652/files#diff-5e04af96d5156e787f28d519a8c99615R148) and so the screenshots are not saved.
+
+Given this fact, we should limit the use of `before(:all)` to only those operations where a screenshot is not
+necessary in case of failure and QA logs would be enough for debugging.