summaryrefslogtreecommitdiff
path: root/doc/development/testing.md
diff options
context:
space:
mode:
authorSimon Knox <psimyn@gmail.com>2017-10-03 08:41:33 +1100
committerSimon Knox <psimyn@gmail.com>2017-10-03 08:41:33 +1100
commitd54983885bee00ff7b49c4352639beca4a082f46 (patch)
tree24c684554dfbfcc1a1fb3aa70b51f1ffdf041a48 /doc/development/testing.md
parent06b31461f34bac86d31d898e4f0e5b573d6b0345 (diff)
parent0781e956e4a4174494aa28b62b8dfc92a92e8e84 (diff)
downloadgitlab-ce-d54983885bee00ff7b49c4352639beca4a082f46.tar.gz
Merge remote-tracking branch 'origin' into 37229-mr-widget-status-icon
Diffstat (limited to 'doc/development/testing.md')
-rw-r--r--doc/development/testing.md30
1 files changed, 20 insertions, 10 deletions
diff --git a/doc/development/testing.md b/doc/development/testing.md
index 83269303005..d856b003353 100644
--- a/doc/development/testing.md
+++ b/doc/development/testing.md
@@ -150,6 +150,16 @@ always in-sync with the codebase.
[GitLab QA]: https://gitlab.com/gitlab-org/gitlab-qa
[part of GitLab Rails]: https://gitlab.com/gitlab-org/gitlab-ce/tree/master/qa
+## Test for what should not be there
+
+This is particularly important for permission calls and might be called a
+negative assertion: make sure only the bare minimum is returned and nothing else.
+
+See an issue about [leaking tokens] as an example of a vulnerability that is
+captured by such a test.
+
+[leaking tokens]: https://gitlab.com/gitlab-org/gitlab-ce/issues/37948
+
## How to test at the correct level?
As many things in life, deciding what to test at each level of testing is a
@@ -493,24 +503,24 @@ Here are some things to keep in mind regarding test performance:
Our current CI parallelization setup is as follows:
-1. The `knapsack` job in the prepare stage that is supposed to ensure we have a
- `knapsack/${CI_PROJECT_NAME}/rspec_report-master.json` file:
+1. The `retrieve-tests-metadata` job in the `prepare` stage ensures that we have
+ a `knapsack/${CI_PROJECT_NAME}/rspec_report-master.json` file:
- The `knapsack/${CI_PROJECT_NAME}/rspec_report-master.json` file is fetched
from S3, if it's not here we initialize the file with `{}`.
-1. Each `rspec x y` job are run with `knapsack rspec` and should have an evenly
- distributed share of tests:
+1. Each `rspec-pg x y`/`rspec-mysql x y` job is run with `knapsack rspec` and
+ should have an evenly distributed share of tests:
- It works because the jobs have access to the
`knapsack/${CI_PROJECT_NAME}/rspec_report-master.json` since the "artifacts
from all previous stages are passed by default". [^1]
- - the jobs set their own report path to
+ - The jobs set their own report path to
`KNAPSACK_REPORT_PATH=knapsack/${CI_PROJECT_NAME}/${JOB_NAME[0]}_node_${CI_NODE_INDEX}_${CI_NODE_TOTAL}_report.json`.
- - if knapsack is doing its job, test files that are run should be listed under
+ - If knapsack is doing its job, test files that are run should be listed under
`Report specs`, not under `Leftover specs`.
-1. The `update-knapsack` job takes all the
+1. The `update-tests-metadata` job takes all the
`knapsack/${CI_PROJECT_NAME}/${JOB_NAME[0]}_node_${CI_NODE_INDEX}_${CI_NODE_TOTAL}_report.json`
- files from the `rspec x y` jobs and merge them all together into a single
- `knapsack/${CI_PROJECT_NAME}/rspec_report-master.json` file that is then
- uploaded to S3.
+ files from the `rspec-pg x y`/`rspec-mysql x y`jobs and merge them all together
+ into a single `knapsack/${CI_PROJECT_NAME}/rspec_report-master.json` file that
+ is then uploaded to S3.
After that, the next pipeline will use the up-to-date
`knapsack/${CI_PROJECT_NAME}/rspec_report-master.json` file. The same strategy