diff options
Diffstat (limited to 'doc/development/testing.md')
| -rw-r--r-- | doc/development/testing.md | 32 |
1 files changed, 21 insertions, 11 deletions
diff --git a/doc/development/testing.md b/doc/development/testing.md index 83269303005..4d5b90de6fc 100644 --- a/doc/development/testing.md +++ b/doc/development/testing.md @@ -150,6 +150,16 @@ always in-sync with the codebase. [GitLab QA]: https://gitlab.com/gitlab-org/gitlab-qa [part of GitLab Rails]: https://gitlab.com/gitlab-org/gitlab-ce/tree/master/qa +## Test for what should not be there + +This is particularly important for permission calls and might be called a +negative assertion: make sure only the bare minimum is returned and nothing else. + +See an issue about [leaking tokens] as an example of a vulnerability that is +captured by such a test. + +[leaking tokens]: https://gitlab.com/gitlab-org/gitlab-ce/issues/37948 + ## How to test at the correct level? As many things in life, deciding what to test at each level of testing is a @@ -292,7 +302,7 @@ range of inputs, might look like this: ```ruby describe "#==" do - using Rspec::Parameterized::TableSyntax + using RSpec::Parameterized::TableSyntax let(:project1) { create(:project) } let(:project2) { create(:project) } @@ -493,24 +503,24 @@ Here are some things to keep in mind regarding test performance: Our current CI parallelization setup is as follows: -1. The `knapsack` job in the prepare stage that is supposed to ensure we have a - `knapsack/${CI_PROJECT_NAME}/rspec_report-master.json` file: +1. The `retrieve-tests-metadata` job in the `prepare` stage ensures that we have + a `knapsack/${CI_PROJECT_NAME}/rspec_report-master.json` file: - The `knapsack/${CI_PROJECT_NAME}/rspec_report-master.json` file is fetched from S3, if it's not here we initialize the file with `{}`. -1. Each `rspec x y` job are run with `knapsack rspec` and should have an evenly - distributed share of tests: +1. Each `rspec-pg x y`/`rspec-mysql x y` job is run with `knapsack rspec` and + should have an evenly distributed share of tests: - It works because the jobs have access to the `knapsack/${CI_PROJECT_NAME}/rspec_report-master.json` since the "artifacts from all previous stages are passed by default". [^1] - - the jobs set their own report path to + - The jobs set their own report path to `KNAPSACK_REPORT_PATH=knapsack/${CI_PROJECT_NAME}/${JOB_NAME[0]}_node_${CI_NODE_INDEX}_${CI_NODE_TOTAL}_report.json`. - - if knapsack is doing its job, test files that are run should be listed under + - If knapsack is doing its job, test files that are run should be listed under `Report specs`, not under `Leftover specs`. -1. The `update-knapsack` job takes all the +1. The `update-tests-metadata` job takes all the `knapsack/${CI_PROJECT_NAME}/${JOB_NAME[0]}_node_${CI_NODE_INDEX}_${CI_NODE_TOTAL}_report.json` - files from the `rspec x y` jobs and merge them all together into a single - `knapsack/${CI_PROJECT_NAME}/rspec_report-master.json` file that is then - uploaded to S3. + files from the `rspec-pg x y`/`rspec-mysql x y`jobs and merge them all together + into a single `knapsack/${CI_PROJECT_NAME}/rspec_report-master.json` file that + is then uploaded to S3. After that, the next pipeline will use the up-to-date `knapsack/${CI_PROJECT_NAME}/rspec_report-master.json` file. The same strategy |