diff options
Diffstat (limited to 'doc/development/testing_guide')
9 files changed, 216 insertions, 34 deletions
diff --git a/doc/development/testing_guide/best_practices.md b/doc/development/testing_guide/best_practices.md index e27d4911158..54d3f368bbd 100644 --- a/doc/development/testing_guide/best_practices.md +++ b/doc/development/testing_guide/best_practices.md @@ -358,6 +358,19 @@ a place to start. The most expensive examples here are in shared examples; any reductions generally have a larger impact as they are called in multiple places. +#### Top slow tests + +We collect information about tests duration in [`rspec_profiling_stats`](https://gitlab.com/gitlab-org/rspec_profiling_stats) project. The data is showed using GitLab Pages in this +[UI](https://gitlab-org.gitlab.io/rspec_profiling_stats/) + +With [issue](https://gitlab.com/gitlab-org/gitlab/-/issues/375983) we defined thresholds for tests duration that can act a guide. + +For tests that are not meeting the thresholds it is recommended to create issues and improve the tests duration. + +| Date | Feature tests | Controllers and Requests tests | Other | +| --- | --- | --- | --- | +| 2023-02-15 | 67.42 seconds | 44.66 seconds | 76.86 seconds | + #### Avoid repeating expensive actions While isolated examples are very clear, and help serve the purpose of specs as @@ -415,7 +428,7 @@ results are available, and not just the first failure. - Don't assert against the absolute value of a sequence-generated attribute (see [Gotchas](../gotchas.md#do-not-assert-against-the-absolute-value-of-a-sequence-generated-attribute)). - Avoid using `expect_any_instance_of` or `allow_any_instance_of` (see - [Gotchas](../gotchas.md#do-not-assert-against-the-absolute-value-of-a-sequence-generated-attribute)). + [Gotchas](../gotchas.md#avoid-using-expect_any_instance_of-or-allow_any_instance_of-in-rspec)). - Don't supply the `:each` argument to hooks because it's the default. - On `before` and `after` hooks, prefer it scoped to `:context` over `:all` - When using `evaluate_script("$('.js-foo').testSomething()")` (or `execute_script`) which acts on a given element, @@ -432,6 +445,12 @@ results are available, and not just the first failure. You must [set feature category metadata for each RSpec example](../feature_categorization/index.md#rspec-examples). +### Tests depending on EE license + +You can use `if: Gitlab.ee?` or `unless: Gitlab.ee?` on context/spec blocks to execute tests depending on whether running with `FOSS_ONLY=1`. + +Example: [SchemaValidator reads a different path depending on the license](https://gitlab.com/gitlab-org/gitlab/-/blob/7cdcf9819cfa02c701d6fa9f18c1e7a8972884ed/spec/lib/gitlab/ci/parsers/security/validators/schema_validator_spec.rb#L571) + ### Coverage [`simplecov`](https://github.com/colszowka/simplecov) is used to generate code test coverage reports. diff --git a/doc/development/testing_guide/contract/consumer_tests.md b/doc/development/testing_guide/contract/consumer_tests.md index 39cc34d6153..60ce71db79d 100644 --- a/doc/development/testing_guide/contract/consumer_tests.md +++ b/doc/development/testing_guide/contract/consumer_tests.md @@ -17,7 +17,7 @@ Then, populate it with the following function and parameters: - [`PactOptions`](#the-pactoptions-parameter) - [`PactFn`](#the-pactfn-parameter) -To learn more about how the contract test directory is structured, see the contract testing [test suite folder structure](index.md#test-suite-folder-structure). +For more information about how the contract test directory is structured, see [Test suite folder structure](index.md#test-suite-folder-structure). ### The `pactWith` function @@ -47,7 +47,7 @@ pactWith( ); ``` -To learn more about how to name the consumers and providers, see contract testing [naming conventions](index.md#naming-conventions). +For more information about how to name consumers and providers, see [Naming conventions](index.md#naming-conventions). ### The `PactFn` parameter diff --git a/doc/development/testing_guide/contract/provider_tests.md b/doc/development/testing_guide/contract/provider_tests.md index 3ce9f91a307..cb3aeae529d 100644 --- a/doc/development/testing_guide/contract/provider_tests.md +++ b/doc/development/testing_guide/contract/provider_tests.md @@ -12,7 +12,7 @@ This tutorial guides you through writing a provider test from scratch. It is a c Provider tests are quite simple. The goal is to set up the test data and then link that with the corresponding contract. Start by creating a file called `get_discussions_helper.rb` under `spec/contracts/provider/pact_helpers/project/merge_request`. Note that the files are called `helpers` to match how they are called by Pact in the Rake tasks, which are set up at the end of this tutorial. -To learn more about how the contract test directory is structured, see the contract testing [test suite folder structure](index.md#test-suite-folder-structure). +For more information about how the contract test directory is structured, see [Test suite folder structure](index.md#test-suite-folder-structure). ### The `service_provider` block @@ -48,7 +48,7 @@ module Provider end ``` -To learn more about how to name the consumers and providers, see contract testing [naming conventions](index.md#naming-conventions). +For more information about how to name consumers and providers, see [Naming conventions](index.md#naming-conventions). ## Configure the test app diff --git a/doc/development/testing_guide/end_to_end/index.md b/doc/development/testing_guide/end_to_end/index.md index 8369dcb0740..77ceb9fa546 100644 --- a/doc/development/testing_guide/end_to_end/index.md +++ b/doc/development/testing_guide/end_to_end/index.md @@ -97,6 +97,9 @@ This problem was discovered in <https://gitlab.com/gitlab-org/gitlab-qa/-/issues work-around was suggested in <https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/4717>. A feature proposal to segregate access control regarding running pipelines from ability to push/merge was also created at <https://gitlab.com/gitlab-org/gitlab/-/issues/24585>. +For more technical details on CI/CD setup and documentation on adding new test jobs to `package-and-test` pipeline, see +[`package_and_test` setup documentation](package_and_test_pipeline.md). + #### With merged results pipelines In a merged results pipeline, the pipeline runs on a new ref that contains the merge result of the source and target branch. @@ -256,8 +259,7 @@ Learn how to perform [tests that require special setup or consideration to run o ## How do you write tests? -In order to write new tests, you first need to learn more about GitLab QA -architecture. See the [documentation about it](https://gitlab.com/gitlab-org/gitlab-qa/blob/master/docs/architecture.md). +Before you write new tests, review the [GitLab QA architecture](https://gitlab.com/gitlab-org/gitlab-qa/blob/master/docs/architecture.md). After you've decided where to put [test environment orchestration scenarios](https://gitlab.com/gitlab-org/gitlab-qa/tree/master/lib/gitlab/qa/scenario) and [instance-level scenarios](https://gitlab.com/gitlab-org/gitlab-foss/tree/master/qa/qa/specs/features), take a look at the [GitLab QA README](https://gitlab.com/gitlab-org/gitlab/-/tree/master/qa/README.md), diff --git a/doc/development/testing_guide/end_to_end/package_and_test_pipeline.md b/doc/development/testing_guide/end_to_end/package_and_test_pipeline.md new file mode 100644 index 00000000000..b0257e7b02c --- /dev/null +++ b/doc/development/testing_guide/end_to_end/package_and_test_pipeline.md @@ -0,0 +1,134 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +--- + +# e2e:package-and-test + +The `e2e:package-and-test` child pipeline is the main executor of E2E testing for the GitLab platform. The pipeline definition has several dynamic +components to reduce the number of tests being executed in merge request pipelines. + +## Setup + +Pipeline setup consists of: + +- The `e2e-test-pipeline-generate` job in the `prepare` stage of the main GitLab pipeline. +- The `e2e:package-and-test` job in the `qa` stage, which triggers the child pipeline that is responsible for building the `omnibus` package and + running E2E tests. + +### e2e-test-pipeline-generate + +This job consists of two components that implement selective test execution: + +- The [`detect_changes`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/qa/tasks/ci.rake) Rake task determines which e2e specs should be executed + in a particular merge request pipeline. This task analyzes changes in a particular merge request and determines which specs must be executed. + Based on that, a `dry-run` of every [scenario](https://gitlab.com/gitlab-org/gitlab/-/tree/master/qa/qa/scenario/test) executes and determines if a + scenario contains any executable tests. Selective test execution uses [these criteria](index.md#selective-test-execution) to determine which specific + tests to execute. +- [`generate-e2e-pipeline`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/scripts/generate-e2e-pipeline) is executed, which generates a child + pipeline YAML definition file with appropriate environment variables. + +### e2e:package-and-test + +E2E test execution pipeline consists of several stages which all support execution of E2E tests. + +#### .pre + +This stage is responsible for the following tasks: + +- Fetching `knapsack` reports that support [parallel test execution](index.md#run-tests-in-parallel). +- Triggering downstream pipeline which builds the [`omnibus-gitlab`](https://gitlab.com/gitlab-org/omnibus-gitlab) Docker image. + +#### test + +This stage runs e2e tests against different types of GitLab configurations. The number of jobs executed is determined dynamically by +[`e2e-test-pipeline-generate`](package_and_test_pipeline.md#e2e-test-pipeline-generate) job. + +#### report + +This stage is responsible for [allure test report](index.md#allure-report) generation. + +## Adding new jobs + +Selective test execution depends on a set of rules present in every job definition. A typical job contains the following attributes: + +```yaml +variables: + QA_SCENARIO: Test::Integration::MyNewJob +rules: + - !reference [.rules:test:qa, rules] + - if: $QA_SUITES =~ /Test::Integration::MyNewJob/ + - !reference [.rules:test:manual, rules] +``` + +In this example: + +- `QA_SCENARIO: Test::Integration::MyNewJob`: name of the scenario class that is passed to the + [`gitlab-qa`](https://gitlab.com/gitlab-org/gitlab-qa/-/blob/master/docs/what_tests_can_be_run.md) executor. +- `!reference [.rules:test:qa, rules]`: main rule definition that is matched for pipelines that should execute all tests. For example, when changes to + `qa` framework are present. +- `if: $QA_SUITES =~ /Test::Integration::MyNewJob/`: main rule responsible for selective test execution. `QA_SUITE` is the name of the scenario + abstraction located in [`qa framework`](https://gitlab.com/gitlab-org/gitlab/-/tree/master/qa/qa/scenario/test). + + `QA_SUITE` is not the same as `QA_SCENARIO`, which is passed to the `gitlab-qa` executor. For consistency, it usually has the same name. `QA_SUITE` + abstraction class usually contains information on what tags to run and optionally some additional setup steps. +- `!reference [.rules:test:manual, rules]`: final rule that is always matched and sets the job to `manual` so it can still be executed on demand, + even if not set to execute by selective test execution. + +Considering example above, perform the following steps to create a new job: + +1. Create new scenario type `my_new_job.rb` in the [`integration`](https://gitlab.com/gitlab-org/gitlab-qa/-/tree/master/lib/gitlab/qa/scenario/test/integration) directory + of the [`gitlab-qa`](https://gitlab.com/gitlab-org/gitlab-qa) project and release new version so it's generally available. +1. Create new scenario `my_new_job.rb` in [`integration`](https://gitlab.com/gitlab-org/gitlab/-/tree/master/qa/qa/scenario/test/integration) directory of the + [`qa`](https://gitlab.com/gitlab-org/gitlab/-/tree/master/qa) framework. In the most simple case, this scenario would define RSpec tags that should be executed: + + ```ruby + module QA + module Scenario + module Test + module Integration + class MyNewJob < Test::Instance::All + tags :some_special_tag + end + end + end + end + end + ``` + +1. Add new job definition in the [`main.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/ci/package-and-test/main.gitlab-ci.yml) pipeline definition: + + ```yaml + ee:my-new-job: + extends: .qa + variables: + QA_SCENARIO: Test::Integration::MyNewJob + rules: + - !reference [.rules:test:qa, rules] + - if: $QA_SUITES =~ /Test::Integration::MyNewJob/ + - !reference [.rules:test:manual, rules] + ``` + +### Parallel jobs + +For selective execution to work correctly with job types that require running multiple parallel jobs, +a job definition typically must be split into parallel and selective variants. Splitting is necessary so that when selective execution +executes only a single spec, multiple unnecessary jobs are not spawned. For example: + +```yaml +ee:my-new-job-selective: + extends: .qa + variables: + QA_SCENARIO: Test::Integration::MyNewJob + rules: + - !reference [.rules:test:qa-selective, rules] + - if: $QA_SUITES =~ /Test::Integration::MyNewJob/ +ee:my-new-job: + extends: + - .parallel + - ee:my-new-job-selective + rules: + - !reference [.rules:test:qa-parallel, rules] + - if: $QA_SUITES =~ /Test::Integration::MyNewJob/ +``` diff --git a/doc/development/testing_guide/end_to_end/resources.md b/doc/development/testing_guide/end_to_end/resources.md index 37d354bf60c..f3e072ffa21 100644 --- a/doc/development/testing_guide/end_to_end/resources.md +++ b/doc/development/testing_guide/end_to_end/resources.md @@ -49,7 +49,7 @@ create the resource via the public GitLab API: - `#api_post_path`: The `POST` path to create a new resource. - `#api_post_body`: The `POST` body (as a Ruby hash) to create a new resource. -> Be aware that many API resources are [paginated](../../../api/index.md#pagination). +> Be aware that many API resources are [paginated](../../../api/rest/index.md#pagination). > If you don't find the results you expect, check if there is more that one page of results. Let's take the `Shirt` resource class, and add these three API methods: diff --git a/doc/development/testing_guide/end_to_end/style_guide.md b/doc/development/testing_guide/end_to_end/style_guide.md index 419942d6b8f..32d8bf339ed 100644 --- a/doc/development/testing_guide/end_to_end/style_guide.md +++ b/doc/development/testing_guide/end_to_end/style_guide.md @@ -8,6 +8,8 @@ info: To determine the technical writer assigned to the Stage/Group associated w This document describes the conventions used at GitLab for writing End-to-end (E2E) tests using the GitLab QA project. +Please note that this guide is an extension of the primary [testing standards and style guidelines](../index.md). If this guide defines a rule that contradicts the primary guide, this guide takes precedence. + ## `click_` versus `go_to_` ### When to use `click_`? diff --git a/doc/development/testing_guide/flaky_tests.md b/doc/development/testing_guide/flaky_tests.md index ef5b75d166f..f476815bf32 100644 --- a/doc/development/testing_guide/flaky_tests.md +++ b/doc/development/testing_guide/flaky_tests.md @@ -154,8 +154,11 @@ When a test frequently fails in `master`, create [a ~"failure::flaky-test" issue](https://about.gitlab.com/handbook/engineering/workflow/#broken-master). If the test cannot be fixed in a timely fashion, there is an impact on the -productivity of all the developers, so it should be quarantined by -assigning the `:quarantine` metadata with the issue URL, and add the `~"quarantined test"` label to the issue. +productivity of all the developers, so it should be quarantined. There are two ways to quarantine tests, depending on the test framework being used: RSpec and Jest. + +### RSpec + +For RSpec tests, you can use the `:quarantine` metadata with the issue URL. ```ruby it 'succeeds', quarantine: 'https://gitlab.com/gitlab-org/gitlab/-/issues/12345' do @@ -169,6 +172,26 @@ This means it is skipped unless run with `--tag quarantine`: bin/rspec --tag quarantine ``` +### Jest + +For Jest specs, you can use the `.skip` method along with the `eslint-disable-next-line` comment to disable the `jest/no-disabled-tests` ESLint rule and include the issue URL. Here's an example: + +```javascript +// https://gitlab.com/gitlab-org/gitlab/-/issues/56789 +// eslint-disable-next-line jest/no-disabled-tests +it.skip('should throw an error', () => { + expect(response).toThrowError(expected_error) +}); +``` + +This means it is skipped unless the test suit is run with `--runInBand` Jest command line option: + +```shell +jest --runInBand +``` + +For both test frameworks, make sure to add the `~"quarantined test"` label to the issue. + Once a test is in quarantine, there are 3 choices: - Fix the test (that is, get rid of its flakiness). diff --git a/doc/development/testing_guide/review_apps.md b/doc/development/testing_guide/review_apps.md index 5edf18725c0..b074a9e34f3 100644 --- a/doc/development/testing_guide/review_apps.md +++ b/doc/development/testing_guide/review_apps.md @@ -21,18 +21,23 @@ For any of the following scenarios, the `start-review-app-pipeline` job would be - for scheduled pipelines - the MR has the `pipeline:run-review-app` label set -## QA runs on review apps +## E2E test runs on review apps -On every [pipeline](https://gitlab.com/gitlab-org/gitlab/pipelines/125315730) in the `qa` stage (which comes after the -`review` stage), the `review-qa-smoke` and `review-qa-reliable` jobs are automatically started. The `review-qa-smoke` runs -the QA smoke suite and the `review-qa-reliable` executes E2E tests identified as [reliable](https://about.gitlab.com/handbook/engineering/quality/quality-engineering/reliable-tests/). +On every pipeline in the `qa` stage (which comes after the `review` stage), the `review-qa-smoke` and `review-qa-blocking` jobs are automatically started. -`review-qa-*` jobs ensure that end-to-end tests for the changes in the merge request pass in a live environment. This shifts the identification of e2e failures from an environment on the path to production to the merge request, to prevent breaking features on GitLab.com or costly GitLab.com deployment blockers. `review-qa-*` failures should be investigated with counterpart SET involvement if needed to help determine the root cause of the error. +`qa` stage consists of following jobs: -You can also manually start the `review-qa-all`: it runs the full QA suite. +- `review-qa-smoke`: small and fast subset of tests to validate core functionality of GitLab. +- `review-qa-blocking`: subset of tests identified as [reliable](https://about.gitlab.com/handbook/engineering/quality/quality-engineering/reliable-tests/). These tests are + considered stable and are not allowed to fail. +- `review-qa-non-blocking`: rest of the e2e tests that can be triggered manually. + +`review-qa-*` jobs ensure that end-to-end tests for the changes in the merge request pass in a live environment. This shifts the identification of e2e failures from an environment +on the path to production to the merge request to prevent breaking features on GitLab.com or costly GitLab.com deployment blockers. If needed, `review-qa-*` failures should be +investigated with an SET (software engineer in test) counterpart to help determine the root cause of the error. After the end-to-end test runs have finished, [Allure reports](https://github.com/allure-framework/allure2) are generated and published by -the `allure-report-qa-smoke`, `allure-report-qa-reliable`, and `allure-report-qa-all` jobs. A comment with links to the reports are added to the merge request. +the `e2e-test-report` job. A comment with links to the reports is added to the merge request. Errors can be found in the `gitlab-review-apps` Sentry project and [filterable by review app URL](https://sentry.gitlab.net/gitlab/gitlab-review-apps/?query=url%3A%22https%3A%2F%2Fgitlab-review-require-ve-u92nn2.gitlab-review.app%2F%22) or [commit SHA](https://sentry.gitlab.net/gitlab/gitlab-review-apps/releases/6095b501da7/all-events/). @@ -129,31 +134,28 @@ resource.labels.pod_name:"review-qa-raise-e-12chm0-migrations" ```mermaid graph TD A["build-qa-image, compile-production-assets<br/>(canonical default refs only)"]; + B1[start-review-app-pipeline]; B[review-build-cng]; - C[review-deploy]; + C["review-deploy<br><br>Helm deploys the review app using the Cloud<br/>Native images built by the CNG-mirror pipeline.<br><br>Cloud Native images are deployed to the `review-apps`<br>Kubernetes (GKE) cluster, in the GCP `gitlab-review-apps` project."]; D[CNG-mirror]; - E[review-qa-smoke, review-qa-reliable]; + E[review-qa-smoke, review-qa-blocking, review-qa-non-blocking<br><br>gitlab-qa runs the e2e tests against the review app.]; - A -->|once the `prepare` stage is done| B - B -.->|triggers a CNG-mirror pipeline and wait for it to be done| D - D -.->|polls until completed| B - B -->|once the `review-build-cng` job is done| C - C -->|once the `review-deploy` job is done| E + A --> B1 + B1 --> B + B -.->|triggers a CNG-mirror pipeline| D + D -.->|depends on the multi-project pipeline| B + B --> C + C --> E -subgraph "1. gitlab `prepare` stage" +subgraph "1. gitlab-org/gitlab parent pipeline" A + B1 end -subgraph "2. gitlab `review-prepare` stage" +subgraph "2. gitlab-org/gitlab child pipeline" B - end - -subgraph "3. gitlab `review` stage" - C["review-deploy<br><br>Helm deploys the review app using the Cloud<br/>Native images built by the CNG-mirror pipeline.<br><br>Cloud Native images are deployed to the `review-apps`<br>Kubernetes (GKE) cluster, in the GCP `gitlab-review-apps` project."] - end - -subgraph "4. gitlab `qa` stage" - E[review-qa-smoke, review-qa-reliable<br><br>gitlab-qa runs the smoke and reliable suites against the review app.] + C + E end subgraph "CNG-mirror pipeline" |