diff options
Diffstat (limited to 'doc/development/testing_guide')
22 files changed, 299 insertions, 225 deletions
diff --git a/doc/development/testing_guide/best_practices.md b/doc/development/testing_guide/best_practices.md index dabb18c1f75..d1b7883451f 100644 --- a/doc/development/testing_guide/best_practices.md +++ b/doc/development/testing_guide/best_practices.md @@ -50,6 +50,22 @@ bundle exec guard When using spring and guard together, use `SPRING=1 bundle exec guard` instead to make use of spring. +### Ruby warnings + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/47767) in GitLab 13.7. + +We've enabled [deprecation warnings](https://ruby-doc.org/core-2.7.2/Warning.html) +by default when running specs. Making these warnings more visible to developers +helps upgrading to newer Ruby versions. + +You can silence deprecation warnings by setting the environment variable +`SILENCE_DEPRECATIONS`, for example: + +```shell +# silence all deprecation warnings +SILENCE_DEPRECATIONS=1 bin/rspec spec/models/project_spec.rb +``` + ### Test speed GitLab has a massive test suite that, without [parallelization](ci.md#test-suite-parallelization-on-the-ci), can take hours @@ -311,7 +327,7 @@ Use the coverage reports to ensure your tests cover 100% of your code. ### System / Feature tests -NOTE: **Note:** +NOTE: Before writing a new system test, [please consider **not** writing one](testing_levels.md#consider-not-writing-a-system-test)! @@ -432,7 +448,7 @@ instead of 30+ seconds in case of a regular `spec_helper`. ### `subject` and `let` variables -GitLab's RSpec suite has made extensive use of `let`(along with its strict, non-lazy +The GitLab RSpec suite has made extensive use of `let`(along with its strict, non-lazy version `let!`) variables to reduce duplication. However, this sometimes [comes at the cost of clarity](https://thoughtbot.com/blog/lets-not), so we need to set some guidelines for their use going forward: @@ -603,6 +619,32 @@ it "really connects to Prometheus", :permit_dns do And if you need more specific control, the DNS blocking is implemented in `spec/support/helpers/dns_helpers.rb` and these methods can be called elsewhere. +#### Stubbing File methods + +In the situations where you need to +[stub](https://relishapp.com/rspec/rspec-mocks/v/3-9/docs/basics/allowing-messages) +methods such as `File.read`, make sure to: + +1. Stub `File.read` for only the filepath you are interested in. +1. Call the original implementation for other filepaths. + +Otherwise `File.read` calls from other parts of the codebase get +stubbed incorrectly. You should use the `stub_file_read`, and +`expect_file_read` helper methods which does the stubbing for +`File.read` correctly. + +```ruby +# bad, all Files will read and return nothing +allow(File).to receive(:read) + +# good +stub_file_read(my_filepath) + +# also OK +allow(File).to receive(:read).and_call_original +allow(File).to receive(:read).with(my_filepath) +``` + #### Filesystem Filesystem data can be roughly split into "repositories", and "everything else". @@ -678,7 +720,7 @@ at all possible. #### Test Snowplow events -CAUTION: **Warning:** +WARNING: Snowplow performs **runtime type checks** by using the [contracts gem](https://rubygems.org/gems/contracts). Since Snowplow is **by default disabled in tests and development**, it can be hard to **catch exceptions** when mocking `Gitlab::Tracking`. @@ -750,7 +792,7 @@ describe "#==" do end ``` -CAUTION: **Caution:** +WARNING: Only use simple values as input in the `where` block. Using procs, stateful objects, FactoryBot-created objects etc. can lead to [unexpected results](https://github.com/tomykaira/rspec-parameterized/issues/8). diff --git a/doc/development/testing_guide/ci.md b/doc/development/testing_guide/ci.md index 618f9010b4d..7318f767219 100644 --- a/doc/development/testing_guide/ci.md +++ b/doc/development/testing_guide/ci.md @@ -1,7 +1,7 @@ --- stage: none group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- # GitLab tests in the Continuous Integration (CI) context @@ -12,8 +12,8 @@ Our current CI parallelization setup is as follows: 1. The `retrieve-tests-metadata` job in the `prepare` stage ensures we have a `knapsack/report-master.json` file: - - The `knapsack/report-master.json` file is fetched from S3, if it's not here - we initialize the file with `{}`. + - The `knapsack/report-master.json` file is fetched from the latest `master` pipeline which runs `update-tests-metadata` + (for now it's the 2-hourly scheduled master pipeline), if it's not here we initialize the file with `{}`. 1. Each `[rspec|rspec-ee] [unit|integration|system|geo] n m` job are run with `knapsack rspec` and should have an evenly distributed share of tests: - It works because the jobs have access to the `knapsack/report-master.json` @@ -25,9 +25,9 @@ Our current CI parallelization setup is as follows: 1. The `update-tests-metadata` job (which only runs on scheduled pipelines for [the canonical project](https://gitlab.com/gitlab-org/gitlab) takes all the `knapsack/rspec*_pg_*.json` files and merge them all together into a single - `knapsack/report-master.json` file that is then uploaded to S3. + `knapsack/report-master.json` file that is saved as artifact. -After that, the next pipeline will use the up-to-date `knapsack/report-master.json` file. +After that, the next pipeline uses the up-to-date `knapsack/report-master.json` file. ## Monitoring diff --git a/doc/development/testing_guide/end_to_end/beginners_guide.md b/doc/development/testing_guide/end_to_end/beginners_guide.md index ef0bd9902e1..d60b780eeea 100644 --- a/doc/development/testing_guide/end_to_end/beginners_guide.md +++ b/doc/development/testing_guide/end_to_end/beginners_guide.md @@ -1,20 +1,20 @@ --- stage: none group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- # Beginner's guide to writing end-to-end tests -In this tutorial, you will learn about the creation of end-to-end (_e2e_) tests +This tutorial walks you through the creation of end-to-end (_e2e_) tests for [GitLab Community Edition](https://about.gitlab.com/install/?version=ce) and [GitLab Enterprise Edition](https://about.gitlab.com/install/). -By the end of this tutorial, you will be able to: +By the end of this tutorial, you can: - Determine whether an end-to-end test is needed. - Understand the directory structure within `qa/`. -- Write a basic end-to-end test that will validate login features. +- Write a basic end-to-end test that validates login features. - Develop any missing [page object](page_objects.md) libraries. ## Before you write a test @@ -29,7 +29,7 @@ must be configured to run the specs. The end-to-end tests: - Create [resources](resources.md) (such as project, issue, user) on an ad-hoc basis. - Test the UI and API interfaces, and use the API to efficiently set up the UI tests. -TIP: **Tip:** +NOTE: For more information, see [End-to-end testing Best Practices](best_practices.md). ## Determine if end-to-end tests are needed @@ -52,7 +52,7 @@ For information about the distribution of tests per level in GitLab, see - Finally, discuss the proposed test with the developer(s) involved in implementing the feature and the lower-level tests. -CAUTION: **Caution:** +WARNING: Check both [GitLab Community Edition](https://gitlab-org.gitlab.io/gitlab-foss/coverage-ruby/#_AllFiles) and [GitLab Enterprise Edition](https://gitlab-org.gitlab.io/gitlab/coverage-ruby/#_AllFiles) coverage projects for previously-written tests for this feature. For analyzing the code coverage, @@ -67,18 +67,18 @@ end-to-end flows, and is easiest to understand. The GitLab QA end-to-end tests are organized by the different [stages in the DevOps lifecycle](https://gitlab.com/gitlab-org/gitlab-foss/tree/master/qa/qa/specs/features/browser_ui). Determine where the test should be placed by -[stage](https://about.gitlab.com/handbook/product/product-categories/#devops-stages), -determine which feature the test will belong to, and then place it in a subdirectory +[stage](https://about.gitlab.com/handbook/product/categories/#devops-stages), +determine which feature the test belongs to, and then place it in a subdirectory under the stage.  -If the test is Enterprise Edition only, the test will be created in the `features/ee` +If the test is Enterprise Edition only, the test is created in the `features/ee` directory, but follow the same DevOps lifecycle format. ## Create a skeleton test -In the first part of this tutorial we will be testing login, which is owned by the +In the first part of this tutorial we are testing login, which is owned by the Manage stage. Inside `qa/specs/features/browser_ui/1_manage/login`, create a file `basic_login_spec.rb`. @@ -86,7 +86,7 @@ file `basic_login_spec.rb`. See the [`RSpec.describe` outer block](#the-outer-rspecdescribe-block) -CAUTION: **Deprecation notice:** +WARNING: The outer `context` [was deprecated](https://gitlab.com/gitlab-org/quality/team-tasks/-/issues/550) in `13.2` in adherence to RSpec 4.0 specifications. Use `RSpec.describe` instead. @@ -286,12 +286,12 @@ end Note the following important points: -- At the start of our example, we will be at the `page/issue/show.rb` [page](page_objects.md). +- At the start of our example, we are at the `page/issue/show.rb` [page](page_objects.md). - Our test fabricates only what it needs, when it needs it. - The issue is fabricated through the API to save time. - GitLab prefers `let()` over instance variables. See [best practices](../best_practices.md#subject-and-let-variables). -- `be_closed` is not implemented in `page/project/issue/show.rb` yet, but will be +- `be_closed` is not implemented in `page/project/issue/show.rb` yet, but is implemented in the next step. The issue is fabricated as a [Resource](resources.md), which is a GitLab entity @@ -349,3 +349,9 @@ Where `<test_file>` is: - `qa/specs/features/browser_ui/1_manage/login/login_spec.rb` when running the Login example. - `qa/specs/features/browser_ui/2_plan/issues/issue_spec.rb` when running the Issue example. + +## End-to-end test merge request template + +When submitting a new end-to-end test, use the ["New End to End Test"](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/merge_request_templates/New%20End%20To%20End%20Test.md) +merge request description template for additional +steps that are required prior a successful merge. diff --git a/doc/development/testing_guide/end_to_end/best_practices.md b/doc/development/testing_guide/end_to_end/best_practices.md index 4d12a0f79cb..b761e33367f 100644 --- a/doc/development/testing_guide/end_to_end/best_practices.md +++ b/doc/development/testing_guide/end_to_end/best_practices.md @@ -1,7 +1,7 @@ --- stage: none group: Development -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- # End-to-end testing Best Practices @@ -278,10 +278,10 @@ In line with [using the API](#prefer-api-over-ui), use a `Commit` resource whene ```ruby # Using a commit resource -Resource::Commit.fabricate_via_api! do |commit| +Resource::Repository::Commit.fabricate_via_api! do |commit| commit.commit_message = 'Initial commit' commit.add_files([ - {file_path: 'README.md', content: 'Hello, GitLab'} + { file_path: 'README.md', content: 'Hello, GitLab' } ]) end @@ -372,8 +372,10 @@ end [See this merge request for a real example of adding a custom matcher](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/46302). -NOTE: **Note:** -We need to create custom negatable matchers only for the predicate methods we've added to the test framework, and only if we're using `not_to`. If we use `to have_no_*` a negatable matcher is not necessary. +We are creating custom negatable matchers in `qa/spec/support/matchers`. + +NOTE: +We need to create custom negatable matchers only for the predicate methods we've added to the test framework, and only if we're using `not_to`. If we use `to have_no_*` a negatable matcher is not necessary but it increases code readability. ### Why we need negatable matchers diff --git a/doc/development/testing_guide/end_to_end/dynamic_element_validation.md b/doc/development/testing_guide/end_to_end/dynamic_element_validation.md index 871b3f80c18..1e7f528f6ff 100644 --- a/doc/development/testing_guide/end_to_end/dynamic_element_validation.md +++ b/doc/development/testing_guide/end_to_end/dynamic_element_validation.md @@ -1,7 +1,7 @@ --- stage: none group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- # Dynamic Element Validation @@ -23,7 +23,7 @@ We interpret user actions on the page to have some sort of effect. These actions ### Navigation -When a page is navigated to, there are elements that will always appear on the page unconditionally. +When a page is navigated to, there are elements that always appear on the page unconditionally. Dynamic element validation is instituted when using @@ -100,7 +100,7 @@ Runtime::Browser.visit(:gitlab, Page::MyPage) execute_stuff ``` -will invoke GitLab QA to scan `MyPage` for `my_element` and `another_element` to be on the page before continuing to +invokes GitLab QA to scan `MyPage` for `my_element` and `another_element` to be on the page before continuing to `execute_stuff` ### Clicking @@ -113,7 +113,7 @@ def open_layer end ``` -will invoke GitLab QA to ensure that `message_content` appears on +invokes GitLab QA to ensure that `message_content` appears on the Layer upon clicking `my_element`. -This will imply that the Layer is indeed rendered before we continue our test. +This implies that the Layer is indeed rendered before we continue our test. diff --git a/doc/development/testing_guide/end_to_end/environment_selection.md b/doc/development/testing_guide/end_to_end/environment_selection.md index f5e3e99b79e..7e34b6c265d 100644 --- a/doc/development/testing_guide/end_to_end/environment_selection.md +++ b/doc/development/testing_guide/end_to_end/environment_selection.md @@ -1,7 +1,7 @@ --- stage: none group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- # Environment selection @@ -19,7 +19,7 @@ We can specify what environments or pipelines to run tests against using the `on | `production` | Match against production | `Static` | | `pipeline` | Match against a pipeline | `Array` or `Static`| -CAUTION: **Caution:** +WARNING: You cannot specify `:production` and `{ <switch>: 'value' }` simultaneously. These options are mutually exclusive. If you want to specify production, you can control the `tld` and `domain` independently. @@ -35,7 +35,7 @@ can control the `tld` and `domain` independently. | `dev.gitlab.org` | `only: { tld: '.org', domain: 'gitlab', subdomain: 'dev' }` | `(dev).gitlab.org` | | `staging.gitlab.com & domain.gitlab.com` | `only: { subdomain: %i[staging domain] }` | `(staging|domain).+.com` | | `nightly` | `only: { pipeline: :nightly }` | "nightly" | -| `nightly`, `canary` | `only_run_in_pipeline: [:nightly, :canary]` | ["nightly"](https://gitlab.com/gitlab-org/quality/nightly) and ["canary"](https://gitlab.com/gitlab-org/quality/canary) | +| `nightly`, `canary` | `only: { pipeline: [:nightly, :canary] }` | ["nightly"](https://gitlab.com/gitlab-org/quality/nightly) and ["canary"](https://gitlab.com/gitlab-org/quality/canary) | ```ruby RSpec.describe 'Area' do @@ -65,4 +65,4 @@ If you want to run an `only: { :pipeline }` tagged test on your local GDK make s Similarly to specifying that a test should only run against a specific environment, it's also possible to quarantine a test only when it runs against a specific environment. The syntax is exactly the same, except that the `only: { ... }` hash is nested in the [`quarantine: { ... }`](https://about.gitlab.com/handbook/engineering/quality/guidelines/debugging-qa-test-failures/#quarantining-tests) hash. -For instance, `quarantine: { only: { subdomain: :staging } }` will only quarantine the test when run against staging. +For instance, `quarantine: { only: { subdomain: :staging } }` only quarantines the test when run against staging. diff --git a/doc/development/testing_guide/end_to_end/feature_flags.md b/doc/development/testing_guide/end_to_end/feature_flags.md index 2ff1c9f6dc3..1e0eda6491a 100644 --- a/doc/development/testing_guide/end_to_end/feature_flags.md +++ b/doc/development/testing_guide/end_to_end/feature_flags.md @@ -1,7 +1,7 @@ --- stage: none group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- # Testing with feature flags @@ -10,14 +10,14 @@ To run a specific test with a feature flag enabled you can use the `QA::Runtime: enable and disable feature flags ([via the API](../../../api/features.md)). Note that administrator authorization is required to change feature flags. `QA::Runtime::Feature` -will automatically authenticate as an administrator as long as you provide an appropriate access +automatically authenticates as an administrator as long as you provide an appropriate access token via `GITLAB_QA_ADMIN_ACCESS_TOKEN` (recommended), or provide `GITLAB_ADMIN_USERNAME` and `GITLAB_ADMIN_PASSWORD`. Please be sure to include the tag `:requires_admin` so that the test can be skipped in environments where admin access is not available. -CAUTION: **Caution:** +WARNING: You are strongly advised to [enable feature flags only for a group, project, user](../../feature_flags/development.md#feature-actors), or [feature group](../../feature_flags/development.md#feature-groups). This makes it possible to test a feature in a shared environment without affecting other users. @@ -60,7 +60,7 @@ feature_group = "a_feature_group" Runtime::Feature.enable(:feature_flag_name, feature_group: feature_group) ``` -If no scope is provided, the feature flag will be set instance-wide: +If no scope is provided, the feature flag is set instance-wide: ```ruby # This will affect all users! diff --git a/doc/development/testing_guide/end_to_end/flows.md b/doc/development/testing_guide/end_to_end/flows.md index 291d8bd5319..b99a1f9deb7 100644 --- a/doc/development/testing_guide/end_to_end/flows.md +++ b/doc/development/testing_guide/end_to_end/flows.md @@ -1,7 +1,7 @@ --- stage: none group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- # Flows in GitLab QA diff --git a/doc/development/testing_guide/end_to_end/index.md b/doc/development/testing_guide/end_to_end/index.md index 1d144359ed8..d981f9bcbba 100644 --- a/doc/development/testing_guide/end_to_end/index.md +++ b/doc/development/testing_guide/end_to_end/index.md @@ -1,7 +1,7 @@ --- stage: none group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- # End-to-end Testing @@ -126,8 +126,8 @@ For example, when we [dequarantine](https://about.gitlab.com/handbook/engineerin a flaky test we first want to make sure that it's no longer flaky. We can do that using the `ce:custom-parallel` and `ee:custom-parallel` jobs. Both are manual jobs that you can configure using custom variables. -When you click the name (not the play icon) of one of the parallel jobs, -you'll be prompted to enter variables. You can use any of [the variables +When clicking the name (not the play icon) of one of the parallel jobs, +you are prompted to enter variables. You can use any of [the variables that can be used with `gitlab-qa`](https://gitlab.com/gitlab-org/gitlab-qa/blob/master/docs/what_tests_can_be_run.md#supported-gitlab-environment-variables) as well as these: @@ -137,7 +137,7 @@ as well as these: | `QA_TESTS` | The test(s) to run (no default, which means run all the tests in the scenario). Use file paths as you would when running tests via RSpec, e.g., `qa/specs/features/ee/browser_ui` would include all the `EE` UI tests. | | `QA_RSPEC_TAGS` | The RSpec tags to add (no default) | -For now [manual jobs with custom variables will not use the same variable +For now [manual jobs with custom variables don't use the same variable when retried](https://gitlab.com/gitlab-org/gitlab/-/issues/31367), so if you want to run the same test(s) multiple times, specify the same variables in each `custom-parallel` job (up to as many of the 10 available jobs that you want to run). diff --git a/doc/development/testing_guide/end_to_end/page_objects.md b/doc/development/testing_guide/end_to_end/page_objects.md index 7eacaf4b08a..939e44cedd9 100644 --- a/doc/development/testing_guide/end_to_end/page_objects.md +++ b/doc/development/testing_guide/end_to_end/page_objects.md @@ -1,7 +1,7 @@ --- stage: none group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- # Page objects in GitLab QA @@ -22,7 +22,7 @@ fields. ## Why do we need that? We need page objects because we need to reduce duplication and avoid problems -whenever someone changes some selectors in GitLab's source code. +whenever someone changes some selectors in the GitLab source code. Imagine that we have a hundred specs in GitLab QA, and we need to sign into GitLab each time, before we make assertions. Without a page object, one would @@ -30,13 +30,13 @@ need to rely on volatile helpers or invoke Capybara methods directly. Imagine invoking `fill_in :user_login` in every `*_spec.rb` file / test example. When someone later changes `t.text_field :login` in the view associated with -this page to `t.text_field :username` it will generate a different field +this page to `t.text_field :username` it generates a different field identifier, what would effectively break all tests. Because we are using `Page::Main::Login.perform(&:sign_in_using_credentials)` everywhere, when we want to sign in to GitLab, the page object is the single -source of truth, and we will need to update `fill_in :user_login` -to `fill_in :user_username` only in a one place. +source of truth, and we must update `fill_in :user_login` +to `fill_in :user_username` only in one place. ## What problems did we have in the past? @@ -44,7 +44,7 @@ We do not run QA tests for every commit, because of performance reasons, and the time it would take to build packages and test everything. That is why when someone changes `t.text_field :login` to -`t.text_field :username` in the _new session_ view we won't know about this +`t.text_field :username` in the _new session_ view we don't know about this change until our GitLab QA nightly pipeline fails, or until someone triggers `package-and-qa` action in their merge request. @@ -56,14 +56,14 @@ problem by introducing coupling between GitLab CE / EE views and GitLab QA. ## How did we solve fragile tests problem? -Currently, when you add a new `Page::Base` derived class, you will also need to +Currently, when you add a new `Page::Base` derived class, you must also define all selectors that your page objects depend on. Whenever you push your code to CE / EE repository, `qa:selectors` sanity test -job is going to be run as a part of a CI pipeline. +job runs as a part of a CI pipeline. -This test is going to validate all page objects that we have implemented in -`qa/page` directory. When it fails, you will be notified about missing +This test validates all page objects that we have implemented in +`qa/page` directory. When it fails, it notifies you about missing or invalid views/selectors definition. ## How to properly implement a page object? @@ -95,10 +95,10 @@ end ### Defining Elements -The `view` DSL method will correspond to the Rails view, partial, or Vue component that renders the elements. +The `view` DSL method corresponds to the Rails view, partial, or Vue component that renders the elements. The `element` DSL method in turn declares an element for which a corresponding -`data-qa-selector=element_name_snaked` data attribute will need to be added to the view file. +`data-qa-selector=element_name_snaked` data attribute must be added to the view file. You can also define a value (String or Regexp) to match to the actual view code but **this is deprecated** in favor of the above method for two reasons: @@ -257,7 +257,7 @@ These modules must: 1. Include/prepend other modules and define their `view`/`elements` in a `base.class_eval` block to ensure they're defined in the class that prepends the module. -These steps ensure the sanity selectors check will detect problems properly. +These steps ensure the sanity selectors check detect problems properly. For example, `qa/qa/ee/page/merge_request/show.rb` adds EE-specific methods to `qa/qa/page/merge_request/show.rb` (with `QA::Page::MergeRequest::Show.prepend_if_ee('QA::EE::Page::MergeRequest::Show')`) and following is how it's implemented diff --git a/doc/development/testing_guide/end_to_end/resources.md b/doc/development/testing_guide/end_to_end/resources.md index d73bae331d5..b6aef123c64 100644 --- a/doc/development/testing_guide/end_to_end/resources.md +++ b/doc/development/testing_guide/end_to_end/resources.md @@ -1,7 +1,7 @@ --- stage: none group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- # Resource class in GitLab QA @@ -94,7 +94,7 @@ needs a group to be created in. To define a resource attribute, you can use the `attribute` method with a block using the other resource class to fabricate the resource. -That will allow access to the other resource from your resource object's +That allows access to the other resource from your resource object's methods. You would usually use it in `#fabricate!`, `#api_get_path`, `#api_post_path`, `#api_post_body`. @@ -144,7 +144,7 @@ end ``` **Note that all the attributes are lazily constructed. This means if you want -a specific attribute to be fabricated first, you'll need to call the +a specific attribute to be fabricated first, you must call the attribute method first even if you're not using it.** #### Product data attributes @@ -185,7 +185,7 @@ end ``` **Note again that all the attributes are lazily constructed. This means if -you call `shirt.brand` after moving to the other page, it'll not properly +you call `shirt.brand` after moving to the other page, it doesn't properly retrieve the data because we're no longer on the expected page.** Consider this: @@ -201,7 +201,7 @@ shirt.project.visit! shirt.brand # => FAIL! ``` -The above example will fail because now we're on the project page, trying to +The above example fails because now we're on the project page, trying to construct the brand data from the shirt page, however we moved to the project page already. There are two ways to solve this, one is that we could try to retrieve the brand before visiting the project again: @@ -219,8 +219,8 @@ shirt.project.visit! shirt.brand # => OK! ``` -The attribute will be stored in the instance therefore all the following calls -will be fine, using the data previously constructed. If we think that this +The attribute is stored in the instance, therefore all the following calls +are fine, using the data previously constructed. If we think that this might be too brittle, we could eagerly construct the data right before ending fabrication: @@ -249,12 +249,12 @@ module QA end ``` -The `populate` method will iterate through its arguments and call each +The `populate` method iterates through its arguments and call each attribute respectively. Here `populate(:brand)` has the same effect as just `brand`. Using the populate method makes the intention clearer. -With this, it will make sure we construct the data right after we create the -shirt. The drawback is that this will always construct the data when the +With this, it ensures we construct the data right after we create the +shirt. The drawback is that this always constructs the data when the resource is fabricated even if we don't need to use the data. Alternatively, we could just make sure we're on the right page before @@ -290,7 +290,7 @@ module QA end ``` -This will make sure it's on the shirt page before constructing brand, and +This ensures it's on the shirt page before constructing brand, and move back to the previous page to avoid breaking the state. #### Define an attribute based on an API response @@ -343,16 +343,16 @@ end - resource instance variables have the highest precedence - attributes from the API response take precedence over attributes from the block (usually from Browser UI) -- attributes without a value will raise a `QA::Resource::Base::NoValueError` error +- attributes without a value raises a `QA::Resource::Base::NoValueError` error ## Creating resources in your tests To create a resource in your tests, you can call the `.fabricate!` method on the resource class. -Note that if the resource class supports API fabrication, this will use this +Note that if the resource class supports API fabrication, this uses this fabrication by default. -Here is an example that will use the API fabrication method under the hood +Here is an example that uses the API fabrication method under the hood since it's supported by the `Shirt` resource class: ```ruby @@ -389,8 +389,7 @@ my_shirt = Resource::Shirt.fabricate_via_api! do |shirt| end ``` -In this case, the result will be similar to calling -`Resource::Shirt.fabricate!`. +In this case, the result is similar to calling `Resource::Shirt.fabricate!`. ## Where to ask for help? diff --git a/doc/development/testing_guide/end_to_end/rspec_metadata_tests.md b/doc/development/testing_guide/end_to_end/rspec_metadata_tests.md index 8e99cf18ea0..6d6f7fbcf8d 100644 --- a/doc/development/testing_guide/end_to_end/rspec_metadata_tests.md +++ b/doc/development/testing_guide/end_to_end/rspec_metadata_tests.md @@ -1,7 +1,7 @@ --- stage: none group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- # RSpec metadata for end-to-end tests @@ -14,16 +14,16 @@ This is a partial list of the [RSpec metadata](https://relishapp.com/rspec/rspec | Tag | Description | |-----|-------------| | `:elasticsearch` | The test requires an Elasticsearch service. It is used by the [instance-level scenario](https://gitlab.com/gitlab-org/gitlab-qa#definitions) [`Test::Integration::Elasticsearch`](https://gitlab.com/gitlab-org/gitlab/-/blob/72b62b51bdf513e2936301cb6c7c91ec27c35b4d/qa/qa/ee/scenario/test/integration/elasticsearch.rb) to include only tests that require Elasticsearch. | -| `:gitaly_cluster` | The test will run against a GitLab instance where repositories are stored on redundant Gitaly nodes behind a Praefect node. All nodes are [separate containers](../../../administration/gitaly/praefect.md#requirements-for-configuring-a-gitaly-cluster). Tests that use this tag have a longer setup time since there are three additional containers that need to be started. | -| `:jira` | The test requires a Jira Server. [GitLab-QA](https://gitlab.com/gitlab-org/gitlab-qa) will provision the Jira Server in a Docker container when the `Test::Integration::Jira` test scenario is run. -| `:kubernetes` | The test includes a GitLab instance that is configured to be run behind an SSH tunnel, allowing a TLS-accessible GitLab. This test will also include provisioning of at least one Kubernetes cluster to test against. _This tag is often be paired with `:orchestrated`._ | +| `:gitaly_cluster` | The test runs against a GitLab instance where repositories are stored on redundant Gitaly nodes behind a Praefect node. All nodes are [separate containers](../../../administration/gitaly/praefect.md#requirements-for-configuring-a-gitaly-cluster). Tests that use this tag have a longer setup time since there are three additional containers that need to be started. | +| `:jira` | The test requires a Jira Server. [GitLab-QA](https://gitlab.com/gitlab-org/gitlab-qa) provisions the Jira Server in a Docker container when the `Test::Integration::Jira` test scenario is run. +| `:kubernetes` | The test includes a GitLab instance that is configured to be run behind an SSH tunnel, allowing a TLS-accessible GitLab. This test also includes provisioning of at least one Kubernetes cluster to test against. _This tag is often be paired with `:orchestrated`._ | | `:only` | The test is only to be run against specific environments or pipelines. See [Environment selection](environment_selection.md) for more information. | -| `:orchestrated` | The GitLab instance under test may be [configured by `gitlab-qa`](https://gitlab.com/gitlab-org/gitlab-qa/-/blob/master/docs/what_tests_can_be_run.md#orchestrated-tests) to be different to the default GitLab configuration, or `gitlab-qa` may launch additional services in separate Docker containers, or both. Tests tagged with `:orchestrated` are excluded when testing environments where we can't dynamically modify GitLab's configuration (for example, Staging). | -| `:quarantine` | The test has been [quarantined](https://about.gitlab.com/handbook/engineering/quality/guidelines/debugging-qa-test-failures/#quarantining-tests), will run in a separate job that only includes quarantined tests, and is allowed to fail. The test will be skipped in its regular job so that if it fails it will not hold up the pipeline. Note that you can also [quarantine a test only when it runs against specific environment](environment_selection.md#quarantining-a-test-for-a-specific-environment). | +| `:orchestrated` | The GitLab instance under test may be [configured by `gitlab-qa`](https://gitlab.com/gitlab-org/gitlab-qa/-/blob/master/docs/what_tests_can_be_run.md#orchestrated-tests) to be different to the default GitLab configuration, or `gitlab-qa` may launch additional services in separate Docker containers, or both. Tests tagged with `:orchestrated` are excluded when testing environments where we can't dynamically modify the GitLab configuration (for example, Staging). | +| `:quarantine` | The test has been [quarantined](https://about.gitlab.com/handbook/engineering/quality/guidelines/debugging-qa-test-failures/#quarantining-tests), runs in a separate job that only includes quarantined tests, and is allowed to fail. The test is skipped in its regular job so that if it fails it doesn't hold up the pipeline. Note that you can also [quarantine a test only when it runs against specific environment](environment_selection.md#quarantining-a-test-for-a-specific-environment). | | `:reliable` | The test has been [promoted to a reliable test](https://about.gitlab.com/handbook/engineering/quality/guidelines/reliable-tests/#promoting-an-existing-test-to-reliable) meaning it passes consistently in all pipelines, including merge requests. | | `:requires_admin` | The test requires an admin account. Tests with the tag are excluded when run against Canary and Production environments. | -| `:runner` | The test depends on and will set up a GitLab Runner instance, typically to run a pipeline. | -| `:skip_live_env` | The test will be excluded when run against live deployed environments such as Staging, Canary, and Production. | +| `:runner` | The test depends on and sets up a GitLab Runner instance, typically to run a pipeline. | +| `:skip_live_env` | The test is excluded when run against live deployed environments such as Staging, Canary, and Production. | | `:testcase` | The link to the test case issue in the [Quality Testcases project](https://gitlab.com/gitlab-org/quality/testcases/). | | `:mattermost` | The test requires a GitLab Mattermost service on the GitLab instance. | | `:ldap_no_server` | The test requires a GitLab instance to be configured to use LDAP. To be used with the `:orchestrated` tag. It does not spin up an LDAP server at orchestration time. Instead, it creates the LDAP server at runtime. | @@ -33,7 +33,7 @@ This is a partial list of the [RSpec metadata](https://relishapp.com/rspec/rspec | `:smtp` | The test requires a GitLab instance to be configured to use an SMTP server. Tests SMTP notification email delivery from GitLab by using MailHog. | | `:group_saml` | The test requires a GitLab instance that has SAML SSO enabled at the group level. Interacts with an external SAML identity provider. Paired with the `:orchestrated` tag. | | `:instance_saml` | The test requires a GitLab instance that has SAML SSO enabled at the instance level. Interacts with an external SAML identity provider. Paired with the `:orchestrated` tag. | -| `:skip_signup_disabled` | The test uses UI to sign up a new user and will be skipped in any environment that does not allow new user registration via the UI. | +| `:skip_signup_disabled` | The test uses UI to sign up a new user and is skipped in any environment that does not allow new user registration via the UI. | | `:smoke` | The test belongs to the test suite which verifies basic functionality of a GitLab instance.| | `:github` | The test requires a GitHub personal access token. | | `:repository_storage` | The test requires a GitLab instance to be configured to use multiple [repository storage paths](../../../administration/repository_storage_paths.md). Paired with the `:orchestrated` tag. | diff --git a/doc/development/testing_guide/end_to_end/running_tests_that_require_special_setup.md b/doc/development/testing_guide/end_to_end/running_tests_that_require_special_setup.md index df4b833526c..8a49c333f9f 100644 --- a/doc/development/testing_guide/end_to_end/running_tests_that_require_special_setup.md +++ b/doc/development/testing_guide/end_to_end/running_tests_that_require_special_setup.md @@ -1,7 +1,7 @@ --- stage: none group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- # Running tests that require special setup @@ -10,7 +10,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w The [`jenkins_build_status_spec`](https://gitlab.com/gitlab-org/gitlab/blob/163c8a8c814db26d11e104d1cb2dcf02eb567dbe/qa/qa/specs/features/ee/browser_ui/3_create/jenkins/jenkins_build_status_spec.rb) spins up a Jenkins instance in a Docker container based on an image stored in the [GitLab-QA container registry](https://gitlab.com/gitlab-org/gitlab-qa/container_registry). The Docker image it uses is preconfigured with some base data and plugins. -The test then configures the GitLab plugin in Jenkins with a URL of the GitLab instance that will be used +The test then configures the GitLab plugin in Jenkins with a URL of the GitLab instance that are used to run the tests. Unfortunately, the GitLab Jenkins plugin does not accept ports so `http://localhost:3000` would not be accepted. Therefore, this requires us to run GitLab on port 80 or inside a Docker container. @@ -30,7 +30,7 @@ To run the tests from the `/qa` directory: CHROME_HEADLESS=false bin/qa Test::Instance::All http://localhost -- qa/specs/features/ee/browser_ui/3_create/jenkins/jenkins_build_status_spec.rb ``` -The test will automatically spin up a Docker container for Jenkins and tear down once the test completes. +The test automatically spins up a Docker container for Jenkins and tear down once the test completes. However, if you need to run Jenkins manually outside of the tests, use this command: @@ -43,7 +43,7 @@ docker run \ registry.gitlab.com/gitlab-org/gitlab-qa/jenkins-gitlab:version1 ``` -Jenkins will be available on `http://localhost:8080`. +Jenkins is available on `http://localhost:8080`. Admin username is `admin` and password is `password`. @@ -59,19 +59,19 @@ the Docker Engine. The tests tagged `:gitaly_ha` are orchestrated tests that can only be run against a set of Docker containers as configured and started by [the `Test::Integration::GitalyCluster` GitLab QA scenario](https://gitlab.com/gitlab-org/gitlab-qa/-/blob/master/docs/what_tests_can_be_run.md#testintegrationgitalycluster-ceeefull-image-address). -As described in the documentation about the scenario noted above, the following command will run the tests: +As described in the documentation about the scenario noted above, the following command runs the tests: ```shell gitlab-qa Test::Integration::GitalyCluster EE ``` -However, that will remove the containers after it finishes running the tests. If you would like to do further testing, for example, if you would like to run a single test via a debugger, you can use [the `--no-tests` option](https://gitlab.com/gitlab-org/gitlab-qa#command-line-options) to make `gitlab-qa` skip running the tests, and to leave the containers running so that you can continue to use them. +However, that removes the containers after it finishes running the tests. If you would like to do further testing, for example, if you would like to run a single test via a debugger, you can use [the `--no-tests` option](https://gitlab.com/gitlab-org/gitlab-qa#command-line-options) to make `gitlab-qa` skip running the tests, and to leave the containers running so that you can continue to use them. ```shell gitlab-qa Test::Integration::GitalyCluster EE --no-tests ``` -When all the containers are running you will see the output of the `docker ps` command, showing on which ports the GitLab container can be accessed. For example: +When all the containers are running, the output of the `docker ps` command shows which ports the GitLab container can be accessed on. For example: ```plaintext CONTAINER ID ... PORTS NAMES @@ -82,7 +82,7 @@ That shows that the GitLab instance running in the `gitlab-gitaly-ha` container Another option is to use NGINX. -In both cases you will need to configure your machine to translate `gitlab-gitlab-ha.test` into an appropriate IP address: +In both cases you must configure your machine to translate `gitlab-gitlab-ha.test` into an appropriate IP address: ```shell echo '127.0.0.1 gitlab-gitaly-ha.test' | sudo tee -a /etc/hosts @@ -205,7 +205,7 @@ You can now search through the logs for *Job log*, which matches delimited secti A Job log is a subsection within these logs, related to app deployment. We use two jobs: `build` and `production`. You can find the root causes of deployment failures in these logs, which can compromise the entire test. -If a `build` job fails, the `production` job won't run, and the test fails. +If a `build` job fails, the `production` job doesn't run, and the test fails. The long test setup does not take screenshots of failures, which is a known [issue](https://gitlab.com/gitlab-org/quality/team-tasks/-/issues/270). However, if the spec fails (after a successful deployment) then you should be able to find screenshots which display the feature failure. @@ -286,7 +286,7 @@ CHROME_HEADLESS=false bundle exec bin/qa QA::EE::Scenario::Test::Geo --primary-a You can use [GitLab-QA Orchestrator](https://gitlab.com/gitlab-org/gitlab-qa) to orchestrate two GitLab containers and configure them as a Geo setup. -Geo requires an EE license. To visit the Geo sites in your browser, you will need a reverse proxy server (for example, [NGINX](https://www.nginx.com/)). +Geo requires an EE license. To visit the Geo sites in your browser, you need a reverse proxy server (for example, [NGINX](https://www.nginx.com/)). 1. Export your EE license @@ -296,7 +296,7 @@ Geo requires an EE license. To visit the Geo sites in your browser, you will nee 1. (Optional) Pull the GitLab image - This step is optional because pulling the Docker image is part of the [`Test::Integration::Geo` orchestrated scenario](https://gitlab.com/gitlab-org/gitlab-qa/-/blob/d8c5c40607c2be0eda58bbca1b9f534b00889a0b/lib/gitlab/qa/scenario/test/integration/geo.rb). However, it's easier to monitor the download progress if you pull the image first, and the scenario will skip this step after checking that the image is up to date. + This step is optional because pulling the Docker image is part of the [`Test::Integration::Geo` orchestrated scenario](https://gitlab.com/gitlab-org/gitlab-qa/-/blob/d8c5c40607c2be0eda58bbca1b9f534b00889a0b/lib/gitlab/qa/scenario/test/integration/geo.rb). However, it's easier to monitor the download progress if you pull the image first, and the scenario skips this step after checking that the image is up to date. ```shell # For the most recent nightly image @@ -309,7 +309,7 @@ Geo requires an EE license. To visit the Geo sites in your browser, you will nee docker pull registry.gitlab.com/gitlab-org/build/omnibus-gitlab-mirror/gitlab-ee:examplesha123456789 ``` -1. Run the [`Test::Integration::Geo` orchestrated scenario](https://gitlab.com/gitlab-org/gitlab-qa/-/blob/d8c5c40607c2be0eda58bbca1b9f534b00889a0b/lib/gitlab/qa/scenario/test/integration/geo.rb) with the `--no-teardown` option to build the GitLab containers, configure the Geo setup, and run Geo end-to-end tests. Running the tests after the Geo setup is complete is optional; the containers will keep running after you stop the tests. +1. Run the [`Test::Integration::Geo` orchestrated scenario](https://gitlab.com/gitlab-org/gitlab-qa/-/blob/d8c5c40607c2be0eda58bbca1b9f534b00889a0b/lib/gitlab/qa/scenario/test/integration/geo.rb) with the `--no-teardown` option to build the GitLab containers, configure the Geo setup, and run Geo end-to-end tests. Running the tests after the Geo setup is complete is optional; the containers keep running after you stop the tests. ```shell # Using the most recent nightly image diff --git a/doc/development/testing_guide/end_to_end/style_guide.md b/doc/development/testing_guide/end_to_end/style_guide.md index e6c96bca93e..ac4d26df794 100644 --- a/doc/development/testing_guide/end_to_end/style_guide.md +++ b/doc/development/testing_guide/end_to_end/style_guide.md @@ -1,7 +1,7 @@ --- stage: none group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- # Style guide for writing end-to-end tests @@ -74,7 +74,8 @@ We follow a simple formula roughly based on Hungarian notation. - `_tab` - `_menu_item` -*Note: If none of the listed types are suitable, please open a merge request to add an appropriate type to the list.* +NOTE: +If none of the listed types are suitable, please open a merge request to add an appropriate type to the list. ### Examples diff --git a/doc/development/testing_guide/flaky_tests.md b/doc/development/testing_guide/flaky_tests.md index 47ed11d76a2..126d8725c21 100644 --- a/doc/development/testing_guide/flaky_tests.md +++ b/doc/development/testing_guide/flaky_tests.md @@ -1,7 +1,7 @@ --- stage: none group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- # Flaky tests @@ -26,14 +26,14 @@ it 'should succeed', quarantine: 'https://gitlab.com/gitlab-org/gitlab/-/issues/ end ``` -This means it will be skipped unless run with `--tag quarantine`: +This means it is skipped unless run with `--tag quarantine`: ```shell bin/rspec --tag quarantine ``` **Before putting a test in quarantine, you should make sure that a -~"master:broken" issue exists for it so it won't stay in quarantine forever.** +~"master:broken" issue exists for it so it doesn't stay in quarantine forever.** Once a test is in quarantine, there are 3 choices: diff --git a/doc/development/testing_guide/frontend_testing.md b/doc/development/testing_guide/frontend_testing.md index 28fe63f1fb4..d83d58d14dd 100644 --- a/doc/development/testing_guide/frontend_testing.md +++ b/doc/development/testing_guide/frontend_testing.md @@ -1,12 +1,12 @@ --- stage: none group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- # Frontend testing standards and style guidelines -There are two types of test suites you'll encounter while developing frontend code +There are two types of test suites encountered while developing frontend code at GitLab. We use Karma with Jasmine and Jest for JavaScript unit and integration testing, and RSpec feature tests with Capybara for e2e (end-to-end) integration testing. @@ -25,17 +25,19 @@ If you are looking for a guide on Vue component testing, you can jump right away ## Jest -We have started to migrate frontend tests to the [Jest](https://jestjs.io) testing framework (see also the corresponding -[epic](https://gitlab.com/groups/gitlab-org/-/epics/895)). - +We use Jest to write frontend unit and integration tests. Jest tests can be found in `/spec/frontend` and `/ee/spec/frontend` in EE. -Most examples have a Jest and Karma example. See the Karma examples only as explanation to what's going on in the code, should you stumble over some use cases during your discovery. The Jest examples are the one you should follow. - ## Karma test suite -While GitLab is switching over to [Jest](https://jestjs.io) you'll still find Karma tests in our application. [Karma](http://karma-runner.github.io/) is a test runner which uses [Jasmine](https://jasmine.github.io/) as its test -framework. Jest also uses Jasmine as foundation, that's why it's looking quite similar. +While GitLab has switched over to [Jest](https://jestjs.io), Karma tests still exist in our +application because some of our specs require a browser and can't be easiliy migrated to Jest. +Those specs intend to eventually drop Karma in favor of either Jest or RSpec. You can track this migration +in the [related epic](https://gitlab.com/groups/gitlab-org/-/epics/4900). + +[Karma](http://karma-runner.github.io/) is a test runner which uses +[Jasmine](https://jasmine.github.io/) as its test framework. Jest also uses Jasmine as foundation, +that's why it's looking quite similar. Karma tests live in `spec/javascripts/` and `/ee/spec/javascripts` in EE. @@ -43,23 +45,10 @@ Karma tests live in `spec/javascripts/` and `/ee/spec/javascripts` in EE. might have a corresponding `spec/javascripts/behaviors/autosize_spec.js` file. Keep in mind that in a CI environment, these tests are run in a headless -browser and you will not have access to certain APIs, such as +browser and you don't have access to certain APIs, such as [`Notification`](https://developer.mozilla.org/en-US/docs/Web/API/notification), which have to be stubbed. -### When should I use Jest over Karma? - -If you need to update an existing Karma test file (found in `spec/javascripts`), you do not -need to migrate the whole spec to Jest. Simply updating the Karma spec to test your change -is fine. It is probably more appropriate to migrate to Jest in a separate merge request. - -If you create a new test file, it needs to be created in Jest. This will -help support our migration and we think you'll love using Jest. - -As always, please use discretion. Jest solves a lot of issues we experienced in Karma and -provides a better developer experience, however there are potentially unexpected issues -which could arise (especially with testing against browser specific features). - ### Differences to Karma - Jest runs in a Node.js environment, not in a browser. Support for running Jest tests in a browser [is planned](https://gitlab.com/gitlab-org/gitlab/-/issues/26982). @@ -71,7 +60,7 @@ which could arise (especially with testing against browser specific features). - No [context object](https://jasmine.github.io/tutorials/your_first_suite#section-The_%3Ccode%3Ethis%3C/code%3E_keyword) is passed to tests in Jest. This means sharing `this.something` between `beforeEach()` and `it()` for example does not work. Instead you should declare shared variables in the context that they are needed (via `const` / `let`). -- The following will cause tests to fail in Jest: +- The following cause tests to fail in Jest: - Unmocked requests. - Unhandled Promise rejections. - Calls to `console.warn`, including warnings from libraries like Vue. @@ -89,14 +78,14 @@ See also the issue for [support running Jest tests in browsers](https://gitlab.c ### Debugging Jest tests -Running `yarn jest-debug` will run Jest in debug mode, allowing you to debug/inspect as described in the [Jest docs](https://jestjs.io/docs/en/troubleshooting#tests-are-failing-and-you-don-t-know-why). +Running `yarn jest-debug` runs Jest in debug mode, allowing you to debug/inspect as described in the [Jest docs](https://jestjs.io/docs/en/troubleshooting#tests-are-failing-and-you-don-t-know-why). ### Timeout error The default timeout for Jest is set in [`/spec/frontend/test_setup.js`](https://gitlab.com/gitlab-org/gitlab/blob/master/spec/frontend/test_setup.js). -If your test exceeds that time, it will fail. +If your test exceeds that time, it fails. If you cannot improve the performance of the tests, you can increase the timeout for a specific test using @@ -115,37 +104,6 @@ describe('Component', () => { Remember that the performance of each test depends on the environment. -### Timout error due to async components - -If your component is fetching some other components asynchroneously based on some conditions, it might happen so that your Jest suite for this component will become flaky timing out from time to time. - -```javascript -// ide.vue -export default { - components: { - 'error-message': () => import('./error_message.vue'), - 'gl-button': () => import('@gitlab/ui/src/components/base/button/button.vue'), - ... -}; -``` - -To address this issue, you can "help" Jest by stubbing the async components so that Jest would not need to fetch those asynchroneously at the run-time. - -```javascript -// ide_spec.js -import { GlButton } from '@gitlab/ui'; -import ErrorMessage from '~/ide/components/error_message.vue'; -... -return shallowMount(ide, { - ... - stubs: { - ErrorMessage, - GlButton, - ... - }, -}) -``` - ## What and how to test Before jumping into more gritty details about Jest-specific workflows like mocks and spies, we should briefly cover what to test with Jest. @@ -227,15 +185,15 @@ For example, it's better to use the generated markup to trigger a button click a ## Common practices -Following you'll find some general common practices you will find as part of our test suite. Should you stumble over something not following this guide, ideally fix it right away. 🎉 +These some general common practices included as part of our test suite. Should you stumble over something not following this guide, ideally fix it right away. 🎉 ### How to query DOM elements When it comes to querying DOM elements in your tests, it is best to uniquely and semantically target the element. -Preferentially, this is done by targeting what the user actually sees using [DOM Testing Library](https://testing-library.com/docs/dom-testing-library/intro). -When selecting by text it is best to use [`getByRole` or `findByRole`](https://testing-library.com/docs/dom-testing-library/api-queries#byrole) +Preferentially, this is done by targeting what the user actually sees using [DOM Testing Library](https://testing-library.com/docs/dom-testing-library/intro/). +When selecting by text it is best to use [`getByRole` or `findByRole`](https://testing-library.com/docs/dom-testing-library/api-queries/#byrole) as these enforce accessibility best practices as well. The examples below demonstrate the order of preference. When writing Vue component unit tests, it can be wise to query children by component, so that the unit test can focus on comprehensive value coverage @@ -246,6 +204,7 @@ possible selectors include: - A semantic attribute like `name` (also verifies that `name` was setup properly) - A `data-testid` attribute ([recommended by maintainers of `@vue/test-utils`](https://github.com/vuejs/vue-test-utils/issues/1498#issuecomment-610133465)) + optionally combined with [`findByTestId`](#extendedwrapper-and-findbytestid) - a Vue `ref` (if using `@vue/test-utils`) ```javascript @@ -262,7 +221,8 @@ it('exists', () => { // Good (especially for unit tests) wrapper.find(FooComponent); wrapper.find('input[name=foo]'); - wrapper.find('[data-testid="foo"]'); + wrapper.find('[data-testid="my-foo-id"]'); + wrapper.findByTestId('my-foo-id'); // with the extendedWrapper utility – check below wrapper.find({ ref: 'foo'}); // Bad @@ -273,6 +233,8 @@ it('exists', () => { }); ``` +It is recommended to use `kebab-case` for `data-testid` attribute. + It is not recommended that you add `.js-*` classes just for testing purposes. Only do this if there are no other feasible options available. Do not use a `.qa-*` class or `data-qa-selector` attribute for any tests other than QA end-to-end testing. @@ -389,7 +351,7 @@ it('tests a promise rejection', () => { ### Manipulating Time -Sometimes we have to test time-sensitive code. For example, recurring events that run every X amount of seconds or similar. Here you'll find some strategies to deal with that: +Sometimes we have to test time-sensitive code. For example, recurring events that run every X amount of seconds or similar. Here are some strategies to deal with that: #### `setTimeout()` / `setInterval()` in application @@ -435,7 +397,7 @@ it('does something', () => { Sometimes a test needs to wait for something to happen in the application before it continues. Avoid using [`setTimeout`](https://developer.mozilla.org/en-US/docs/Web/API/WindowOrWorkerGlobalScope/setTimeout) -because it makes the reason for waiting unclear and if used within Karma with a time larger than zero it will slow down our test suite. +because it makes the reason for waiting unclear and if used within Karma with a time larger than zero it slows down our test suite. Instead use one of the following approaches. #### Promises and Ajax calls @@ -599,7 +561,7 @@ Jest has [`toBe`](https://jestjs.io/docs/en/expect#tobevalue) and As [`toBe`](https://jestjs.io/docs/en/expect#tobevalue) uses [`Object.is`](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Object/is) to compare values, it's faster (by default) than using `toEqual`. -While the latter will eventually fallback to leverage [`Object.is`](https://github.com/facebook/jest/blob/master/packages/expect/src/jasmineUtils.ts#L91), +While the latter eventually falls back to leverage [`Object.is`](https://github.com/facebook/jest/blob/master/packages/expect/src/jasmineUtils.ts#L91), for primitive values, it should only be used when complex objects need a comparison. Examples: @@ -745,10 +707,10 @@ a [Jest mock for the package `monaco-editor`](https://gitlab.com/gitlab-org/gitl If a manual mock is needed for a CE module, please place it in `spec/frontend/mocks/ce`. -- Files in `spec/frontend/mocks/ce` will mock the corresponding CE module from `app/assets/javascripts`, mirroring the source module's path. - - Example: `spec/frontend/mocks/ce/lib/utils/axios_utils` will mock the module `~/lib/utils/axios_utils`. +- Files in `spec/frontend/mocks/ce` mocks the corresponding CE module from `app/assets/javascripts`, mirroring the source module's path. + - Example: `spec/frontend/mocks/ce/lib/utils/axios_utils` mocks the module `~/lib/utils/axios_utils`. - We don't support mocking EE modules yet. -- If a mock is found for which a source module doesn't exist, the test suite will fail. 'Virtual' mocks, or mocks that don't have a 1-to-1 association with a source module, are not supported yet. +- If a mock is found for which a source module doesn't exist, the test suite fails. 'Virtual' mocks, or mocks that don't have a 1-to-1 association with a source module, are not supported yet. #### Manual mock examples @@ -776,11 +738,10 @@ Please consult the [official Jest docs](https://jestjs.io/docs/en/jest-object#mo For running the frontend tests, you need the following commands: -- `rake frontend:fixtures` (re-)generates [fixtures](#frontend-test-fixtures). -- `yarn test` executes the tests. -- `yarn jest` executes only the Jest tests. - -As long as the fixtures don't change, `yarn test` is sufficient (and saves you some time). +- `rake frontend:fixtures` (re-)generates [fixtures](#frontend-test-fixtures). Make sure that + fixtures are up-to-date before running tests that require them. +- `yarn jest` runs Jest tests. +- `yarn karma` runs Karma tests. ### Live testing and focused testing -- Jest @@ -809,12 +770,12 @@ yarn jest term Karma allows something similar, but it's way more costly. -Running Karma with `yarn run karma-start` will compile the JavaScript -assets and run a server at `http://localhost:9876/` where it will automatically -run the tests on any browser which connects to it. You can enter that URL on +Running Karma with `yarn run karma-start` compiles the JavaScript +assets and runs a server at `http://localhost:9876/` where it automatically +runs the tests on any browser which connects to it. You can enter that URL on multiple browsers at once to have it run the tests on each in parallel. -While Karma is running, any changes you make will instantly trigger a recompile +While Karma is running, any changes you make instantly trigger a recompile and retest of the **entire test suite**, so you can see instantly if you've broken a test with your changes. You can use [Jasmine focused](https://jasmine.github.io/2.5/focused_specs.html) or excluded tests (with `fdescribe` or `xdescribe`) to get Karma to run only the @@ -866,8 +827,8 @@ You can find generated fixtures are in `tmp/tests/frontend/fixtures-ee`. #### Creating new fixtures For each fixture, you can find the content of the `response` variable in the output file. -For example, test named `"merge_requests/diff_discussion.json"` in `spec/frontend/fixtures/merge_requests.rb` -will produce output file `tmp/tests/frontend/fixtures-ee/merge_requests/diff_discussion.json`. +For example, a test named `"merge_requests/diff_discussion.json"` in `spec/frontend/fixtures/merge_requests.rb` +produces an output file `tmp/tests/frontend/fixtures-ee/merge_requests/diff_discussion.json`. The `response` variable gets automatically set if the test is marked as `type: :request` or `type: :controller`. When creating a new fixture, it often makes sense to take a look at the corresponding tests for the @@ -977,12 +938,12 @@ describe.each` ### RSpec errors due to JavaScript -By default RSpec unit tests will not run JavaScript in the headless browser -and will simply rely on inspecting the HTML generated by rails. +By default RSpec unit tests don't run JavaScript in the headless browser +and rely on inspecting the HTML generated by rails. If an integration test depends on JavaScript to run correctly, you need to make sure the spec is configured to enable JavaScript when the tests are run. If you -don't do this you'll see vague error messages from the spec runner. +don't do this, the spec runner displays vague error messages. To enable a JavaScript driver in an `rspec` test, add `:js` to the individual spec or the context block containing multiple specs that need @@ -1004,6 +965,45 @@ describe "Admin::AbuseReports", :js do end ``` +### Jest test timeout due to async imports + +If a module asynchronously imports some other modules at runtime, these modules must be +transpiled by the Jest loaders at runtime. It's possible that this can cause [Jest to timeout](https://gitlab.com/gitlab-org/gitlab/-/issues/280809). + +If you run into this issue, consider eager importing the module so that Jest compiles +and caches it at compile-time, fixing the runtime timeout. + +Consider the following example: + +```javascript +// the_subject.js + +export default { + components: { + // Async import Thing because it is large and isn't always needed. + Thing: () => import(/* webpackChunkName: 'thing' */ './path/to/thing.vue'), + } +}; +``` + +Jest doesn't automatically transpile the `thing.vue` module, and depending on its size, could +cause Jest to time out. We can force Jest to transpile and cache this module by eagerly importing +it like so: + +```javascript +// the_subject_spec.js + +import Subject from '~/feature/the_subject.vue'; + +// Force Jest to transpile and cache +// eslint-disable-next-line import/order, no-unused-vars +import _Thing from '~/feature/path/to/thing.vue'; +``` + +**PLEASE NOTE:** Do not simply disregard test timeouts. This could be a sign that there's +actually a production problem. Use this opportunity to analyze the production webpack bundles and +chunks and confirm that there is not a production issue with the async imports. + ## Overview of Frontend Testing Levels Main information on frontend testing levels can be found in the [Testing Levels page](testing_levels.md). @@ -1016,7 +1016,7 @@ Tests relevant for frontend development can be found at the following places: RSpec runs complete [feature tests](testing_levels.md#frontend-feature-tests), while the Jest and Karma directories contain [frontend unit tests](testing_levels.md#frontend-unit-tests), [frontend component tests](testing_levels.md#frontend-component-tests), and [frontend integration tests](testing_levels.md#frontend-integration-tests). -All tests in `spec/javascripts/` will eventually be migrated to `spec/frontend/` (see also [#52483](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/52483)). +All tests in `spec/javascripts/` are intended to be migrated to `spec/frontend/` (see also [#52483](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/52483)). Before May 2018, `features/` also contained feature tests run by Spinach. These tests were removed from the codebase in May 2018 ([#23036](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/23036)). @@ -1045,7 +1045,7 @@ testAction( ); ``` -Check an example in [`spec/javascripts/ide/stores/actions_spec.jsspec/javascripts/ide/stores/actions_spec.js`](https://gitlab.com/gitlab-org/gitlab/blob/master/spec/javascripts/ide/stores/actions_spec.js). +Check an example in [`spec/frontend/ide/stores/actions_spec.js`](https://gitlab.com/gitlab-org/gitlab/-/blob/fdc7197609dfa7caeb1d962042a26248e49f27da/spec/frontend/ide/stores/actions_spec.js#L392). ### Wait until Axios requests finish @@ -1057,6 +1057,29 @@ These are very useful if you don't have a handle to the request's Promise, for e Both functions run `callback` on the next tick after the requests finish (using `setImmediate()`), to allow any `.then()` or `.catch()` handlers to run. +### `extendedWrapper` and `findByTestId` + +Using `data-testid` is one of the [recommended ways to query DOM elements](#how-to-query-dom-elements). +You can use the `extendedWrapper` utility on the `wrapper` returned by `shalowMount`/`mount`. +By doing so, the `wrapper` provides you with the ability to perform a `findByTestId`, +which is a shortcut to the more verbose `wrapper.find('[data-testid="my-test-id"]');` + +```javascript +import { extendedWrapper } from 'jest/helpers/vue_test_utils_helper'; + +describe('FooComponent', () => { + const wrapper = extendedWrapper(shallowMount({ + template: `<div data-testid="my-test-id"></div>`, + })); + + it('exists', () => { + expect(wrapper.findByTestId('my-test-id').exists()).toBe(true); + }); +}); +``` + +Check an example in [`spec/frontend/alert_management/components/alert_details_spec.js`](https://gitlab.com/gitlab-org/gitlab/-/blob/ac1c9fa4c5b3b45f9566147b1c88fd1339cd7c25/spec/frontend/alert_management/components/alert_details_spec.js#L32). + ## Testing with older browsers Some regressions only affect a specific browser version. We can install and test in particular browsers with either Firefox or BrowserStack using the following steps: @@ -1065,7 +1088,7 @@ Some regressions only affect a specific browser version. We can install and test [BrowserStack](https://www.browserstack.com/) allows you to test more than 1200 mobile devices and browsers. You can use it directly through the [live app](https://www.browserstack.com/live) or you can install the [chrome extension](https://chrome.google.com/webstore/detail/browserstack/nkihdmlheodkdfojglpcjjmioefjahjb) for easy access. -Sign in to BrowserStack with the credentials saved in the **Engineering** vault of GitLab's +Sign in to BrowserStack with the credentials saved in the **Engineering** vault of the GitLab [shared 1Password account](https://about.gitlab.com/handbook/security/#1password-guide). ### Firefox @@ -1076,7 +1099,7 @@ You can download any older version of Firefox from the releases FTP server, <htt 1. From the website, select a version, in this case `50.0.1`. 1. Go to the mac folder. -1. Select your preferred language, you will find the DMG package inside, download it. +1. Select your preferred language. The DMG package is inside. Download it. 1. Drag and drop the application to any other folder but the `Applications` folder. 1. Rename the application to something like `Firefox_Old`. 1. Move the application to the `Applications` folder. diff --git a/doc/development/testing_guide/index.md b/doc/development/testing_guide/index.md index 840c8c9206c..68326879dd0 100644 --- a/doc/development/testing_guide/index.md +++ b/doc/development/testing_guide/index.md @@ -1,7 +1,7 @@ --- stage: none group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- # Testing standards and style guidelines diff --git a/doc/development/testing_guide/review_apps.md b/doc/development/testing_guide/review_apps.md index 5dcc7e7091e..a5294be40a9 100644 --- a/doc/development/testing_guide/review_apps.md +++ b/doc/development/testing_guide/review_apps.md @@ -1,7 +1,7 @@ --- stage: none group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- # Review Apps @@ -102,8 +102,8 @@ subgraph "CNG-mirror pipeline" - The manual `review-stop` can be used to stop a Review App manually, and is also started by GitLab once a merge request's branch is deleted after being merged. -- The Kubernetes cluster is connected to the `gitlab` projects using - [GitLab's Kubernetes integration](../../user/project/clusters/index.md). This basically +- The Kubernetes cluster is connected to the `gitlab` projects using the + [GitLab Kubernetes integration](../../user/project/clusters/index.md). This basically allows to have a link to the Review App directly from the merge request widget. ### Auto-stopping of Review Apps @@ -164,7 +164,7 @@ used by the `review-deploy` and `review-stop` jobs. You need to [open an access request (internal link)](https://gitlab.com/gitlab-com/access-requests/-/issues/new) for the `gcp-review-apps-dev` GCP group and role. -This will grant you the following permissions for: +This grants you the following permissions for: - [Retrieving pod logs](#dig-into-a-pods-logs). Granted by [Viewer (`roles/viewer`)](https://cloud.google.com/iam/docs/understanding-roles#kubernetes-engine-roles). - [Running a Rails console](#run-a-rails-console). Granted by [Kubernetes Engine Developer (`roles/container.pods.exec`)](https://cloud.google.com/iam/docs/understanding-roles#kubernetes-engine-roles). @@ -317,7 +317,7 @@ kubectl get cm --sort-by='{.metadata.creationTimestamp}' | grep 'review-' | grep ### Using K9s -[K9s](https://github.com/derailed/k9s) is a powerful command line dashboard which allows you to filter by labels. This can help identify trends with apps exceeding the [review-app resource requests](https://gitlab.com/gitlab-org/gitlab/-/blob/master/scripts/review_apps/base-config.yaml). Kubernetes will schedule pods to nodes based on resource requests and allow for CPU usage up to the limits. +[K9s](https://github.com/derailed/k9s) is a powerful command line dashboard which allows you to filter by labels. This can help identify trends with apps exceeding the [review-app resource requests](https://gitlab.com/gitlab-org/gitlab/-/blob/master/scripts/review_apps/base-config.yaml). Kubernetes schedules pods to nodes based on resource requests and allow for CPU usage up to the limits. - In K9s you can sort or add filters by typing the `/` character - `-lrelease=<review-app-slug>` - filters down to all pods for a release. This aids in determining what is having issues in a single deployment @@ -395,8 +395,8 @@ helm ls -d | grep "Jun 4" | cut -f1 | xargs helm delete --purge #### Mitigation steps taken to avoid this problem in the future -We've created a new node pool with smaller machines so that it's less likely -that a machine will hit the "too many mount points" problem in the future. +We've created a new node pool with smaller machines to reduce the risk +that a machine reaches the "too many mount points" problem in the future. ## Frequently Asked Questions diff --git a/doc/development/testing_guide/smoke.md b/doc/development/testing_guide/smoke.md index 76484dd193b..4fd2729a4d3 100644 --- a/doc/development/testing_guide/smoke.md +++ b/doc/development/testing_guide/smoke.md @@ -1,17 +1,17 @@ --- stage: none group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- # Smoke Tests It is imperative in any testing suite that we have Smoke Tests. In short, smoke -tests will run quick sanity end-to-end functional tests from GitLab QA and are +tests run quick end-to-end functional tests from GitLab QA and are designed to run against the specified environment to ensure that basic functionality is working. -Currently, our suite consists of this basic functionality coverage: +Our suite consists of this basic functionality coverage: - User standard authentication - SSH Key creation and addition to a user diff --git a/doc/development/testing_guide/testing_levels.md b/doc/development/testing_guide/testing_levels.md index d9e7edfa0c8..14d4ee82f75 100644 --- a/doc/development/testing_guide/testing_levels.md +++ b/doc/development/testing_guide/testing_levels.md @@ -1,7 +1,7 @@ --- stage: none group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- # Testing levels @@ -129,14 +129,14 @@ graph RL - **All server requests**: When running frontend unit tests, the backend may not be reachable, so all outgoing requests need to be mocked. - **Asynchronous background operations**: - Background operations cannot be stopped or waited on, so they will continue running in the following tests and cause side effects. + Background operations cannot be stopped or waited on, so they continue running in the following tests and cause side effects. #### What *not* to mock in unit tests - **Non-exported functions or classes**: - Everything that is not exported can be considered private to the module, and will be implicitly tested through the exported classes and functions. + Everything that is not exported can be considered private to the module, and is implicitly tested through the exported classes and functions. - **Methods of the class under test**: - By mocking methods of the class under test, the mocks will be tested and not the real methods. + By mocking methods of the class under test, the mocks are tested and not the real methods. - **Utility functions (pure functions, or those that only modify parameters)**: If a function has no side effects because it has no state, it is safe to not mock it in tests. - **Full HTML pages**: @@ -206,7 +206,7 @@ graph RL - **All server requests**: Similar to unit tests, when running component tests, the backend may not be reachable, so all outgoing requests need to be mocked. - **Asynchronous background operations**: - Similar to unit tests, background operations cannot be stopped or waited on. This means they will continue running in the following tests and cause side effects. + Similar to unit tests, background operations cannot be stopped or waited on. This means they continue running in the following tests and cause side effects. - **Child components**: Every component is tested individually, so child components are mocked. See also [`shallowMount()`](https://vue-test-utils.vuejs.org/api/#shallowmount) @@ -214,7 +214,7 @@ graph RL #### What *not* to mock in component tests - **Methods or computed properties of the component under test**: - By mocking part of the component under test, the mocks will be tested and not the real component. + By mocking part of the component under test, the mocks are tested and not the real component. - **Functions and classes independent from Vue**: All plain JavaScript code is already covered by unit tests and needs not to be mocked in component tests. @@ -295,7 +295,7 @@ graph RL Similar to unit and component tests, when running component tests, the backend may not be reachable, so all outgoing requests must be mocked. - **Asynchronous background operations that are not perceivable on the page**: Background operations that affect the page must be tested on this level. - All other background operations cannot be stopped or waited on, so they will continue running in the following tests and cause side effects. + All other background operations cannot be stopped or waited on, so they continue running in the following tests and cause side effects. #### What *not* to mock in integration tests @@ -360,7 +360,7 @@ possible). | Tests path | Testing engine | Notes | | ---------- | -------------- | ----- | -| `spec/features/` | [Capybara](https://github.com/teamcapybara/capybara) + [RSpec](https://github.com/rspec/rspec-rails#feature-specs) | If your test has the `:js` metadata, the browser driver will be [Poltergeist](https://github.com/teamcapybara/capybara#poltergeist), otherwise it's using [RackTest](https://github.com/teamcapybara/capybara#racktest). | +| `spec/features/` | [Capybara](https://github.com/teamcapybara/capybara) + [RSpec](https://github.com/rspec/rspec-rails#feature-specs) | If your test has the `:js` metadata, the browser driver is [Poltergeist](https://github.com/teamcapybara/capybara#poltergeist), otherwise it's using [RackTest](https://github.com/teamcapybara/capybara#racktest). | ### Frontend feature tests @@ -453,13 +453,13 @@ should take care of not introducing too many (slow and duplicated) tests. The reasons why we should follow these best practices are as follows: -- System tests are slow to run since they spin up the entire application stack +- System tests are slow to run because they spin up the entire application stack in a headless browser, and even slower when they integrate a JS driver - When system tests run with a JavaScript driver, the tests are run in a different thread than the application. This means it does not share a - database connection and your test will have to commit the transactions in + database connection and your test must commit the transactions in order for the running application to see the data (and vice-versa). In that - case we need to truncate the database after each spec instead of simply + case we need to truncate the database after each spec instead of rolling back a transaction (the faster strategy that's in use for other kind of tests). This is slower than transactions, however, so we want to use truncation only when necessary. @@ -488,7 +488,7 @@ Every new feature should come with a [test plan](https://gitlab.com/gitlab-org/g | Tests path | Testing engine | Notes | | ---------- | -------------- | ----- | -| `qa/qa/specs/features/` | [Capybara](https://github.com/teamcapybara/capybara) + [RSpec](https://github.com/rspec/rspec-rails#feature-specs) + Custom QA framework | Tests should be placed under their corresponding [Product category](https://about.gitlab.com/handbook/product/product-categories/) | +| `qa/qa/specs/features/` | [Capybara](https://github.com/teamcapybara/capybara) + [RSpec](https://github.com/rspec/rspec-rails#feature-specs) + Custom QA framework | Tests should be placed under their corresponding [Product category](https://about.gitlab.com/handbook/product/categories/) | > See [end-to-end tests](end_to_end/index.md) for more information. diff --git a/doc/development/testing_guide/testing_migrations_guide.md b/doc/development/testing_guide/testing_migrations_guide.md index b944c7ed406..31054d0ffb2 100644 --- a/doc/development/testing_guide/testing_migrations_guide.md +++ b/doc/development/testing_guide/testing_migrations_guide.md @@ -1,7 +1,7 @@ --- stage: none group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: reference --- @@ -24,15 +24,15 @@ Adding a `:migration` tag to a test signature enables some custom RSpec [`spec/support/migration.rb`](https://gitlab.com/gitlab-org/gitlab/-/blob/f81fa6ab1dd788b70ef44b85aaba1f31ffafae7d/spec/support/migration.rb) to run. -A `before` hook will revert all migrations to the point that a migration +A `before` hook reverts all migrations to the point that a migration under test is not yet migrated. -In other words, our custom RSpec hooks will find a previous migration, and +In other words, our custom RSpec hooks finds a previous migration, and migrate the database **down** to the previous migration version. With this approach you can test a migration against a database schema. -An `after` hook will migrate the database **up** and reinstitute the latest +An `after` hook migrates the database **up** and reinstitutes the latest schema version, so that the process does not affect subsequent specs and ensures proper isolation. @@ -40,7 +40,7 @@ ensures proper isolation. To test an `ActiveRecord::Migration` class (i.e., a regular migration `db/migrate` or a post-migration `db/post_migrate`), you -will need to load the migration file by using the `require_migration!` helper +must load the migration file by using the `require_migration!` helper method because it is not autoloaded by Rails. Example: @@ -57,15 +57,15 @@ RSpec.describe ... #### `require_migration!` -Since the migration files are not autoloaded by Rails, you will need to manually +Since the migration files are not autoloaded by Rails, you must manually load the migration file. To do so, you can use the `require_migration!` helper method -which can automatically load the correct migration file based on the spec file name. +which can automatically load the correct migration file based on the spec filename. For example, if your spec file is named as `populate_foo_column_spec.rb` then the -helper method will try to load `${schema_version}_populate_foo_column.rb` migration file. +helper method tries to load `${schema_version}_populate_foo_column.rb` migration file. In case there is no pattern between your spec file and the actual migration file, -you can provide the migration file name without the schema version, like so: +you can provide the migration filename without the schema version, like so: ```ruby require_migration!('populate_foo_column') @@ -75,8 +75,9 @@ require_migration!('populate_foo_column') Use the `table` helper to create a temporary `ActiveRecord::Base`-derived model for a table. [FactoryBot](best_practices.md#factories) -**should not** be used to create data for migration specs. For example, to -create a record in the `projects` table: +**should not** be used to create data for migration specs because it relies on +application code which can change after the migration has run, and cause the test +to fail. For example, to create a record in the `projects` table: ```ruby project = table(:projects).create!(id: 1, name: 'gitlab1', path: 'gitlab1') @@ -84,8 +85,8 @@ project = table(:projects).create!(id: 1, name: 'gitlab1', path: 'gitlab1') #### `migrate!` -Use the `migrate!` helper to run the migration that is under test. It will not only -run the migration, but will also bump the schema version in the `schema_migrations` +Use the `migrate!` helper to run the migration that is under test. It +runs the migration and bumps the schema version in the `schema_migrations` table. It is necessary because in the `after` hook we trigger the rest of the migrations, and we need to know where to start. Example: @@ -102,7 +103,7 @@ end #### `reversible_migration` Use the `reversible_migration` helper to test migrations with either a -`change` or both `up` and `down` hooks. This will test that the state of +`change` or both `up` and `down` hooks. This tests that the state of the application and its data after the migration becomes reversed is the same as it was before the migration ran in the first place. The helper: @@ -183,7 +184,7 @@ end ## Testing a non-`ActiveRecord::Migration` class To test a non-`ActiveRecord::Migration` test (a background migration), -you will need to manually provide a required schema version. Please add a +you must manually provide a required schema version. Please add a `schema` tag to a context that you want to switch the database schema within. If not set, `schema` defaults to `:latest`. diff --git a/doc/development/testing_guide/testing_rake_tasks.md b/doc/development/testing_guide/testing_rake_tasks.md index 384739d1adb..dc754721e24 100644 --- a/doc/development/testing_guide/testing_rake_tasks.md +++ b/doc/development/testing_guide/testing_rake_tasks.md @@ -1,7 +1,7 @@ --- stage: none group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- # Testing Rake tasks @@ -11,7 +11,7 @@ in lieu of the standard Spec helper. Instead of `require 'spec_helper'`, use `require 'rake_helper'`. The helper includes `spec_helper` for you, and configures a few other things to make testing Rake tasks easier. -At a minimum, requiring the Rake helper will redirect `stdout`, include the +At a minimum, requiring the Rake helper redirects `stdout`, include the runtime task helpers, and include the `RakeHelpers` Spec support module. The `RakeHelpers` module exposes a `run_rake_task(<task>)` method to make |