diff options
Diffstat (limited to 'doc/development/testing_guide/end_to_end')
12 files changed, 96 insertions, 88 deletions
diff --git a/doc/development/testing_guide/end_to_end/beginners_guide.md b/doc/development/testing_guide/end_to_end/beginners_guide.md index ef0bd9902e1..d60b780eeea 100644 --- a/doc/development/testing_guide/end_to_end/beginners_guide.md +++ b/doc/development/testing_guide/end_to_end/beginners_guide.md @@ -1,20 +1,20 @@ --- stage: none group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- # Beginner's guide to writing end-to-end tests -In this tutorial, you will learn about the creation of end-to-end (_e2e_) tests +This tutorial walks you through the creation of end-to-end (_e2e_) tests for [GitLab Community Edition](https://about.gitlab.com/install/?version=ce) and [GitLab Enterprise Edition](https://about.gitlab.com/install/). -By the end of this tutorial, you will be able to: +By the end of this tutorial, you can: - Determine whether an end-to-end test is needed. - Understand the directory structure within `qa/`. -- Write a basic end-to-end test that will validate login features. +- Write a basic end-to-end test that validates login features. - Develop any missing [page object](page_objects.md) libraries. ## Before you write a test @@ -29,7 +29,7 @@ must be configured to run the specs. The end-to-end tests: - Create [resources](resources.md) (such as project, issue, user) on an ad-hoc basis. - Test the UI and API interfaces, and use the API to efficiently set up the UI tests. -TIP: **Tip:** +NOTE: For more information, see [End-to-end testing Best Practices](best_practices.md). ## Determine if end-to-end tests are needed @@ -52,7 +52,7 @@ For information about the distribution of tests per level in GitLab, see - Finally, discuss the proposed test with the developer(s) involved in implementing the feature and the lower-level tests. -CAUTION: **Caution:** +WARNING: Check both [GitLab Community Edition](https://gitlab-org.gitlab.io/gitlab-foss/coverage-ruby/#_AllFiles) and [GitLab Enterprise Edition](https://gitlab-org.gitlab.io/gitlab/coverage-ruby/#_AllFiles) coverage projects for previously-written tests for this feature. For analyzing the code coverage, @@ -67,18 +67,18 @@ end-to-end flows, and is easiest to understand. The GitLab QA end-to-end tests are organized by the different [stages in the DevOps lifecycle](https://gitlab.com/gitlab-org/gitlab-foss/tree/master/qa/qa/specs/features/browser_ui). Determine where the test should be placed by -[stage](https://about.gitlab.com/handbook/product/product-categories/#devops-stages), -determine which feature the test will belong to, and then place it in a subdirectory +[stage](https://about.gitlab.com/handbook/product/categories/#devops-stages), +determine which feature the test belongs to, and then place it in a subdirectory under the stage.  -If the test is Enterprise Edition only, the test will be created in the `features/ee` +If the test is Enterprise Edition only, the test is created in the `features/ee` directory, but follow the same DevOps lifecycle format. ## Create a skeleton test -In the first part of this tutorial we will be testing login, which is owned by the +In the first part of this tutorial we are testing login, which is owned by the Manage stage. Inside `qa/specs/features/browser_ui/1_manage/login`, create a file `basic_login_spec.rb`. @@ -86,7 +86,7 @@ file `basic_login_spec.rb`. See the [`RSpec.describe` outer block](#the-outer-rspecdescribe-block) -CAUTION: **Deprecation notice:** +WARNING: The outer `context` [was deprecated](https://gitlab.com/gitlab-org/quality/team-tasks/-/issues/550) in `13.2` in adherence to RSpec 4.0 specifications. Use `RSpec.describe` instead. @@ -286,12 +286,12 @@ end Note the following important points: -- At the start of our example, we will be at the `page/issue/show.rb` [page](page_objects.md). +- At the start of our example, we are at the `page/issue/show.rb` [page](page_objects.md). - Our test fabricates only what it needs, when it needs it. - The issue is fabricated through the API to save time. - GitLab prefers `let()` over instance variables. See [best practices](../best_practices.md#subject-and-let-variables). -- `be_closed` is not implemented in `page/project/issue/show.rb` yet, but will be +- `be_closed` is not implemented in `page/project/issue/show.rb` yet, but is implemented in the next step. The issue is fabricated as a [Resource](resources.md), which is a GitLab entity @@ -349,3 +349,9 @@ Where `<test_file>` is: - `qa/specs/features/browser_ui/1_manage/login/login_spec.rb` when running the Login example. - `qa/specs/features/browser_ui/2_plan/issues/issue_spec.rb` when running the Issue example. + +## End-to-end test merge request template + +When submitting a new end-to-end test, use the ["New End to End Test"](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/merge_request_templates/New%20End%20To%20End%20Test.md) +merge request description template for additional +steps that are required prior a successful merge. diff --git a/doc/development/testing_guide/end_to_end/best_practices.md b/doc/development/testing_guide/end_to_end/best_practices.md index 4d12a0f79cb..b761e33367f 100644 --- a/doc/development/testing_guide/end_to_end/best_practices.md +++ b/doc/development/testing_guide/end_to_end/best_practices.md @@ -1,7 +1,7 @@ --- stage: none group: Development -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- # End-to-end testing Best Practices @@ -278,10 +278,10 @@ In line with [using the API](#prefer-api-over-ui), use a `Commit` resource whene ```ruby # Using a commit resource -Resource::Commit.fabricate_via_api! do |commit| +Resource::Repository::Commit.fabricate_via_api! do |commit| commit.commit_message = 'Initial commit' commit.add_files([ - {file_path: 'README.md', content: 'Hello, GitLab'} + { file_path: 'README.md', content: 'Hello, GitLab' } ]) end @@ -372,8 +372,10 @@ end [See this merge request for a real example of adding a custom matcher](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/46302). -NOTE: **Note:** -We need to create custom negatable matchers only for the predicate methods we've added to the test framework, and only if we're using `not_to`. If we use `to have_no_*` a negatable matcher is not necessary. +We are creating custom negatable matchers in `qa/spec/support/matchers`. + +NOTE: +We need to create custom negatable matchers only for the predicate methods we've added to the test framework, and only if we're using `not_to`. If we use `to have_no_*` a negatable matcher is not necessary but it increases code readability. ### Why we need negatable matchers diff --git a/doc/development/testing_guide/end_to_end/dynamic_element_validation.md b/doc/development/testing_guide/end_to_end/dynamic_element_validation.md index 871b3f80c18..1e7f528f6ff 100644 --- a/doc/development/testing_guide/end_to_end/dynamic_element_validation.md +++ b/doc/development/testing_guide/end_to_end/dynamic_element_validation.md @@ -1,7 +1,7 @@ --- stage: none group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- # Dynamic Element Validation @@ -23,7 +23,7 @@ We interpret user actions on the page to have some sort of effect. These actions ### Navigation -When a page is navigated to, there are elements that will always appear on the page unconditionally. +When a page is navigated to, there are elements that always appear on the page unconditionally. Dynamic element validation is instituted when using @@ -100,7 +100,7 @@ Runtime::Browser.visit(:gitlab, Page::MyPage) execute_stuff ``` -will invoke GitLab QA to scan `MyPage` for `my_element` and `another_element` to be on the page before continuing to +invokes GitLab QA to scan `MyPage` for `my_element` and `another_element` to be on the page before continuing to `execute_stuff` ### Clicking @@ -113,7 +113,7 @@ def open_layer end ``` -will invoke GitLab QA to ensure that `message_content` appears on +invokes GitLab QA to ensure that `message_content` appears on the Layer upon clicking `my_element`. -This will imply that the Layer is indeed rendered before we continue our test. +This implies that the Layer is indeed rendered before we continue our test. diff --git a/doc/development/testing_guide/end_to_end/environment_selection.md b/doc/development/testing_guide/end_to_end/environment_selection.md index f5e3e99b79e..7e34b6c265d 100644 --- a/doc/development/testing_guide/end_to_end/environment_selection.md +++ b/doc/development/testing_guide/end_to_end/environment_selection.md @@ -1,7 +1,7 @@ --- stage: none group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- # Environment selection @@ -19,7 +19,7 @@ We can specify what environments or pipelines to run tests against using the `on | `production` | Match against production | `Static` | | `pipeline` | Match against a pipeline | `Array` or `Static`| -CAUTION: **Caution:** +WARNING: You cannot specify `:production` and `{ <switch>: 'value' }` simultaneously. These options are mutually exclusive. If you want to specify production, you can control the `tld` and `domain` independently. @@ -35,7 +35,7 @@ can control the `tld` and `domain` independently. | `dev.gitlab.org` | `only: { tld: '.org', domain: 'gitlab', subdomain: 'dev' }` | `(dev).gitlab.org` | | `staging.gitlab.com & domain.gitlab.com` | `only: { subdomain: %i[staging domain] }` | `(staging|domain).+.com` | | `nightly` | `only: { pipeline: :nightly }` | "nightly" | -| `nightly`, `canary` | `only_run_in_pipeline: [:nightly, :canary]` | ["nightly"](https://gitlab.com/gitlab-org/quality/nightly) and ["canary"](https://gitlab.com/gitlab-org/quality/canary) | +| `nightly`, `canary` | `only: { pipeline: [:nightly, :canary] }` | ["nightly"](https://gitlab.com/gitlab-org/quality/nightly) and ["canary"](https://gitlab.com/gitlab-org/quality/canary) | ```ruby RSpec.describe 'Area' do @@ -65,4 +65,4 @@ If you want to run an `only: { :pipeline }` tagged test on your local GDK make s Similarly to specifying that a test should only run against a specific environment, it's also possible to quarantine a test only when it runs against a specific environment. The syntax is exactly the same, except that the `only: { ... }` hash is nested in the [`quarantine: { ... }`](https://about.gitlab.com/handbook/engineering/quality/guidelines/debugging-qa-test-failures/#quarantining-tests) hash. -For instance, `quarantine: { only: { subdomain: :staging } }` will only quarantine the test when run against staging. +For instance, `quarantine: { only: { subdomain: :staging } }` only quarantines the test when run against staging. diff --git a/doc/development/testing_guide/end_to_end/feature_flags.md b/doc/development/testing_guide/end_to_end/feature_flags.md index 2ff1c9f6dc3..1e0eda6491a 100644 --- a/doc/development/testing_guide/end_to_end/feature_flags.md +++ b/doc/development/testing_guide/end_to_end/feature_flags.md @@ -1,7 +1,7 @@ --- stage: none group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- # Testing with feature flags @@ -10,14 +10,14 @@ To run a specific test with a feature flag enabled you can use the `QA::Runtime: enable and disable feature flags ([via the API](../../../api/features.md)). Note that administrator authorization is required to change feature flags. `QA::Runtime::Feature` -will automatically authenticate as an administrator as long as you provide an appropriate access +automatically authenticates as an administrator as long as you provide an appropriate access token via `GITLAB_QA_ADMIN_ACCESS_TOKEN` (recommended), or provide `GITLAB_ADMIN_USERNAME` and `GITLAB_ADMIN_PASSWORD`. Please be sure to include the tag `:requires_admin` so that the test can be skipped in environments where admin access is not available. -CAUTION: **Caution:** +WARNING: You are strongly advised to [enable feature flags only for a group, project, user](../../feature_flags/development.md#feature-actors), or [feature group](../../feature_flags/development.md#feature-groups). This makes it possible to test a feature in a shared environment without affecting other users. @@ -60,7 +60,7 @@ feature_group = "a_feature_group" Runtime::Feature.enable(:feature_flag_name, feature_group: feature_group) ``` -If no scope is provided, the feature flag will be set instance-wide: +If no scope is provided, the feature flag is set instance-wide: ```ruby # This will affect all users! diff --git a/doc/development/testing_guide/end_to_end/flows.md b/doc/development/testing_guide/end_to_end/flows.md index 291d8bd5319..b99a1f9deb7 100644 --- a/doc/development/testing_guide/end_to_end/flows.md +++ b/doc/development/testing_guide/end_to_end/flows.md @@ -1,7 +1,7 @@ --- stage: none group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- # Flows in GitLab QA diff --git a/doc/development/testing_guide/end_to_end/index.md b/doc/development/testing_guide/end_to_end/index.md index 1d144359ed8..d981f9bcbba 100644 --- a/doc/development/testing_guide/end_to_end/index.md +++ b/doc/development/testing_guide/end_to_end/index.md @@ -1,7 +1,7 @@ --- stage: none group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- # End-to-end Testing @@ -126,8 +126,8 @@ For example, when we [dequarantine](https://about.gitlab.com/handbook/engineerin a flaky test we first want to make sure that it's no longer flaky. We can do that using the `ce:custom-parallel` and `ee:custom-parallel` jobs. Both are manual jobs that you can configure using custom variables. -When you click the name (not the play icon) of one of the parallel jobs, -you'll be prompted to enter variables. You can use any of [the variables +When clicking the name (not the play icon) of one of the parallel jobs, +you are prompted to enter variables. You can use any of [the variables that can be used with `gitlab-qa`](https://gitlab.com/gitlab-org/gitlab-qa/blob/master/docs/what_tests_can_be_run.md#supported-gitlab-environment-variables) as well as these: @@ -137,7 +137,7 @@ as well as these: | `QA_TESTS` | The test(s) to run (no default, which means run all the tests in the scenario). Use file paths as you would when running tests via RSpec, e.g., `qa/specs/features/ee/browser_ui` would include all the `EE` UI tests. | | `QA_RSPEC_TAGS` | The RSpec tags to add (no default) | -For now [manual jobs with custom variables will not use the same variable +For now [manual jobs with custom variables don't use the same variable when retried](https://gitlab.com/gitlab-org/gitlab/-/issues/31367), so if you want to run the same test(s) multiple times, specify the same variables in each `custom-parallel` job (up to as many of the 10 available jobs that you want to run). diff --git a/doc/development/testing_guide/end_to_end/page_objects.md b/doc/development/testing_guide/end_to_end/page_objects.md index 7eacaf4b08a..939e44cedd9 100644 --- a/doc/development/testing_guide/end_to_end/page_objects.md +++ b/doc/development/testing_guide/end_to_end/page_objects.md @@ -1,7 +1,7 @@ --- stage: none group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- # Page objects in GitLab QA @@ -22,7 +22,7 @@ fields. ## Why do we need that? We need page objects because we need to reduce duplication and avoid problems -whenever someone changes some selectors in GitLab's source code. +whenever someone changes some selectors in the GitLab source code. Imagine that we have a hundred specs in GitLab QA, and we need to sign into GitLab each time, before we make assertions. Without a page object, one would @@ -30,13 +30,13 @@ need to rely on volatile helpers or invoke Capybara methods directly. Imagine invoking `fill_in :user_login` in every `*_spec.rb` file / test example. When someone later changes `t.text_field :login` in the view associated with -this page to `t.text_field :username` it will generate a different field +this page to `t.text_field :username` it generates a different field identifier, what would effectively break all tests. Because we are using `Page::Main::Login.perform(&:sign_in_using_credentials)` everywhere, when we want to sign in to GitLab, the page object is the single -source of truth, and we will need to update `fill_in :user_login` -to `fill_in :user_username` only in a one place. +source of truth, and we must update `fill_in :user_login` +to `fill_in :user_username` only in one place. ## What problems did we have in the past? @@ -44,7 +44,7 @@ We do not run QA tests for every commit, because of performance reasons, and the time it would take to build packages and test everything. That is why when someone changes `t.text_field :login` to -`t.text_field :username` in the _new session_ view we won't know about this +`t.text_field :username` in the _new session_ view we don't know about this change until our GitLab QA nightly pipeline fails, or until someone triggers `package-and-qa` action in their merge request. @@ -56,14 +56,14 @@ problem by introducing coupling between GitLab CE / EE views and GitLab QA. ## How did we solve fragile tests problem? -Currently, when you add a new `Page::Base` derived class, you will also need to +Currently, when you add a new `Page::Base` derived class, you must also define all selectors that your page objects depend on. Whenever you push your code to CE / EE repository, `qa:selectors` sanity test -job is going to be run as a part of a CI pipeline. +job runs as a part of a CI pipeline. -This test is going to validate all page objects that we have implemented in -`qa/page` directory. When it fails, you will be notified about missing +This test validates all page objects that we have implemented in +`qa/page` directory. When it fails, it notifies you about missing or invalid views/selectors definition. ## How to properly implement a page object? @@ -95,10 +95,10 @@ end ### Defining Elements -The `view` DSL method will correspond to the Rails view, partial, or Vue component that renders the elements. +The `view` DSL method corresponds to the Rails view, partial, or Vue component that renders the elements. The `element` DSL method in turn declares an element for which a corresponding -`data-qa-selector=element_name_snaked` data attribute will need to be added to the view file. +`data-qa-selector=element_name_snaked` data attribute must be added to the view file. You can also define a value (String or Regexp) to match to the actual view code but **this is deprecated** in favor of the above method for two reasons: @@ -257,7 +257,7 @@ These modules must: 1. Include/prepend other modules and define their `view`/`elements` in a `base.class_eval` block to ensure they're defined in the class that prepends the module. -These steps ensure the sanity selectors check will detect problems properly. +These steps ensure the sanity selectors check detect problems properly. For example, `qa/qa/ee/page/merge_request/show.rb` adds EE-specific methods to `qa/qa/page/merge_request/show.rb` (with `QA::Page::MergeRequest::Show.prepend_if_ee('QA::EE::Page::MergeRequest::Show')`) and following is how it's implemented diff --git a/doc/development/testing_guide/end_to_end/resources.md b/doc/development/testing_guide/end_to_end/resources.md index d73bae331d5..b6aef123c64 100644 --- a/doc/development/testing_guide/end_to_end/resources.md +++ b/doc/development/testing_guide/end_to_end/resources.md @@ -1,7 +1,7 @@ --- stage: none group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- # Resource class in GitLab QA @@ -94,7 +94,7 @@ needs a group to be created in. To define a resource attribute, you can use the `attribute` method with a block using the other resource class to fabricate the resource. -That will allow access to the other resource from your resource object's +That allows access to the other resource from your resource object's methods. You would usually use it in `#fabricate!`, `#api_get_path`, `#api_post_path`, `#api_post_body`. @@ -144,7 +144,7 @@ end ``` **Note that all the attributes are lazily constructed. This means if you want -a specific attribute to be fabricated first, you'll need to call the +a specific attribute to be fabricated first, you must call the attribute method first even if you're not using it.** #### Product data attributes @@ -185,7 +185,7 @@ end ``` **Note again that all the attributes are lazily constructed. This means if -you call `shirt.brand` after moving to the other page, it'll not properly +you call `shirt.brand` after moving to the other page, it doesn't properly retrieve the data because we're no longer on the expected page.** Consider this: @@ -201,7 +201,7 @@ shirt.project.visit! shirt.brand # => FAIL! ``` -The above example will fail because now we're on the project page, trying to +The above example fails because now we're on the project page, trying to construct the brand data from the shirt page, however we moved to the project page already. There are two ways to solve this, one is that we could try to retrieve the brand before visiting the project again: @@ -219,8 +219,8 @@ shirt.project.visit! shirt.brand # => OK! ``` -The attribute will be stored in the instance therefore all the following calls -will be fine, using the data previously constructed. If we think that this +The attribute is stored in the instance, therefore all the following calls +are fine, using the data previously constructed. If we think that this might be too brittle, we could eagerly construct the data right before ending fabrication: @@ -249,12 +249,12 @@ module QA end ``` -The `populate` method will iterate through its arguments and call each +The `populate` method iterates through its arguments and call each attribute respectively. Here `populate(:brand)` has the same effect as just `brand`. Using the populate method makes the intention clearer. -With this, it will make sure we construct the data right after we create the -shirt. The drawback is that this will always construct the data when the +With this, it ensures we construct the data right after we create the +shirt. The drawback is that this always constructs the data when the resource is fabricated even if we don't need to use the data. Alternatively, we could just make sure we're on the right page before @@ -290,7 +290,7 @@ module QA end ``` -This will make sure it's on the shirt page before constructing brand, and +This ensures it's on the shirt page before constructing brand, and move back to the previous page to avoid breaking the state. #### Define an attribute based on an API response @@ -343,16 +343,16 @@ end - resource instance variables have the highest precedence - attributes from the API response take precedence over attributes from the block (usually from Browser UI) -- attributes without a value will raise a `QA::Resource::Base::NoValueError` error +- attributes without a value raises a `QA::Resource::Base::NoValueError` error ## Creating resources in your tests To create a resource in your tests, you can call the `.fabricate!` method on the resource class. -Note that if the resource class supports API fabrication, this will use this +Note that if the resource class supports API fabrication, this uses this fabrication by default. -Here is an example that will use the API fabrication method under the hood +Here is an example that uses the API fabrication method under the hood since it's supported by the `Shirt` resource class: ```ruby @@ -389,8 +389,7 @@ my_shirt = Resource::Shirt.fabricate_via_api! do |shirt| end ``` -In this case, the result will be similar to calling -`Resource::Shirt.fabricate!`. +In this case, the result is similar to calling `Resource::Shirt.fabricate!`. ## Where to ask for help? diff --git a/doc/development/testing_guide/end_to_end/rspec_metadata_tests.md b/doc/development/testing_guide/end_to_end/rspec_metadata_tests.md index 8e99cf18ea0..6d6f7fbcf8d 100644 --- a/doc/development/testing_guide/end_to_end/rspec_metadata_tests.md +++ b/doc/development/testing_guide/end_to_end/rspec_metadata_tests.md @@ -1,7 +1,7 @@ --- stage: none group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- # RSpec metadata for end-to-end tests @@ -14,16 +14,16 @@ This is a partial list of the [RSpec metadata](https://relishapp.com/rspec/rspec | Tag | Description | |-----|-------------| | `:elasticsearch` | The test requires an Elasticsearch service. It is used by the [instance-level scenario](https://gitlab.com/gitlab-org/gitlab-qa#definitions) [`Test::Integration::Elasticsearch`](https://gitlab.com/gitlab-org/gitlab/-/blob/72b62b51bdf513e2936301cb6c7c91ec27c35b4d/qa/qa/ee/scenario/test/integration/elasticsearch.rb) to include only tests that require Elasticsearch. | -| `:gitaly_cluster` | The test will run against a GitLab instance where repositories are stored on redundant Gitaly nodes behind a Praefect node. All nodes are [separate containers](../../../administration/gitaly/praefect.md#requirements-for-configuring-a-gitaly-cluster). Tests that use this tag have a longer setup time since there are three additional containers that need to be started. | -| `:jira` | The test requires a Jira Server. [GitLab-QA](https://gitlab.com/gitlab-org/gitlab-qa) will provision the Jira Server in a Docker container when the `Test::Integration::Jira` test scenario is run. -| `:kubernetes` | The test includes a GitLab instance that is configured to be run behind an SSH tunnel, allowing a TLS-accessible GitLab. This test will also include provisioning of at least one Kubernetes cluster to test against. _This tag is often be paired with `:orchestrated`._ | +| `:gitaly_cluster` | The test runs against a GitLab instance where repositories are stored on redundant Gitaly nodes behind a Praefect node. All nodes are [separate containers](../../../administration/gitaly/praefect.md#requirements-for-configuring-a-gitaly-cluster). Tests that use this tag have a longer setup time since there are three additional containers that need to be started. | +| `:jira` | The test requires a Jira Server. [GitLab-QA](https://gitlab.com/gitlab-org/gitlab-qa) provisions the Jira Server in a Docker container when the `Test::Integration::Jira` test scenario is run. +| `:kubernetes` | The test includes a GitLab instance that is configured to be run behind an SSH tunnel, allowing a TLS-accessible GitLab. This test also includes provisioning of at least one Kubernetes cluster to test against. _This tag is often be paired with `:orchestrated`._ | | `:only` | The test is only to be run against specific environments or pipelines. See [Environment selection](environment_selection.md) for more information. | -| `:orchestrated` | The GitLab instance under test may be [configured by `gitlab-qa`](https://gitlab.com/gitlab-org/gitlab-qa/-/blob/master/docs/what_tests_can_be_run.md#orchestrated-tests) to be different to the default GitLab configuration, or `gitlab-qa` may launch additional services in separate Docker containers, or both. Tests tagged with `:orchestrated` are excluded when testing environments where we can't dynamically modify GitLab's configuration (for example, Staging). | -| `:quarantine` | The test has been [quarantined](https://about.gitlab.com/handbook/engineering/quality/guidelines/debugging-qa-test-failures/#quarantining-tests), will run in a separate job that only includes quarantined tests, and is allowed to fail. The test will be skipped in its regular job so that if it fails it will not hold up the pipeline. Note that you can also [quarantine a test only when it runs against specific environment](environment_selection.md#quarantining-a-test-for-a-specific-environment). | +| `:orchestrated` | The GitLab instance under test may be [configured by `gitlab-qa`](https://gitlab.com/gitlab-org/gitlab-qa/-/blob/master/docs/what_tests_can_be_run.md#orchestrated-tests) to be different to the default GitLab configuration, or `gitlab-qa` may launch additional services in separate Docker containers, or both. Tests tagged with `:orchestrated` are excluded when testing environments where we can't dynamically modify the GitLab configuration (for example, Staging). | +| `:quarantine` | The test has been [quarantined](https://about.gitlab.com/handbook/engineering/quality/guidelines/debugging-qa-test-failures/#quarantining-tests), runs in a separate job that only includes quarantined tests, and is allowed to fail. The test is skipped in its regular job so that if it fails it doesn't hold up the pipeline. Note that you can also [quarantine a test only when it runs against specific environment](environment_selection.md#quarantining-a-test-for-a-specific-environment). | | `:reliable` | The test has been [promoted to a reliable test](https://about.gitlab.com/handbook/engineering/quality/guidelines/reliable-tests/#promoting-an-existing-test-to-reliable) meaning it passes consistently in all pipelines, including merge requests. | | `:requires_admin` | The test requires an admin account. Tests with the tag are excluded when run against Canary and Production environments. | -| `:runner` | The test depends on and will set up a GitLab Runner instance, typically to run a pipeline. | -| `:skip_live_env` | The test will be excluded when run against live deployed environments such as Staging, Canary, and Production. | +| `:runner` | The test depends on and sets up a GitLab Runner instance, typically to run a pipeline. | +| `:skip_live_env` | The test is excluded when run against live deployed environments such as Staging, Canary, and Production. | | `:testcase` | The link to the test case issue in the [Quality Testcases project](https://gitlab.com/gitlab-org/quality/testcases/). | | `:mattermost` | The test requires a GitLab Mattermost service on the GitLab instance. | | `:ldap_no_server` | The test requires a GitLab instance to be configured to use LDAP. To be used with the `:orchestrated` tag. It does not spin up an LDAP server at orchestration time. Instead, it creates the LDAP server at runtime. | @@ -33,7 +33,7 @@ This is a partial list of the [RSpec metadata](https://relishapp.com/rspec/rspec | `:smtp` | The test requires a GitLab instance to be configured to use an SMTP server. Tests SMTP notification email delivery from GitLab by using MailHog. | | `:group_saml` | The test requires a GitLab instance that has SAML SSO enabled at the group level. Interacts with an external SAML identity provider. Paired with the `:orchestrated` tag. | | `:instance_saml` | The test requires a GitLab instance that has SAML SSO enabled at the instance level. Interacts with an external SAML identity provider. Paired with the `:orchestrated` tag. | -| `:skip_signup_disabled` | The test uses UI to sign up a new user and will be skipped in any environment that does not allow new user registration via the UI. | +| `:skip_signup_disabled` | The test uses UI to sign up a new user and is skipped in any environment that does not allow new user registration via the UI. | | `:smoke` | The test belongs to the test suite which verifies basic functionality of a GitLab instance.| | `:github` | The test requires a GitHub personal access token. | | `:repository_storage` | The test requires a GitLab instance to be configured to use multiple [repository storage paths](../../../administration/repository_storage_paths.md). Paired with the `:orchestrated` tag. | diff --git a/doc/development/testing_guide/end_to_end/running_tests_that_require_special_setup.md b/doc/development/testing_guide/end_to_end/running_tests_that_require_special_setup.md index df4b833526c..8a49c333f9f 100644 --- a/doc/development/testing_guide/end_to_end/running_tests_that_require_special_setup.md +++ b/doc/development/testing_guide/end_to_end/running_tests_that_require_special_setup.md @@ -1,7 +1,7 @@ --- stage: none group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- # Running tests that require special setup @@ -10,7 +10,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w The [`jenkins_build_status_spec`](https://gitlab.com/gitlab-org/gitlab/blob/163c8a8c814db26d11e104d1cb2dcf02eb567dbe/qa/qa/specs/features/ee/browser_ui/3_create/jenkins/jenkins_build_status_spec.rb) spins up a Jenkins instance in a Docker container based on an image stored in the [GitLab-QA container registry](https://gitlab.com/gitlab-org/gitlab-qa/container_registry). The Docker image it uses is preconfigured with some base data and plugins. -The test then configures the GitLab plugin in Jenkins with a URL of the GitLab instance that will be used +The test then configures the GitLab plugin in Jenkins with a URL of the GitLab instance that are used to run the tests. Unfortunately, the GitLab Jenkins plugin does not accept ports so `http://localhost:3000` would not be accepted. Therefore, this requires us to run GitLab on port 80 or inside a Docker container. @@ -30,7 +30,7 @@ To run the tests from the `/qa` directory: CHROME_HEADLESS=false bin/qa Test::Instance::All http://localhost -- qa/specs/features/ee/browser_ui/3_create/jenkins/jenkins_build_status_spec.rb ``` -The test will automatically spin up a Docker container for Jenkins and tear down once the test completes. +The test automatically spins up a Docker container for Jenkins and tear down once the test completes. However, if you need to run Jenkins manually outside of the tests, use this command: @@ -43,7 +43,7 @@ docker run \ registry.gitlab.com/gitlab-org/gitlab-qa/jenkins-gitlab:version1 ``` -Jenkins will be available on `http://localhost:8080`. +Jenkins is available on `http://localhost:8080`. Admin username is `admin` and password is `password`. @@ -59,19 +59,19 @@ the Docker Engine. The tests tagged `:gitaly_ha` are orchestrated tests that can only be run against a set of Docker containers as configured and started by [the `Test::Integration::GitalyCluster` GitLab QA scenario](https://gitlab.com/gitlab-org/gitlab-qa/-/blob/master/docs/what_tests_can_be_run.md#testintegrationgitalycluster-ceeefull-image-address). -As described in the documentation about the scenario noted above, the following command will run the tests: +As described in the documentation about the scenario noted above, the following command runs the tests: ```shell gitlab-qa Test::Integration::GitalyCluster EE ``` -However, that will remove the containers after it finishes running the tests. If you would like to do further testing, for example, if you would like to run a single test via a debugger, you can use [the `--no-tests` option](https://gitlab.com/gitlab-org/gitlab-qa#command-line-options) to make `gitlab-qa` skip running the tests, and to leave the containers running so that you can continue to use them. +However, that removes the containers after it finishes running the tests. If you would like to do further testing, for example, if you would like to run a single test via a debugger, you can use [the `--no-tests` option](https://gitlab.com/gitlab-org/gitlab-qa#command-line-options) to make `gitlab-qa` skip running the tests, and to leave the containers running so that you can continue to use them. ```shell gitlab-qa Test::Integration::GitalyCluster EE --no-tests ``` -When all the containers are running you will see the output of the `docker ps` command, showing on which ports the GitLab container can be accessed. For example: +When all the containers are running, the output of the `docker ps` command shows which ports the GitLab container can be accessed on. For example: ```plaintext CONTAINER ID ... PORTS NAMES @@ -82,7 +82,7 @@ That shows that the GitLab instance running in the `gitlab-gitaly-ha` container Another option is to use NGINX. -In both cases you will need to configure your machine to translate `gitlab-gitlab-ha.test` into an appropriate IP address: +In both cases you must configure your machine to translate `gitlab-gitlab-ha.test` into an appropriate IP address: ```shell echo '127.0.0.1 gitlab-gitaly-ha.test' | sudo tee -a /etc/hosts @@ -205,7 +205,7 @@ You can now search through the logs for *Job log*, which matches delimited secti A Job log is a subsection within these logs, related to app deployment. We use two jobs: `build` and `production`. You can find the root causes of deployment failures in these logs, which can compromise the entire test. -If a `build` job fails, the `production` job won't run, and the test fails. +If a `build` job fails, the `production` job doesn't run, and the test fails. The long test setup does not take screenshots of failures, which is a known [issue](https://gitlab.com/gitlab-org/quality/team-tasks/-/issues/270). However, if the spec fails (after a successful deployment) then you should be able to find screenshots which display the feature failure. @@ -286,7 +286,7 @@ CHROME_HEADLESS=false bundle exec bin/qa QA::EE::Scenario::Test::Geo --primary-a You can use [GitLab-QA Orchestrator](https://gitlab.com/gitlab-org/gitlab-qa) to orchestrate two GitLab containers and configure them as a Geo setup. -Geo requires an EE license. To visit the Geo sites in your browser, you will need a reverse proxy server (for example, [NGINX](https://www.nginx.com/)). +Geo requires an EE license. To visit the Geo sites in your browser, you need a reverse proxy server (for example, [NGINX](https://www.nginx.com/)). 1. Export your EE license @@ -296,7 +296,7 @@ Geo requires an EE license. To visit the Geo sites in your browser, you will nee 1. (Optional) Pull the GitLab image - This step is optional because pulling the Docker image is part of the [`Test::Integration::Geo` orchestrated scenario](https://gitlab.com/gitlab-org/gitlab-qa/-/blob/d8c5c40607c2be0eda58bbca1b9f534b00889a0b/lib/gitlab/qa/scenario/test/integration/geo.rb). However, it's easier to monitor the download progress if you pull the image first, and the scenario will skip this step after checking that the image is up to date. + This step is optional because pulling the Docker image is part of the [`Test::Integration::Geo` orchestrated scenario](https://gitlab.com/gitlab-org/gitlab-qa/-/blob/d8c5c40607c2be0eda58bbca1b9f534b00889a0b/lib/gitlab/qa/scenario/test/integration/geo.rb). However, it's easier to monitor the download progress if you pull the image first, and the scenario skips this step after checking that the image is up to date. ```shell # For the most recent nightly image @@ -309,7 +309,7 @@ Geo requires an EE license. To visit the Geo sites in your browser, you will nee docker pull registry.gitlab.com/gitlab-org/build/omnibus-gitlab-mirror/gitlab-ee:examplesha123456789 ``` -1. Run the [`Test::Integration::Geo` orchestrated scenario](https://gitlab.com/gitlab-org/gitlab-qa/-/blob/d8c5c40607c2be0eda58bbca1b9f534b00889a0b/lib/gitlab/qa/scenario/test/integration/geo.rb) with the `--no-teardown` option to build the GitLab containers, configure the Geo setup, and run Geo end-to-end tests. Running the tests after the Geo setup is complete is optional; the containers will keep running after you stop the tests. +1. Run the [`Test::Integration::Geo` orchestrated scenario](https://gitlab.com/gitlab-org/gitlab-qa/-/blob/d8c5c40607c2be0eda58bbca1b9f534b00889a0b/lib/gitlab/qa/scenario/test/integration/geo.rb) with the `--no-teardown` option to build the GitLab containers, configure the Geo setup, and run Geo end-to-end tests. Running the tests after the Geo setup is complete is optional; the containers keep running after you stop the tests. ```shell # Using the most recent nightly image diff --git a/doc/development/testing_guide/end_to_end/style_guide.md b/doc/development/testing_guide/end_to_end/style_guide.md index e6c96bca93e..ac4d26df794 100644 --- a/doc/development/testing_guide/end_to_end/style_guide.md +++ b/doc/development/testing_guide/end_to_end/style_guide.md @@ -1,7 +1,7 @@ --- stage: none group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- # Style guide for writing end-to-end tests @@ -74,7 +74,8 @@ We follow a simple formula roughly based on Hungarian notation. - `_tab` - `_menu_item` -*Note: If none of the listed types are suitable, please open a merge request to add an appropriate type to the list.* +NOTE: +If none of the listed types are suitable, please open a merge request to add an appropriate type to the list. ### Examples |