diff options
author | GitLab Bot <gitlab-bot@gitlab.com> | 2020-05-20 14:34:42 +0000 |
---|---|---|
committer | GitLab Bot <gitlab-bot@gitlab.com> | 2020-05-20 14:34:42 +0000 |
commit | 9f46488805e86b1bc341ea1620b866016c2ce5ed (patch) | |
tree | f9748c7e287041e37d6da49e0a29c9511dc34768 /doc/development/testing_guide | |
parent | dfc92d081ea0332d69c8aca2f0e745cb48ae5e6d (diff) | |
download | gitlab-ce-9f46488805e86b1bc341ea1620b866016c2ce5ed.tar.gz |
Add latest changes from gitlab-org/gitlab@13-0-stable-ee
Diffstat (limited to 'doc/development/testing_guide')
-rw-r--r-- | doc/development/testing_guide/best_practices.md | 60 | ||||
-rw-r--r-- | doc/development/testing_guide/end_to_end/beginners_guide.md | 340 | ||||
-rw-r--r-- | doc/development/testing_guide/end_to_end/img/gl-devops-lifecycle-by-stage-numbers_V12_10.png | bin | 0 -> 28571 bytes | |||
-rw-r--r-- | doc/development/testing_guide/end_to_end/index.md | 2 | ||||
-rw-r--r-- | doc/development/testing_guide/end_to_end/page_objects.md | 47 | ||||
-rw-r--r-- | doc/development/testing_guide/end_to_end/quick_start_guide.md | 621 | ||||
-rw-r--r-- | doc/development/testing_guide/end_to_end/rspec_metadata_tests.md | 2 | ||||
-rw-r--r-- | doc/development/testing_guide/end_to_end/style_guide.md | 1 | ||||
-rw-r--r-- | doc/development/testing_guide/flaky_tests.md | 2 | ||||
-rw-r--r-- | doc/development/testing_guide/frontend_testing.md | 60 | ||||
-rw-r--r-- | doc/development/testing_guide/index.md | 12 | ||||
-rw-r--r-- | doc/development/testing_guide/review_apps.md | 41 | ||||
-rw-r--r-- | doc/development/testing_guide/testing_levels.md | 56 |
13 files changed, 495 insertions, 749 deletions
diff --git a/doc/development/testing_guide/best_practices.md b/doc/development/testing_guide/best_practices.md index 62767180077..f0137e542cc 100644 --- a/doc/development/testing_guide/best_practices.md +++ b/doc/development/testing_guide/best_practices.md @@ -17,24 +17,22 @@ our test design. We can find some helpful heuristics documented in the Handbook ## Test speed -GitLab has a massive test suite that, without [parallelization], can take hours +GitLab has a massive test suite that, without [parallelization](ci.md#test-suite-parallelization-on-the-ci), can take hours to run. It's important that we make an effort to write tests that are accurate and effective _as well as_ fast. Here are some things to keep in mind regarding test performance: -- `double` and `spy` are faster than `FactoryBot.build(...)` +- `instance_double` and `spy` are faster than `FactoryBot.build(...)` - `FactoryBot.build(...)` and `.build_stubbed` are faster than `.create`. - Don't `create` an object when `build`, `build_stubbed`, `attributes_for`, - `spy`, or `double` will do. Database persistence is slow! + `spy`, or `instance_double` will do. Database persistence is slow! - Don't mark a feature as requiring JavaScript (through `:js` in RSpec) unless it's _actually_ required for the test to be valid. Headless browser testing is slow! -[parallelization]: ci.md#test-suite-parallelization-on-the-ci - ## RSpec -To run rspec tests: +To run RSpec tests: ```shell # run all tests @@ -71,6 +69,8 @@ FDOC=1 bin/rspec spec/[path]/[to]/[spec].rb - Use `Gitlab.config.gitlab.host` rather than hard coding `'localhost'` - Don't assert against the absolute value of a sequence-generated attribute (see [Gotchas](../gotchas.md#do-not-assert-against-the-absolute-value-of-a-sequence-generated-attribute)). +- Avoid using `expect_any_instance_of` or `allow_any_instance_of` (see + [Gotchas](../gotchas.md#do-not-assert-against-the-absolute-value-of-a-sequence-generated-attribute)). - Don't supply the `:each` argument to hooks since it's the default. - On `before` and `after` hooks, prefer it scoped to `:context` over `:all` - When using `evaluate_script("$('.js-foo').testSomething()")` (or `execute_script`) which acts on a given element, @@ -320,26 +320,48 @@ stub_feature_flags(ci_live_trace: false) Feature.enabled?(:ci_live_trace) # => false ``` -If you wish to set up a test where a feature flag is disabled for some -actors and not others, you can specify this in options passed to the -helper. For example, to disable the `ci_live_trace` feature flag for a -specifc project: +If you wish to set up a test where a feature flag is enabled only +for some actors and not others, you can specify this in options +passed to the helper. For example, to enable the `ci_live_trace` +feature flag for a specifc project: ```ruby project1, project2 = build_list(:project, 2) -# Feature will only be disabled for project1 -stub_feature_flags(ci_live_trace: { enabled: false, thing: project1 }) +# Feature will only be enabled for project1 +stub_feature_flags(ci_live_trace: project1) + +Feature.enabled?(:ci_live_trace) # => false +Feature.enabled?(:ci_live_trace, project1) # => true +Feature.enabled?(:ci_live_trace, project2) # => false +``` + +This represents an actual behavior of FlipperGate: -Feature.enabled?(:ci_live_trace, project1) # => false -Feature.enabled?(:ci_live_trace, project2) # => true +1. You can enable an override for a specified actor to be enabled +1. You can disable (remove) an override for a specified actor, + fallbacking to default state +1. There's no way to model that you explicitly disable a specified actor + +```ruby +Feature.enable(:my_feature) +Feature.disable(:my_feature, project1) +Feature.enabled?(:my_feature) # => true +Feature.enabled?(:my_feature, project1) # => true +``` + +```ruby +Feature.disable(:my_feature2) +Feature.enable(:my_feature2, project1) +Feature.enabled?(:my_feature2) # => false +Feature.enabled?(:my_feature2, project1) # => true ``` ### Pristine test environments The code exercised by a single GitLab test may access and modify many items of data. Without careful preparation before a test runs, and cleanup afterward, -data can be changed by a test in such a way that it affects the behaviour of +data can be changed by a test in such a way that it affects the behavior of following tests. This should be avoided at all costs! Fortunately, the existing test framework handles most cases already. @@ -493,7 +515,7 @@ range of inputs. By specifying the test case once, alongside a table of inputs and the expected output for each, your tests can be made easier to read and more compact. -We use the [rspec-parameterized](https://github.com/tomykaira/rspec-parameterized) +We use the [RSpec::Parameterized](https://github.com/tomykaira/rspec-parameterized) gem. A short example, using the table syntax and checking Ruby equality for a range of inputs, might look like this: @@ -526,7 +548,7 @@ objects, FactoryBot-created objects etc. can lead to ### Prometheus tests Prometheus metrics may be preserved from one test run to another. To ensure that metrics are -reset before each example, add the `:prometheus` tag to the Rspec test. +reset before each example, add the `:prometheus` tag to the RSpec test. ### Matchers @@ -651,7 +673,7 @@ end ### Factories -GitLab uses [factory_bot] as a test fixture replacement. +GitLab uses [factory_bot](https://github.com/thoughtbot/factory_bot) as a test fixture replacement. - Factory definitions live in `spec/factories/`, named using the pluralization of their corresponding model (`User` factories are defined in `users.rb`). @@ -666,8 +688,6 @@ GitLab uses [factory_bot] as a test fixture replacement. - Factories don't have to be limited to `ActiveRecord` objects. [See example](https://gitlab.com/gitlab-org/gitlab-foss/commit/0b8cefd3b2385a21cfed779bd659978c0402766d). -[factory_bot]: https://github.com/thoughtbot/factory_bot - ### Fixtures All fixtures should be placed under `spec/fixtures/`. diff --git a/doc/development/testing_guide/end_to_end/beginners_guide.md b/doc/development/testing_guide/end_to_end/beginners_guide.md new file mode 100644 index 00000000000..73960a2f74d --- /dev/null +++ b/doc/development/testing_guide/end_to_end/beginners_guide.md @@ -0,0 +1,340 @@ +# Beginner's guide to writing end-to-end tests + +In this tutorial, you will learn about the creation of end-to-end (_e2e_) tests +for [GitLab Community Edition](https://about.gitlab.com/install/?version=ce) and +[GitLab Enterprise Edition](https://about.gitlab.com/install/). + +By the end of this tutorial, you will be able to: + +- Determine whether an end-to-end test is needed. +- Understand the directory structure within `qa/`. +- Write a basic end-to-end test that will validate login features. +- Develop any missing [page object](page_objects.md) libraries. + +## Before you write a test + +Before you write tests, your +[GitLab Development Kit (GDK)](https://gitlab.com/gitlab-org/gitlab-development-kit) +must be configured to run the specs. The end-to-end tests: + +- Are contained within the `qa/` directory. +- Should be independent and + [idempotent](https://en.wikipedia.org/wiki/Idempotence#Computer_science_meaning). +- Create [resources](resources.md) (such as project, issue, user) on an ad-hoc basis. +- Test the UI and API interfaces, and use the API to efficiently set up the UI tests. + +TIP: **Tip:** +For more information, see [End-to-end testing Best Practices](best_practices.md). + +## Determine if end-to-end tests are needed + +Check the code coverage of a specific feature before writing end-to-end tests, +for both [GitLab Community Edition](https://gitlab-org.gitlab.io/gitlab-foss/coverage-ruby/#_AllFiles) +and [GitLab Enterprise Edition](https://gitlab-org.gitlab.io/gitlab/coverage-ruby/#_AllFiles) projects. +Does sufficient test coverage exist at the unit, feature, or integration levels? +If you answered *yes*, then you *don't* need an end-to-end test. + +For information about the distribution of tests per level in GitLab, see +[Testing Levels](https://gitlab.com/gitlab-org/gitlab/blob/master/doc/development/testing_guide/testing_levels.md). + +- See the + [How to test at the correct level?](https://gitlab.com/gitlab-org/gitlab/blob/master/doc/development/testing_guide/testing_levels.md#how-to-test-at-the-correct-level) + section of the [Testing levels](https://gitlab.com/gitlab-org/gitlab/blob/master/doc/development/testing_guide/testing_levels.md) document. +- Review how often the feature changes. Stable features that don't change very often + might not be worth covering with end-to-end tests if they are already covered + in lower level tests. +- Finally, discuss the proposed test with the developer(s) involved in implementing + the feature and the lower-level tests. + +CAUTION: **Caution:** +Check both [GitLab Community Edition](https://gitlab-org.gitlab.io/gitlab-foss/coverage-ruby/#_AllFiles) and +[GitLab Enterprise Edition](https://gitlab-org.gitlab.io/gitlab/coverage-ruby/#_AllFiles) coverage projects +for previously-written tests for this feature. For analyzing the code coverage, +you must understand which application files implement specific features. + +NOTE: **Note:** +In this tutorial we're writing a login end-to-end test, even though it has been +sufficiently covered by lower-level testing, because it's the first step for most +end-to-end flows, and is easiest to understand. + +## Identify the DevOps stage + +The GitLab QA end-to-end tests are organized by the different +[stages in the DevOps lifecycle](https://gitlab.com/gitlab-org/gitlab-foss/tree/master/qa/qa/specs/features/browser_ui). +Determine where the test should be placed by +[stage](https://about.gitlab.com/handbook/product/categories/#devops-stages), +determine which feature the test will belong to, and then place it in a subdirectory +under the stage. + +![DevOps lifecycle by stages](img/gl-devops-lifecycle-by-stage-numbers_V12_10.png) + +NOTE: **Note:** +If the test is Enterprise Edition only, the test will be created in the `features/ee` +directory, but follow the same DevOps lifecycle format. + +## Create a skeleton test + +In the first part of this tutorial we will be testing login, which is owned by the +Manage stage. Inside `qa/specs/features/browser_ui/1_manage/login`, create a +file `basic_login_spec.rb`. + +### The outer `context` block + +Specs have an outer `context` indicating the DevOps stage. + +```ruby +# frozen_string_literal: true + +module QA + context 'Manage' do + + end +end +``` + +### The `describe` block + +Inside of our outer `context`, describe the feature to test. In this case, `Login`. + +```ruby +# frozen_string_literal: true + +module QA + context 'Manage' do + describe 'Login' do + + end + end +end +``` + +### The `it` blocks (examples) + +Every test suite contains at least one `it` block (example). A good way to start +writing end-to-end tests is to write test case descriptions as `it` blocks: + +```ruby +module QA + context 'Manage' do + describe 'Login' do + it 'can login' do + + end + + it 'can logout' do + + end + end + end +end +``` + +## Write the test + +An important question is "What do we test?" and even more importantly, "How do we test?" + +Begin by logging in. + +```ruby +# frozen_string_literal: true + +module QA + context 'Manage' do + describe 'Login' do + it 'can login' do + Flow::Login.sign_in + + end + + it 'can logout' do + Flow::Login.sign_in + + end + end + end +end +``` + +After [running the spec](#run-the-spec), our test should login and end; then we +should answer the question "What do we test?" + +```ruby +# frozen_string_literal: true + +module QA + context 'Manage' do + describe 'Login' do + it 'can login' do + Flow::Login.sign_in + + Page::Main::Menu.perform do |menu| + expect(menu).to be_signed_in + end + end + + it 'can logout' do + Flow::Login.sign_in + + Page::Main::Menu.perform do |menu| + menu.sign_out + + expect(menu).not_to be_signed_in + end + end + end + end +end +``` + +**What do we test?** + +1. Can we log in? +1. Can we log out? + +**How do we test?** + +1. Check if the user avatar appears in the top navigation. +1. Check if the user avatar *does not* appear in the top navigation. + +NOTE: **Note:** +Behind the scenes, `be_signed_in` is a +[predicate matcher](https://relishapp.com/rspec/rspec-expectations/v/3-8/docs/built-in-matchers/predicate-matchers) +that [implements checking the user avatar](https://gitlab.com/gitlab-org/gitlab/-/blob/master/qa/qa/page/main/menu.rb#L74). + +## De-duplicate your code + +Refactor your test to use a `before` block for test setup, since it's duplicating +a call to `sign_in`. + +```ruby +# frozen_string_literal: true + +module QA + context 'Manage' do + describe 'Login' do + before do + Flow::Login.sign_in + end + + it 'can login' do + Page::Main::Menu.perform do |menu| + expect(menu).to be_signed_in + end + end + + it 'can logout' do + Page::Main::Menu.perform do |menu| + menu.sign_out + + expect(menu).not_to be_signed_in + end + end + end + end +end +``` + +The `before` block is essentially a `before(:each)` and is run before each example, +ensuring we now log in at the beginning of each test. + +## Test setup using resources and page objects + +Next, let's test something other than Login. Let's test Issues, which are owned by the Plan +stage, so [create a file](#identify-the-devops-stage) in +`qa/specs/features/browser_ui/3_create/issues` called `issues_spec.rb`. + +```ruby +# frozen_string_literal: true + +module QA + context 'Plan' do + describe 'Issues' do + let(:issue) do + Resource::Issue.fabricate_via_api! do |issue| + issue.title = 'My issue' + issue.description = 'This is an issue specific to this test' + end + end + + before do + Flow::Login.sign_in + issue.visit! + end + + it 'can close an issue' do + Page::Project::Issue::Show.perform do |show| + show.click_close_issue_button + + expect(show).to be_closed + end + end + end + end +end +``` + +Note the following important points: + +- At the start of our example, we will be at the `page/issue/show.rb` [page](page_objects.md). +- Our test fabricates only what it needs, when it needs it. +- The issue is fabricated through the API to save time. +- GitLab prefers `let()` over instance variables. See + [best practices](../best_practices.md#let-variables). +- `be_closed` is not implemented in `page/project/issue/show.rb` yet, but will be + implemented in the next step. + +The issue is fabricated as a [Resource](resources.md), which is a GitLab entity +you can create through the UI or API. Other examples include: + +- A [Merge Request](https://gitlab.com/gitlab-org/gitlab/-/blob/master/qa/qa/resource/merge_request.rb). +- A [User](https://gitlab.com/gitlab-org/gitlab/-/blob/master/qa/qa/resource/user.rb). +- A [Project](https://gitlab.com/gitlab-org/gitlab/-/blob/master/qa/qa/resource/project.rb). +- A [Group](https://gitlab.com/gitlab-org/gitlab/-/blob/master/qa/qa/resource/group.rb). + +## Write the page object + +A [Page Object](page_objects.md) is a class in our suite that represents a page +within GitLab. The **Login** page would be one example. Since our page object for +the **Issue Show** page already exists, add the `closed?` method. + +```ruby +module Page::Project::Issue + class Show + view 'app/views/projects/issues/show.html.haml' do + element :closed_status_box + end + + def closed? + has_element?(:closed_status_box) + end + end +end +``` + +Next, define the element `closed_status_box` within your view, so your Page Object +can see it. + +```haml +-#=> app/views/projects/issues/show.html.haml +.issuable-status-box.status-box.status-box-issue-closed{ ..., data: { qa_selector: 'closed_status_box' } } +``` + +## Run the spec + +Before running the spec, confirm: + +- The GDK is installed. +- The GDK is running on port 3000 locally. +- No additional [RSpec metadata tags](rspec_metadata_tests.md) have been applied. +- Your working directory is `qa/` within your GDK GitLab installation. + +To run the spec, run the following command: + +```ruby +bundle exec bin/qa Test::Instance::All http://localhost:3000 -- <test_file> +``` + +Where `<test_file>` is: + +- `qa/specs/features/browser_ui/1_manage/login/login_spec.rb` when running the Login example. +- `qa/specs/features/browser_ui/2_plan/issues/issue_spec.rb` when running the Issue example. diff --git a/doc/development/testing_guide/end_to_end/img/gl-devops-lifecycle-by-stage-numbers_V12_10.png b/doc/development/testing_guide/end_to_end/img/gl-devops-lifecycle-by-stage-numbers_V12_10.png Binary files differnew file mode 100644 index 00000000000..d11305c3686 --- /dev/null +++ b/doc/development/testing_guide/end_to_end/img/gl-devops-lifecycle-by-stage-numbers_V12_10.png diff --git a/doc/development/testing_guide/end_to_end/index.md b/doc/development/testing_guide/end_to_end/index.md index 443b7b06a24..e6a683e9148 100644 --- a/doc/development/testing_guide/end_to_end/index.md +++ b/doc/development/testing_guide/end_to_end/index.md @@ -180,7 +180,7 @@ instance-level scenarios](https://gitlab.com/gitlab-org/gitlab-foss/tree/master/ Continued reading: -- [Quick Start Guide](quick_start_guide.md) +- [Beginner's Guide](beginners_guide.md) - [Style Guide](style_guide.md) - [Best Practices](best_practices.md) - [Testing with feature flags](feature_flags.md) diff --git a/doc/development/testing_guide/end_to_end/page_objects.md b/doc/development/testing_guide/end_to_end/page_objects.md index 22e7375be1f..9d4fa5316c4 100644 --- a/doc/development/testing_guide/end_to_end/page_objects.md +++ b/doc/development/testing_guide/end_to_end/page_objects.md @@ -238,6 +238,53 @@ the view for code in a library. In such rare cases it's reasonable to use CSS selectors in page object methods, with a comment explaining why an `element` can't be added. +### Define Page concerns + +Some pages share common behaviors, and/or are prepended with EE-specific modules that adds EE-specific methods. + +These modules must: + +1. Extend from the `QA::Page::PageConcern` module, with `extend QA::Page::PageConcern`. +1. Override the `self.prepended` method if they need to `include`/`prepend` other modules themselves, and/or define + `view` or `elements`. +1. Call `super` as the first thing in `self.prepended`. +1. Include/prepend other modules and define their `view`/`elements` in a `base.class_eval` block to ensure they're + defined in the class that prepends the module. + +These steps ensure the sanity selectors check will detect problems properly. + +For example, `qa/qa/ee/page/merge_request/show.rb` adds EE-specific methods to `qa/qa/page/merge_request/show.rb` (with +`QA::Page::MergeRequest::Show.prepend_if_ee('QA::EE::Page::MergeRequest::Show')`) and following is how it's implemented +(only showing the relevant part and refering to the 4 steps described above with inline comments): + +```ruby +module QA + module EE + module Page + module MergeRequest + module Show + extend QA::Page::PageConcern # 1. + + def self.prepended(base) # 2. + super # 3. + + base.class_eval do # 4. + prepend Page::Component::LicenseManagement + + view 'app/assets/javascripts/vue_merge_request_widget/components/states/sha_mismatch.vue' do + element :head_mismatch, "The source branch HEAD has recently changed." + end + + [...] + end + end + end + end + end + end +end +``` + ## Running the test locally During development, you can run the `qa:selectors` test by running diff --git a/doc/development/testing_guide/end_to_end/quick_start_guide.md b/doc/development/testing_guide/end_to_end/quick_start_guide.md deleted file mode 100644 index 0ae3f375284..00000000000 --- a/doc/development/testing_guide/end_to_end/quick_start_guide.md +++ /dev/null @@ -1,621 +0,0 @@ -# Writing end-to-end tests step-by-step - -In this tutorial, you will find different examples, and the steps involved, in the creation of end-to-end (_e2e_) tests for GitLab CE and GitLab EE, using GitLab QA. - -When referring to end-to-end tests in this document, this means testing a specific feature end-to-end such as: - -- A user logging in. -- The creation of a project. -- The management of labels. -- Breaking down epics into sub-epics and issues. - -## Important information before we start writing tests - -It's important to understand that end-to-end tests of isolated features, such as the ones described in the above note, doesn't mean that everything needs to happen through the GUI. - -If you don't exactly understand what we mean by **not everything needs to happen through the GUI,** please make sure you've read the [best practices](best_practices.md) before moving on. - -## This document covers the following items - -- [0.](#0-are-end-to-end-tests-needed) Identifying if end-to-end tests are really needed -- [1.](#1-identifying-the-devops-stage) Identifying the [DevOps stage](https://about.gitlab.com/stages-devops-lifecycle/) of the feature that you are going to cover with end-to-end tests -- [2.](#2-test-skeleton) Creating the skeleton of the test file (`*_spec.rb`) -- [3.](#3-test-cases-mvc) The [MVC](https://about.gitlab.com/handbook/values/#minimum-viable-change-mvc) of the test cases' logic -- [4.](#4-extracting-duplicated-code) Extracting duplicated code into methods -- [5.](#5-tests-pre-conditions-using-resources-and-page-objects) Tests' pre-conditions (`before :context` and `before`) using resources and [Page Objects](page_objects.md) -- [6.](#6-optimization) Optimizing the test suite -- [7.](#7-resources) Using and implementing resources -- [8.](#8-page-objects) Moving element definitions and methods to [Page Objects](page_objects.md) - -### 0. Are end-to-end tests needed? - -At GitLab we respect the [test pyramid](https://gitlab.com/gitlab-org/gitlab/blob/master/doc/development/testing_guide/testing_levels.md), and so, we recommend you check the code coverage of a specific feature before writing end-to-end tests, for both [CE](https://gitlab-org.gitlab.io/gitlab-foss/coverage-ruby/#_AllFiles) and [EE](https://gitlab-org.gitlab.io/gitlab/coverage-ruby/#_AllFiles) projects. - -Sometimes you may notice that there is already good coverage in lower test levels, and we can stay confident that if we break a feature, we will still have quick feedback about it, even without having end-to-end tests. - -> For analyzing the code coverage, you will also need to understand which application files implement specific functionalities. - -#### Some other guidelines are as follows - -- Take a look at the [How to test at the correct level?](https://gitlab.com/gitlab-org/gitlab/blob/master/doc/development/testing_guide/testing_levels.md#how-to-test-at-the-correct-level) section of the [Testing levels](https://gitlab.com/gitlab-org/gitlab/blob/master/doc/development/testing_guide/testing_levels.md) document - -- Look into the frequency in which such a feature is changed (_Stable features that don't change very often might not be worth covering with end-to-end tests if they're already covered in lower levels_) - -- Finally, discuss with the developer(s) involved in developing the feature and the tests themselves, to get their feeling - -If after this analysis you still think that end-to-end tests are needed, keep reading. - -### 1. Identifying the DevOps stage - -The GitLab QA end-to-end tests are organized by the different [stages in the DevOps lifecycle](https://gitlab.com/gitlab-org/gitlab-foss/tree/master/qa/qa/specs/features/browser_ui), and so, if you are creating tests for issue creation, for instance, you would locate the spec files under the `qa/qa/specs/features/browser_ui/2_plan/` directory since issue creation is part of the Plan stage. - - In another case of a test for listing merged merge requests (MRs), the test should go under the `qa/qa/specs/features/browser_ui/3_create/` directory since merge requests are a feature from the Create stage. - -> There may be sub-directories inside the stages directories, for different features. For example: `.../browser_ui/2_plan/ee_epics/` and `.../browser_ui/2_plan/issues/`. - -Now, let's say we want to create tests for the [scoped labels](https://about.gitlab.com/releases/2019/04/22/gitlab-11-10-released/#scoped-labels) feature, available on GitLab EE Premium (this feature is part of the Plan stage.) - -> Because these tests are for a feature available only on GitLab EE, we need to create them in the [EE repository](https://gitlab.com/gitlab-org/gitlab). - -Since [there is no specific directory for this feature](https://gitlab.com/gitlab-org/gitlab/tree/master/qa/qa/specs/features/browser_ui/2_plan), we should create a sub-directory for it. - -Under `.../browser_ui/2_plan/`, let's create a sub-directory called `ee_scoped_labels/`. - -> Notice that since this feature is only available for GitLab EE we prefix the sub-directory with `ee_`. - -### 2. Test skeleton - -Inside the newly created sub-directory, let's create a file describing the test suite (e.g. `editing_scoped_labels_spec.rb`.) - -#### The `context` and `describe` blocks - -Specs have an outer `context` that indicates the DevOps stage. The next level is the `describe` block, that briefly states the subject of the test suite. See the following example: - -```ruby -module QA - context 'Plan' do - describe 'Editing scoped labels on issues' do - end - end -end -``` - -#### The `it` blocks - -Every test suite is composed of at least one `it` block, and a good way to start writing end-to-end tests is by writing test cases descriptions as `it` blocks. These might help you to think of different test scenarios. Take a look at the following example: - -```ruby -module QA - context 'Plan' do - describe 'Editing scoped labels on issues' do - it 'replaces an existing label if it has the same key' do - end - - it 'keeps both scoped labels when adding a label with a different key' do - end - end - end -end -``` - -### 3. Test cases MVC - -For the [MVC](https://about.gitlab.com/handbook/values/#minimum-viable-change-mvc) of our test cases, let's say that we already have the application in the state needed for the tests, and then let's focus on the logic of the test cases only. - -To evolve the test cases drafted on step 2, let's imagine that the user is already logged into a GitLab EE instance, they already have at least a Premium license in use, there is already a project created, there is already an issue opened in the project, the issue already has a scoped label (e.g. `animal::fox`), there are other scoped labels (for the same scope and for a different scope (e.g. `animal::dolphin` and `plant::orchid`), and finally, the user is already on the issue's page. Let's also suppose that for every test case the application is in a clean state, meaning that one test case won't affect another. - -> Note: there are different approaches to creating an application state for end-to-end tests. Some of them are very time consuming and subject to failures, such as when using the GUI for all the pre-conditions of the tests. On the other hand, other approaches are more efficient, such as using the public APIs. The latter is more efficient since it doesn't depend on the GUI. We won't focus on this part yet, but it's good to keep it in mind. - -Let's now focus on the first test case. - -```ruby -it 'replaces an existing label if it has the same key' do - # This implementation is only for tutorial purposes. We normally encapsulate elements in Page Objects (which we cover on section 8). - page.find('.block.labels .edit-link').click - page.find('.dropdown-menu-labels .dropdown-input-field').send_keys ['animal::dolphin', :enter] - page.find('#content-body').click - page.refresh - - labels_block = page.find(%q([data-qa-selector="labels_block"])) - - expect(labels_block).to have_content('animal::dolphin') - expect(labels_block).not_to have_content('animal::fox') - expect(page).to have_content('added animal::dolphin label and removed animal::fox') -end -``` - -> Notice that the test itself is simple. The most challenging part is the creation of the application state, which will be covered later. -> -> The exemplified test case's MVC is not enough for the change to be merged, but it helps to build up the test logic. The reason is that we do not want to use locators directly in the tests, and tests **must** use [Page Objects](page_objects.md) before they can be merged. This way we better separate the responsibilities, where the Page Objects encapsulate elements and methods that allow us to interact with pages, while the spec files describe the test cases in more business-related language. - -Below are the steps that the test covers: - -1. The test finds the 'Edit' link for the labels and clicks on it. -1. Then it fills in the 'Assign labels' input field with the value 'animal::dolphin' and press enters. -1. Then it clicks in the content body to apply the label and refreshes the page. -1. Finally, the expectations check that the previous scoped label was removed and that the new one was added. - -Let's now see how the second test case would look. - -```ruby -it 'keeps both scoped labels when adding a label with a different key' do - # This implementation is only for tutorial purposes. We normally encapsulate elements in Page Objects (which we cover on section 8). - page.find('.block.labels .edit-link').click - page.find('.dropdown-menu-labels .dropdown-input-field').send_keys ['plant::orchid', :enter] - page.find('#content-body').click - page.refresh - - labels_block = page.find(%q([data-qa-selector="labels_block"])) - - expect(labels_block).to have_content('animal::fox') - expect(labels_block).to have_content('plant::orchid') - expect(page).to have_content('added animal::fox') - expect(page).to have_content('added plant::orchid') -end -``` - -> Note that elements are always located using CSS selectors, and a good practice is to add test-specific selectors (this is called "testability"). For example, the `labels_block` element uses the CSS selector [`data-qa-selector="labels_block"`](page_objects.md#data-qa-selector-vs-qa-selector), which was added specifically for testing purposes. - -Below are the steps that the test covers: - -1. The test finds the 'Edit' link for the labels and clicks on it. -1. Then it fills in the 'Assign labels' input field with the value 'plant::orchid' and press enters. -1. Then it clicks in the content body to apply the label and refreshes the page. -1. Finally, the expectations check that both scoped labels are present. - -> Similar to the previous test, this one is also very straightforward, but there is some code duplication. Let's address it. - -### 4. Extracting duplicated code - -If we refactor the tests created on step 3 we could come up with something like this: - -```ruby -before do - ... - - @initial_label = 'animal::fox' - @new_label_same_scope = 'animal::dolphin' - @new_label_different_scope = 'plant::orchid' - - ... -end - -it 'replaces an existing label if it has the same key' do - select_label_and_refresh @new_label_same_scope - - labels_block = page.find(%q([data-qa-selector="labels_block"])) - - expect(labels_block).to have_content(@new_label_same_scope) - expect(labels_block).not_to have_content(@initial_label) - expect(page).to have_content("added #{@new_label_same_scope}") - expect(page).to have_content("and removed #{@initial_label}") -end - -it 'keeps both scoped label when adding a label with a different key' do - select_label_and_refresh @new_label_different_scope - - labels_block = page.find(%q([data-qa-selector="labels_block"])) - - expect(labels_blocks).to have_content(@new_label_different_scope) - expect(labels_blocks).to have_content(@initial_label) - expect(page).to have_content("added #{@new_label_different_scope}") - expect(page).to have_content("added #{@initial_label}") -end - -def select_label_and_refresh(label) - page.find('.block.labels .edit-link').click - page.find('.dropdown-menu-labels .dropdown-input-field').send_keys [label, :enter] - page.find('#content-body').click - page.refresh -end -``` - -First, we remove the duplication of strings by defining the global variables `@initial_label`, `@new_label_same_scope` and `@new_label_different_scope` in the `before` block, and by using them in the expectations. - -Then, by creating a reusable `select_label_and_refresh` method, we remove the code duplication of this action, and later we can move this method to a Page Object class that will be created for easier maintenance purposes. - -Notice that the reusable method is created at the bottom of the file. This helps readability, -where reading the code should be similar to reading a newspaper: - -- High-level information is at the top, like the title and summary of the news. -- Low level, or more specific information, is at the bottom. - -### 5. Tests' pre-conditions using resources and Page Objects - -In this section, we will address the previously mentioned subject of creating the application state for the tests, using the `before :context` and `before` blocks, together with resources and Page Objects. - -#### `before :context` - -A pre-condition for the entire test suite is defined in the `before :context` block. - -> For our test suite, due to the need of the tests being completely independent of each other, we won't use the `before :context` block. The `before :context` block would make the tests dependent on each other because the first test changes the label of the issue, and the second one depends on the `'animal::fox'` label being set. - -TIP: **Tip:** In case of a test suite with only one `it` block it's ok to use only the `before` block (see below) with all the test's pre-conditions. - -#### `before` - -As the pre-conditions for our test suite, the things that needs to happen before each test starts are: - -- The user logging in; -- A premium license already being set; -- A project being created with an issue and labels already set; -- The issue page being opened with only one scoped label applied to it. - -> When running end-to-end tests as part of the GitLab's continuous integration process [a license is already set as an environment variable](https://gitlab.com/gitlab-org/gitlab/blob/1a60d926740db10e3b5724713285780a4f470531/qa/qa/ee/strategy.rb#L20). For running tests locally you can set up such license by following the document [what tests can be run?](https://gitlab.com/gitlab-org/gitlab-qa/blob/master/docs/what_tests_can_be_run.md), based on the [supported GitLab environment variables](https://gitlab.com/gitlab-org/gitlab-qa/blob/master/docs/what_tests_can_be_run.md#supported-gitlab-environment-variables). - -#### Implementation - -In the following code we will focus only on the test suite's pre-conditions: - -```ruby -module QA - context 'Plan' do - describe 'Editing scoped labels on issues' do - before do - Runtime::Browser.visit(:gitlab, Page::Main::Login) - Page::Main::Login.perform(&:sign_in_using_credentials) - - @initial_label = 'animal::fox' - @new_label_same_scope = 'animal::dolphin' - @new_label_different_scope = 'plant::orchid' - - issue = Resource::Issue.fabricate_via_api! do |issue| - issue.title = 'Issue to test the scoped labels' - issue.labels = [@initial_label] - end - - [@new_label_same_scope, @new_label_different_scope].each do |label| - Resource::Label.fabricate_via_api! do |l| - l.project = issue.project - l.title = label - end - end - - issue.visit! - end - - it 'replaces an existing label if it has the same key' do - ... - end - - it 'keeps both scoped labels when adding a label with a different key' do - ... - end - - def select_label_and_refresh(label) - ... - end - end - end -end -``` - -In the `before` block we create all the application state needed for the tests to run. We do that by using the `Runtime::Browser.visit` method to go to the login page, by performing a `sign_in_using_credentials` from the `Login` Page Object, by fabricating resources via APIs (`issue`, and `Resource::Label`), and by using the `issue.visit!` to visit the issue page. - -> A project is created in the background by creating the `issue` resource. -> -> When creating the [Resources](resources.md), notice that when calling the `fabricate_via_api` method, we pass some attribute:values, like `title`, and `labels` for the `issue` resource; and `project` and `title` for the `label` resource. -> -> What's important to understand here is that by creating the application state mostly using the public APIs we save a lot of time in the test suite setup stage. -> -> Soon we will cover the use of the already existing resources' methods and the creation of your own `fabricate_via_api` methods for resources where this is still not available, but first, let's optimize our implementation. - -### 6. Optimization - -As already mentioned in the [best practices](best_practices.md) document, end-to-end tests are very costly in terms of execution time, and it's our responsibility as software engineers to ensure that we optimize them as much as possible. - -> Note that end-to-end tests are slow to run and so they can have several actions and assertions in a single test, which helps us get feedback from the tests sooner. In comparison, unit tests are much faster to run and can exercise every little piece of the application in isolation, and so they usually have only one assertion per test. - -Some improvements that we could make in our test suite to optimize its time to run are: - -1. Having a single test case (an `it` block) that exercises both scenarios to avoid "wasting" time in the tests' pre-conditions, instead of having two different test cases. -1. Making the selection of labels more performant by allowing for the selection of more than one label in the same reusable method. - -Let's look at a suggestion that addresses the above points, one by one: - -```ruby -module QA - context 'Plan' do - describe 'Editing scoped labels on issues' do - before do - ... - end - - it 'correctly applies scoped labels depending on if they are from the same or a different scope' do - select_labels_and_refresh [@new_label_same_scope, @new_label_different_scope] - - labels_block = page.all(%q([data-qa-selector="labels_block"])) - - expect(labels_block).to have_content(@new_label_same_scope) - expect(labels_block).to have_content(@new_label_different_scope) - expect(labels_block).not_to have_content(@initial_label) - expect(page).to have_content("added #{@initial_label}") - expect(page).to have_content("added #{@new_label_same_scope} #{@new_label_different_scope} labels and removed #{@initial_label}") - end - - def select_labels_and_refresh(labels) - find('.block.labels .edit-link').click - labels.each do |label| - find('.dropdown-menu-labels .dropdown-input-field').send_keys [label, :enter] - end - find('#content-body').click - refresh - end - end - end - end -``` - -To address point 1, we changed the test implementation from two `it` blocks into a single one that exercises both scenarios. Now the new test description is: `'correctly applies the scoped labels depending if they are from the same or a different scope'`. It's a long description, but it describes well what the test does. - -> Notice that the implementation of the new and unique `it` block had to change a little bit. Below we describe in details what it does. - -1. It selects two scoped labels simultaneously, one from the same scope of the one already applied in the issue during the setup phase (in the `before` block), and another one from a different scope. -1. It asserts that the correct labels are visible in the `labels_block`, and that the labels were correctly added and removed; -1. Finally, the `select_label_and_refresh` method is changed to `select_labels_and_refresh`, which accepts an array of labels instead of a single label, and it iterates on them for faster label selection (this is what is used in step 1 explained above.) - -### 7. Resources - -**Note:** When writing this document, some code that is now merged to master was not implemented yet, but we left them here for the readers to understand the whole process of end-to-end test creation. - -You can think of [Resources](resources.md) as anything that can be created on GitLab CE or EE, either through the GUI, the API, or the CLI. - -With that in mind, resources can be a project, an epic, an issue, a label, a commit, etc. - -As you saw in the tests' pre-conditions and the optimization sections, we're already creating some of these resources. We are doing that by calling the `fabricate_via_api!` method. - -> We could be using the `fabricate!` method instead, which would use the `fabricate_via_api!` method if it exists, and fallback to GUI fabrication otherwise, but we recommend being explicit to make it clear what the test does. Also, we always recommend fabricating resources via API since this makes tests faster and more reliable. - -For our test suite example, the resources that we need to create don't have the necessary code for the `fabricate_via_api!` method to correctly work (e.g., the issue and label resources), so we will have to create them. - -#### Implementation - -In the following we describe the changes needed in each of the resource files mentioned above. - -**Issue resource** - -Now, let's make it possible to create an issue resource through the API. - -First, in the [issue resource](https://gitlab.com/gitlab-org/gitlab/blob/d3584e80b4236acdf393d815d604801573af72cc/qa/qa/resource/issue.rb), let's expose its id and labels attributes. - -Add the following `attribute :id` and `attribute :labels` right above the [`attribute :title`](https://gitlab.com/gitlab-org/gitlab/blob/d3584e80b4236acdf393d815d604801573af72cc/qa/qa/resource/issue.rb#L15). - -> This line is needed to allow for the issue fabrication, and for labels to be automatically added to the issue when fabricating it via API. -> -> We add the attributes above the existing attribute to keep them alphabetically organized. - -Then, let's initialize an instance variable for labels to allow an empty array as default value when such information is not passed during the resource fabrication, since this optional. [Between the attributes and the `fabricate!` method](https://gitlab.com/gitlab-org/gitlab/blob/1a1f1408728f19b2aa15887cd20bddab7e70c8bd/qa/qa/resource/issue.rb#L18), add the following: - -```ruby -def initialize - @labels = [] -end -``` - -Next, add the following code right below the [`fabricate!`](https://gitlab.com/gitlab-org/gitlab/blob/d3584e80b4236acdf393d815d604801573af72cc/qa/qa/resource/issue.rb#L27) method. - -```ruby -def api_get_path - "/projects/#{project.id}/issues/#{id}" -end - -def api_post_path - "/projects/#{project.id}/issues" -end - -def api_post_body - { - labels: labels, - title: title - } -end -``` - -By defining the `api_get_path` method, we allow the [`ApiFabricator`](https://gitlab.com/gitlab-org/gitlab/blob/master/qa/qa/resource/api_fabricator.rb) module to know which path to use to get a single issue. - -> This `GET` path can be found in the [public API documentation](../../../api/issues.md#single-issue). - -By defining the `api_post_path` method, we allow the [`ApiFabricator`](https://gitlab.com/gitlab-org/gitlab/blob/master/qa/qa/resource/api_fabricator.rb) module to know which path to use to create a new issue in a specific project. - -> This `POST` path can be found in the [public API documentation](../../../api/issues.md#new-issue). - -By defining the `api_post_body` method, we allow the [`ApiFabricator.api_post`](https://gitlab.com/gitlab-org/gitlab/blob/a9177ca1812bac57e2b2fa4560e1d5dd8ffac38b/qa/qa/resource/api_fabricator.rb#L68) method to know which data to send when making the `POST` request. - -> Notice that we pass both `labels` and `title` attributes in the `api_post_body`, where `labels` receives an array of labels, and [`title` is required](../../../api/issues.md#new-issue). Also, notice that we keep them alphabetically organized. - -**Label resource** - -Finally, let's make it possible to create label resources through the API. - -Add the following code right below the [`fabricate!`](https://gitlab.com/gitlab-org/gitlab/blob/a9177ca1812bac57e2b2fa4560e1d5dd8ffac38b/qa/qa/resource/label.rb#L36) method. - -```ruby -def resource_web_url(resource) - super -rescue ResourceURLMissingError - # this particular resource does not expose a web_url property -end - -def api_get_path - raise NotImplementedError, "The Labels API doesn't expose a single-resource endpoint so this method cannot be properly implemented." -end - -def api_post_path - "/projects/#{project.id}/labels" -end - -def api_post_body - { - color: @color, - name: @title - } -end -``` - -By defining the `resource_web_url(resource)` method, we override the one from the [`ApiFabricator`](https://gitlab.com/gitlab-org/gitlab/blob/master/qa/qa/resource/api_fabricator.rb#L44) module. We do that to avoid failing the test due to this particular resource not exposing a `web_url` property. - -By defining the `api_get_path` method, we **would** allow for the [`ApiFabricator`](https://gitlab.com/gitlab-org/gitlab/blob/master/qa/qa/resource/api_fabricator.rb) module to know which path to use to get a single label, but since there's no path available for that in the public API, we raise a `NotImplementedError` instead. - -By defining the `api_post_path` method, we allow for the [`ApiFabricator`](https://gitlab.com/gitlab-org/gitlab/blob/master/qa/qa/resource/api_fabricator.rb) module to know which path to use to create a new label in a specific project. - -By defining the `api_post_body` method, we allow for the [`ApiFabricator.api_post`](https://gitlab.com/gitlab-org/gitlab/blob/a9177ca1812bac57e2b2fa4560e1d5dd8ffac38b/qa/qa/resource/api_fabricator.rb#L68) method to know which data to send when making the `POST` request. - -> Notice that we pass both `color` and `name` attributes in the `api_post_body` since [those are required](../../../api/labels.md#create-a-new-label). Also, notice that we keep them alphabetically organized. - -### 8. Page Objects - -Page Objects are used in end-to-end tests for maintenance reasons, where a page's elements and methods are defined to be reused in any test. - -> Page Objects are auto-loaded in the [`qa/qa.rb`](https://gitlab.com/gitlab-org/gitlab/blob/master/qa/qa.rb) file and available in all the test files (`*_spec.rb`). - -Take a look at the [Page Objects](page_objects.md) documentation. - -Now, let's go back to our example. - -As you may have noticed, we are defining elements with CSS selectors and the `select_labels_and_refresh` method directly in the test file, and this is an anti-pattern since we need to better separate the responsibilities. - -To address this issue, we will move the implementation to Page Objects, and the test suite will only focus on the business rules that we are testing. - -#### Updates in the test file - -As in a test-driven development approach, let's start changing the test file even before the Page Object implementation is in place. - -Replace the code of the `it` block in the test file by the following: - -```ruby -module QA - context 'Plan' do - describe 'Editing scoped labels on issues' do - before do - ... - end - - it 'correctly applies scoped labels depending on if they are from the same or a different scope' do - Page::Project::Issue::Show.perform do |issue_page| - issue_page.select_labels_and_refresh [@new_label_same_scope, @new_label_different_scope] - - expect(page).to have_content("added #{@initial_label}") - expect(page).to have_content("added #{@new_label_same_scope} #{@new_label_different_scope} labels and removed #{@initial_label}") - expect(issue_page.text_of_labels_block).to have_content(@new_label_same_scope) - expect(issue_page.text_of_labels_block).to have_content(@new_label_different_scope) - expect(issue_page.text_of_labels_block).not_to have_content(@initial_label) - end - end - end - end -end -``` - -Notice that `select_labels_and_refresh` is now a method from the issue Page Object (which is not yet implemented), and that we verify the labels' text by using `text_of_labels_block`, instead of via the `labels_block` element. The `text_of_labels_block` method will also be implemented in the issue Page Object. - -Let's now update the Issue Page Object. - -#### Updates in the Issue Page Object - -> Page Objects are located in the `qa/qa/page/` directory, and its sub-directories. - -The file we will have to change is the [Issue Page Object](https://gitlab.com/gitlab-org/gitlab/blob/master/qa/qa/page/project/issue/show.rb). - -First, add the following code right below the definition of an already implemented view (keep in mind that view's definitions and their elements should be alphabetically ordered): - -```ruby -view 'app/helpers/dropdowns_helper.rb' do - element :dropdown_input_field -end - -view 'app/views/shared/issuable/_sidebar.html.haml' do - element :dropdown_menu_labels - element :edit_link_labels - element :labels_block -end -``` - -Similarly to what we did before, let's first change the Page Object even without the elements being defined in the view (`_sidebar.html.haml`) and the `dropdowns_helper.rb` files, and later we will update them by adding the appropriate CSS selectors. - -Now, let's implement the methods `select_labels_and_refresh` and `text_of_labels_block`. - -Somewhere between the definition of the views and the private methods, add the following snippet of code (these should also be alphabetically ordered for organization reasons): - -```ruby -def select_labels_and_refresh(labels) - click_element(:edit_link_labels) - labels.each do |label| - within_element(:dropdown_menu_labels, text: label) do - send_keys_to_element(:dropdown_input_field, [label, :enter]) - end - end - click_body - labels.each do |label| - has_element?(:labels_block, text: label) - end - refresh -end - -def text_of_labels_block - find_element(:labels_block) -end -``` - -##### Details of `select_labels_and_refresh` - -Notice that we have not only moved the `select_labels_and_refresh` method, but we have also changed its implementation to: - -1. Click the `:edit_link_labels` element previously defined, instead of using `find('.block.labels .edit-link').click` -1. Use `within_element(:dropdown_menu_labels, text: label)`, and inside of it, we call `send_keys_to_element(:dropdown_input_field, [label, :enter])`, which is a method that we will implement in the `QA::Page::Base` class to replace `find('.dropdown-menu-labels .dropdown-input-field').send_keys [label, :enter]` -1. Use `click_body` after iterating on each label, instead of using `find('#content-body').click` -1. Iterate on every label again, and then we use `has_element?(:labels_block, text: label)` after clicking the page body (which applies the labels), and before refreshing the page, to avoid test flakiness due to refreshing too fast. - -##### Details of `text_of_labels_block` - -The `text_of_labels_block` method is a simple method that returns the `:labels_block` element (`find_element(:labels_block)`). - -#### Updates in the view (*.html.haml) and `dropdowns_helper.rb` files - -Now let's change the view and the `dropdowns_helper` files to add the selectors that relate to the [Page Objects](page_objects.md). - -In [`app/views/shared/issuable/_sidebar.html.haml:105`](https://gitlab.com/gitlab-org/gitlab/blob/7ca12defc7a965987b162a6ebef302f95dc8867f/app/views/shared/issuable/_sidebar.html.haml#L105), add a `data: { qa_selector: 'edit_link_labels' }` data attribute. - -The code should look like this: - -```haml -= link_to _('Edit'), '#', class: 'js-sidebar-dropdown-toggle edit-link float-right', data: { qa_selector: 'edit_link_labels' } -``` - -In the same file, on [line 121](https://gitlab.com/gitlab-org/gitlab/blob/7ca12defc7a965987b162a6ebef302f95dc8867f/app/views/shared/issuable/_sidebar.html.haml#L121), add a `data: { qa_selector: 'dropdown_menu_labels' }` data attribute. - -The code should look like this: - -```haml -.dropdown-menu.dropdown-select.dropdown-menu-paging.dropdown-menu-labels.dropdown-menu-selectable.dropdown-extended-height{ data: { qa_selector: 'dropdown_menu_labels' } } -``` - -In [`app/helpers/dropdowns_helper.rb:94`](https://gitlab.com/gitlab-org/gitlab/blob/7ca12defc7a965987b162a6ebef302f95dc8867f/app/helpers/dropdowns_helper.rb#L94), add a `data: { qa_selector: 'dropdown_input_field' }` data attribute. - -The code should look like this: - -```ruby -filter_output = search_field_tag search_id, nil, class: "dropdown-input-field", placeholder: placeholder, autocomplete: 'off', data: { qa_selector: 'dropdown_input_field' } -``` - -> `data-qa-*` data attributes and CSS classes starting with `qa-` are used solely for the purpose of QA and testing. -> By defining these, we add **testability** to the application. -> -> When defining a data attribute like: `qa_selector: 'labels_block'`, it should match the element definition: `element :labels_block`. We use a [sanity test](https://gitlab.com/gitlab-org/gitlab-foss/tree/master/doc/development/testing_guide/end_to_end/page_objects.md#how-did-we-solve-fragile-tests-problem) to check that defined elements have their respective selectors in the specified views. - -#### Updates in the `QA::Page::Base` class - -The last thing that we have to do is to update `QA::Page::Base` class to add the `send_keys_to_element` method on it. - -Add the following snippet of code somewhere where class methods are defined (remember to organize methods alphabetically, and if you see a place where this standard is not being followed, it would be helpful if you could rearrange it): - -```ruby -def send_keys_to_element(name, keys) - find_element(name).send_keys(keys) -end -``` - -This method receives an element (`name`) and the `keys` that it will send to that element, and the keys are an array that can receive strings, or "special" keys, like `:enter`. - -As you might remember, in the Issue Page Object we call this method like this: `send_keys_to_element(:dropdown_input_field, [label, :enter])`. - -With that, you should be able to start writing end-to-end tests yourself. *Congratulations!* diff --git a/doc/development/testing_guide/end_to_end/rspec_metadata_tests.md b/doc/development/testing_guide/end_to_end/rspec_metadata_tests.md index f8dc3366904..b7c93d205a3 100644 --- a/doc/development/testing_guide/end_to_end/rspec_metadata_tests.md +++ b/doc/development/testing_guide/end_to_end/rspec_metadata_tests.md @@ -14,3 +14,5 @@ This is a partial list of the [RSpec metadata](https://relishapp.com/rspec/rspec | `:reliable` | The test has been [promoted to a reliable test](https://about.gitlab.com/handbook/engineering/quality/guidelines/reliable-tests/#promoting-an-existing-test-to-reliable) meaning it passes consistently in all pipelines, including merge requests. | | `:requires_admin` | The test requires an admin account. Tests with the tag are excluded when run against Canary and Production environments. | | `:runner` | The test depends on and will set up a GitLab Runner instance, typically to run a pipeline. | +| `:gitaly_ha` | The test will run against a GitLab instance where repositories are stored on redundant Gitaly nodes behind a Praefect node. All nodes are [separate containers](../../../administration/gitaly/praefect.md#requirements-for-configuring-a-gitaly-cluster). Tests that use this tag have a longer setup time since there are three additional containers that need to be started. | +| `:skip_live_env` | The test will be excluded when run against live deployed environments such as Staging, Canary, and Production. | diff --git a/doc/development/testing_guide/end_to_end/style_guide.md b/doc/development/testing_guide/end_to_end/style_guide.md index 7f4616f394b..9c02af12d5d 100644 --- a/doc/development/testing_guide/end_to_end/style_guide.md +++ b/doc/development/testing_guide/end_to_end/style_guide.md @@ -66,6 +66,7 @@ We follow a simple formula roughly based on hungarian notation. - `_placeholder`: a temporary element that appears while content is loading. For example, the elements that are displayed instead of discussions while the discussions are being fetched. - `_radio` - `_tab` + - `_menu_item` *Note: If none of the listed types are suitable, please open a merge request to add an appropriate type to the list.* diff --git a/doc/development/testing_guide/flaky_tests.md b/doc/development/testing_guide/flaky_tests.md index 1e53e92fad5..6c1b06ce59a 100644 --- a/doc/development/testing_guide/flaky_tests.md +++ b/doc/development/testing_guide/flaky_tests.md @@ -40,7 +40,7 @@ Quarantined tests are run on the CI in dedicated jobs that are allowed to fail: ## Automatic retries and flaky tests detection -On our CI, we use [rspec-retry](https://github.com/NoRedInk/rspec-retry) to automatically retry a failing example a few +On our CI, we use [RSpec::Retry](https://github.com/NoRedInk/rspec-retry) to automatically retry a failing example a few times (see [`spec/spec_helper.rb`](https://gitlab.com/gitlab-org/gitlab/blob/master/spec/spec_helper.rb) for the precise retries count). We also use a home-made `RspecFlaky::Listener` listener which records flaky diff --git a/doc/development/testing_guide/frontend_testing.md b/doc/development/testing_guide/frontend_testing.md index 2a3fcf122a6..52d538c7159 100644 --- a/doc/development/testing_guide/frontend_testing.md +++ b/doc/development/testing_guide/frontend_testing.md @@ -5,7 +5,7 @@ at GitLab. We use Karma with Jasmine and Jest for JavaScript unit and integratio and RSpec feature tests with Capybara for e2e (end-to-end) integration testing. Unit and feature tests need to be written for all new features. -Most of the time, you should use [RSpec] for your feature tests. +Most of the time, you should use [RSpec](https://github.com/rspec/rspec-rails#feature-specs) for your feature tests. Regression tests should be written for bug fixes to prevent them from recurring in the future. @@ -15,7 +15,7 @@ information on general testing practices at GitLab. ## Vue.js testing -If you are looking for a guide on Vue component testing, you can jump right away to this [section][vue-test]. +If you are looking for a guide on Vue component testing, you can jump right away to this [section](../fe_guide/vue.md#testing-vue-components). ## Jest @@ -30,7 +30,7 @@ Jest tests can be found in `/spec/frontend` and `/ee/spec/frontend` in EE. ## Karma test suite -While GitLab is switching over to [Jest][jest] you'll still find Karma tests in our application. [Karma][karma] is a test runner which uses [Jasmine] as its test +While GitLab is switching over to [Jest](https://jestjs.io) you'll still find Karma tests in our application. [Karma](http://karma-runner.github.io/) is a test runner which uses [Jasmine](https://jasmine.github.io/) as its test framework. Jest also uses Jasmine as foundation, that's why it's looking quite similar. Karma tests live in `spec/javascripts/` and `/ee/spec/javascripts` in EE. @@ -549,7 +549,8 @@ TBU Jasmine provides stubbing and mocking capabilities. There are some subtle differences in how to use it within Karma and Jest. -Stubs or spies are often used synonymously. In Jest it's quite easy thanks to the `.spyOn` method. [Official docs][jestspy] +Stubs or spies are often used synonymously. In Jest it's quite easy thanks to the `.spyOn` method. +[Official docs](https://jestjs.io/docs/en/jest-object#jestspyonobject-methodname) The more challenging part are mocks, which can be used for functions or even dependencies. ### Manual module mocks @@ -637,12 +638,12 @@ Karma allows something similar, but it's way more costly. Running Karma with `yarn run karma-start` will compile the JavaScript assets and run a server at `http://localhost:9876/` where it will automatically -run the tests on any browser which connects to it. You can enter that url on +run the tests on any browser which connects to it. You can enter that URL on multiple browsers at once to have it run the tests on each in parallel. While Karma is running, any changes you make will instantly trigger a recompile and retest of the **entire test suite**, so you can see instantly if you've broken -a test with your changes. You can use [Jasmine focused][jasmine-focus] or +a test with your changes. You can use [Jasmine focused](https://jasmine.github.io/2.5/focused_specs.html) or excluded tests (with `fdescribe` or `xdescribe`) to get Karma to run only the tests you want while you're working on a specific feature, but make sure to remove these directives when you commit your code. @@ -831,43 +832,6 @@ testAction( Check an example in [spec/javascripts/ide/stores/actions_spec.jsspec/javascripts/ide/stores/actions_spec.js](https://gitlab.com/gitlab-org/gitlab/blob/master/spec/javascripts/ide/stores/actions_spec.js). -### Vue Helper: `mountComponent` - -To make mounting a Vue component easier and more readable, we have a few helpers available in `spec/helpers/vue_mount_component_helper`: - -- `createComponentWithStore` -- `mountComponentWithStore` - -Examples of usage: - -```javascript -beforeEach(() => { - vm = createComponentWithStore(Component, store); - - vm.$store.state.currentBranchId = 'master'; - - vm.$mount(); -}); -``` - -```javascript -beforeEach(() => { - vm = mountComponentWithStore(Component, { - el: '#dummy-element', - store, - props: { badge }, - }); -}); -``` - -Don't forget to clean up: - -```javascript -afterEach(() => { - vm.$destroy(); -}); -``` - ### Wait until axios requests finish The axios utils mock module located in `spec/frontend/mocks/ce/lib/utils/axios_utils.js` contains two helper methods for Jest tests that spawn HTTP requests. @@ -906,13 +870,3 @@ You can download any older version of Firefox from the releases FTP server, <htt --- [Return to Testing documentation](index.md) - -<!-- URL References --> - -[jasmine-focus]: https://jasmine.github.io/2.5/focused_specs.html -[karma]: http://karma-runner.github.io/ -[vue-test]: ../fe_guide/vue.md#testing-vue-components -[rspec]: https://github.com/rspec/rspec-rails#feature-specs -[jasmine]: https://jasmine.github.io/ -[jest]: https://jestjs.io -[jestspy]: https://jestjs.io/docs/en/jest-object#jestspyonobject-methodname diff --git a/doc/development/testing_guide/index.md b/doc/development/testing_guide/index.md index 25da72b6b6a..f0fb06910f8 100644 --- a/doc/development/testing_guide/index.md +++ b/doc/development/testing_guide/index.md @@ -4,15 +4,15 @@ This document describes various guidelines and best practices for automated testing of the GitLab project. It is meant to be an _extension_ of the [thoughtbot testing -styleguide](https://github.com/thoughtbot/guides/tree/master/style/testing). If +style guide](https://github.com/thoughtbot/guides/tree/master/style/testing). If this guide defines a rule that contradicts the thoughtbot guide, this guide takes precedence. Some guidelines may be repeated verbatim to stress their importance. ## Overview -GitLab is built on top of [Ruby on Rails](https://rubyonrails.org/), and we're using [RSpec] for all -the backend tests, with [Capybara] for end-to-end integration testing. +GitLab is built on top of [Ruby on Rails](https://rubyonrails.org/), and we're using [RSpec](https://github.com/rspec/rspec-rails#feature-specs) for all +the backend tests, with [Capybara](https://github.com/teamcapybara/capybara) for end-to-end integration testing. On the frontend side, we're using [Jest](https://jestjs.io/) and [Karma](http://karma-runner.github.io/)/[Jasmine](https://jasmine.github.io/) for JavaScript unit and integration testing. @@ -58,14 +58,10 @@ Everything you should know about how to test Rake tasks. ## [End-to-end tests](end_to_end/index.md) Everything you should know about how to run end-to-end tests using -[GitLab QA][gitlab-qa] testing framework. +[GitLab QA](ttps://gitlab.com/gitlab-org/gitlab-qa) testing framework. ## [Migrations tests](testing_migrations_guide.md) Everything you should know about how to test migrations. [Return to Development documentation](../README.md) - -[RSpec]: https://github.com/rspec/rspec-rails#feature-specs -[Capybara]: https://github.com/teamcapybara/capybara -[gitlab-qa]: https://gitlab.com/gitlab-org/gitlab-qa diff --git a/doc/development/testing_guide/review_apps.md b/doc/development/testing_guide/review_apps.md index 9eb5d5add8a..58acf937d77 100644 --- a/doc/development/testing_guide/review_apps.md +++ b/doc/development/testing_guide/review_apps.md @@ -102,10 +102,10 @@ subgraph "CNG-mirror pipeline" ### Auto-stopping of Review Apps Review Apps are automatically stopped 2 days after the last deployment thanks to -the [Environment auto-stop](../../ci/environments.md#environments-auto-stop) feature. +the [Environment auto-stop](../../ci/environments/index.md#environments-auto-stop) feature. If you need your Review App to stay up for a longer time, you can -[pin its environment](../../ci/environments.md#auto-stop-example) or retry the +[pin its environment](../../ci/environments/index.md#auto-stop-example) or retry the `review-deploy` job to update the "latest deployed at" time. The `review-cleanup` job that automatically runs in scheduled @@ -153,7 +153,13 @@ used by the `review-deploy` and `review-stop` jobs. ### Get access to the GCP Review Apps cluster You need to [open an access request (internal link)](https://gitlab.com/gitlab-com/access-requests/issues/new) -for the `gcp-review-apps-sg` GCP group. +for the `gcp-review-apps-sg` GCP group. In order to join a group, you must specify the desired GCP role in your access request. +The role is what will grant you specific permissions in order to engage with Review App containers. + +Here are some permissions you may want to have, and the roles that grant them: + +- `container.pods.getLogs` - Required to [retrieve pod logs](#dig-into-a-pods-logs). Granted by [Viewer (`roles/viewer`)](https://cloud.google.com/iam/docs/understanding-roles#kubernetes-engine-roles). +- `container.pods.exec` - Required to [run a Rails console](#run-a-rails-console). Granted by [Kubernetes Engine Developer (`roles/container.developer`)](https://cloud.google.com/iam/docs/understanding-roles#kubernetes-engine-roles). ### Log into my Review App @@ -175,7 +181,7 @@ secure note named `gitlab-{ce,ee} Review App's root password`. ### Run a Rails console -1. Make sure you [have access to the cluster](#get-access-to-the-gcp-review-apps-cluster) first. +1. Make sure you [have access to the cluster](#get-access-to-the-gcp-review-apps-cluster) and the `container.pods.exec` permission first. 1. [Filter Workloads by your Review App slug](https://console.cloud.google.com/kubernetes/workload?project=gitlab-review-apps), e.g. `review-qa-raise-e-12chm0`. 1. Find and open the `task-runner` Deployment, e.g. `review-qa-raise-e-12chm0-task-runner`. @@ -191,7 +197,7 @@ secure note named `gitlab-{ce,ee} Review App's root password`. ### Dig into a Pod's logs -1. Make sure you [have access to the cluster](#get-access-to-the-gcp-review-apps-cluster) first. +1. Make sure you [have access to the cluster](#get-access-to-the-gcp-review-apps-cluster) and the `container.pods.getLogs` permission first. 1. [Filter Workloads by your Review App slug](https://console.cloud.google.com/kubernetes/workload?project=gitlab-review-apps), e.g. `review-qa-raise-e-12chm0`. 1. Find and open the `migrations` Deployment, e.g. @@ -209,6 +215,31 @@ Leading indicators may be health check failures leading to restarts or majority The [Review Apps Overview dashboard](https://app.google.stackdriver.com/dashboards/6798952013815386466?project=gitlab-review-apps&timeDomain=1d) aids in identifying load spikes on the cluster, and if nodes are problematic or the entire cluster is trending towards unhealthy. +### Release failed with `ImagePullBackOff` + +**Potential cause:** + +If you see an `ImagePullBackoff` status, check for a missing Docker image. + +**Where to look for further debugging:** + +To check that the Docker images were created, run the following Docker command: + +```shell +`DOCKER_CLI_EXPERIMENTAL=enabled docker manifest repository:tag` +``` + +The output of this command indicates if the Docker image exists. For example: + +```shell +DOCKER_CLI_EXPERIMENTAL=enabled docker manifest inspect registry.gitlab.com/gitlab-org/build/cng-mirror/gitlab-rails-ee:39467-allow-a-release-s-associated-milestones-to-be-edited-thro +``` + +If the Docker image does not exist: + +- Verify the `image.repository` and `image.tag` options in the `helm upgrade --install` command match the repository names used by CNG-mirror pipeline. +- Look further in the corresponding downstream CNG-mirror pipeline in `review-build-cng` job. + ### Node count is always increasing (i.e. never stabilizing or decreasing) **Potential cause:** diff --git a/doc/development/testing_guide/testing_levels.md b/doc/development/testing_guide/testing_levels.md index 295aa6609a8..9285a910ecf 100644 --- a/doc/development/testing_guide/testing_levels.md +++ b/doc/development/testing_guide/testing_levels.md @@ -312,9 +312,7 @@ of a controller test. Testing a fat controller usually involves a lot of stubbin controller.instance_variable_set(:@user, user) ``` -and use methods which are deprecated in Rails 5 ([#23768]). - -[#23768]: https://gitlab.com/gitlab-org/gitlab/-/issues/16260 +and use methods which are deprecated in Rails 5 ([#23768](https://gitlab.com/gitlab-org/gitlab/-/issues/16260)). ### About Karma @@ -356,7 +354,7 @@ possible). | Tests path | Testing engine | Notes | | ---------- | -------------- | ----- | -| `spec/features/` | [Capybara] + [RSpec] | If your test has the `:js` metadata, the browser driver will be [Poltergeist], otherwise it's using [RackTest]. | +| `spec/features/` | [Capybara](https://github.com/teamcapybara/capybara) + [RSpec](https://github.com/rspec/rspec-rails#feature-specs) | If your test has the `:js` metadata, the browser driver will be [Poltergeist](https://github.com/teamcapybara/capybara#poltergeist), otherwise it's using [RackTest](https://github.com/teamcapybara/capybara#racktest). | ### Frontend feature tests @@ -460,9 +458,6 @@ The reasons why we should follow these best practices are as follows: of tests). This is slower than transactions, however, so we want to use truncation only when necessary. -[Poltergeist]: https://github.com/teamcapybara/capybara#poltergeist -[RackTest]: https://github.com/teamcapybara/capybara#racktest - ## Black-box tests at the system level, aka end-to-end tests Formal definitions: @@ -470,11 +465,11 @@ Formal definitions: - <https://en.wikipedia.org/wiki/System_testing> - <https://en.wikipedia.org/wiki/Black-box_testing> -GitLab consists of [multiple pieces] such as [GitLab Shell], [GitLab Workhorse], -[Gitaly], [GitLab Pages], [GitLab Runner], and GitLab Rails. All theses pieces -are configured and packaged by [GitLab Omnibus]. +GitLab consists of [multiple pieces](../architecture.md#components) such as [GitLab Shell](https://gitlab.com/gitlab-org/gitlab-shell), [GitLab Workhorse](https://gitlab.com/gitlab-org/gitlab-workhorse), +[Gitaly](https://gitlab.com/gitlab-org/gitaly), [GitLab Pages](https://gitlab.com/gitlab-org/gitlab-pages), [GitLab Runner](https://gitlab.com/gitlab-org/gitlab-runner), and GitLab Rails. All theses pieces +are configured and packaged by [Omnibus GitLab](https://gitlab.com/gitlab-org/omnibus-gitlab). -The QA framework and instance-level scenarios are [part of GitLab Rails] so that +The QA framework and instance-level scenarios are [part of GitLab Rails](https://gitlab.com/gitlab-org/gitlab-foss/tree/master/qa) so that they're always in-sync with the codebase (especially the views). Note that: @@ -483,11 +478,11 @@ Note that: - data needed for the tests can only be created using the GUI or the API - expectations can only be made against the browser page and API responses -Every new feature should come with a [test plan]. +Every new feature should come with a [test plan](https://gitlab.com/gitlab-org/gitlab/tree/master/.gitlab/issue_templates/Test%20plan.md). | Tests path | Testing engine | Notes | | ---------- | -------------- | ----- | -| `qa/qa/specs/features/` | [Capybara] + [RSpec] + Custom QA framework | Tests should be placed under their corresponding [Product category] | +| `qa/qa/specs/features/` | [Capybara](https://github.com/teamcapybara/capybara) + [RSpec](https://github.com/rspec/rspec-rails#feature-specs) + Custom QA framework | Tests should be placed under their corresponding [Product category](https://about.gitlab.com/handbook/product/categories/) | > See [end-to-end tests](end_to_end/index.md) for more information. @@ -495,17 +490,6 @@ Note that `qa/spec` contains unit tests of the QA framework itself, not to be confused with the application's [unit tests](#unit-tests) or [end-to-end tests](#black-box-tests-at-the-system-level-aka-end-to-end-tests). -[multiple pieces]: ../architecture.md#components -[GitLab Shell]: https://gitlab.com/gitlab-org/gitlab-shell -[GitLab Workhorse]: https://gitlab.com/gitlab-org/gitlab-workhorse -[Gitaly]: https://gitlab.com/gitlab-org/gitaly -[GitLab Pages]: https://gitlab.com/gitlab-org/gitlab-pages -[GitLab Runner]: https://gitlab.com/gitlab-org/gitlab-runner -[GitLab Omnibus]: https://gitlab.com/gitlab-org/omnibus-gitlab -[part of GitLab Rails]: https://gitlab.com/gitlab-org/gitlab-foss/tree/master/qa -[test plan]: https://gitlab.com/gitlab-org/gitlab/tree/master/.gitlab/issue_templates/Test%20plan.md -[Product category]: https://about.gitlab.com/handbook/product/categories/ - ### Smoke tests Smoke tests are quick tests that may be run at any time (especially after the @@ -517,14 +501,11 @@ These tests run against the UI and ensure that basic functionality is working. ### GitLab QA orchestrator -[GitLab QA orchestrator] is a tool that allows to test that all these pieces +[GitLab QA orchestrator](https://gitlab.com/gitlab-org/gitlab-qa) is a tool that allows to test that all these pieces integrate well together by building a Docker image for a given version of GitLab Rails and running end-to-end tests (i.e. using Capybara) against it. -Learn more in the [GitLab QA orchestrator README][gitlab-qa-readme]. - -[GitLab QA orchestrator]: https://gitlab.com/gitlab-org/gitlab-qa -[gitlab-qa-readme]: https://gitlab.com/gitlab-org/gitlab-qa/tree/master/README.md +Learn more in the [GitLab QA orchestrator README](https://gitlab.com/gitlab-org/gitlab-qa/tree/master/README.md). ## EE-specific tests @@ -538,7 +519,9 @@ trade-off: - Unit tests are usually cheap, and you should consider them like the basement of your house: you need them to be confident that your code is behaving correctly. However if you run only unit tests without integration / system - tests, you might [miss] the [big] / [picture] ! + tests, you might [miss](https://twitter.com/ThePracticalDev/status/850748070698651649) the + [big](https://twitter.com/timbray/status/822470746773409794) / + [picture](https://twitter.com/withzombies/status/829716565834752000) ! - Integration tests are a bit more expensive, but don't abuse them. A system test is often better than an integration test that is stubbing a lot of internals. - System tests are expensive (compared to unit tests), even more if they require @@ -546,8 +529,8 @@ trade-off: section. Another way to see it is to think about the "cost of tests", this is well -explained [in this article][tests-cost] and the basic idea is that the cost of a -test includes: +explained [in this article](https://medium.com/table-xi/high-cost-tests-and-high-value-tests-a86e27a54df#.2ulyh3a4e) +and the basic idea is that the cost of a test includes: - The time it takes to write the test - The time it takes to run the test every time the suite runs @@ -557,18 +540,11 @@ test includes: ### Frontend-related tests -There are cases where the behaviour you are testing is not worth the time spent +There are cases where the behavior you are testing is not worth the time spent running the full application, for example, if you are testing styling, animation, edge cases or small actions that don't involve the backend, you should write an integration test using Jasmine. -[miss]: https://twitter.com/ThePracticalDev/status/850748070698651649 -[big]: https://twitter.com/timbray/status/822470746773409794 -[picture]: https://twitter.com/withzombies/status/829716565834752000 -[tests-cost]: https://medium.com/table-xi/high-cost-tests-and-high-value-tests-a86e27a54df#.2ulyh3a4e -[RSpec]: https://github.com/rspec/rspec-rails#feature-specs -[Capybara]: https://github.com/teamcapybara/capybara - --- [Return to Testing documentation](index.md) |