diff options
Diffstat (limited to 'doc/development/testing_guide')
10 files changed, 157 insertions, 42 deletions
diff --git a/doc/development/testing_guide/best_practices.md b/doc/development/testing_guide/best_practices.md index 6ef9be381b4..2c1d70a005e 100644 --- a/doc/development/testing_guide/best_practices.md +++ b/doc/development/testing_guide/best_practices.md @@ -81,7 +81,7 @@ browser is much slower than parsing the HTML response from the app. A common cause of slow tests is excessive creation of objects, and thus computation and DB time. Factories are essential to development, but they can -make inserting data into the DB so easy that we may be able to optimize. +make inserting data into the DB so easy that we may be able to optimize. The two basic techniques to bear in mind here are: @@ -151,11 +151,14 @@ In order to reuse a single object for all calls to a named factory in implicit p can be used: ```ruby +RSpec.describe API::Search, factory_default: :keep do let_it_be(:namespace) { create_default(:namespace) } ``` Then every project we create will use this `namespace`, without us having to pass -it as `namespace: namespace`. +it as `namespace: namespace`. In order to make it work along with `let_it_be`, `factory_default: :keep` +must be explicitly specified. That will keep the default factory for every example in a suite instead of +recreating it for each example. Maybe we don't need to create 208 different projects - we can create one and reuse it. In addition, we can see that only about 1/3 of the @@ -237,7 +240,7 @@ end it 'schedules a background job' do expect(BackgroundJob).to receive(:perform_async) - + subject.execute end ``` @@ -249,7 +252,7 @@ combining the examples: ```ruby it 'performs the expected side-effects' do expect(BackgroundJob).to receive(:perform_async) - + expect { subject.execute } .to change(Event, :count).by(1) .and change { arg_0.frobulance }.to('wibble') @@ -481,26 +484,30 @@ This will result in only one `Project`, `User`, and `ProjectMember` created for is handled automatically using a transaction rollback. Note that if you modify an object defined inside a `let_it_be` block, -then you will need to reload the object as needed, or specify the `reload` -option to reload for every example. +then you must do one of the following: + +- Reload the object as needed. +- Use the `let_it_be_with_reload` alias. +- Specify the `reload` option to reload for every example. ```ruby +let_it_be_with_reload(:project) { create(:project) } let_it_be(:project, reload: true) { create(:project) } ``` -You can also specify the `refind` option as well to completely load a -new object. +You can also use the `let_it_be_with_refind` alias, or specify the `refind` +option as well to completely load a new object. ```ruby +let_it_be_with_refind(:project) { create(:project) } let_it_be(:project, refind: true) { create(:project) } ``` ### Time-sensitive tests -[Timecop](https://github.com/travisjeffery/timecop) is available in our -Ruby-based tests for verifying things that are time-sensitive. Any test that -exercises or verifies something time-sensitive should make use of Timecop to -prevent transient test failures. +[`ActiveSupport::Testing::TimeHelpers`](https://api.rubyonrails.org/v6.0.3.1/classes/ActiveSupport/Testing/TimeHelpers.html) +can be used to verify things that are time-sensitive. Any test that exercises or verifies something time-sensitive +should make use of these helpers to prevent transient test failures. Example: @@ -508,7 +515,7 @@ Example: it 'is overdue' do issue = build(:issue, due_date: Date.tomorrow) - Timecop.freeze(3.days.from_now) do + travel_to(3.days.from_now) do expect(issue).to be_overdue end end @@ -888,6 +895,10 @@ GitLab uses [factory_bot](https://github.com/thoughtbot/factory_bot) as a test f resulting record to pass validation. - When instantiating from a factory, don't supply attributes that aren't required by the test. +- Prefer [implicit](https://github.com/thoughtbot/factory_bot/blob/master/GETTING_STARTED.md#implicit-definition) + or [explicit](https://github.com/thoughtbot/factory_bot/blob/master/GETTING_STARTED.md#explicit-definition) + association definitions instead of using `create` / `build` for association setup. + See [issue #262624](https://gitlab.com/gitlab-org/gitlab/-/issues/262624) for further context. - Factories don't have to be limited to `ActiveRecord` objects. [See example](https://gitlab.com/gitlab-org/gitlab-foss/commit/0b8cefd3b2385a21cfed779bd659978c0402766d). diff --git a/doc/development/testing_guide/ci.md b/doc/development/testing_guide/ci.md index 6917639454c..8091142410c 100644 --- a/doc/development/testing_guide/ci.md +++ b/doc/development/testing_guide/ci.md @@ -28,9 +28,6 @@ After that, the next pipeline will use the up-to-date `knapsack/report-master.js The GitLab test suite is [monitored](../performance.md#rspec-profiling) for the `master` branch, and any branch that includes `rspec-profile` in their name. -A [public dashboard](https://redash.gitlab.com/public/dashboards/l1WhHXaxrCWM5Ai9D7YDqHKehq6OU3bx5gssaiWe?org_slug=default) is available for everyone to see. Feel free to look at the -slowest test files and try to improve them. - ## CI setup - Rails logging to `log/test.log` is disabled by default in CI [for diff --git a/doc/development/testing_guide/end_to_end/beginners_guide.md b/doc/development/testing_guide/end_to_end/beginners_guide.md index c552c44c864..a1883f44170 100644 --- a/doc/development/testing_guide/end_to_end/beginners_guide.md +++ b/doc/development/testing_guide/end_to_end/beginners_guide.md @@ -196,8 +196,8 @@ end **What do we test?** -1. Can we log in? -1. Can we log out? +1. Can we sign in? +1. Can we sign out? **How do we test?** diff --git a/doc/development/testing_guide/end_to_end/best_practices.md b/doc/development/testing_guide/end_to_end/best_practices.md index 36cb49256a6..58bae749dc5 100644 --- a/doc/development/testing_guide/end_to_end/best_practices.md +++ b/doc/development/testing_guide/end_to_end/best_practices.md @@ -241,7 +241,11 @@ All tests expect to be able to log in at the start of the test. For an example see: <https://gitlab.com/gitlab-org/gitlab/-/issues/34736> -Ideally, any actions performed in an `after(:context)` (or [`before(:context)`](#limit-the-use-of-the-ui-in-beforecontext-and-after-hooks)) block would be performed via the API. But if it's necessary to do so via the UI (e.g., if API functionality doesn't exist), make sure to log out at the end of the block. +Ideally, actions performed in an `after(:context)` (or +[`before(:context)`](#limit-the-use-of-the-ui-in-beforecontext-and-after-hooks)) +block are performed using the API. If it's necessary to do so with the user +interface (for example, if API functionality doesn't exist), be sure to sign +out at the end of the block. ```ruby after(:all) do @@ -310,3 +314,56 @@ end # Using native mouse click events in the case of a mask/overlay click_element_coordinates(:title) ``` + +## Ensure `expect` statements wait efficiently + +In general, we use an `expect` statement to check that something _is_ as we expect it. For example: + +```ruby +Page::Project::Pipeline::Show.perform do |pipeline| + expect(pipeline).to have_job("a_job") +end +``` + +### Ensure `expect` checks for negation efficiently + +However, sometimes we want to check that something is _not_ as we _don't_ want it to be. In other +words, we want to make sure something is absent. In such a case we should use an appropriate +predicate method that returns quickly, rather than waiting for a state that won't appear. + +It's most efficient to use a predicate method that returns immediately when there is no job, or waits +until it disappears: + +```ruby +# Good +Page::Project::Pipeline::Show.perform do |pipeline| + expect(pipeline).to have_no_job("a_job") +end +``` + +### Problematic alternatives + +Alternatively, if we want to check that a job doesn't exist it might be tempting to use `not_to`: + +```ruby +# Bad +Page::Project::Pipeline::Show.perform do |pipeline| + expect(pipeline).not_to have_job("a_job") +end +``` + +For this statement to pass, `have_job("a_job")` has to return `false` so that `not_to` can negate it. +The problem is that `have_job("a_job")` waits up to ten seconds for `"a job"` to appear before +returning `false`. Under the expected condition this test will take ten seconds longer than it needs to. + +Instead, we could force no wait: + +```ruby +# Not as bad but potentially flaky +Page::Project::Pipeline::Show.perform do |pipeline| + expect(pipeline).not_to have_job("a_job", wait: 0) +end +``` + +The problem is that if `"a_job"` is present and we're waiting for it to disappear, this statement +will fail. diff --git a/doc/development/testing_guide/end_to_end/environment_selection.md b/doc/development/testing_guide/end_to_end/environment_selection.md index 9eb7db72a51..325f251b280 100644 --- a/doc/development/testing_guide/end_to_end/environment_selection.md +++ b/doc/development/testing_guide/end_to_end/environment_selection.md @@ -1,7 +1,7 @@ # Environment selection -Some tests are designed to be run against specific environments. We can specify -what environments to run tests against using the `only` metadata. +Some tests are designed to be run against specific environments or [pipelines](https://about.gitlab.com/handbook/engineering/quality/guidelines/debugging-qa-test-failures/#scheduled-qa-test-pipelines). +We can specify what environments or pipelines to run tests against using the `only` metadata. ## Available switches @@ -11,15 +11,16 @@ what environments to run tests against using the `only` metadata. | `subdomain` | Set the subdomain matcher | `Array` or `String` | | `domain` | Set the domain matcher | `String` | | `production` | Match against production | `Static` | +| `pipeline` | Match against a pipeline | `Array` or `Static`| CAUTION: **Caution:** -You cannot specify `:production` and `{ <switch>: 'value' }` simultaneously. +You cannot specify `:production` and `{ <switch>: 'value' }` simultaneously. These options are mutually exclusive. If you want to specify production, you can control the `tld` and `domain` independently. ## Examples -| Environment | Key | Matches (regex) | +| Environment or pipeline | Key | Matches (regex for environments, string matching for pipelines) | | ---------------- | --- | --------------- | | `any` | `` | `.+.com` | | `gitlab.com` | `only: :production` | `gitlab.com` | @@ -27,18 +28,24 @@ can control the `tld` and `domain` independently. | `gitlab.com and staging.gitlab.com` | `only: { subdomain: /(staging.)?/, domain: 'gitlab' }` | `(staging.)?gitlab.com` | | `dev.gitlab.org` | `only: { tld: '.org', domain: 'gitlab', subdomain: 'dev' }` | `(dev).gitlab.org` | | `staging.gitlab.com & domain.gitlab.com` | `only: { subdomain: %i[staging domain] }` | `(staging|domain).+.com` | +| `nightly` | `only: { pipeline: :nightly }` | "nightly" | +| `nightly`, `canary` | `only_run_in_pipeline: [:nightly, :canary]` | ["nightly"](https://gitlab.com/gitlab-org/quality/nightly) and ["canary"](https://gitlab.com/gitlab-org/quality/canary) | ```ruby RSpec.describe 'Area' do - it 'runs in any environment' do; end + it 'runs in any environment or pipeline' do; end - it 'runs only in production', only: :production do; end + it 'runs only in production environment', only: :production do; end - it 'runs only in staging', only: { subdomain: :staging } do; end + it 'runs only in staging environment', only: { subdomain: :staging } do; end - it 'runs in dev', only: { tld: '.org', domain: 'gitlab', subdomain: 'dev' } do; end + it 'runs in dev environment', only: { tld: '.org', domain: 'gitlab', subdomain: 'dev' } do; end - it 'runs in prod and staging', only: { subdomain: /(staging.)?/, domain: 'gitlab' } {} + it 'runs in prod and staging environments', only: { subdomain: /(staging.)?/, domain: 'gitlab' } {} + + it 'runs only in nightly pipeline', only: { pipeline: :nightly } do; end + + it 'runs in nightly and canary pipelines', only: { pipeline: [:nightly, :canary] } do; end end ``` @@ -46,6 +53,8 @@ NOTE: **Note:** If the test has a `before` or `after`, you must add the `only` metadata to the outer `RSpec.describe`. +If you want to run an `only: { :pipeline }` tagged test on your local GDK make sure either the `CI_PROJECT_NAME` environment variable is unset, or that the `CI_PROJECT_NAME` environment variable matches the specified pipeline in the `only: { :pipeline }` tag, or just delete the `only: { :pipeline }` tag. + ## Quarantining a test for a specific environment Similarly to specifying that a test should only run against a specific environment, it's also possible to quarantine a diff --git a/doc/development/testing_guide/end_to_end/feature_flags.md b/doc/development/testing_guide/end_to_end/feature_flags.md index 87a9738b313..e571774167d 100644 --- a/doc/development/testing_guide/end_to_end/feature_flags.md +++ b/doc/development/testing_guide/end_to_end/feature_flags.md @@ -1,29 +1,70 @@ # Testing with feature flags -To run a specific test with a feature flag enabled you can use the `QA::Runtime::Feature` class to enable and disable feature flags ([via the API](../../../api/features.md)). +To run a specific test with a feature flag enabled you can use the `QA::Runtime::Feature` class to +enable and disable feature flags ([via the API](../../../api/features.md)). -Note that administrator authorization is required to change feature flags. `QA::Runtime::Feature` will automatically authenticate as an administrator as long as you provide an appropriate access token via `GITLAB_QA_ADMIN_ACCESS_TOKEN` (recommended), or provide `GITLAB_ADMIN_USERNAME` and `GITLAB_ADMIN_PASSWORD`. +Note that administrator authorization is required to change feature flags. `QA::Runtime::Feature` +will automatically authenticate as an administrator as long as you provide an appropriate access +token via `GITLAB_QA_ADMIN_ACCESS_TOKEN` (recommended), or provide `GITLAB_ADMIN_USERNAME` +and `GITLAB_ADMIN_PASSWORD`. -Please be sure to include the tag `:requires_admin` so that the test can be skipped in environments where admin access is not available. +Please be sure to include the tag `:requires_admin` so that the test can be skipped in environments +where admin access is not available. + +CAUTION: **Caution:** +You are strongly advised to [enable feature flags only for a group, project, user](../../feature_flags/development.md#feature-actors), +or [feature group](../../feature_flags/development.md#feature-groups). This makes it possible to +test a feature in a shared environment without affecting other users. + +For example, the code below would enable a feature flag named `:feature_flag_name` for the project +created by the test: ```ruby RSpec.describe "with feature flag enabled", :requires_admin do + let(:project) { Resource::Project.fabricate_via_api! } + before do - Runtime::Feature.enable('feature_flag_name') + Runtime::Feature.enable(:feature_flag_name, project: project) end it "feature flag test" do - # Execute a test with a feature flag enabled + # Execute the test with the feature flag enabled. + # It will only affect the project created in this test. end after do - Runtime::Feature.disable('feature_flag_name') + Runtime::Feature.disable(:feature_flag_name, project: project) end end ``` +Note that the `enable` and `disable` methods first set the flag and then check that the updated +value is returned by the API. + +Similarly, you can enable a feature for a group, user, or feature group: + +```ruby +group = Resource::Group.fabricate_via_api! +Runtime::Feature.enable(:feature_flag_name, group: group) + +user = Resource::User.fabricate_via_api! +Runtime::Feature.enable(:feature_flag_name, user: user) + +feature_group = "a_feature_group" +Runtime::Feature.enable(:feature_flag_name, feature_group: feature_group) +``` + +If no scope is provided, the feature flag will be set instance-wide: + +```ruby +# This will affect all users! +Runtime::Feature.enable(:feature_flag_name) +``` + ## Running a scenario with a feature flag enabled -It's also possible to run an entire scenario with a feature flag enabled, without having to edit existing tests or write new ones. +It's also possible to run an entire scenario with a feature flag enabled, without having to edit +existing tests or write new ones. -Please see the [QA README](https://gitlab.com/gitlab-org/gitlab/tree/master/qa#running-tests-with-a-feature-flag-enabled) for details. +Please see the [QA README](https://gitlab.com/gitlab-org/gitlab/tree/master/qa#running-tests-with-a-feature-flag-enabled) +for details. diff --git a/doc/development/testing_guide/end_to_end/rspec_metadata_tests.md b/doc/development/testing_guide/end_to_end/rspec_metadata_tests.md index a9f54b53e5a..3a1303d9c0c 100644 --- a/doc/development/testing_guide/end_to_end/rspec_metadata_tests.md +++ b/doc/development/testing_guide/end_to_end/rspec_metadata_tests.md @@ -11,7 +11,7 @@ This is a partial list of the [RSpec metadata](https://relishapp.com/rspec/rspec | `:gitaly_cluster` | The test will run against a GitLab instance where repositories are stored on redundant Gitaly nodes behind a Praefect node. All nodes are [separate containers](../../../administration/gitaly/praefect.md#requirements-for-configuring-a-gitaly-cluster). Tests that use this tag have a longer setup time since there are three additional containers that need to be started. | | `:jira` | The test requires a Jira Server. [GitLab-QA](https://gitlab.com/gitlab-org/gitlab-qa) will provision the Jira Server in a Docker container when the `Test::Integration::Jira` test scenario is run. | `:kubernetes` | The test includes a GitLab instance that is configured to be run behind an SSH tunnel, allowing a TLS-accessible GitLab. This test will also include provisioning of at least one Kubernetes cluster to test against. *This tag is often be paired with `:orchestrated`.* | -| `:only` | The test is only to be run against specific environments. See [Environment selection](environment_selection.md) for more information. | +| `:only` | The test is only to be run against specific environments or pipelines. See [Environment selection](environment_selection.md) for more information. | | `:orchestrated` | The GitLab instance under test may be [configured by `gitlab-qa`](https://gitlab.com/gitlab-org/gitlab-qa/-/blob/master/docs/what_tests_can_be_run.md#orchestrated-tests) to be different to the default GitLab configuration, or `gitlab-qa` may launch additional services in separate Docker containers, or both. Tests tagged with `:orchestrated` are excluded when testing environments where we can't dynamically modify GitLab's configuration (for example, Staging). | | `:quarantine` | The test has been [quarantined](https://about.gitlab.com/handbook/engineering/quality/guidelines/debugging-qa-test-failures/#quarantining-tests), will run in a separate job that only includes quarantined tests, and is allowed to fail. The test will be skipped in its regular job so that if it fails it will not hold up the pipeline. Note that you can also [quarantine a test only when it runs against specific environment](environment_selection.md#quarantining-a-test-for-a-specific-environment). | | `:reliable` | The test has been [promoted to a reliable test](https://about.gitlab.com/handbook/engineering/quality/guidelines/reliable-tests/#promoting-an-existing-test-to-reliable) meaning it passes consistently in all pipelines, including merge requests. | diff --git a/doc/development/testing_guide/end_to_end/running_tests_that_require_special_setup.md b/doc/development/testing_guide/end_to_end/running_tests_that_require_special_setup.md index 7ac0a00fcff..658839fcf96 100644 --- a/doc/development/testing_guide/end_to_end/running_tests_that_require_special_setup.md +++ b/doc/development/testing_guide/end_to_end/running_tests_that_require_special_setup.md @@ -274,7 +274,7 @@ CHROME_HEADLESS=false bundle exec bin/qa QA::EE::Scenario::Test::Geo --primary-a ### Using Geo in Docker -You can use [GitLab-QA Orchestrator](https://gitlab.com/gitlab-org/gitlab-qa) to orchestrate two GitLab containers and configure them as a Geo setup. +You can use [GitLab-QA Orchestrator](https://gitlab.com/gitlab-org/gitlab-qa) to orchestrate two GitLab containers and configure them as a Geo setup. Geo requires an EE license. To visit the Geo sites in your browser, you will need a reverse proxy server (for example, [NGINX](https://www.nginx.com/)). @@ -319,7 +319,7 @@ Geo requires an EE license. To visit the Geo sites in your browser, you will nee _Map the hostnames to the local IP in `/etc/hosts` file on your machine:_ ```plaintext - 127.0.0.1 gitlab-primary.geo gitlab-secondary.geo + 127.0.0.1 gitlab-primary.geo gitlab-secondary.geo ``` _Note the assigned ports:_ diff --git a/doc/development/testing_guide/end_to_end/style_guide.md b/doc/development/testing_guide/end_to_end/style_guide.md index 7e9f097f624..645c2633831 100644 --- a/doc/development/testing_guide/end_to_end/style_guide.md +++ b/doc/development/testing_guide/end_to_end/style_guide.md @@ -141,7 +141,7 @@ end ``` ```ruby -Page::Project::New.peform do |new_page| +Page::Project::New.perform do |new_page| new_page.do_something end ``` @@ -155,7 +155,7 @@ end ``` ```ruby -Page::Project::New.peform do |page| +Page::Project::New.perform do |page| page.do_something end ``` diff --git a/doc/development/testing_guide/frontend_testing.md b/doc/development/testing_guide/frontend_testing.md index 30e78766dde..730f8d5ad7d 100644 --- a/doc/development/testing_guide/frontend_testing.md +++ b/doc/development/testing_guide/frontend_testing.md @@ -229,7 +229,7 @@ beforeEach(() => { it('exists', () => { // Best - // NOTE: both mount and shallowMount work as long as a DOM element is available + // NOTE: both mount and shallowMount work as long as a DOM element is available // Finds a properly formatted link with an accessible name of "Click Me" getByRole(el, 'link', { name: /Click Me/i }) getByRole(el, 'link', { name: 'Click Me' }) |