summaryrefslogtreecommitdiff
path: root/doc/development/testing_guide
diff options
context:
space:
mode:
Diffstat (limited to 'doc/development/testing_guide')
-rw-r--r--doc/development/testing_guide/best_practices.md14
-rw-r--r--doc/development/testing_guide/ci.md3
-rw-r--r--doc/development/testing_guide/end_to_end/best_practices.md53
-rw-r--r--doc/development/testing_guide/end_to_end/feature_flags.md57
4 files changed, 110 insertions, 17 deletions
diff --git a/doc/development/testing_guide/best_practices.md b/doc/development/testing_guide/best_practices.md
index 6ef9be381b4..53aa84cffcb 100644
--- a/doc/development/testing_guide/best_practices.md
+++ b/doc/development/testing_guide/best_practices.md
@@ -151,11 +151,14 @@ In order to reuse a single object for all calls to a named factory in implicit p
can be used:
```ruby
+RSpec.describe API::Search, factory_default: :keep do
let_it_be(:namespace) { create_default(:namespace) }
```
Then every project we create will use this `namespace`, without us having to pass
-it as `namespace: namespace`.
+it as `namespace: namespace`. In order to make it work along with `let_it_be`, `factory_default: :keep`
+must be explicitly specified. That will keep the default factory for every example in a suite instead of
+recreating it for each example.
Maybe we don't need to create 208 different projects - we
can create one and reuse it. In addition, we can see that only about 1/3 of the
@@ -497,10 +500,9 @@ let_it_be(:project, refind: true) { create(:project) }
### Time-sensitive tests
-[Timecop](https://github.com/travisjeffery/timecop) is available in our
-Ruby-based tests for verifying things that are time-sensitive. Any test that
-exercises or verifies something time-sensitive should make use of Timecop to
-prevent transient test failures.
+[`ActiveSupport::Testing::TimeHelpers`](https://api.rubyonrails.org/v6.0.3.1/classes/ActiveSupport/Testing/TimeHelpers.html)
+can be used to verify things that are time-sensitive. Any test that exercises or verifies something time-sensitive
+should make use of these helpers to prevent transient test failures.
Example:
@@ -508,7 +510,7 @@ Example:
it 'is overdue' do
issue = build(:issue, due_date: Date.tomorrow)
- Timecop.freeze(3.days.from_now) do
+ travel_to(3.days.from_now) do
expect(issue).to be_overdue
end
end
diff --git a/doc/development/testing_guide/ci.md b/doc/development/testing_guide/ci.md
index 6917639454c..8091142410c 100644
--- a/doc/development/testing_guide/ci.md
+++ b/doc/development/testing_guide/ci.md
@@ -28,9 +28,6 @@ After that, the next pipeline will use the up-to-date `knapsack/report-master.js
The GitLab test suite is [monitored](../performance.md#rspec-profiling) for the `master` branch, and any branch
that includes `rspec-profile` in their name.
-A [public dashboard](https://redash.gitlab.com/public/dashboards/l1WhHXaxrCWM5Ai9D7YDqHKehq6OU3bx5gssaiWe?org_slug=default) is available for everyone to see. Feel free to look at the
-slowest test files and try to improve them.
-
## CI setup
- Rails logging to `log/test.log` is disabled by default in CI [for
diff --git a/doc/development/testing_guide/end_to_end/best_practices.md b/doc/development/testing_guide/end_to_end/best_practices.md
index 36cb49256a6..866a949d795 100644
--- a/doc/development/testing_guide/end_to_end/best_practices.md
+++ b/doc/development/testing_guide/end_to_end/best_practices.md
@@ -310,3 +310,56 @@ end
# Using native mouse click events in the case of a mask/overlay
click_element_coordinates(:title)
```
+
+## Ensure `expect` statements wait efficiently
+
+In general, we use an `expect` statement to check that something _is_ as we expect it. For example:
+
+```ruby
+Page::Project::Pipeline::Show.perform do |pipeline|
+ expect(pipeline).to have_job("a_job")
+end
+```
+
+### Ensure `expect` checks for negation efficiently
+
+However, sometimes we want to check that something is _not_ as we _don't_ want it to be. In other
+words, we want to make sure something is absent. In such a case we should use an appropriate
+predicate method that returns quickly, rather than waiting for a state that won't appear.
+
+It's most efficient to use a predicate method that returns immediately when there is no job, or waits
+until it disappears:
+
+```ruby
+# Good
+Page::Project::Pipeline::Show.perform do |pipeline|
+ expect(pipeline).to have_no_job("a_job")
+end
+```
+
+### Problematic alternatives
+
+Alternatively, if we want to check that a job doesn't exist it might be tempting to use `not_to`:
+
+```ruby
+# Bad
+Page::Project::Pipeline::Show.perform do |pipeline|
+ expect(pipeline).not_to have_job("a_job")
+end
+```
+
+For this statement to pass, `have_job("a_job")` has to return `false` so that `not_to` can negate it.
+The problem is that `have_job("a_job")` waits up to ten seconds for `"a job"` to appear before
+returning `false`. Under the expected condition this test will take ten seconds longer than it needs to.
+
+Instead, we could force no wait:
+
+```ruby
+# Not as bad but potentially flaky
+Page::Project::Pipeline::Show.perform do |pipeline|
+ expect(pipeline).not_to have_job("a_job", wait: 0)
+end
+```
+
+The problem is that if `"a_job"` is present and we're waiting for it to disappear, this statement
+will fail.
diff --git a/doc/development/testing_guide/end_to_end/feature_flags.md b/doc/development/testing_guide/end_to_end/feature_flags.md
index 87a9738b313..e571774167d 100644
--- a/doc/development/testing_guide/end_to_end/feature_flags.md
+++ b/doc/development/testing_guide/end_to_end/feature_flags.md
@@ -1,29 +1,70 @@
# Testing with feature flags
-To run a specific test with a feature flag enabled you can use the `QA::Runtime::Feature` class to enable and disable feature flags ([via the API](../../../api/features.md)).
+To run a specific test with a feature flag enabled you can use the `QA::Runtime::Feature` class to
+enable and disable feature flags ([via the API](../../../api/features.md)).
-Note that administrator authorization is required to change feature flags. `QA::Runtime::Feature` will automatically authenticate as an administrator as long as you provide an appropriate access token via `GITLAB_QA_ADMIN_ACCESS_TOKEN` (recommended), or provide `GITLAB_ADMIN_USERNAME` and `GITLAB_ADMIN_PASSWORD`.
+Note that administrator authorization is required to change feature flags. `QA::Runtime::Feature`
+will automatically authenticate as an administrator as long as you provide an appropriate access
+token via `GITLAB_QA_ADMIN_ACCESS_TOKEN` (recommended), or provide `GITLAB_ADMIN_USERNAME`
+and `GITLAB_ADMIN_PASSWORD`.
-Please be sure to include the tag `:requires_admin` so that the test can be skipped in environments where admin access is not available.
+Please be sure to include the tag `:requires_admin` so that the test can be skipped in environments
+where admin access is not available.
+
+CAUTION: **Caution:**
+You are strongly advised to [enable feature flags only for a group, project, user](../../feature_flags/development.md#feature-actors),
+or [feature group](../../feature_flags/development.md#feature-groups). This makes it possible to
+test a feature in a shared environment without affecting other users.
+
+For example, the code below would enable a feature flag named `:feature_flag_name` for the project
+created by the test:
```ruby
RSpec.describe "with feature flag enabled", :requires_admin do
+ let(:project) { Resource::Project.fabricate_via_api! }
+
before do
- Runtime::Feature.enable('feature_flag_name')
+ Runtime::Feature.enable(:feature_flag_name, project: project)
end
it "feature flag test" do
- # Execute a test with a feature flag enabled
+ # Execute the test with the feature flag enabled.
+ # It will only affect the project created in this test.
end
after do
- Runtime::Feature.disable('feature_flag_name')
+ Runtime::Feature.disable(:feature_flag_name, project: project)
end
end
```
+Note that the `enable` and `disable` methods first set the flag and then check that the updated
+value is returned by the API.
+
+Similarly, you can enable a feature for a group, user, or feature group:
+
+```ruby
+group = Resource::Group.fabricate_via_api!
+Runtime::Feature.enable(:feature_flag_name, group: group)
+
+user = Resource::User.fabricate_via_api!
+Runtime::Feature.enable(:feature_flag_name, user: user)
+
+feature_group = "a_feature_group"
+Runtime::Feature.enable(:feature_flag_name, feature_group: feature_group)
+```
+
+If no scope is provided, the feature flag will be set instance-wide:
+
+```ruby
+# This will affect all users!
+Runtime::Feature.enable(:feature_flag_name)
+```
+
## Running a scenario with a feature flag enabled
-It's also possible to run an entire scenario with a feature flag enabled, without having to edit existing tests or write new ones.
+It's also possible to run an entire scenario with a feature flag enabled, without having to edit
+existing tests or write new ones.
-Please see the [QA README](https://gitlab.com/gitlab-org/gitlab/tree/master/qa#running-tests-with-a-feature-flag-enabled) for details.
+Please see the [QA README](https://gitlab.com/gitlab-org/gitlab/tree/master/qa#running-tests-with-a-feature-flag-enabled)
+for details.