diff options
Diffstat (limited to 'doc/development/testing_guide/end_to_end')
4 files changed, 22 insertions, 14 deletions
diff --git a/doc/development/testing_guide/end_to_end/best_practices.md b/doc/development/testing_guide/end_to_end/best_practices.md index 2200069ecfd..042879b47aa 100644 --- a/doc/development/testing_guide/end_to_end/best_practices.md +++ b/doc/development/testing_guide/end_to_end/best_practices.md @@ -53,3 +53,15 @@ In summary: - **Do**: Split tests across separate files, unless the tests share expensive setup. - **Don't**: Put new tests in an existing file without considering the impact on parallelization. + +## Limit the use of `before(:all)` hook + +Limit the use of `before(:all)` to perform setup tasks with only API calls, non UI operations +or basic UI operations such as login. + +We use [`capybara-screenshot`](https://github.com/mattheworiordan/capybara-screenshot) library to automatically save screenshots on failures. +This library [saves the screenshots in the RSpec's `after` hook](https://github.com/mattheworiordan/capybara-screenshot/blob/master/lib/capybara-screenshot/rspec.rb#L97). +[If there is a failure in `before(:all)`, the `after` hook is not called](https://github.com/rspec/rspec-core/pull/2652/files#diff-5e04af96d5156e787f28d519a8c99615R148) and so the screenshots are not saved. + +Given this fact, we should limit the use of `before(:all)` to only those operations where a screenshot is not +necessary in case of failure and QA logs would be enough for debugging. diff --git a/doc/development/testing_guide/end_to_end/index.md b/doc/development/testing_guide/end_to_end/index.md index 9685a61d0c1..a9fb4be284e 100644 --- a/doc/development/testing_guide/end_to_end/index.md +++ b/doc/development/testing_guide/end_to_end/index.md @@ -16,14 +16,14 @@ a black-box testing framework for the API and the UI. ### Testing nightly builds We run scheduled pipeline each night to test nightly builds created by Omnibus. -You can find these nightly pipelines at [gitlab-org/quality/nightly/pipelines][quality-nightly-pipelines]. -Results are reported in the `#qa-nightly` Slack channel. +You can find these nightly pipelines at `https://gitlab.com/gitlab-org/quality/nightly/pipelines` +(need Developer access permissions). Results are reported in the `#qa-nightly` Slack channel. ### Testing staging We run scheduled pipeline each night to test staging. -You can find these nightly pipelines at [gitlab-org/quality/staging/pipelines][quality-staging-pipelines]. -Results are reported in the `#qa-staging` Slack channel. +You can find these nightly pipelines at `https://gitlab.com/gitlab-org/quality/staging/pipelines` +(need developer access permissions). Results are reported in the `#qa-staging` Slack channel. ### Testing code in merge requests @@ -52,7 +52,7 @@ graph LR A1 -.->|1. Triggers an omnibus-gitlab pipeline and wait for it to be done| A2 B2[`Trigger-qa` stage<br>`Trigger:qa-test` job] -.->|2. Triggers a gitlab-qa pipeline and wait for it to be done| A3 -subgraph "gitlab-ce/ee pipeline" +subgraph "gitlab-foss/gitlab pipeline" A1[`test` stage<br>`package-and-qa-manual` job] end @@ -135,20 +135,16 @@ Continued reading: You can ask question in the `#quality` channel on Slack (GitLab internal) or you can find an issue you would like to work on in -[the `gitlab-ce` issue tracker][gitlab-ce-issues], -[the `gitlab-ee` issue tracker][gitlab-ce-issues], or +[the `gitlab` issue tracker][gitlab-issues], or [the `gitlab-qa` issue tracker][gitlab-qa-issues]. [omnibus-gitlab]: https://gitlab.com/gitlab-org/omnibus-gitlab [gitlab-qa]: https://gitlab.com/gitlab-org/gitlab-qa [gitlab-qa-readme]: https://gitlab.com/gitlab-org/gitlab-qa/tree/master/README.md -[quality-nightly-pipelines]: https://gitlab.com/gitlab-org/quality/nightly/pipelines -[quality-staging-pipelines]: https://gitlab.com/gitlab-org/quality/staging/pipelines [review-apps]: ../review_apps.md [gitlab-qa-architecture]: https://gitlab.com/gitlab-org/gitlab-qa/blob/master/docs/architecture.md [gitlab-qa-issues]: https://gitlab.com/gitlab-org/gitlab-qa/issues?label_name%5B%5D=new+scenario -[gitlab-ce-issues]: https://gitlab.com/gitlab-org/gitlab-foss/issues?label_name[]=QA&label_name[]=test -[gitlab-ee-issues]: https://gitlab.com/gitlab-org/gitlab/issues?label_name[]=QA&label_name[]=test +[gitlab-issues]: https://gitlab.com/gitlab-org/gitlab/issues?label_name[]=QA&label_name[]=test [test environment orchestration scenarios]: https://gitlab.com/gitlab-org/gitlab-qa/tree/master/lib/gitlab/qa/scenario [instance-level scenarios]: https://gitlab.com/gitlab-org/gitlab-foss/tree/master/qa/qa/specs/features [Page objects documentation]: https://gitlab.com/gitlab-org/gitlab/tree/master/qa/qa/page/README.md diff --git a/doc/development/testing_guide/end_to_end/page_objects.md b/doc/development/testing_guide/end_to_end/page_objects.md index 8820b54fa87..28111c18378 100644 --- a/doc/development/testing_guide/end_to_end/page_objects.md +++ b/doc/development/testing_guide/end_to_end/page_objects.md @@ -2,7 +2,7 @@ In GitLab QA we are using a known pattern, called _Page Objects_. -This means that we have built an abstraction for all GitLab pages that we use +This means that we have built an abstraction for all pages in GitLab that we use to drive GitLab QA scenarios. Whenever we do something on a page, like filling in a form, or clicking a button, we do that only through a page object associated with this area of GitLab. diff --git a/doc/development/testing_guide/end_to_end/quick_start_guide.md b/doc/development/testing_guide/end_to_end/quick_start_guide.md index f5a46d574b0..2457d8ada5a 100644 --- a/doc/development/testing_guide/end_to_end/quick_start_guide.md +++ b/doc/development/testing_guide/end_to_end/quick_start_guide.md @@ -24,7 +24,7 @@ If you don't exactly understand what we mean by **not everything needs to happen ### 0. Are end-to-end tests needed? -At GitLab we respect the [test pyramid](https://gitlab.com/gitlab-org/gitlab/blob/master/doc/development/testing_guide/testing_levels.md), and so, we recommend you check the code coverage of a specific feature before writing end-to-end tests, for both [CE](https://gitlab-org.gitlab.io/gitlab-ce/coverage-ruby/#_AllFiles) and [EE](https://gitlab-org.gitlab.io/gitlab-ee/coverage-ruby/#_AllFiles) projects. +At GitLab we respect the [test pyramid](https://gitlab.com/gitlab-org/gitlab/blob/master/doc/development/testing_guide/testing_levels.md), and so, we recommend you check the code coverage of a specific feature before writing end-to-end tests, for both [CE](https://gitlab-org.gitlab.io/gitlab-foss/coverage-ruby/#_AllFiles) and [EE](https://gitlab-org.gitlab.io/gitlab/coverage-ruby/#_AllFiles) projects. Sometimes you may notice that there is already good coverage in other test levels, and we can stay confident that if we break a feature, we will still have quick feedback about it, even without having end-to-end tests. @@ -38,7 +38,7 @@ The GitLab QA end-to-end tests are organized by the different [stages in the Dev > There may be sub-directories inside the stages directories, for different features. For example: `.../browser_ui/2_plan/ee_epics/` and `.../browser_ui/2_plan/issues/`. -Now, let's say we want to create tests for the [scoped labels](https://about.gitlab.com/2019/04/22/gitlab-11-10-released/#scoped-labels) feature, available on GitLab EE Premium (this feature is part of the Plan stage.) +Now, let's say we want to create tests for the [scoped labels](https://about.gitlab.com/blog/2019/04/22/gitlab-11-10-released/#scoped-labels) feature, available on GitLab EE Premium (this feature is part of the Plan stage.) > Because these tests are for a feature available only on GitLab EE, we need to create them in the [EE repository](https://gitlab.com/gitlab-org/gitlab). |