diff options
author | GitLab Bot <gitlab-bot@gitlab.com> | 2019-09-18 14:02:45 +0000 |
---|---|---|
committer | GitLab Bot <gitlab-bot@gitlab.com> | 2019-09-18 14:02:45 +0000 |
commit | 80f61b4035607d7cd87de993b8f5e996bde3481f (patch) | |
tree | 06b12f51e97d87192e3dd0e05edf55143645b894 /doc/development/testing_guide | |
parent | 4ab54c2233e91f60a80e5b6fa2181e6899fdcc3e (diff) | |
download | gitlab-ce-80f61b4035607d7cd87de993b8f5e996bde3481f.tar.gz |
Add latest changes from gitlab-org/gitlab@master
Diffstat (limited to 'doc/development/testing_guide')
-rw-r--r-- | doc/development/testing_guide/best_practices.md | 4 | ||||
-rw-r--r-- | doc/development/testing_guide/end_to_end/index.md | 14 | ||||
-rw-r--r-- | doc/development/testing_guide/end_to_end/quick_start_guide.md | 46 | ||||
-rw-r--r-- | doc/development/testing_guide/flaky_tests.md | 42 | ||||
-rw-r--r-- | doc/development/testing_guide/frontend_testing.md | 31 | ||||
-rw-r--r-- | doc/development/testing_guide/review_apps.md | 29 | ||||
-rw-r--r-- | doc/development/testing_guide/smoke.md | 1 | ||||
-rw-r--r-- | doc/development/testing_guide/testing_levels.md | 8 |
8 files changed, 92 insertions, 83 deletions
diff --git a/doc/development/testing_guide/best_practices.md b/doc/development/testing_guide/best_practices.md index 0f982c3a48b..a8cbf3aaa5b 100644 --- a/doc/development/testing_guide/best_practices.md +++ b/doc/development/testing_guide/best_practices.md @@ -148,7 +148,7 @@ Add `screenshot_and_open_image` in a `:js` spec to screenshot what Capybara The HTML dumps created by this are missing CSS. This results in them looking very different from the actual application. -There is a [small hack](https://gitlab.com/gitlab-org/gitlab-ce/snippets/1718469) to add CSS which makes debugging easier. +There is a [small hack](https://gitlab.com/gitlab-org/gitlab-foss/snippets/1718469) to add CSS which makes debugging easier. ### Fast unit tests @@ -538,7 +538,7 @@ GitLab uses [factory_bot] as a test fixture replacement. - When instantiating from a factory, don't supply attributes that aren't required by the test. - Factories don't have to be limited to `ActiveRecord` objects. - [See example](https://gitlab.com/gitlab-org/gitlab-ce/commit/0b8cefd3b2385a21cfed779bd659978c0402766d). + [See example](https://gitlab.com/gitlab-org/gitlab-foss/commit/0b8cefd3b2385a21cfed779bd659978c0402766d). [factory_bot]: https://github.com/thoughtbot/factory_bot [traits]: http://www.rubydoc.info/gems/factory_bot/file/GETTING_STARTED.md#Traits diff --git a/doc/development/testing_guide/end_to_end/index.md b/doc/development/testing_guide/end_to_end/index.md index c00be77ce8c..9685a61d0c1 100644 --- a/doc/development/testing_guide/end_to_end/index.md +++ b/doc/development/testing_guide/end_to_end/index.md @@ -112,7 +112,7 @@ of the test scenarios you can run via the orchestrator](https://gitlab.com/gitla On the other hand, if you would like to run against a local development GitLab environment, you can use the [GitLab Development Kit (GDK)](https://gitlab.com/gitlab-org/gitlab-development-kit/). -Please refer to the instructions in the [QA README](https://gitlab.com/gitlab-org/gitlab-ce/tree/master/qa/README.md#how-can-i-use-it) +Please refer to the instructions in the [QA README](https://gitlab.com/gitlab-org/gitlab/tree/master/qa/README.md#how-can-i-use-it) and the section below. ## How do I write tests? @@ -147,10 +147,10 @@ you can find an issue you would like to work on in [review-apps]: ../review_apps.md [gitlab-qa-architecture]: https://gitlab.com/gitlab-org/gitlab-qa/blob/master/docs/architecture.md [gitlab-qa-issues]: https://gitlab.com/gitlab-org/gitlab-qa/issues?label_name%5B%5D=new+scenario -[gitlab-ce-issues]: https://gitlab.com/gitlab-org/gitlab-ce/issues?label_name[]=QA&label_name[]=test -[gitlab-ee-issues]: https://gitlab.com/gitlab-org/gitlab-ee/issues?label_name[]=QA&label_name[]=test +[gitlab-ce-issues]: https://gitlab.com/gitlab-org/gitlab-foss/issues?label_name[]=QA&label_name[]=test +[gitlab-ee-issues]: https://gitlab.com/gitlab-org/gitlab/issues?label_name[]=QA&label_name[]=test [test environment orchestration scenarios]: https://gitlab.com/gitlab-org/gitlab-qa/tree/master/lib/gitlab/qa/scenario -[instance-level scenarios]: https://gitlab.com/gitlab-org/gitlab-ce/tree/master/qa/qa/specs/features -[Page objects documentation]: https://gitlab.com/gitlab-org/gitlab-ce/tree/master/qa/qa/page/README.md -[instance-qa-readme]: https://gitlab.com/gitlab-org/gitlab-ce/tree/master/qa/README.md -[instance-qa-examples]: https://gitlab.com/gitlab-org/gitlab-ce/tree/master/qa/qa +[instance-level scenarios]: https://gitlab.com/gitlab-org/gitlab-foss/tree/master/qa/qa/specs/features +[Page objects documentation]: https://gitlab.com/gitlab-org/gitlab/tree/master/qa/qa/page/README.md +[instance-qa-readme]: https://gitlab.com/gitlab-org/gitlab/tree/master/qa/README.md +[instance-qa-examples]: https://gitlab.com/gitlab-org/gitlab-foss/tree/master/qa/qa diff --git a/doc/development/testing_guide/end_to_end/quick_start_guide.md b/doc/development/testing_guide/end_to_end/quick_start_guide.md index d52d6db38b9..f5a46d574b0 100644 --- a/doc/development/testing_guide/end_to_end/quick_start_guide.md +++ b/doc/development/testing_guide/end_to_end/quick_start_guide.md @@ -24,7 +24,7 @@ If you don't exactly understand what we mean by **not everything needs to happen ### 0. Are end-to-end tests needed? -At GitLab we respect the [test pyramid](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/development/testing_guide/testing_levels.md), and so, we recommend you check the code coverage of a specific feature before writing end-to-end tests, for both [CE](https://gitlab-org.gitlab.io/gitlab-ce/coverage-ruby/#_AllFiles) and [EE](https://gitlab-org.gitlab.io/gitlab-ee/coverage-ruby/#_AllFiles) projects. +At GitLab we respect the [test pyramid](https://gitlab.com/gitlab-org/gitlab/blob/master/doc/development/testing_guide/testing_levels.md), and so, we recommend you check the code coverage of a specific feature before writing end-to-end tests, for both [CE](https://gitlab-org.gitlab.io/gitlab-ce/coverage-ruby/#_AllFiles) and [EE](https://gitlab-org.gitlab.io/gitlab-ee/coverage-ruby/#_AllFiles) projects. Sometimes you may notice that there is already good coverage in other test levels, and we can stay confident that if we break a feature, we will still have quick feedback about it, even without having end-to-end tests. @@ -32,7 +32,7 @@ If after this analysis you still think that end-to-end tests are needed, keep re ### 1. Identifying the DevOps stage -The GitLab QA end-to-end tests are organized by the different [stages in the DevOps lifecycle](https://gitlab.com/gitlab-org/gitlab-ce/tree/master/qa/qa/specs/features/browser_ui), and so, if you are creating tests for issue creation, for instance, you would locate the spec files under the `qa/qa/specs/features/browser_ui/2_plan/` directory since issue creation is part of the Plan stage. +The GitLab QA end-to-end tests are organized by the different [stages in the DevOps lifecycle](https://gitlab.com/gitlab-org/gitlab-foss/tree/master/qa/qa/specs/features/browser_ui), and so, if you are creating tests for issue creation, for instance, you would locate the spec files under the `qa/qa/specs/features/browser_ui/2_plan/` directory since issue creation is part of the Plan stage. In another case of a test for listing merged merge requests (MRs), the test should go under the `qa/qa/specs/features/browser_ui/3_create/` directory since merge requests are a feature from the Create stage. @@ -40,9 +40,9 @@ The GitLab QA end-to-end tests are organized by the different [stages in the Dev Now, let's say we want to create tests for the [scoped labels](https://about.gitlab.com/2019/04/22/gitlab-11-10-released/#scoped-labels) feature, available on GitLab EE Premium (this feature is part of the Plan stage.) -> Because these tests are for a feature available only on GitLab EE, we need to create them in the [EE repository](https://gitlab.com/gitlab-org/gitlab-ee). +> Because these tests are for a feature available only on GitLab EE, we need to create them in the [EE repository](https://gitlab.com/gitlab-org/gitlab). -Since [there is no specific directory for this feature](https://gitlab.com/gitlab-org/gitlab-ee/tree/master/qa/qa/specs/features/browser_ui/2_plan), we should create a sub-directory for it. +Since [there is no specific directory for this feature](https://gitlab.com/gitlab-org/gitlab/tree/master/qa/qa/specs/features/browser_ui/2_plan), we should create a sub-directory for it. Under `.../browser_ui/2_plan/`, let's create a sub-directory called `ee_scoped_labels/`. @@ -222,7 +222,7 @@ As the pre-conditions for our test suite, the things that needs to happen before - A project being created with an issue and labels already set; - The issue page being opened with only one scoped label applied to it. -> When running end-to-end tests as part of the GitLab's continuous integration process [a license is already set as an environment variable](https://gitlab.com/gitlab-org/gitlab-ee/blob/1a60d926740db10e3b5724713285780a4f470531/qa/qa/ee/strategy.rb#L20). For running tests locally you can set up such license by following the document [what tests can be run?](https://gitlab.com/gitlab-org/gitlab-qa/blob/master/docs/what_tests_can_be_run.md), based on the [supported GitLab environment variables](https://gitlab.com/gitlab-org/gitlab-qa/blob/master/docs/what_tests_can_be_run.md#supported-gitlab-environment-variables). +> When running end-to-end tests as part of the GitLab's continuous integration process [a license is already set as an environment variable](https://gitlab.com/gitlab-org/gitlab/blob/1a60d926740db10e3b5724713285780a4f470531/qa/qa/ee/strategy.rb#L20). For running tests locally you can set up such license by following the document [what tests can be run?](https://gitlab.com/gitlab-org/gitlab-qa/blob/master/docs/what_tests_can_be_run.md), based on the [supported GitLab environment variables](https://gitlab.com/gitlab-org/gitlab-qa/blob/master/docs/what_tests_can_be_run.md#supported-gitlab-environment-variables). #### Implementation @@ -357,15 +357,15 @@ In the following we describe the changes needed in each of the resource files me Now, let's make it possible to create an issue resource through the API. -First, in the [issue resource](https://gitlab.com/gitlab-org/gitlab-ee/blob/d3584e80b4236acdf393d815d604801573af72cc/qa/qa/resource/issue.rb), let's expose its id and labels attributes. +First, in the [issue resource](https://gitlab.com/gitlab-org/gitlab/blob/d3584e80b4236acdf393d815d604801573af72cc/qa/qa/resource/issue.rb), let's expose its id and labels attributes. -Add the following `attribute :id` and `attribute :labels` right above the [`attribute :title`](https://gitlab.com/gitlab-org/gitlab-ee/blob/d3584e80b4236acdf393d815d604801573af72cc/qa/qa/resource/issue.rb#L15). +Add the following `attribute :id` and `attribute :labels` right above the [`attribute :title`](https://gitlab.com/gitlab-org/gitlab/blob/d3584e80b4236acdf393d815d604801573af72cc/qa/qa/resource/issue.rb#L15). > This line is needed to allow for the issue fabrication, and for labels to be automatically added to the issue when fabricating it via API. > > We add the attributes above the existing attribute to keep them alphabetically organized. -Then, let's initialize an instance variable for labels to allow an empty array as default value when such information is not passed during the resource fabrication, since this optional. [Between the attributes and the `fabricate!` method](https://gitlab.com/gitlab-org/gitlab-ee/blob/1a1f1408728f19b2aa15887cd20bddab7e70c8bd/qa/qa/resource/issue.rb#L18), add the following: +Then, let's initialize an instance variable for labels to allow an empty array as default value when such information is not passed during the resource fabrication, since this optional. [Between the attributes and the `fabricate!` method](https://gitlab.com/gitlab-org/gitlab/blob/1a1f1408728f19b2aa15887cd20bddab7e70c8bd/qa/qa/resource/issue.rb#L18), add the following: ```ruby def initialize @@ -373,7 +373,7 @@ def initialize end ``` -Next, add the following code right below the [`fabricate!`](https://gitlab.com/gitlab-org/gitlab-ee/blob/d3584e80b4236acdf393d815d604801573af72cc/qa/qa/resource/issue.rb#L27) method. +Next, add the following code right below the [`fabricate!`](https://gitlab.com/gitlab-org/gitlab/blob/d3584e80b4236acdf393d815d604801573af72cc/qa/qa/resource/issue.rb#L27) method. ```ruby def api_get_path @@ -392,15 +392,15 @@ def api_post_body end ``` -By defining the `api_get_path` method, we allow the [`ApiFabricator`](https://gitlab.com/gitlab-org/gitlab-ee/blob/master/qa/qa/resource/api_fabricator.rb) module to know which path to use to get a single issue. +By defining the `api_get_path` method, we allow the [`ApiFabricator`](https://gitlab.com/gitlab-org/gitlab/blob/master/qa/qa/resource/api_fabricator.rb) module to know which path to use to get a single issue. > This `GET` path can be found in the [public API documentation](../../../api/issues.md#single-issue). -By defining the `api_post_path` method, we allow the [`ApiFabricator`](https://gitlab.com/gitlab-org/gitlab-ee/blob/master/qa/qa/resource/api_fabricator.rb) module to know which path to use to create a new issue in a specific project. +By defining the `api_post_path` method, we allow the [`ApiFabricator`](https://gitlab.com/gitlab-org/gitlab/blob/master/qa/qa/resource/api_fabricator.rb) module to know which path to use to create a new issue in a specific project. > This `POST` path can be found in the [public API documentation](../../../api/issues.md#new-issue). -By defining the `api_post_body` method, we allow the [`ApiFabricator.api_post`](https://gitlab.com/gitlab-org/gitlab-ee/blob/a9177ca1812bac57e2b2fa4560e1d5dd8ffac38b/qa/qa/resource/api_fabricator.rb#L68) method to know which data to send when making the `POST` request. +By defining the `api_post_body` method, we allow the [`ApiFabricator.api_post`](https://gitlab.com/gitlab-org/gitlab/blob/a9177ca1812bac57e2b2fa4560e1d5dd8ffac38b/qa/qa/resource/api_fabricator.rb#L68) method to know which data to send when making the `POST` request. > Notice that we pass both `labels` and `title` attributes in the `api_post_body`, where `labels` receives an array of labels, and [`title` is required](../../../api/issues.md#new-issue). Also, notice that we keep them alphabetically organized. @@ -408,7 +408,7 @@ By defining the `api_post_body` method, we allow the [`ApiFabricator.api_post`]( Finally, let's make it possible to create label resources through the API. -Add the following code right below the [`fabricate!`](https://gitlab.com/gitlab-org/gitlab-ee/blob/a9177ca1812bac57e2b2fa4560e1d5dd8ffac38b/qa/qa/resource/label.rb#L36) method. +Add the following code right below the [`fabricate!`](https://gitlab.com/gitlab-org/gitlab/blob/a9177ca1812bac57e2b2fa4560e1d5dd8ffac38b/qa/qa/resource/label.rb#L36) method. ```ruby def resource_web_url(resource) @@ -433,13 +433,13 @@ def api_post_body end ``` -By defining the `resource_web_url(resource)` method, we override the one from the [`ApiFabricator`](https://gitlab.com/gitlab-org/gitlab-ee/blob/master/qa/qa/resource/api_fabricator.rb#L44) module. We do that to avoid failing the test due to this particular resource not exposing a `web_url` property. +By defining the `resource_web_url(resource)` method, we override the one from the [`ApiFabricator`](https://gitlab.com/gitlab-org/gitlab/blob/master/qa/qa/resource/api_fabricator.rb#L44) module. We do that to avoid failing the test due to this particular resource not exposing a `web_url` property. -By defining the `api_get_path` method, we **would** allow for the [`ApiFabricator`](https://gitlab.com/gitlab-org/gitlab-ee/blob/master/qa/qa/resource/api_fabricator.rb) module to know which path to use to get a single label, but since there's no path available for that in the publich API, we raise a `NotImplementedError` instead. +By defining the `api_get_path` method, we **would** allow for the [`ApiFabricator`](https://gitlab.com/gitlab-org/gitlab/blob/master/qa/qa/resource/api_fabricator.rb) module to know which path to use to get a single label, but since there's no path available for that in the publich API, we raise a `NotImplementedError` instead. -By defining the `api_post_path` method, we allow for the [`ApiFabricator`](https://gitlab.com/gitlab-org/gitlab-ee/blob/master/qa/qa/resource/api_fabricator.rb) module to know which path to use to create a new label in a specific project. +By defining the `api_post_path` method, we allow for the [`ApiFabricator`](https://gitlab.com/gitlab-org/gitlab/blob/master/qa/qa/resource/api_fabricator.rb) module to know which path to use to create a new label in a specific project. -By defining the `api_post_body` method, we we allow for the [`ApiFabricator.api_post`](https://gitlab.com/gitlab-org/gitlab-ee/blob/a9177ca1812bac57e2b2fa4560e1d5dd8ffac38b/qa/qa/resource/api_fabricator.rb#L68) method to know which data to send when making the `POST` request. +By defining the `api_post_body` method, we we allow for the [`ApiFabricator.api_post`](https://gitlab.com/gitlab-org/gitlab/blob/a9177ca1812bac57e2b2fa4560e1d5dd8ffac38b/qa/qa/resource/api_fabricator.rb#L68) method to know which data to send when making the `POST` request. > Notice that we pass both `color` and `name` attributes in the `api_post_body` since [those are required](../../../api/labels.md#create-a-new-label). Also, notice that we keep them alphabetically organized. @@ -447,7 +447,7 @@ By defining the `api_post_body` method, we we allow for the [`ApiFabricator.api_ Page Objects are used in end-to-end tests for maintenance reasons, where a page's elements and methods are defined to be reused in any test. -> Page Objects are auto-loaded in the [`qa/qa.rb`](https://gitlab.com/gitlab-org/gitlab-ee/blob/master/qa/qa.rb) file and available in all the test files (`*_spec.rb`). +> Page Objects are auto-loaded in the [`qa/qa.rb`](https://gitlab.com/gitlab-org/gitlab/blob/master/qa/qa.rb) file and available in all the test files (`*_spec.rb`). Take a look at the [Page Objects] documentation. @@ -495,7 +495,7 @@ Let's now update the Issue Page Object. > Page Objects are located in the `qa/qa/page/` directory, and its sub-directories. -The file we will have to change is the [Issue Page Object](https://gitlab.com/gitlab-org/gitlab-ee/blob/master/qa/qa/page/project/issue/show.rb). +The file we will have to change is the [Issue Page Object](https://gitlab.com/gitlab-org/gitlab/blob/master/qa/qa/page/project/issue/show.rb). First, add the following code right below the definition of an already implemented view (keep in mind that view's definitions and their elements should be alphabetically ordered): @@ -554,7 +554,7 @@ The `text_of_labels_block` method is a simple method that returns the `:labels_b Now let's change the view and the `dropdowns_helper` files to add the selectors that relate to the [Page Objects]. -In [`app/views/shared/issuable/_sidebar.html.haml:105`](https://gitlab.com/gitlab-org/gitlab-ee/blob/7ca12defc7a965987b162a6ebef302f95dc8867f/app/views/shared/issuable/_sidebar.html.haml#L105), add a `data: { qa_selector: 'edit_link_labels' }` data attribute. +In [`app/views/shared/issuable/_sidebar.html.haml:105`](https://gitlab.com/gitlab-org/gitlab/blob/7ca12defc7a965987b162a6ebef302f95dc8867f/app/views/shared/issuable/_sidebar.html.haml#L105), add a `data: { qa_selector: 'edit_link_labels' }` data attribute. The code should look like this: @@ -562,7 +562,7 @@ The code should look like this: = link_to _('Edit'), '#', class: 'js-sidebar-dropdown-toggle edit-link float-right', data: { qa_selector: 'edit_link_labels' } ``` -In the same file, on [line 121](https://gitlab.com/gitlab-org/gitlab-ee/blob/7ca12defc7a965987b162a6ebef302f95dc8867f/app/views/shared/issuable/_sidebar.html.haml#L121), add a `data: { qa_selector: 'dropdown_menu_labels' }` data attribute. +In the same file, on [line 121](https://gitlab.com/gitlab-org/gitlab/blob/7ca12defc7a965987b162a6ebef302f95dc8867f/app/views/shared/issuable/_sidebar.html.haml#L121), add a `data: { qa_selector: 'dropdown_menu_labels' }` data attribute. The code should look like this: @@ -570,7 +570,7 @@ The code should look like this: .dropdown-menu.dropdown-select.dropdown-menu-paging.dropdown-menu-labels.dropdown-menu-selectable.dropdown-extended-height{ data: { qa_selector: 'dropdown_menu_labels' } } ``` -In [`app/helpers/dropdowns_helper.rb:94`](https://gitlab.com/gitlab-org/gitlab-ee/blob/7ca12defc7a965987b162a6ebef302f95dc8867f/app/helpers/dropdowns_helper.rb#L94), add a `data: { qa_selector: 'dropdown_input_field' }` data attribute. +In [`app/helpers/dropdowns_helper.rb:94`](https://gitlab.com/gitlab-org/gitlab/blob/7ca12defc7a965987b162a6ebef302f95dc8867f/app/helpers/dropdowns_helper.rb#L94), add a `data: { qa_selector: 'dropdown_input_field' }` data attribute. The code should look like this: @@ -581,7 +581,7 @@ filter_output = search_field_tag search_id, nil, class: "dropdown-input-field", > `data-qa-*` data attributes and CSS classes starting with `qa-` are used solely for the purpose of QA and testing. > By defining these, we add **testability** to the application. > -> When defining a data attribute like: `qa_selector: 'labels_block'`, it should match the element definition: `element :labels_block`. We use a [sanity test](https://gitlab.com/gitlab-org/gitlab-ce/tree/master/qa/qa/page#how-did-we-solve-fragile-tests-problem) to check that defined elements have their respective selectors in the specified views. +> When defining a data attribute like: `qa_selector: 'labels_block'`, it should match the element definition: `element :labels_block`. We use a [sanity test](https://gitlab.com/gitlab-org/gitlab-foss/tree/master/qa/qa/page#how-did-we-solve-fragile-tests-problem) to check that defined elements have their respective selectors in the specified views. #### Updates in the `QA::Page::Base` class diff --git a/doc/development/testing_guide/flaky_tests.md b/doc/development/testing_guide/flaky_tests.md index eb0bf6fc563..8e73b3d9ae9 100644 --- a/doc/development/testing_guide/flaky_tests.md +++ b/doc/development/testing_guide/flaky_tests.md @@ -48,51 +48,51 @@ examples in a JSON report file on `master` (`retrieve-tests-metadata` and `updat is detected in any other branch (`flaky-examples-check` job). In the future, the `flaky-examples-check` job will not be allowed to fail. -This was originally implemented in: <https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/13021>. +This was originally implemented in: <https://gitlab.com/gitlab-org/gitlab-foss/merge_requests/13021>. [rspec-retry]: https://github.com/NoRedInk/rspec-retry -[`spec/spec_helper.rb`]: https://gitlab.com/gitlab-org/gitlab-ce/blob/master/spec/spec_helper.rb +[`spec/spec_helper.rb`]: https://gitlab.com/gitlab-org/gitlab-foss/blob/master/spec/spec_helper.rb ## Problems we had in the past at GitLab -- [`rspec-retry` is bitting us when some API specs fail](https://gitlab.com/gitlab-org/gitlab-ce/issues/29242): <https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/9825> -- [Sporadic RSpec failures due to `PG::UniqueViolation`](https://gitlab.com/gitlab-org/gitlab-ce/issues/28307#note_24958837): <https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/9846> - - Follow-up: <https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/10688> - - [Capybara.reset_session! should be called before requests are blocked](https://gitlab.com/gitlab-org/gitlab-ce/issues/33779): <https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/12224> +- [`rspec-retry` is bitting us when some API specs fail](https://gitlab.com/gitlab-org/gitlab-foss/issues/29242): <https://gitlab.com/gitlab-org/gitlab-foss/merge_requests/9825> +- [Sporadic RSpec failures due to `PG::UniqueViolation`](https://gitlab.com/gitlab-org/gitlab-foss/issues/28307#note_24958837): <https://gitlab.com/gitlab-org/gitlab-foss/merge_requests/9846> + - Follow-up: <https://gitlab.com/gitlab-org/gitlab-foss/merge_requests/10688> + - [Capybara.reset_session! should be called before requests are blocked](https://gitlab.com/gitlab-org/gitlab-foss/issues/33779): <https://gitlab.com/gitlab-org/gitlab-foss/merge_requests/12224> - FFaker generates funky data that tests are not ready to handle (and tests should be predictable so that's bad!): - - [Make `spec/mailers/notify_spec.rb` more robust](https://gitlab.com/gitlab-org/gitlab-ce/issues/20121): <https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/10015> - - [Transient failure in spec/requests/api/commits_spec.rb](https://gitlab.com/gitlab-org/gitlab-ce/issues/27988#note_25342521): <https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/9944> - - [Replace FFaker factory data with sequences](https://gitlab.com/gitlab-org/gitlab-ce/issues/29643): <https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/10184> - - [Transient failure in spec/finders/issues_finder_spec.rb](https://gitlab.com/gitlab-org/gitlab-ce/issues/30211#note_26707685): <https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/10404> + - [Make `spec/mailers/notify_spec.rb` more robust](https://gitlab.com/gitlab-org/gitlab-foss/issues/20121): <https://gitlab.com/gitlab-org/gitlab-foss/merge_requests/10015> + - [Transient failure in spec/requests/api/commits_spec.rb](https://gitlab.com/gitlab-org/gitlab-foss/issues/27988#note_25342521): <https://gitlab.com/gitlab-org/gitlab-foss/merge_requests/9944> + - [Replace FFaker factory data with sequences](https://gitlab.com/gitlab-org/gitlab-foss/issues/29643): <https://gitlab.com/gitlab-org/gitlab-foss/merge_requests/10184> + - [Transient failure in spec/finders/issues_finder_spec.rb](https://gitlab.com/gitlab-org/gitlab-foss/issues/30211#note_26707685): <https://gitlab.com/gitlab-org/gitlab-foss/merge_requests/10404> ### Time-sensitive flaky tests -- <https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/10046> -- <https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/10306> +- <https://gitlab.com/gitlab-org/gitlab-foss/merge_requests/10046> +- <https://gitlab.com/gitlab-org/gitlab-foss/merge_requests/10306> ### Array order expectation -- <https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/10148> +- <https://gitlab.com/gitlab-org/gitlab-foss/merge_requests/10148> ### Feature tests -- [Be sure to create all the data the test need before starting exercize](https://gitlab.com/gitlab-org/gitlab-ce/issues/32622#note_31128195): <https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/12059> -- [Bis](https://gitlab.com/gitlab-org/gitlab-ce/issues/34609#note_34048715): <https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/12604> -- [Bis](https://gitlab.com/gitlab-org/gitlab-ce/issues/34698#note_34276286): <https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/12664> -- [Assert against the underlying database state instead of against a page's content](https://gitlab.com/gitlab-org/gitlab-ce/issues/31437): <https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/10934> +- [Be sure to create all the data the test need before starting exercize](https://gitlab.com/gitlab-org/gitlab-foss/issues/32622#note_31128195): <https://gitlab.com/gitlab-org/gitlab-foss/merge_requests/12059> +- [Bis](https://gitlab.com/gitlab-org/gitlab-foss/issues/34609#note_34048715): <https://gitlab.com/gitlab-org/gitlab-foss/merge_requests/12604> +- [Bis](https://gitlab.com/gitlab-org/gitlab-foss/issues/34698#note_34276286): <https://gitlab.com/gitlab-org/gitlab-foss/merge_requests/12664> +- [Assert against the underlying database state instead of against a page's content](https://gitlab.com/gitlab-org/gitlab-foss/issues/31437): <https://gitlab.com/gitlab-org/gitlab-foss/merge_requests/10934> #### Capybara viewport size related issues -- [Transient failure of spec/features/issues/filtered_search/filter_issues_spec.rb](https://gitlab.com/gitlab-org/gitlab-ce/issues/29241#note_26743936): <https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/10411> +- [Transient failure of spec/features/issues/filtered_search/filter_issues_spec.rb](https://gitlab.com/gitlab-org/gitlab-foss/issues/29241#note_26743936): <https://gitlab.com/gitlab-org/gitlab-foss/merge_requests/10411> #### Capybara JS driver related issues -- [Don't wait for AJAX when no AJAX request is fired](https://gitlab.com/gitlab-org/gitlab-ce/issues/30461): <https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/10454> -- [Bis](https://gitlab.com/gitlab-org/gitlab-ce/issues/34647): <https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/12626> +- [Don't wait for AJAX when no AJAX request is fired](https://gitlab.com/gitlab-org/gitlab-foss/issues/30461): <https://gitlab.com/gitlab-org/gitlab-foss/merge_requests/10454> +- [Bis](https://gitlab.com/gitlab-org/gitlab-foss/issues/34647): <https://gitlab.com/gitlab-org/gitlab-foss/merge_requests/12626> #### PhantomJS / WebKit related issues -- Memory is through the roof! (TL;DR: Load images but block images requests!): <https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/12003> +- Memory is through the roof! (TL;DR: Load images but block images requests!): <https://gitlab.com/gitlab-org/gitlab-foss/merge_requests/12003> ## Resources diff --git a/doc/development/testing_guide/frontend_testing.md b/doc/development/testing_guide/frontend_testing.md index 91004bf177d..da843218d8b 100644 --- a/doc/development/testing_guide/frontend_testing.md +++ b/doc/development/testing_guide/frontend_testing.md @@ -35,10 +35,10 @@ which could arise (especially with testing against browser specific features). ### Differences to Karma -- Jest runs in a Node.js environment, not in a browser. Support for running Jest tests in a browser [is planned](https://gitlab.com/gitlab-org/gitlab-ce/issues/58205). +- Jest runs in a Node.js environment, not in a browser. Support for running Jest tests in a browser [is planned](https://gitlab.com/gitlab-org/gitlab-foss/issues/58205). - Because Jest runs in a Node.js environment, it uses [jsdom](https://github.com/jsdom/jsdom) by default. See also its [limitations](#limitations-of-jsdom) below. - Jest does not have access to Webpack loaders or aliases. - The aliases used by Jest are defined in its [own config](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/jest.config.js). + The aliases used by Jest are defined in its [own config](https://gitlab.com/gitlab-org/gitlab-foss/blob/master/jest.config.js). - All calls to `setTimeout` and `setInterval` are mocked away. See also [Jest Timer Mocks](https://jestjs.io/docs/en/timer-mocks). - `rewire` is not required because Jest supports mocking modules. See also [Manual Mocks](https://jestjs.io/docs/en/manual-mocks). - No [context object](https://jasmine.github.io/tutorials/your_first_suite#section-The_%3Ccode%3Ethis%3C/code%3E_keyword) is passed to tests in Jest. @@ -58,7 +58,7 @@ This comes with a number of limitations, namely: - [No element sizes or positions](https://github.com/jsdom/jsdom/blob/15.1.1/lib/jsdom/living/nodes/Element-impl.js#L334-L371) - [No layout engine](https://github.com/jsdom/jsdom/issues/1322) in general -See also the issue for [support running Jest tests in browsers](https://gitlab.com/gitlab-org/gitlab-ce/issues/58205). +See also the issue for [support running Jest tests in browsers](https://gitlab.com/gitlab-org/gitlab-foss/issues/58205). ### Debugging Jest tests @@ -67,13 +67,13 @@ Running `yarn jest-debug` will run Jest in debug mode, allowing you to debug/ins ### Timeout error The default timeout for Jest is set in -[`/spec/frontend/test_setup.js`](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/spec/frontend/test_setup.js). +[`/spec/frontend/test_setup.js`](https://gitlab.com/gitlab-org/gitlab-foss/blob/master/spec/frontend/test_setup.js). If your test exceeds that time, it will fail. If you cannot improve the performance of the tests, you can increase the timeout for a specific test using -[`setTestTimeout`](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/spec/frontend/helpers/timeout.js). +[`setTestTimeout`](https://gitlab.com/gitlab-org/gitlab-foss/blob/master/spec/frontend/helpers/timeout.js). ```javascript import { setTestTimeout } from 'helpers/timeout'; @@ -321,8 +321,7 @@ it('waits for an Ajax call', done => { }); ``` -If you are not able to register handlers to the `Promise`—for example because it is executed in a synchronous Vue life -cycle hook—you can flush all pending `Promise`s: +If you are not able to register handlers to the `Promise`, for example because it is executed in a synchronous Vue life cycle hook, please take a look at the [waitFor](#wait-until-axios-requests-finish) helpers or you can flush all pending `Promise`s: **in Jest:** @@ -389,7 +388,7 @@ it('renders something', done => { ##### `setTimeout()` / `setInterval()` in application If the application itself is waiting for some time, mock await the waiting. In Jest this is already -[done by default](https://gitlab.com/gitlab-org/gitlab-ce/blob/a2128edfee799e49a8732bfa235e2c5e14949c68/jest.config.js#L47) +[done by default](https://gitlab.com/gitlab-org/gitlab-foss/blob/a2128edfee799e49a8732bfa235e2c5e14949c68/jest.config.js#L47) (see also [Jest Timer Mocks](https://jestjs.io/docs/en/timer-mocks)). In Karma you can use the [Jasmine mock clock](https://jasmine.github.io/api/2.9/Clock.html). @@ -617,10 +616,10 @@ Tests relevant for frontend development can be found at the following places: - `spec/features/` which are run by RSpec and contain - [feature tests](#feature-tests) -All tests in `spec/javascripts/` will eventually be migrated to `spec/frontend/` (see also [#52483](https://gitlab.com/gitlab-org/gitlab-ce/issues/52483)). +All tests in `spec/javascripts/` will eventually be migrated to `spec/frontend/` (see also [#52483](https://gitlab.com/gitlab-org/gitlab-foss/issues/52483)). In addition, there used to be feature tests in `features/`, run by Spinach. -These were removed from the codebase in May 2018 ([#23036](https://gitlab.com/gitlab-org/gitlab-ce/issues/23036)). +These were removed from the codebase in May 2018 ([#23036](https://gitlab.com/gitlab-org/gitlab-foss/issues/23036)). See also [Notes on testing Vue components](../fe_guide/vue.html#testing-vue-components). @@ -1049,7 +1048,7 @@ testAction( ); ``` -Check an example in [spec/javascripts/ide/stores/actions_spec.jsspec/javascripts/ide/stores/actions_spec.js](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/spec/javascripts/ide/stores/actions_spec.js). +Check an example in [spec/javascripts/ide/stores/actions_spec.jsspec/javascripts/ide/stores/actions_spec.js](https://gitlab.com/gitlab-org/gitlab-foss/blob/master/spec/javascripts/ide/stores/actions_spec.js). ### Vue Helper: `mountComponent` @@ -1088,6 +1087,16 @@ afterEach(() => { }); ``` +### Wait until axios requests finish + +The axios utils mock module located in `spec/frontend/mocks/ce/lib/utils/axios_utils.js` contains two helper methods for Jest tests that spawn HTTP requests. +These are very useful if you don't have a handle to the request's Promise, for example when a Vue component does a request as part of its life cycle. + +- `waitFor(url, callback)`: Runs `callback` after a request to `url` finishes (either successfully or unsuccessfully). +- `waitForAll(callback)`: Runs `callback` once all pending requests have finished. If no requests are pending, runs `callback` on the next tick. + +Both functions run `callback` on the next tick after the requests finish (using `setImmediate()`), to allow any `.then()` or `.catch()` handlers to run. + ## Testing with older browsers Some regressions only affect a specific browser version. We can install and test in particular browsers with either Firefox or Browserstack using the following steps: diff --git a/doc/development/testing_guide/review_apps.md b/doc/development/testing_guide/review_apps.md index 8698a1e4c2d..8ce25376a05 100644 --- a/doc/development/testing_guide/review_apps.md +++ b/doc/development/testing_guide/review_apps.md @@ -1,8 +1,7 @@ # Review Apps -Review Apps are automatically deployed by each pipeline, both in -[CE](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/22010) and -[EE](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/6665). +Review Apps are automatically deployed by [the +pipeline](https://gitlab.com/gitlab-org/gitlab/merge_requests/6665). ## How does it work? @@ -190,8 +189,8 @@ that the `review-apps-ce/ee` cluster is unhealthy. Leading indicators may be hea The following items may help diagnose this: -- [Instance group CPU Utilization in GCP](https://console.cloud.google.com/compute/instanceGroups/details/us-central1-a/gke-review-apps-ce-preemp-n1-standard-a4c9571c-grp?project=gitlab-review-apps&tab=monitoring&graph=GCE_CPU&duration=PT12H) - helpful to identify if nodes are problematic or the entire cluster is trending towards unhealthy -- [Instance Group size in GCP](https://console.cloud.google.com/compute/instanceGroups/details/us-central1-a/gke-review-apps-ce-preemp-n1-standard-a4c9571c-grp?project=gitlab-review-apps&tab=monitoring&graph=GCE_SIZE&duration=PT12H) - aids in identifying load spikes on the cluster. Kubernetes will add nodes up to 220 based on total resource requests. +- [Instance group CPU Utilization in GCP](https://console.cloud.google.com/compute/instanceGroups/details/us-central1-b/gke-review-apps-ee-preemp-n1-standard-8affc0f5-grp?project=gitlab-review-apps&tab=monitoring&graph=GCE_CPU&duration=P30D) - helpful to identify if nodes are problematic or the entire cluster is trending towards unhealthy +- [Instance Group size in GCP](https://console.cloud.google.com/compute/instanceGroups/details/us-central1-b/gke-review-apps-ee-preemp-n1-standard-8affc0f5-grp?project=gitlab-review-apps&tab=monitoring&graph=GCE_SIZE&duration=P30D) - aids in identifying load spikes on the cluster. Kubernetes will add nodes up to 220 based on total resource requests. - `kubectl top nodes --sort-by=cpu` - can identify if node spikes are common or load on specific nodes which may get rebalanced by the Kubernetes scheduler. - `kubectl top pods --sort-by=cpu` - - [K9s] - K9s is a powerful command line dashboard which allows you to filter by labels. This can help identify trends with apps exceeding the [review-app resource requests](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/scripts/review_apps/base-config.yaml). Kubernetes will schedule pods to nodes based on resource requests and allow for CPU usage up to the limits. @@ -206,7 +205,7 @@ The following items may help diagnose this: #### Finding the problem -[In the past](https://gitlab.com/gitlab-org/gitlab-ce/issues/62834), it happened +[In the past](https://gitlab.com/gitlab-org/gitlab-foss/issues/62834), it happened that the `dns-gitlab-review-app-external-dns` Deployment was in a pending state, effectively preventing all the Review Apps from getting a DNS record assigned, making them unreachable via domain name. @@ -299,10 +298,10 @@ find a way to limit it to only us.** - [Stern](https://github.com/wercker/stern) - enables cross pod log tailing based on label/field selectors [charts-1068]: https://gitlab.com/gitlab-org/charts/gitlab/issues/1068 -[gitlab-pipeline]: https://gitlab.com/gitlab-org/gitlab-ce/pipelines/44362587 -[gitlab:assets:compile]: https://gitlab.com/gitlab-org/gitlab-ce/-/jobs/149511610 -[review-build-cng]: https://gitlab.com/gitlab-org/gitlab-ce/-/jobs/149511623 -[review-deploy]: https://gitlab.com/gitlab-org/gitlab-ce/-/jobs/149511624 +[gitlab-pipeline]: https://gitlab.com/gitlab-org/gitlab-foss/pipelines/44362587 +[gitlab:assets:compile]: https://gitlab.com/gitlab-org/gitlab-foss/-/jobs/149511610 +[review-build-cng]: https://gitlab.com/gitlab-org/gitlab-foss/-/jobs/149511623 +[review-deploy]: https://gitlab.com/gitlab-org/gitlab-foss/-/jobs/149511624 [cng-mirror]: https://gitlab.com/gitlab-org/build/CNG-mirror [cng]: https://gitlab.com/gitlab-org/build/CNG [cng-mirror-pipeline]: https://gitlab.com/gitlab-org/build/CNG-mirror/pipelines/44364657 @@ -310,13 +309,13 @@ find a way to limit it to only us.** [helm-chart]: https://gitlab.com/gitlab-org/charts/gitlab/ [review-apps-ce]: https://console.cloud.google.com/kubernetes/clusters/details/us-central1-a/review-apps-ce?project=gitlab-review-apps [review-apps-ee]: https://console.cloud.google.com/kubernetes/clusters/details/us-central1-b/review-apps-ee?project=gitlab-review-apps -[review-apps.sh]: https://gitlab.com/gitlab-org/gitlab-ee/blob/master/scripts/review_apps/review-apps.sh -[automated_cleanup.rb]: https://gitlab.com/gitlab-org/gitlab-ee/blob/master/scripts/review_apps/automated_cleanup.rb -[Auto-DevOps.gitlab-ci.yml]: https://gitlab.com/gitlab-org/gitlab-ce/blob/master/lib/gitlab/ci/templates/Auto-DevOps.gitlab-ci.yml -[gitlab-ci-yml]: https://gitlab.com/gitlab-org/gitlab-ce/blob/master/.gitlab-ci.yml +[review-apps.sh]: https://gitlab.com/gitlab-org/gitlab/blob/master/scripts/review_apps/review-apps.sh +[automated_cleanup.rb]: https://gitlab.com/gitlab-org/gitlab/blob/master/scripts/review_apps/automated_cleanup.rb +[Auto-DevOps.gitlab-ci.yml]: https://gitlab.com/gitlab-org/gitlab-foss/blob/master/lib/gitlab/ci/templates/Auto-DevOps.gitlab-ci.yml +[gitlab-ci-yml]: https://gitlab.com/gitlab-org/gitlab-foss/blob/master/.gitlab-ci.yml [gitlab-k8s-integration]: ../../user/project/clusters/index.md -[password-bug]: https://gitlab.com/gitlab-org/gitlab-ce/issues/53621 [K9s]: https://github.com/derailed/k9s +[password-bug]: https://gitlab.com/gitlab-org/gitlab-foss/issues/53621 --- diff --git a/doc/development/testing_guide/smoke.md b/doc/development/testing_guide/smoke.md index c9d3238fbe9..5a144b43478 100644 --- a/doc/development/testing_guide/smoke.md +++ b/doc/development/testing_guide/smoke.md @@ -12,6 +12,7 @@ Currently, our suite consists of this basic functionality coverage: - Project simple creation - Project creation with Auto-DevOps enabled - Issue creation +- Issue user mentions - Merge Request creation - Snippet creation diff --git a/doc/development/testing_guide/testing_levels.md b/doc/development/testing_guide/testing_levels.md index 1aee306f492..b9436abf856 100644 --- a/doc/development/testing_guide/testing_levels.md +++ b/doc/development/testing_guide/testing_levels.md @@ -44,7 +44,7 @@ records should use stubs/doubles as much as possible. | `config/routes.rb`, `config/routes/` | `spec/routing/` | RSpec | | | `config/puma.example.development.rb`, `config/unicorn.rb.example` | `spec/rack_servers/` | RSpec | | | `db/` | `spec/db/` | RSpec | | -| `db/{post_,}migrate/` | `spec/migrations/` | RSpec | More details at [`spec/migrations/README.md`](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/spec/migrations/README.md). | +| `db/{post_,}migrate/` | `spec/migrations/` | RSpec | More details at [`spec/migrations/README.md`](https://gitlab.com/gitlab-org/gitlab/blob/master/spec/migrations/README.md). | | `Gemfile` | `spec/dependencies/`, `spec/sidekiq/` | RSpec | | | `lib/` | `spec/lib/` | RSpec | | | `lib/tasks/` | `spec/tasks/` | RSpec | | @@ -81,7 +81,7 @@ controller.instance_variable_set(:@user, user) and use methods which are deprecated in Rails 5 ([#23768]). -[#23768]: https://gitlab.com/gitlab-org/gitlab-ce/issues/23768 +[#23768]: https://gitlab.com/gitlab-org/gitlab-foss/issues/23768 ### About Karma @@ -190,8 +190,8 @@ confused with the application's [unit tests](#unit-tests) or [GitLab Pages]: https://gitlab.com/gitlab-org/gitlab-pages [GitLab Runner]: https://gitlab.com/gitlab-org/gitlab-runner [GitLab Omnibus]: https://gitlab.com/gitlab-org/omnibus-gitlab -[part of GitLab Rails]: https://gitlab.com/gitlab-org/gitlab-ce/tree/master/qa -[test plan]: https://gitlab.com/gitlab-org/gitlab-ce/tree/master/.gitlab/issue_templates/Test%20plan.md +[part of GitLab Rails]: https://gitlab.com/gitlab-org/gitlab-foss/tree/master/qa +[test plan]: https://gitlab.com/gitlab-org/gitlab/tree/master/.gitlab/issue_templates/Test%20plan.md [Product category]: https://about.gitlab.com/handbook/product/categories/ ### Smoke tests |