diff options
Diffstat (limited to 'qa')
24 files changed, 202 insertions, 1214 deletions
diff --git a/qa/Gemfile b/qa/Gemfile index 64215b24cf1..12994b85322 100644 --- a/qa/Gemfile +++ b/qa/Gemfile @@ -10,3 +10,4 @@ gem 'airborne', '~> 0.2.13' gem 'nokogiri', '~> 1.10.3' gem 'rspec-retry', '~> 0.6.1' gem 'faker', '~> 1.6', '>= 1.6.6' +gem 'knapsack', '~> 1.17' diff --git a/qa/Gemfile.lock b/qa/Gemfile.lock index a06c88b6f0a..6b0635ed0e2 100644 --- a/qa/Gemfile.lock +++ b/qa/Gemfile.lock @@ -39,6 +39,8 @@ GEM domain_name (~> 0.5) i18n (0.9.1) concurrent-ruby (~> 1.0) + knapsack (1.17.1) + rake launchy (2.4.3) addressable (~> 2.3) method_source (0.9.0) @@ -102,6 +104,7 @@ DEPENDENCIES capybara (~> 2.16.1) capybara-screenshot (~> 1.0.18) faker (~> 1.6, >= 1.6.6) + knapsack (~> 1.17) nokogiri (~> 1.10.3) pry-byebug (~> 3.5.1) rake (~> 12.3.0) diff --git a/qa/README.md b/qa/README.md index f75205133e6..ef6f202464d 100644 --- a/qa/README.md +++ b/qa/README.md @@ -1,6 +1,6 @@ # GitLab QA - End-to-end tests for GitLab -This directory contains [end-to-end tests](doc/development/testing_guide/end_to_end_tests.md) +This directory contains [end-to-end tests](../../../doc/development/testing_guide/end_to_end/index.md) for GitLab. It includes the test framework and the tests themselves. The tests can be found in `qa/specs/features` (not to be confused with the unit @@ -29,7 +29,7 @@ verify coupling between page objects implemented as a part of GitLab QA and corresponding views / partials / selectors in CE / EE. Whenever `qa:selectors` job fails in your merge request, you are supposed to -fix [page objects](qa/page/README.md). You should also trigger end-to-end tests +fix [page objects](../doc/development/testing_guide/end_to_end/page_objects.md). You should also trigger end-to-end tests using `package-and-qa` manual action, to test if everything works fine. ## How can I use it? @@ -49,10 +49,10 @@ will need to [modify your GDK setup](https://gitlab.com/gitlab-org/gitlab-qa/blo ### Writing tests -- [Writing tests from scratch tutorial](docs/writing_tests_from_scratch.md) - - [Best practices](docs/best_practices.md) - - [Using page objects](qa/page/README.md) - - [Guidelines](docs/guidelines.md) +- [Writing tests from scratch tutorial](../doc/development/testing_guide/end_to_end/quick_start_guide.md) + - [Best practices](../doc/development/testing_guide/best_practices.md) + - [Using page objects](../doc/development/testing_guide/end_to_end/page_objects.md) + - [Guidelines](../doc/development/testing_guide/index.md) ### Running specific tests diff --git a/qa/docs/best_practices.md b/qa/docs/best_practices.md deleted file mode 100644 index d6e5350b0c8..00000000000 --- a/qa/docs/best_practices.md +++ /dev/null @@ -1,38 +0,0 @@ -# Best practices when writing end-to-end tests - -The majority of the end-to-end tests require some state to be built in the application for the tests to happen. - -A good example is a user being logged in as a pre-condition for testing the feature. - -But if the login feature is already covered with end-to-end tests through the GUI, there is no reason to perform such an expensive task to test the functionality of creating a project, or importing a repo, even if this features depend on a user being logged in. Let's see an example to make things clear. - -Let's say that, on average, the process to perform a successful login through the GUI takes 2 seconds. - -Now, realize that almost all tests need the user to be logged in, and that we need every test to run in isolation, meaning that tests cannot interfere with each other. This would mean that for every test the user needs to log in, and "waste 2 seconds". - -Now, multiply the number of tests per 2 seconds, and as your test suite grows, the time to run it grows with it, and this is not sustainable. - -An alternative to perform a login in a cheaper way would be having an endpoint (available only for testing) where we could pass the user's credentials as encrypted values as query strings, and then we would be redirected to the logged in home page if the credentials are valid. Let's say that, on average, this process takes only 200 miliseconds. - -You see the point right? - -Performing a login through the GUI for every test would cost a lot in terms of tests' execution. - -And there is another reason. - -Let's say that you don't follow the above suggestion, and depend on the GUI for the creation of every application state in order to test a specific feature. In this case we could be talking about the **Issues** feature, that depends on a project to exist, and the user to be logged in. - -What would happen if there was a bug in the project creation page, where the 'Create' button is disabled, not allowing for the creation of a project through the GUI, but the API logic is still working? - -In this case, instead of having only the project creation test failing, we would have many tests that depend on a project to be failing too. - -But, if we were following the best practices, only one test would be failing, and tests for other features that depend on a project to exist would continue to pass, since they could be creating the project behind the scenes interacting directly with the public APIs, ensuring a more reliable metric of test failure rate. - -Finally, interacting with the application only by its GUI generates a higher rate of test flakiness, and we want to avoid that at max. - -**The takeaways here are:** - -- Building state through the GUI is time consuming and it's not sustainable as the test suite grows. -- When depending only on the GUI to create the application's state and tests fail due to front-end issues, we can't rely on the test failures rate, and we generates a higher rate of test flakiness. - -Now that we are aware of all of it, [let's go create some tests](writing_tests_from_scratch.md). diff --git a/qa/docs/guidelines.md b/qa/docs/guidelines.md deleted file mode 100644 index cd4b939fd71..00000000000 --- a/qa/docs/guidelines.md +++ /dev/null @@ -1,97 +0,0 @@ -# Style guide for writing E2E tests - -This document describes the conventions used at GitLab for writing E2E tests using the GitLab QA project. - -## `click_` versus `go_to_` - -### When to use `click_`? - -When clicking in a single link to navigate, use `click_`. - -E.g.: - -```ruby -def click_ci_cd_pipelines - within_sidebar do - click_element :link_pipelines - end -end -``` - -From a testing perspective, if we want to check that clicking a link, or a button (a single interaction) is working as intended, we would want the test to read as: - -- Click a certain element -- Verify the action took place - -### When to use `go_to_`? - -When interacting with multiple elements to go to a page, use `go_to_`. - -E.g.: - -```ruby -def go_to_operations_environments - hover_operations do - within_submenu do - click_element(:operations_environments_link) - end - end -end -``` - -`go_to_` fits the definition of interacting with multiple elements very well given it's more of a meta-navigation action that includes multiple interactions. - -Notice that in the above example, before clicking the `:operations_environments_link`, another element is hovered over. - -> We can create these methods as helpers to abstract multi-step navigation. - -### Element Naming Convention - -When adding new elements to a page, it's important that we have a uniform element naming convention. - -We follow a simple formula roughly based on hungarian notation. - -*Formula*: `element :<descriptor>_<type>` - -- `descriptor`: The natural-language description of what the element is. On the login page, this could be `username`, or `password`. -- `type`: A physical control on the page that can be seen by a user. - - `_button` - - `_link` - - `_tab` - - `_dropdown` - - `_field` - - `_checkbox` - - `_radio` - - `_content` - -*Note: This list is a work in progress. This list will eventually be the end-all enumeration of all available types. - I.e., any element that does not end with something in this list is bad form.* - -#### Examples - -**Good** -```ruby -view '...' do - element :edit_button - element :notes_tab - element :squash_checkbox - element :username_field - element :issue_title_content -end -``` - -**Bad** -```ruby -view '...' do - # `_confirmation` should be `_field`. what sort of confirmation? a checkbox confirmation? no real way to disambiguate. - # an appropriate replacement would be `element :password_confirmation_field` - element :password_confirmation - - # `clone_options` is too vague. If it's a dropdown menu, it should be `clone_dropdown`. - # If it's a checkbox, it should be `clone_checkbox` - element :clone_options - - # how is this url being displayed? is it a textbox? a simple span? - element :ssh_clone_url -end -``` diff --git a/qa/docs/writing_tests_from_scratch.md b/qa/docs/writing_tests_from_scratch.md deleted file mode 100644 index 65e7a78a8b5..00000000000 --- a/qa/docs/writing_tests_from_scratch.md +++ /dev/null @@ -1,482 +0,0 @@ -# Writing end-to-end tests step-by-step - -In this tutorial, you will find different examples, and the steps involved, in the creation of end-to-end (_e2e_) tests for GitLab CE and GitLab EE, using GitLab QA. - -> When referring to end-to-end tests in this document, this means testing a specific feature end-to-end, such as a user logging in, the creation of a project, the management of labels, breaking down epics into sub-epics and issues, etc. - -## Important information before we start writing tests - -It's important to understand that end-to-end tests of isolated features, such as the ones described in the above note, doesn't mean that everything needs to happen through the GUI. - -If you don't exactly understand what we mean by **not everything needs to happen through the GUI,** please make sure you've read the [best practices](best_practices.md) before moving on. - -## This document covers the following items: - -0. Identifying if end-to-end tests are really needed -1. Identifying the [DevOps stage](https://about.gitlab.com/stages-devops-lifecycle/) of the feature that you are going to cover with end-to-end tests -2. Creating the skeleton of the test file (`*_spec.rb`) -3. The [MVC](https://about.gitlab.com/handbook/values/#minimum-viable-change-mvc) of the test cases logic -4. Extracting duplicated code into methods -5. Tests' pre-conditions (`before :all` and `before`) using resources and [Page Objects](./qa/page/README.md) -6. Optimizing the test suite -7. Using and implementing resources -8. Moving elements definitions and its methods to [Page Objects](./qa/page/README.md) - - Adding testability to the application - -### 0. Are end-to-end tests needed? - -At GitLab we respect the [test pyramid](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/development/testing_guide/testing_levels.md), and so, we recommend to check the code coverage of a specific feature before writing end-to-end tests. - -Sometimes you may notice that there is already a good coverage in other test levels, and we can stay confident that if we break a feature, we will still have quick feedback about it, even without having end-to-end tests. - -If after this analysis you still think that end-to-end tests are needed, keep reading. - -### 1. Identifying the DevOps stage - -The GitLab QA end-to-end tests are organized by the different [stages in the DevOps lifecycle](https://gitlab.com/gitlab-org/gitlab-ce/tree/master/qa/qa/specs/features/browser_ui), and so, if you are creating tests for issue creation, for instance, you would locate the spec files under the `qa/qa/specs/features/browser_ui/2_plan/` directory since issue creation is part of the Plan stage. - - In another case of a test for listing merged merge requests (MRs), the test should go under the `qa/qa/specs/features/browser_ui/3_create/` directory since merge request is a feature from the Create stage. - -> There may be sub-directories inside the stages directories, for different features. For example: `.../browser_ui/2_plan/ee_epics/` and `.../browser_ui/2_plan/issues/`. - -Now, let's say we want to create tests for the [scoped labels](https://about.gitlab.com/2019/04/22/gitlab-11-10-released/#scoped-labels) feature, available on GitLab EE Premium (this feature is part of the Plan stage.) - -> Because these tests are for a feature available only on GitLab EE, we need to create them in the [EE repository](https://gitlab.com/gitlab-org/gitlab-ee). - -Since [there is no specific directory for this feature](https://gitlab.com/gitlab-org/gitlab-ee/tree/master/qa/qa/specs/features/browser_ui/2_plan), we should create a sub-directory for it. - -Under `.../browser_ui/2_plan/`, let's create a sub-directory called `ee_scoped_labels/`. - -> Notice that since this feature is only available for GitLab EE we prefix the sub-directory with `ee_`. - -### 2. Test skeleton - -Inside the newly created sub-directory, let's create a file describing the test suite (e.g. `editing_scoped_labels_spec.rb`.) - -#### The `context` and `describe` blocks - -Specs have an outer `context` that indicates the DevOps stage. The next level is the `describe` block, that briefly states the subject of the test suite. See the following example: - -```ruby -module QA - context 'Plan' do - describe 'Editing scoped labels properties on issues' do - end - end -end -``` - -#### The `it` blocks - -Every test suite is composed by at least one `it` block, and a good way to start writing end-to-end tests is by typing test cases descriptions as `it` blocks. Take a look at the following example: - -```ruby -module QA - context 'Plan' do - describe 'Editing scoped labels properties on issues' do - it 'replaces an existing label if it has the same key' do - end - - it 'keeps both scoped labels when adding a label with a different key' do - end - end - end -end -``` - -### 3. Test cases MVC - -For the [MVC](https://about.gitlab.com/handbook/values/#minimum-viable-change-mvc) of our test cases, let's say that we already have the application in the state needed for the tests, and then let's focus on the logic of the test cases only. - -To evolve the test cases drafted on step 2, let's imagine that the user is already logged in a GitLab EE instance, they already have at least a Premium license in use, there is already a project created, there is already an issue opened in the project, the issue already has a scoped label (e.g. `foo::bar`), there are other scoped labels (for the same scope and for a different scope, e.g. `foo::baz` and `bar::bah`), and finally, the user is already on the issue's page. Let's also suppose that for every test case the application is in a clean state, meaning that one test case won't affect another. - -> Note: there are different approaches to create an application state for end-to-end tests. Some of them are very time consuming and subject to failures, such as when using the GUI for all the pre-conditions of the tests. On the other hand, other approaches are more efficient, such as using the public APIs. The latter is more efficient since it doesn't depend on the GUI. We won't focus on this part yet, but it's good to keep it in mind. - -Let's now focus on the first test case. - -```ruby -it 'keeps the latest scoped label when adding a label with the same key of an existing one, but with a different value' do - # This implementation is only for tutorial purposes. We normally encapsulate elements in Page Objects. - page.find('.block.labels .edit-link').click - page.find('.dropdown-menu-labels .dropdown-input-field').send_keys ['foo::baz', :enter] - page.find('#content-body').click - page.refresh - - scoped_label = page.find('.qa-labels-block .scoped-label-wrapper') - - expect(scoped_label).to have_content('foo::baz') - expect(scoped_label).not_to have_content('foo::bar') - expect(page).to have_content('added foo::baz label and removed foo::bar') -end -``` - -> Notice that the test itself is simple. The most challenging part is the creation of the application state, which will be covered later. - -> The exemplified test cases' MVC is not enough for the change to be submitted in an MR, but they help on building up the test logic. The reason is that we do not want to use locators directly in the tests, and tests **must** use [Page Objects](./qa/page/README.md) before they can be merged. - -Below are the steps that the test covers: - -1. The test finds the 'Edit' link for the labels and clicks on it -2. Then it fills in the 'Assign labels' input field with the value 'foo::baz' and press enter -3. Then it clicks in the content body to apply the label and refreshes the page -4. Finally the expectation that the previous scoped label was removed and that the new one was added happens - -Let's now see how the second test case would look like. - -```ruby -it 'keeps both scoped labels when adding a label with a different key' do - # This implementation is only for tutorial purposes. We normally encapsulate elements in Page Objects. - page.find('.block.labels .edit-link').click - page.find('.dropdown-menu-labels .dropdown-input-field').send_keys ['bar::bah', :enter] - page.find('#content-body').click - page.refresh - - scoped_labels = page.all('.qa-labels-block .scoped-label-wrapper') - - expect(scoped_labels.first).to have_content('bar::bah') - expect(scoped_labels.last).to have_content('foo::ba') - expect(page).to have_content('added bar::bah') - expect(page).to have_content('added foo::ba') -end -``` - -> Note that elements are always located using CSS selectors, and a good practice is to add test specific attribute:value for elements (this is called adding testability to the application and we will talk more about it later.) - -Below are the steps that the test covers: - -1. The test finds the 'Edit' link for the labels and clicks on it -2. Then it fills in the 'Assign labels' input field with the value 'bar::bah' and press enter -3. Then it clicks in the content body to apply the label and refreshes the page -4. Finally the expectation that the both scoped labels are present happens - -> Similar to the previous test, this one is also very straight forward, but there is some code duplication. Let's address it. - -### 4. Extracting duplicated code - -If we refactor the tests created on step 3 we could come up with something like this: - -```ruby -it 'keeps the latest scoped label when adding a label with the same key of an existing one, but with a different value' do - select_label_and_refresh 'foo::baz' - - expect(page).to have_content('added foo::baz') - expect(page).to have_content('and removed foo::bar') - - scoped_label = page.find('.qa-labels-block .scoped-label-wrapper') - - expect(scoped_label).to have_content('foo::baz') - expect(scoped_label).not_to have_content('foo::bar') -end - -it 'keeps both scoped label when adding a label with a different key' do - select_label_and_refresh 'bar::bah' - - expect(page).to have_content('added bar::bah') - expect(page).to have_content('added foo::ba') - - scoped_labels = page.all('.qa-labels-block .scoped-label-wrapper') - - expect(scoped_labels.first).to have_content('bar::bah') - expect(scoped_labels.last).to have_content('foo::ba') -end - -def select_label_and_refresh(label) - page.find('.block.labels .edit-link').click - page.find('.dropdown-menu-labels .dropdown-input-field').send_keys [label, :enter] - page.find('#content-body').click - page.refresh -end -``` - -By creating a reusable `select_label_and_refresh` method, we remove the code duplication, and later we can move this method to a Page Object class that will be created for easier maintenance purposes. - -> Notice that the reusable method is created in the bottom of the file. The reason for that is that reading the code should be similar to reading a newspaper, where high-level information is at the top, like the title and summary of the news, while low level, or more specific information, is at the bottom. - -### 5. Tests' pre-conditions using resources and Page Objects - -In this section, we will address the previously mentioned subject of creating the application state for the tests, using the `before :all` and `before` blocks, together with resources and Page Objects. - -#### `before :all` - -A pre-condition for the entire test suite is defined in the `before :all` block. - -For our test suite example, some things that could happen before the entire test suite starts are: - -- The user logging in; -- A premium license already being set up; -- A project being created with an issue and labels already setup. - -> In case of a test suite with only one `it` block it's ok to use only the `before` block (see below) with all the test's pre-conditions. - -#### `before` - -A pre-condition for each test case is defined in the `before` block. - -For our test cases samples, what we need is that for every test the issue page is opened, and there is only one scoped label applied to it. - -#### Implementation - -In the following code we will focus on the test suite and the test cases' pre-conditions only: - -```ruby -module QA - context 'Plan' do - describe 'Editing scoped labels properties on issues' do - before :all do - project = Resource::Project.fabricate_via_api! do |resource| - resource.name = 'scoped-labels-project' - end - - @foo_bar_scoped_label = 'foo::bar' - - @issue = Resource::Issue.fabricate_via_api! do |issue| - issue.project = project - issue.title = 'Issue to test the scoped labels' - issue.labels = @foo_bar_scoped_label - end - - @labels = ['foo::baz', 'bar::bah'] - @labels.each do |label| - Resource::Label.fabricate_via_api! do |l| - l.project = project.id - l.title = label - end - end - - Runtime::Browser.visit(:gitlab, Page::Main::Login) - Page::Main::Login.perform(&:sign_in_using_credentials) - end - - before do - Page::Project::Issue::Show.perform do |issue_page| - @issue.visit! - end - end - - it 'keeps the latest scoped label when adding a label with the same key of an existing one, but with a different value' do - ... - end - - it 'keeps both scoped labels when adding a label with a different key' do - ... - end - - def select_label_and_refresh(label) - ... - end - end - end -end -``` - -In the `before :all` block we create all the application state needed for the tests to run. We do that by fabricating resources via APIs (`project`, `@issue`, and `@labels`), by using the `Runtime::Browser.visit` method to go to the login page, and by performing a `sign_in_using_credentials` from the `Login` Page Object. - -> When creating the resources, notice that when calling the `fabricate_via_api` method, we pass some attribute:values, like `name` for the `project` resource; `project`, `title`, and `labels` for the `issue` resource; and `project`, and `title` for `label` resources. - -> What's important to understand here is that by creating the application state mostly using the public APIs we save a lot of time in the test suite setup stage. - -> Soon we will cover the use of the already existing resources' methods and the creation of your own `fabricate_via_api` methods for resources where this is still not available, but first, let's optimize our implementation. - -### 6. Optimization - -As already mentioned in the [best practices](./BEST_PRACTICES.md) document, end-to-end tests are very costly in terms of execution time, and it's our responsibility as software engineers to ensure that we optimize them as much as possible. - -> Differently than unit tests, that exercise every little piece of the application in isolation, usually having only one assertion per test, and being very fast to run, end-to-end tests can have more actions and assertions in a single test to help on speeding up the test's feedback since they are much slower when comparing to unit tests. - -Some improvements that we could make in our test suite to optimize its time to run are: - -1. Having a single test case (an `it` block) that exercise both scenarios to avoid "wasting" time in the tests' pre-conditions, instead of having two different test cases. -2. Moving all the pre-conditions to the `before` block since there will be only one `it` block. -3. Making the selection of labels more performant by allowing for the selection of more than one label in the same reusable method. - -Let's look at a suggestion that addresses the above points, one by one: - -```ruby -module QA - context 'Plan' do - describe 'Editing scoped labels properties on issues' do - before do - project = Resource::Project.fabricate_via_api! do |resource| - resource.name = 'scoped-labels-project' - end - - @foo_bar_scoped_label = 'foo::bar' - - @issue = Resource::Issue.fabricate_via_api! do |issue| - issue.project = project - issue.title = 'Issue to test the scoped labels' - issue.labels = @foo_bar_scoped_label - end - - @labels = ['foo::baz', 'bar::bah'] - @labels.each do |label| - Resource::Label.fabricate_via_api! do |l| - l.project = project.id - l.title = label - end - end - - Runtime::Browser.visit(:gitlab, Page::Main::Login) - Page::Main::Login.perform(&:sign_in_using_credentials) - Page::Project::Issue::Show.perform do |issue_page| - @issue.visit! - end - end - - it 'correctly applies the scoped labels depending if they are from the same or a different scope' do - select_labels_and_refresh @labels - - scoped_labels = page.all('.qa-labels-block .scoped-label-wrapper') - - expect(page).to have_content("added #{@foo_bar_scoped_label}") - expect(page).to have_content("added #{@labels[1]} #{@labels[0]} labels and removed #{@foo_bar_scoped_label}") - expect(scoped_labels.count).to eq(2) - expect(scoped_labels.first).to have_content(@labels[1]) - expect(scoped_labels.last).to have_content(@labels[0]) - end - - def select_labels_and_refresh(labels) - find('.block.labels .edit-link').click - labels.each do |label| - find('.dropdown-menu-labels .dropdown-input-field').send_keys [label, :enter] - end - find('#content-body').click - refresh - end - end - end - end -``` - -As you can see, now all the pre-conditions from the `before :all` block were moved to the `before` block, addressing point 2. - -To address point 1, we changed the test implementation from two `it` blocks into a single one that exercises both scenarios. Now the new test description is: `'correctly applies the scoped labels depending if they are from the same or a different scope'`. It's a long description, but it describes well what the test does. - -> Notice that the implementation of the new and unique `it` block had to change a little bit. Below we describe in details what it does. - -1. At the same time, it selects two scoped labels, one from the same scope of the one already applied in the issue during the setup phase (in the `before` block), and another one from a different scope. -2. It runs the assertions that the labels where correctly added and removed; that only two labels are applied; and that those are the correct ones, and that they are shown in the right order. - -Finally, the `select_label_and_refresh` method is changed to `select_labels_and_refresh`, which accepts an array of labels instead of a single label, and it iterates on them for faster label selection (this is what is used in step 1 explained above.) - -### 7. Resources - -You can think of resources as anything that can be created on GitLab CE or EE, either through the GUI, the API, or the CLI. - -With that in mind, resources can be a project, an epic, an issue, a label, a commit, etc. - -As you saw in the tests' pre-conditions and the optimization sections, we're already creating some of these resources, and we are doing that by calling the `fabricate_via_api!` method. - -> We could be using the `fabricate!` method instead, which would use the `fabricate_via_api!` method if it exists, and fallback to GUI fabrication otherwise, but we recommend being explicit to make it clear what the test does. Also, we always recommend fabricating resources via API since this makes tests faster and more reliable. - -For our test suite example, the [project resource](https://gitlab.com/gitlab-org/gitlab-ee/blob/d3584e80b4236acdf393d815d604801573af72cc/qa/qa/resource/project.rb#L55) already had a `fabricate_via_api!` method available, while other resources don't have it, so we will have to create them, like for the issue and label resources. Also, we will have to make a small change in the project resource to expose its `id` attribute so that we can refer to it when fabricating the issue. - -#### Implementation - -Following we describe the changes needed in every of the before-mentioned resource files. - -**Project resource** - -Let's start with the smallest change. - -In the [project resource](https://gitlab.com/gitlab-org/gitlab-ee/blob/master/qa/qa/resource/project.rb), let's expose its `id` attribute. - -Add the following `attribute :id` right below the [`attribute :description`](https://gitlab.com/gitlab-org/gitlab-ee/blob/d3584e80b4236acdf393d815d604801573af72cc/qa/qa/resource/project.rb#L11). - -> This line is needed to allow for issues and labels to be automatically added to a project when fabricating them via API. - -**Issue resource** - -Now, let's make it possible to create an issue resource through the API. - -First, in the [issue resource](https://gitlab.com/gitlab-org/gitlab-ee/blob/d3584e80b4236acdf393d815d604801573af72cc/qa/qa/resource/issue.rb), let's expose its labels attribute. - -Add the following `attribute :labels` right below the [`attribute :title`](https://gitlab.com/gitlab-org/gitlab-ee/blob/d3584e80b4236acdf393d815d604801573af72cc/qa/qa/resource/issue.rb#L15). - -> This line is needed to allow for labels to be automatically added to an issue when fabricating it via API. - -Next, add the following code right below the [`fabricate!`](https://gitlab.com/gitlab-org/gitlab-ee/blob/d3584e80b4236acdf393d815d604801573af72cc/qa/qa/resource/issue.rb#L27) method. - -```ruby -def api_get_path - "/projects/#{project.id}/issues/#{id}" -end - -def api_post_path - "/projects/#{project.id}/issues" -end - -def api_post_body - { - title: title, - labels: [labels] - } -end -``` - -By defining the `api_get_path` method, we allow the [`ApiFabricator`](https://gitlab.com/gitlab-org/gitlab-ee/blob/master/qa/qa/resource/api_fabricator.rb) module to know which path to use to get a single issue. - -> This `GET` path can be found in the [public API documentation](https://docs.gitlab.com/ee/api/issues.html#single-issue). - -By defining the `api_post_path` method, we allow the [`ApiFabricator`](https://gitlab.com/gitlab-org/gitlab-ee/blob/master/qa/qa/resource/api_fabricator.rb) module to know which path to use to create a new issue in a specific project. - -> This `POST` path can be found in the [public API documentation](https://docs.gitlab.com/ee/api/issues.html#new-issue). - -By defining the `api_post_body` method, we allow the [`ApiFabricator.api_post`](https://gitlab.com/gitlab-org/gitlab-ee/blob/a9177ca1812bac57e2b2fa4560e1d5dd8ffac38b/qa/qa/resource/api_fabricator.rb#L68) method to know which data to send when making the `POST` request. - -> Notice that we pass both `title` and `labels` attributes in the `api_post_body`, where `labels` receives an array of labels, and [`title` is required](https://docs.gitlab.com/ee/api/issues.html#new-issue). - -**Label resource** - -Finally, let's make it possible to create label resources through the API. - -Add the following code right below the [`fabricate!`](https://gitlab.com/gitlab-org/gitlab-ee/blob/a9177ca1812bac57e2b2fa4560e1d5dd8ffac38b/qa/qa/resource/label.rb#L36) method. - -```ruby -def resource_web_url(resource) - super -rescue ResourceURLMissingError - # this particular resource does not expose a web_url property -end - -def api_get_path - raise NotImplementedError, "The Labels API doesn't expose a single-resource endpoint so this method cannot be properly implemented." -end - -def api_post_path - "/projects/#{project}/labels" -end - -def api_post_body - { - name: @title, - color: @color - } -end -``` - -By defining the `resource_web_url(resource)` method, we override the one from the [`ApiFabricator`](https://gitlab.com/gitlab-org/gitlab-ee/blob/master/qa/qa/resource/api_fabricator.rb#L44) module. We do that to avoid failing the test due to this particular resource not exposing a `web_url` property. - -By defining the `api_get_path` method, we **would** allow for the [`ApiFabricator`](https://gitlab.com/gitlab-org/gitlab-ee/blob/master/qa/qa/resource/api_fabricator.rb) module to know which path to use to get a single label, but since there's no path available for that in the publich API, we raise a `NotImplementedError` instead. - -By defining the `api_post_path` method, we allow for the [`ApiFabricator `](https://gitlab.com/gitlab-org/gitlab-ee/blob/master/qa/qa/resource/api_fabricator.rb) module to know which path to use to create a new label in a specific project. - -By defining the `api_post_body` method, we we allow for the [`ApiFabricator.api_post`](https://gitlab.com/gitlab-org/gitlab-ee/blob/a9177ca1812bac57e2b2fa4560e1d5dd8ffac38b/qa/qa/resource/api_fabricator.rb#L68) method to know which data to send when making the `POST` request. - -> Notice that we pass both `name` and `color` attributes in the `api_post_body` since [those are required](https://docs.gitlab.com/ee/api/labels.html#create-a-new-label). - -### 8. Page Objects - -> Page Objects are auto-loaded in the `qa/qa.rb` file and available in all the test files (`*_spec.rb`). - -Page Objects are used in end-to-end tests for maintenance reasons, where page's elements and methods are defined to be reused in any test. - -Take a look at [this document that specifically details the usage of Page Objects](./qa/page/README.md). - -Now, let's go back to our examples. - -... - -#### Adding testability - -TBD. diff --git a/qa/knapsack/gitlab-ce/review-qa-all_master_report.json b/qa/knapsack/gitlab-ce/review-qa-all_master_report.json new file mode 100644 index 00000000000..f147346ba0f --- /dev/null +++ b/qa/knapsack/gitlab-ce/review-qa-all_master_report.json @@ -0,0 +1,42 @@ +{ + "qa/specs/features/browser_ui/1_manage/project/add_project_member_spec.rb": 9.697327613830566, + "qa/specs/features/browser_ui/3_create/merge_request/rebase_merge_request_spec.rb": 46.54227638244629, + "qa/specs/features/browser_ui/1_manage/project/create_project_spec.rb": 10.214765310287476, + "qa/specs/features/browser_ui/1_manage/project/view_project_activity_spec.rb": 7.882027864456177, + "qa/specs/features/api/3_create/repository/files_spec.rb": 5.015859127044678, + "qa/specs/features/browser_ui/2_plan/issue/filter_issue_comments_spec.rb": 12.772682905197144, + "qa/specs/features/browser_ui/3_create/repository/push_protected_branch_spec.rb": 29.76174831390381, + "qa/specs/features/browser_ui/3_create/repository/use_ssh_key_spec.rb": 22.800872802734375, + "qa/specs/features/browser_ui/1_manage/login/register_spec.rb": 22.320587396621704, + "qa/specs/features/api/1_manage/users_spec.rb": 0.6089541912078857, + "qa/specs/features/browser_ui/3_create/repository/clone_spec.rb": 0.9618203639984131, + "qa/specs/features/browser_ui/2_plan/issue/collapse_comments_in_discussions_spec.rb": 13.403101205825806, + "qa/specs/features/browser_ui/non_devops/performance_bar_spec.rb": 8.810423135757446, + "qa/specs/features/browser_ui/3_create/repository/push_over_http_spec.rb": 7.730542182922363, + "qa/specs/features/browser_ui/3_create/repository/add_ssh_key_spec.rb": 16.18057894706726, + "qa/specs/features/browser_ui/4_verify/ci_variable/add_ci_variable_spec.rb": 8.31815505027771, + "qa/specs/features/browser_ui/3_create/repository/push_http_private_token_spec.rb": 9.48607873916626, + "qa/specs/features/browser_ui/2_plan/issue/create_issue_spec.rb": 19.552733182907104, + "qa/specs/features/browser_ui/3_create/wiki/create_edit_clone_push_wiki_spec.rb": 17.273863554000854, + "qa/specs/features/browser_ui/7_configure/auto_devops/create_project_with_auto_devops_spec.rb": 8.281434059143066, + "qa/specs/features/browser_ui/3_create/repository/create_edit_delete_file_via_web_spec.rb": 18.047621726989746, + "qa/specs/features/browser_ui/6_release/deploy_token/add_deploy_token_spec.rb": 7.422840595245361, + "qa/specs/features/browser_ui/1_manage/login/log_in_spec.rb": 3.438166856765747, + "qa/specs/features/browser_ui/4_verify/runner/register_runner_spec.rb": 18.679633855819702, + "qa/specs/features/browser_ui/3_create/merge_request/squash_merge_request_spec.rb": 27.943300485610962, + "qa/specs/features/browser_ui/3_create/repository/add_list_delete_branches_spec.rb": 39.17585229873657, + "qa/specs/features/browser_ui/3_create/repository/add_file_template_spec.rb": 40.09336972236633, + "qa/specs/features/browser_ui/3_create/merge_request/view_merge_request_diff_patch_spec.rb": 3.705310821533203, + "qa/specs/features/browser_ui/3_create/snippet/create_snippet_spec.rb": 5.812374591827393, + "qa/specs/features/browser_ui/6_release/deploy_key/clone_using_deploy_key_spec.rb": 92.46774697303772, + "qa/specs/features/browser_ui/1_manage/project/import_github_repo_spec.rb": 100.28881478309631, + "qa/specs/features/browser_ui/3_create/repository/push_mirroring_over_http_spec.rb": 23.710937023162842, + "qa/specs/features/browser_ui/1_manage/login/login_via_oauth_spec.rb": 20.58603596687317, + "qa/specs/features/browser_ui/3_create/web_ide/add_file_template_spec.rb": 25.460349321365356, + "qa/specs/features/browser_ui/3_create/repository/push_over_http_file_size_spec.rb": 19.459370374679565, + "qa/specs/features/browser_ui/3_create/repository/user_views_commit_diff_patch_spec.rb": 6.731764793395996, + "qa/specs/features/browser_ui/6_release/deploy_key/add_deploy_key_spec.rb": 15.342933893203735, + "qa/specs/features/browser_ui/2_plan/issue/issue_suggestions_spec.rb": 11.280649185180664, + "qa/specs/features/browser_ui/3_create/merge_request/create_merge_request_spec.rb": 57.48992609977722, + "qa/specs/features/browser_ui/3_create/merge_request/merge_merge_request_from_fork_spec.rb": 32.5517954826355 +}
\ No newline at end of file diff --git a/qa/qa/fixtures/auto_devops_rack/Dockerfile b/qa/qa/fixtures/auto_devops_rack/Dockerfile new file mode 100644 index 00000000000..1f59c23ea88 --- /dev/null +++ b/qa/qa/fixtures/auto_devops_rack/Dockerfile @@ -0,0 +1,9 @@ +FROM ruby:2.6.3-alpine +ADD ./ /app/ +WORKDIR /app +ENV RACK_ENV production +ENV PORT 5000 +EXPOSE 5000 + +RUN bundle install +CMD ["bundle","exec", "rackup", "-p", "5000"] diff --git a/qa/qa/page/README.md b/qa/qa/page/README.md deleted file mode 100644 index d0de33892c4..00000000000 --- a/qa/qa/page/README.md +++ /dev/null @@ -1,134 +0,0 @@ -# Page objects in GitLab QA - -In GitLab QA we are using a known pattern, called _Page Objects_. - -This means that we have built an abstraction for all GitLab pages that we use -to drive GitLab QA scenarios. Whenever we do something on a page, like filling -in a form, or clicking a button, we do that only through a page object -associated with this area of GitLab. - -For example, when GitLab QA test harness signs in into GitLab, it needs to fill -in a user login and user password. In order to do that, we have a class, called -`Page::Main::Login` and `sign_in_using_credentials` methods, that is the only -piece of the code, that has knowledge about `user_login` and `user_password` -fields. - -## Why do we need that? - -We need page objects, because we need to reduce duplication and avoid problems -whenever someone changes some selectors in GitLab's source code. - -Imagine that we have a hundred specs in GitLab QA, and we need to sign into -GitLab each time, before we make assertions. Without a page object one would -need to rely on volatile helpers or invoke Capybara methods directly. Imagine -invoking `fill_in :user_login` in every `*_spec.rb` file / test example. - -When someone later changes `t.text_field :login` in the view associated with -this page to `t.text_field :username` it will generate a different field -identifier, what would effectively break all tests. - -Because we are using `Page::Main::Login.act { sign_in_using_credentials }` -everywhere, when we want to sign into GitLab, the page object is the single -source of truth, and we will need to update `fill_in :user_login` -to `fill_in :user_username` only in a one place. - -## What problems did we have in the past? - -We do not run QA tests for every commit, because of performance reasons, and -the time it would take to build packages and test everything. - -That is why when someone changes `t.text_field :login` to -`t.text_field :username` in the _new session_ view we won't know about this -change until our GitLab QA nightly pipeline fails, or until someone triggers -`package-and-qa` action in their merge request. - -Obviously such a change would break all tests. We call this problem a _fragile -tests problem_. - -In order to make GitLab QA more reliable and robust, we had to solve this -problem by introducing coupling between GitLab CE / EE views and GitLab QA. - -## How did we solve fragile tests problem? - -Currently, when you add a new `Page::Base` derived class, you will also need to -define all selectors that your page objects depends on. - -Whenever you push your code to CE / EE repository, `qa:selectors` sanity test -job is going to be run as a part of a CI pipeline. - -This test is going to validate all page objects that we have implemented in -`qa/page` directory. When it fails, you will be notified about missing -or invalid views / selectors definition. - -## How to properly implement a page object? - -We have built a DSL to define coupling between a page object and GitLab views -it is actually implemented by. See an example below. - -```ruby -module Page - module Main - class Login < Page::Base - view 'app/views/devise/passwords/edit.html.haml' do - element :password_field - element :password_confirmation - element :change_password_button - end - - view 'app/views/devise/sessions/_new_base.html.haml' do - element :login_field - element :password_field - element :sign_in_button - end - - # ... - end -end -``` - -The `view` DSL method declares the filename of the view where an -`element` is implemented. - -The `element` DSL method in turn declares an element for which a corresponding -`qa-element-name-dasherized` CSS class need to be added to the view file. - -You can also define a value (String or Regexp) to match to the actual view -code but **this is deprecated** in favor of the above method for two reasons: - -- Consistency: there is only one way to define an element -- Separation of concerns: QA uses dedicated CSS classes instead of reusing code - or classes used by other components (e.g. `js-*` classes etc.) - -```ruby -view 'app/views/my/view.html.haml' do - # Implicitly require `.qa-logout-button` CSS class to be present in the view - element :logout_button - - ## This is deprecated and forbidden by the `QA/ElementWithPattern` RuboCop cop. - # Require `f.submit "Sign in"` to be present in `my/view.html.haml - element :my_button, 'f.submit "Sign in"' # rubocop:disable QA/ElementWithPattern - - ## This is deprecated and forbidden by the `QA/ElementWithPattern` RuboCop cop. - # Match every line in `my/view.html.haml` against - # `/link_to .* "My Profile"/` regexp. - element :profile_link, /link_to .* "My Profile"/ # rubocop:disable QA/ElementWithPattern -end -``` - -## Running the test locally - -During development, you can run the `qa:selectors` test by running - -```shell -bin/qa Test::Sanity::Selectors -``` - -from within the `qa` directory. - -## Where to ask for help? - -If you need more information, ask for help on `#quality` channel on Slack -(internal, GitLab Team only). - -If you are not a Team Member, and you still need help to contribute, please -open an issue in GitLab CE issue tracker with the `~QA` label. diff --git a/qa/qa/page/project/branches/show.rb b/qa/qa/page/project/branches/show.rb index 762eacdab15..480fc7d78cb 100644 --- a/qa/qa/page/project/branches/show.rb +++ b/qa/qa/page/project/branches/show.rb @@ -28,9 +28,11 @@ module QA finished_loading? end - def has_no_branch?(branch_name) - within_element(:all_branches) do - has_no_element?(:branch_name, text: branch_name, wait: Support::Waiter::DEFAULT_MAX_WAIT_TIME) + def has_no_branch?(branch_name, reload: false) + wait(reload: reload) do + within_element(:all_branches) do + has_no_element?(:branch_name, text: branch_name) + end end end diff --git a/qa/qa/resource/README.md b/qa/qa/resource/README.md deleted file mode 100644 index 2c8859b6599..00000000000 --- a/qa/qa/resource/README.md +++ /dev/null @@ -1,392 +0,0 @@ -# Resource class in GitLab QA - -Resources are primarily created using Browser UI steps, but can also -be created via the API or the CLI. - -## How to properly implement a resource class? - -All resource classes should inherit from [`Resource::Base`](./base.rb). - -There is only one mandatory method to implement to define a resource class. -This is the `#fabricate!` method, which is used to build the resource via the -browser UI. Note that you should only use [Page objects](../page/README.md) to -interact with a Web page in this method. - -Here is an imaginary example: - -```ruby -module QA - module Resource - class Shirt < Base - attr_accessor :name - - def fabricate! - Page::Dashboard::Index.perform do |dashboard_index| - dashboard_index.go_to_new_shirt - end - - Page::Shirt::New.perform do |shirt_new| - shirt_new.set_name(name) - shirt_new.create_shirt! - end - end - end - end -end -``` - -### Define API implementation - -A resource class may also implement the three following methods to be able to -create the resource via the public GitLab API: - -- `#api_get_path`: The `GET` path to fetch an existing resource. -- `#api_post_path`: The `POST` path to create a new resource. -- `#api_post_body`: The `POST` body (as a Ruby hash) to create a new resource. - -Let's take the `Shirt` resource class, and add these three API methods: - -```ruby -module QA - module Resource - class Shirt < Base - attr_accessor :name - - def fabricate! - # ... same as before - end - - def api_get_path - "/shirt/#{name}" - end - - def api_post_path - "/shirts" - end - - def api_post_body - { - name: name - } - end - end - end -end -``` - -The [`Project` resource](./project.rb) is a good real example of Browser -UI and API implementations. - -#### Resource attributes - -A resource may need another resource to exist first. For instance, a project -needs a group to be created in. - -To define a resource attribute, you can use the `attribute` method with a -block using the other resource class to fabricate the resource. - -That will allow access to the other resource from your resource object's -methods. You would usually use it in `#fabricate!`, `#api_get_path`, -`#api_post_path`, `#api_post_body`. - -Let's take the `Shirt` resource class, and add a `project` attribute to it: - -```ruby -module QA - module Resource - class Shirt < Base - attr_accessor :name - - attribute :project do - Project.fabricate! do |resource| - resource.name = 'project-to-create-a-shirt' - end - end - - def fabricate! - project.visit! - - Page::Project::Show.perform do |project_show| - project_show.go_to_new_shirt - end - - Page::Shirt::New.perform do |shirt_new| - shirt_new.set_name(name) - shirt_new.create_shirt! - end - end - - def api_get_path - "/project/#{project.path}/shirt/#{name}" - end - - def api_post_path - "/project/#{project.path}/shirts" - end - - def api_post_body - { - name: name - } - end - end - end -end -``` - -**Note that all the attributes are lazily constructed. This means if you want -a specific attribute to be fabricated first, you'll need to call the -attribute method first even if you're not using it.** - -#### Product data attributes - -Once created, you may want to populate a resource with attributes that can be -found in the Web page, or in the API response. -For instance, once you create a project, you may want to store its repository -SSH URL as an attribute. - -Again we could use the `attribute` method with a block, using a page object -to retrieve the data on the page. - -Let's take the `Shirt` resource class, and define a `:brand` attribute: - -```ruby -module QA - module Resource - class Shirt < Base - attr_accessor :name - - attribute :project do - Project.fabricate! do |resource| - resource.name = 'project-to-create-a-shirt' - end - end - - # Attribute populated from the Browser UI (using the block) - attribute :brand do - Page::Shirt::Show.perform do |shirt_show| - shirt_show.fetch_brand_from_page - end - end - - # ... same as before - end - end -end -``` - -**Note again that all the attributes are lazily constructed. This means if -you call `shirt.brand` after moving to the other page, it'll not properly -retrieve the data because we're no longer on the expected page.** - -Consider this: - -```ruby -shirt = - QA::Resource::Shirt.fabricate! do |resource| - resource.name = "GitLab QA" - end - -shirt.project.visit! - -shirt.brand # => FAIL! -``` - -The above example will fail because now we're on the project page, trying to -construct the brand data from the shirt page, however we moved to the project -page already. There are two ways to solve this, one is that we could try to -retrieve the brand before visiting the project again: - -```ruby -shirt = - QA::Resource::Shirt.fabricate! do |resource| - resource.name = "GitLab QA" - end - -shirt.brand # => OK! - -shirt.project.visit! - -shirt.brand # => OK! -``` - -The attribute will be stored in the instance therefore all the following calls -will be fine, using the data previously constructed. If we think that this -might be too brittle, we could eagerly construct the data right before -ending fabrication: - -```ruby -module QA - module Resource - class Shirt < Base - # ... same as before - - def fabricate! - project.visit! - - Page::Project::Show.perform do |project_show| - project_show.go_to_new_shirt - end - - Page::Shirt::New.perform do |shirt_new| - shirt_new.set_name(name) - shirt_new.create_shirt! - end - - populate(:brand) # Eagerly construct the data - end - end - end -end -``` - -The `populate` method will iterate through its arguments and call each -attribute respectively. Here `populate(:brand)` has the same effect as -just `brand`. Using the populate method makes the intention clearer. - -With this, it will make sure we construct the data right after we create the -shirt. The drawback is that this will always construct the data when the -resource is fabricated even if we don't need to use the data. - -Alternatively, we could just make sure we're on the right page before -constructing the brand data: - -```ruby -module QA - module Resource - class Shirt < Base - attr_accessor :name - - attribute :project do - Project.fabricate! do |resource| - resource.name = 'project-to-create-a-shirt' - end - end - - # Attribute populated from the Browser UI (using the block) - attribute :brand do - back_url = current_url - visit! - - Page::Shirt::Show.perform do |shirt_show| - shirt_show.fetch_brand_from_page - end - - visit(back_url) - end - - # ... same as before - end - end -end -``` - -This will make sure it's on the shirt page before constructing brand, and -move back to the previous page to avoid breaking the state. - -#### Define an attribute based on an API response - -Sometimes, you want to define a resource attribute based on the API response -from its `GET` or `POST` request. For instance, if the creation of a shirt via -the API returns - -```ruby -{ - brand: 'a-brand-new-brand', - style: 't-shirt', - materials: [[:cotton, 80], [:polyamide, 20]] -} -``` - -you may want to store `style` as-is in the resource, and fetch the first value -of the first `materials` item in a `main_fabric` attribute. - -Let's take the `Shirt` resource class, and define a `:style` and a -`:main_fabric` attributes: - -```ruby -module QA - module Resource - class Shirt < Base - # ... same as before - - # @style from the instance if present, - # or fetched from the API response if present, - # or a QA::Resource::Base::NoValueError is raised otherwise - attribute :style - - # If @main_fabric is not present, - # and if the API does not contain this field, this block will be - # used to construct the value based on the API response, and - # store the result in @main_fabric - attribute :main_fabric do - api_response.&dig(:materials, 0, 0) - end - - # ... same as before - end - end -end -``` - -**Notes on attributes precedence:** - -- resource instance variables have the highest precedence -- attributes from the API response take precedence over attributes from the - block (usually from Browser UI) -- attributes without a value will raise a `QA::Resource::Base::NoValueError` error - -## Creating resources in your tests - -To create a resource in your tests, you can call the `.fabricate!` method on -the resource class. -Note that if the resource class supports API fabrication, this will use this -fabrication by default. - -Here is an example that will use the API fabrication method under the hood -since it's supported by the `Shirt` resource class: - -```ruby -my_shirt = Resource::Shirt.fabricate! do |shirt| - shirt.name = 'my-shirt' -end - -expect(page).to have_text(my_shirt.name) # => "my-shirt" from the resource's instance variable -expect(page).to have_text(my_shirt.brand) # => "a-brand-new-brand" from the API response -expect(page).to have_text(my_shirt.style) # => "t-shirt" from the API response -expect(page).to have_text(my_shirt.main_fabric) # => "cotton" from the API response via the block -``` - -If you explicitly want to use the Browser UI fabrication method, you can call -the `.fabricate_via_browser_ui!` method instead: - -```ruby -my_shirt = Resource::Shirt.fabricate_via_browser_ui! do |shirt| - shirt.name = 'my-shirt' -end - -expect(page).to have_text(my_shirt.name) # => "my-shirt" from the resource's instance variable -expect(page).to have_text(my_shirt.brand) # => the brand name fetched from the `Page::Shirt::Show` page via the block -expect(page).to have_text(my_shirt.style) # => QA::Resource::Base::NoValueError will be raised because no API response nor a block is provided -expect(page).to have_text(my_shirt.main_fabric) # => QA::Resource::Base::NoValueError will be raised because no API response and the block didn't provide a value (because it's also based on the API response) -``` - -You can also explicitly use the API fabrication method, by calling the -`.fabricate_via_api!` method: - -```ruby -my_shirt = Resource::Shirt.fabricate_via_api! do |shirt| - shirt.name = 'my-shirt' -end -``` - -In this case, the result will be similar to calling -`Resource::Shirt.fabricate!`. - -## Where to ask for help? - -If you need more information, ask for help on `#quality` channel on Slack -(internal, GitLab Team only). - -If you are not a Team Member, and you still need help to contribute, please -open an issue in GitLab CE issue tracker with the `~QA` label. diff --git a/qa/qa/resource/base.rb b/qa/qa/resource/base.rb index 523d92c7ef3..283fc6cdbcb 100644 --- a/qa/qa/resource/base.rb +++ b/qa/qa/resource/base.rb @@ -15,6 +15,38 @@ module QA def_delegators :evaluator, :attribute + def self.fabricate!(*args, &prepare_block) + fabricate_via_api!(*args, &prepare_block) + rescue NotImplementedError + fabricate_via_browser_ui!(*args, &prepare_block) + end + + def self.fabricate_via_browser_ui!(*args, &prepare_block) + options = args.extract_options! + resource = options.fetch(:resource) { new } + parents = options.fetch(:parents) { [] } + + do_fabricate!(resource: resource, prepare_block: prepare_block, parents: parents) do + log_fabrication(:browser_ui, resource, parents, args) { resource.fabricate!(*args) } + + current_url + end + end + + def self.fabricate_via_api!(*args, &prepare_block) + options = args.extract_options! + resource = options.fetch(:resource) { new } + parents = options.fetch(:parents) { [] } + + raise NotImplementedError unless resource.api_support? + + resource.eager_load_api_client! + + do_fabricate!(resource: resource, prepare_block: prepare_block, parents: parents) do + log_fabrication(:api, resource, parents, args) { resource.fabricate_via_api! } + end + end + def fabricate!(*_args) raise NotImplementedError end @@ -55,38 +87,6 @@ module QA QA::Runtime::Logger.info "<#{self.class}> Attribute #{name.inspect} has both API response `#{api_value}` and a block. API response will be picked. Block will be ignored." end - def self.fabricate!(*args, &prepare_block) - fabricate_via_api!(*args, &prepare_block) - rescue NotImplementedError - fabricate_via_browser_ui!(*args, &prepare_block) - end - - def self.fabricate_via_browser_ui!(*args, &prepare_block) - options = args.extract_options! - resource = options.fetch(:resource) { new } - parents = options.fetch(:parents) { [] } - - do_fabricate!(resource: resource, prepare_block: prepare_block, parents: parents) do - log_fabrication(:browser_ui, resource, parents, args) { resource.fabricate!(*args) } - - current_url - end - end - - def self.fabricate_via_api!(*args, &prepare_block) - options = args.extract_options! - resource = options.fetch(:resource) { new } - parents = options.fetch(:parents) { [] } - - raise NotImplementedError unless resource.api_support? - - resource.eager_load_api_client! - - do_fabricate!(resource: resource, prepare_block: prepare_block, parents: parents) do - log_fabrication(:api, resource, parents, args) { resource.fabricate_via_api! } - end - end - def self.do_fabricate!(resource:, prepare_block:, parents: []) prepare_block.call(resource) if prepare_block diff --git a/qa/qa/resource/file.rb b/qa/qa/resource/file.rb index 57e82ac19ad..ca74654bf90 100644 --- a/qa/qa/resource/file.rb +++ b/qa/qa/resource/file.rb @@ -3,9 +3,12 @@ module QA module Resource class File < Base - attr_accessor :name, + attr_accessor :author_email, + :author_name, + :branch, :content, - :commit_message + :commit_message, + :name attribute :project do Project.fabricate! do |resource| @@ -31,6 +34,30 @@ module QA page.commit_changes end end + + def resource_web_url(resource) + super + rescue ResourceURLMissingError + # this particular resource does not expose a web_url property + end + + def api_get_path + "/projects/#{CGI.escape(project.path_with_namespace)}/repository/files/#{CGI.escape(@name)}" + end + + def api_post_path + api_get_path + end + + def api_post_body + { + branch: @branch || "master", + author_email: @author_email || Runtime::User.default_email, + author_name: @author_name || Runtime::User.username, + content: content, + commit_message: commit_message + } + end end end end diff --git a/qa/qa/runtime/api/client.rb b/qa/qa/runtime/api/client.rb index 58de01705d7..40a3bc85195 100644 --- a/qa/qa/runtime/api/client.rb +++ b/qa/qa/runtime/api/client.rb @@ -25,15 +25,12 @@ module QA private def create_personal_access_token - if @is_new_session - Runtime::Browser.visit(@address, Page::Main::Login) { do_create_personal_access_token } - else - do_create_personal_access_token - end + Runtime::Browser.visit(@address, Page::Main::Login) if @is_new_session + do_create_personal_access_token end def do_create_personal_access_token - Page::Main::Login.act { sign_in_using_credentials } + Page::Main::Login.perform(&:sign_in_using_credentials) Resource::PersonalAccessToken.fabricate!.access_token end end diff --git a/qa/qa/runtime/env.rb b/qa/qa/runtime/env.rb index 03cae3c1fe6..82510dfa03c 100644 --- a/qa/qa/runtime/env.rb +++ b/qa/qa/runtime/env.rb @@ -136,6 +136,10 @@ module QA ENV['GITLAB_QA_PASSWORD_2'] end + def knapsack? + !!(ENV['KNAPSACK_GENERATE_REPORT'] || ENV['KNAPSACK_REPORT_PATH'] || ENV['KNAPSACK_TEST_FILE_PATTERN']) + end + def ldap_username @ldap_username ||= ENV['GITLAB_LDAP_USERNAME'] end diff --git a/qa/qa/specs/features/browser_ui/2_plan/issue/issue_suggestions_spec.rb b/qa/qa/specs/features/browser_ui/2_plan/issue/issue_suggestions_spec.rb index f97b0e56ca2..530fc684437 100644 --- a/qa/qa/specs/features/browser_ui/2_plan/issue/issue_suggestions_spec.rb +++ b/qa/qa/specs/features/browser_ui/2_plan/issue/issue_suggestions_spec.rb @@ -1,8 +1,7 @@ # frozen_string_literal: true module QA - # Failure issue: https://gitlab.com/gitlab-org/quality/staging/issues/53 - context 'Plan', :quarantine do + context 'Plan' do describe 'issue suggestions' do let(:issue_title) { 'Issue Lists are awesome' } @@ -10,12 +9,12 @@ module QA Runtime::Browser.visit(:gitlab, Page::Main::Login) Page::Main::Login.perform(&:sign_in_using_credentials) - project = Resource::Project.fabricate! do |resource| + project = Resource::Project.fabricate_via_api! do |resource| resource.name = 'project-for-issue-suggestions' resource.description = 'project for issue suggestions' end - Resource::Issue.fabricate! do |issue| + Resource::Issue.fabricate_via_browser_ui! do |issue| issue.title = issue_title issue.project = project end diff --git a/qa/qa/specs/features/browser_ui/3_create/repository/add_list_delete_branches_spec.rb b/qa/qa/specs/features/browser_ui/3_create/repository/add_list_delete_branches_spec.rb index cf6c24fa873..37a784248d4 100644 --- a/qa/qa/specs/features/browser_ui/3_create/repository/add_list_delete_branches_spec.rb +++ b/qa/qa/specs/features/browser_ui/3_create/repository/add_list_delete_branches_spec.rb @@ -84,7 +84,7 @@ module QA page.refresh Page::Project::Branches::Show.perform do |branches_view| - expect(branches_view).to have_no_branch(second_branch) + expect(branches_view).to have_no_branch(second_branch, reload: true) end end end diff --git a/qa/qa/specs/features/browser_ui/3_create/repository/create_edit_delete_file_via_web_spec.rb b/qa/qa/specs/features/browser_ui/3_create/repository/create_edit_delete_file_via_web_spec.rb index 46346d1b984..d345fbfe995 100644 --- a/qa/qa/specs/features/browser_ui/3_create/repository/create_edit_delete_file_via_web_spec.rb +++ b/qa/qa/specs/features/browser_ui/3_create/repository/create_edit_delete_file_via_web_spec.rb @@ -12,7 +12,7 @@ module QA file_content = 'QA Test - File content' commit_message_for_create = 'QA Test - Create new file' - Resource::File.fabricate! do |file| + Resource::File.fabricate_via_browser_ui! do |file| file.name = file_name file.content = file_content file.commit_message = commit_message_for_create diff --git a/qa/qa/specs/features/browser_ui/3_create/repository/push_over_http_file_size_spec.rb b/qa/qa/specs/features/browser_ui/3_create/repository/push_over_http_file_size_spec.rb index a544efb35ee..5bfafdfa041 100644 --- a/qa/qa/specs/features/browser_ui/3_create/repository/push_over_http_file_size_spec.rb +++ b/qa/qa/specs/features/browser_ui/3_create/repository/push_over_http_file_size_spec.rb @@ -26,6 +26,8 @@ module QA Page::Main::Login.perform(&:sign_in_using_credentials) set_file_size_limit('') + + Page::Main::Menu.perform(&:sign_out) end it 'push successful when the file size is under the limit' do diff --git a/qa/qa/specs/features/browser_ui/3_create/repository/user_views_commit_diff_patch_spec.rb b/qa/qa/specs/features/browser_ui/3_create/repository/user_views_commit_diff_patch_spec.rb index b7400cdca97..680c5e21fa4 100644 --- a/qa/qa/specs/features/browser_ui/3_create/repository/user_views_commit_diff_patch_spec.rb +++ b/qa/qa/specs/features/browser_ui/3_create/repository/user_views_commit_diff_patch_spec.rb @@ -3,13 +3,16 @@ module QA context 'Create' do # failure reported: https://gitlab.com/gitlab-org/quality/nightly/issues/42 - # also failing in staging until the fix is picked into the next release: - # https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/24533 describe 'Commit data', :quarantine do before(:context) do Runtime::Browser.visit(:gitlab, Page::Main::Login) Page::Main::Login.perform(&:sign_in_using_credentials) + # Get the user's details to confirm they're included in the email patch + @user = Resource::User.fabricate_via_api! do |user| + user.username = Runtime::User.username + end + project_push = Resource::Repository::ProjectPush.fabricate! do |push| push.file_name = 'README.md' push.file_content = '# This is a test project' @@ -21,12 +24,13 @@ module QA # add second file to repo to enable diff from initial commit @commit_message = 'Add second file' - Page::Project::Show.perform(&:create_new_file!) - Page::File::Form.perform do |f| - f.add_name('second') - f.add_content('second file content') - f.add_commit_message(@commit_message) - f.commit_changes + Resource::File.fabricate_via_api! do |file| + file.project = @project + file.name = 'second' + file.content = 'second file content' + file.commit_message = @commit_message + file.author_name = @user.name + file.author_email = @user.public_email end end @@ -42,15 +46,11 @@ module QA end it 'user views raw email patch' do - user = Resource::User.fabricate_via_api! do |user| - user.username = Runtime::User.username - end - view_commit Page::Project::Commit::Show.perform(&:select_email_patches) - expect(page).to have_content("From: #{user.name} <#{user.public_email}>") + expect(page).to have_content("From: #{@user.name} <#{@user.public_email}>") expect(page).to have_content('Subject: [PATCH] Add second file') expect(page).to have_content('diff --git a/second b/second') end diff --git a/qa/qa/specs/features/browser_ui/7_configure/auto_devops/create_project_with_auto_devops_spec.rb b/qa/qa/specs/features/browser_ui/7_configure/auto_devops/create_project_with_auto_devops_spec.rb index 9201a05337f..86ba5e819ba 100644 --- a/qa/qa/specs/features/browser_ui/7_configure/auto_devops/create_project_with_auto_devops_spec.rb +++ b/qa/qa/specs/features/browser_ui/7_configure/auto_devops/create_project_with_auto_devops_spec.rb @@ -9,8 +9,7 @@ module QA Page::Main::Login.perform(&:sign_in_using_credentials) end - # Transient failure issue: https://gitlab.com/gitlab-org/quality/nightly/issues/68 - describe 'Auto DevOps support', :orchestrated, :kubernetes, :quarantine do + describe 'Auto DevOps support', :orchestrated, :kubernetes do context 'when rbac is enabled' do before(:all) do @cluster = Service::KubernetesCluster.new.create! diff --git a/qa/qa/specs/runner.rb b/qa/qa/specs/runner.rb index 306913dafa6..f1cb9378de8 100644 --- a/qa/qa/specs/runner.rb +++ b/qa/qa/specs/runner.rb @@ -1,6 +1,8 @@ # frozen_string_literal: true require 'rspec/core' +require 'rspec/expectations' +require 'knapsack' module QA module Specs @@ -32,10 +34,25 @@ module QA end args.push(options) - args.push(DEFAULT_TEST_PATH_ARGS) unless options.any? { |opt| opt =~ %r{/features/} } Runtime::Browser.configure! + if Runtime::Env.knapsack? + allocator = Knapsack::AllocatorBuilder.new(Knapsack::Adapters::RSpecAdapter).allocator + + QA::Runtime::Logger.info '' + QA::Runtime::Logger.info 'Report specs:' + QA::Runtime::Logger.info allocator.report_node_tests.join(', ') + QA::Runtime::Logger.info '' + QA::Runtime::Logger.info 'Leftover specs:' + QA::Runtime::Logger.info allocator.leftover_node_tests.join(', ') + QA::Runtime::Logger.info '' + + args.push(['--', allocator.node_tests]) + else + args.push(DEFAULT_TEST_PATH_ARGS) unless options.any? { |opt| opt =~ %r{/features/} } + end + RSpec::Core::Runner.run(args.flatten, $stderr, $stdout).tap do |status| abort if status.nonzero? end diff --git a/qa/spec/runtime/env_spec.rb b/qa/spec/runtime/env_spec.rb index 04085efe2ce..2560695ef2e 100644 --- a/qa/spec/runtime/env_spec.rb +++ b/qa/spec/runtime/env_spec.rb @@ -168,6 +168,30 @@ describe QA::Runtime::Env do end end + describe '.knapsack?' do + it 'returns true if KNAPSACK_GENERATE_REPORT is defined' do + stub_env('KNAPSACK_GENERATE_REPORT', 'true') + + expect(described_class.knapsack?).to be_truthy + end + + it 'returns true if KNAPSACK_REPORT_PATH is defined' do + stub_env('KNAPSACK_REPORT_PATH', '/a/path') + + expect(described_class.knapsack?).to be_truthy + end + + it 'returns true if KNAPSACK_TEST_FILE_PATTERN is defined' do + stub_env('KNAPSACK_TEST_FILE_PATTERN', '/a/**/pattern') + + expect(described_class.knapsack?).to be_truthy + end + + it 'returns false if neither KNAPSACK_GENERATE_REPORT nor KNAPSACK_REPORT_PATH nor KNAPSACK_TEST_FILE_PATTERN are defined' do + expect(described_class.knapsack?).to be_falsey + end + end + describe '.require_github_access_token!' do it 'raises ArgumentError if GITHUB_ACCESS_TOKEN is not defined' do stub_env('GITHUB_ACCESS_TOKEN', nil) diff --git a/qa/spec/spec_helper.rb b/qa/spec/spec_helper.rb index a368ffba711..f25dbf3a8ab 100644 --- a/qa/spec/spec_helper.rb +++ b/qa/spec/spec_helper.rb @@ -3,6 +3,11 @@ require_relative '../qa' require 'rspec/retry' +if ENV['CI'] && QA::Runtime::Env.knapsack? && !ENV['NO_KNAPSACK'] + require 'knapsack' + Knapsack::Adapters::RSpecAdapter.bind +end + %w[helpers shared_examples].each do |d| Dir[::File.join(__dir__, d, '**', '*.rb')].each { |f| require f } end |