diff options
Diffstat (limited to 'doc/development/testing_guide/end_to_end')
7 files changed, 28 insertions, 27 deletions
diff --git a/doc/development/testing_guide/end_to_end/best_practices.md b/doc/development/testing_guide/end_to_end/best_practices.md index 57cfcf34726..7df3cd614c7 100644 --- a/doc/development/testing_guide/end_to_end/best_practices.md +++ b/doc/development/testing_guide/end_to_end/best_practices.md @@ -91,7 +91,7 @@ point of failure and so the screenshot would not be captured at the right moment All tests expect to be able to log in at the start of the test. -For an example see: <https://gitlab.com/gitlab-org/gitlab/issues/34736> +For an example see: <https://gitlab.com/gitlab-org/gitlab/-/issues/34736> Ideally, any actions performed in an `after(:context)` (or [`before(:context)`](#limit-the-use-of-the-ui-in-beforecontext-and-after-hooks)) block would be performed via the API. But if it's necessary to do so via the UI (e.g., if API functionality doesn't exist), make sure to log out at the end of the block. diff --git a/doc/development/testing_guide/end_to_end/feature_flags.md b/doc/development/testing_guide/end_to_end/feature_flags.md index 3bd07f17207..ada20cc9dad 100644 --- a/doc/development/testing_guide/end_to_end/feature_flags.md +++ b/doc/development/testing_guide/end_to_end/feature_flags.md @@ -26,4 +26,4 @@ end It's also possible to run an entire scenario with a feature flag enabled, without having to edit existing tests or write new ones. -Please see the [QA readme](https://gitlab.com/gitlab-org/gitlab/tree/master/qa#running-tests-with-a-feature-flag-enabled) for details. +Please see the [QA README](https://gitlab.com/gitlab-org/gitlab/tree/master/qa#running-tests-with-a-feature-flag-enabled) for details. diff --git a/doc/development/testing_guide/end_to_end/index.md b/doc/development/testing_guide/end_to_end/index.md index e6a683e9148..ac051b827d2 100644 --- a/doc/development/testing_guide/end_to_end/index.md +++ b/doc/development/testing_guide/end_to_end/index.md @@ -65,18 +65,18 @@ subgraph "gitlab-qa-mirror pipeline" end ``` -1. Developer triggers a manual action, that can be found in CE / EE merge +1. Developer triggers a manual action, that can be found in GitLab merge requests. This starts a chain of pipelines in multiple projects. 1. The script being executed triggers a pipeline in - [Omnibus GitLab Mirror](https://gitlab.com/gitlab-org/omnibus-gitlab-mirror) + [Omnibus GitLab Mirror](https://gitlab.com/gitlab-org/build/omnibus-gitlab-mirror) and waits for the resulting status. We call this a _status attribution_. -1. GitLab packages are being built in the [Omnibus GitLab](https://gitlab.com/gitlab-org/omnibus-gitlab) +1. GitLab packages are being built in the [Omnibus GitLab Mirror](https://gitlab.com/gitlab-org/build/omnibus-gitlab-mirror) pipeline. Packages are then pushed to its Container Registry. 1. When packages are ready, and available in the registry, a final step in the - [Omnibus GitLab](https://gitlab.com/gitlab-org/omnibus-gitlab) pipeline, triggers a new + [Omnibus GitLab Mirror](https://gitlab.com/gitlab-org/build/omnibus-gitlab-mirror) pipeline, triggers a new GitLab QA pipeline (those with access can view them at `https://gitlab.com/gitlab-org/gitlab-qa-mirror/pipelines`). It also waits for a resulting status. 1. GitLab QA pulls images from the registry, spins-up containers and runs tests @@ -84,9 +84,9 @@ subgraph "gitlab-qa-mirror pipeline" tool. 1. The result of the GitLab QA pipeline is being - propagated upstream, through Omnibus, back to the CE / EE merge request. + propagated upstream, through Omnibus, back to the GitLab merge request. -Please note, we plan to [add more specific information](https://gitlab.com/gitlab-org/quality/team-tasks/issues/156) +Please note, we plan to [add more specific information](https://gitlab.com/gitlab-org/quality/team-tasks/-/issues/156) about the tests included in each job/scenario that runs in `gitlab-qa-mirror`. #### With Pipeline for Merged Results @@ -132,7 +132,7 @@ as well as these: | `QA_RSPEC_TAGS` | The RSpec tags to add (no default) | For now [manual jobs with custom variables will not use the same variable -when retried](https://gitlab.com/gitlab-org/gitlab/issues/31367), so if you want to run the same test(s) multiple times, +when retried](https://gitlab.com/gitlab-org/gitlab/-/issues/31367), so if you want to run the same test(s) multiple times, specify the same variables in each `custom-parallel` job (up to as many of the 10 available jobs that you want to run). @@ -155,7 +155,7 @@ See [Review Apps](../review_apps.md) for more details about Review Apps. If you are not [testing code in a merge request](#testing-code-in-merge-requests), there are two main options for running the tests. If you simply want to run -the existing tests against a live GitLab instance or against a pre-built docker image +the existing tests against a live GitLab instance or against a pre-built Docker image you can use the [GitLab QA orchestrator](https://gitlab.com/gitlab-org/gitlab-qa/tree/master/README.md). See also [examples of the test scenarios you can run via the orchestrator](https://gitlab.com/gitlab-org/gitlab-qa/blob/master/docs/what_tests_can_be_run.md#examples). @@ -191,5 +191,5 @@ Continued reading: You can ask question in the `#quality` channel on Slack (GitLab internal) or you can find an issue you would like to work on in -[the `gitlab` issue tracker](https://gitlab.com/gitlab-org/gitlab/issues?label_name%5B%5D=QA&label_name%5B%5D=test), or -[the `gitlab-qa` issue tracker](https://gitlab.com/gitlab-org/gitlab-qa/issues?label_name%5B%5D=new+scenario). +[the `gitlab` issue tracker](https://gitlab.com/gitlab-org/gitlab/-/issues?label_name%5B%5D=QA&label_name%5B%5D=test), or +[the `gitlab-qa` issue tracker](https://gitlab.com/gitlab-org/gitlab-qa/-/issues?label_name%5B%5D=new+scenario). diff --git a/doc/development/testing_guide/end_to_end/page_objects.md b/doc/development/testing_guide/end_to_end/page_objects.md index 9d4fa5316c4..d43d88779c7 100644 --- a/doc/development/testing_guide/end_to_end/page_objects.md +++ b/doc/development/testing_guide/end_to_end/page_objects.md @@ -89,7 +89,7 @@ end ### Defining Elements -The `view` DSL method will correspond to the rails View, partial, or vue component that renders the elements. +The `view` DSL method will correspond to the rails View, partial, or Vue component that renders the elements. The `element` DSL method in turn declares an element for which a corresponding `data-qa-selector=element_name_snaked` data attribute will need to be added to the view file. @@ -134,7 +134,7 @@ view 'app/views/my/view.html.haml' do end ``` -To add these elements to the view, you must change the rails View, partial, or vue component by adding a `data-qa-selector` attribute +To add these elements to the view, you must change the rails View, partial, or Vue component by adding a `data-qa-selector` attribute for each element defined. In our case, `data-qa-selector="login_field"`, `data-qa-selector="password_field"` and `data-qa-selector="sign_in_button"` @@ -149,7 +149,7 @@ In our case, `data-qa-selector="login_field"`, `data-qa-selector="password_field Things to note: -- The name of the element and the qa_selector must match and be snake_cased +- The name of the element and the `qa_selector` must match and be snake_cased - If the element appears on the page unconditionally, add `required: true` to the element. See [Dynamic element validation](dynamic_element_validation.md) - You may see `.qa-selector` classes in existing Page Objects. We should prefer the [`data-qa-selector`](#data-qa-selector-vs-qa-selector) @@ -255,7 +255,7 @@ These steps ensure the sanity selectors check will detect problems properly. For example, `qa/qa/ee/page/merge_request/show.rb` adds EE-specific methods to `qa/qa/page/merge_request/show.rb` (with `QA::Page::MergeRequest::Show.prepend_if_ee('QA::EE::Page::MergeRequest::Show')`) and following is how it's implemented -(only showing the relevant part and refering to the 4 steps described above with inline comments): +(only showing the relevant part and referring to the 4 steps described above with inline comments): ```ruby module QA diff --git a/doc/development/testing_guide/end_to_end/rspec_metadata_tests.md b/doc/development/testing_guide/end_to_end/rspec_metadata_tests.md index b7c93d205a3..b1a8a14163c 100644 --- a/doc/development/testing_guide/end_to_end/rspec_metadata_tests.md +++ b/doc/development/testing_guide/end_to_end/rspec_metadata_tests.md @@ -9,10 +9,11 @@ This is a partial list of the [RSpec metadata](https://relishapp.com/rspec/rspec |-----|-------------| | `:elasticsearch` | The test requires an Elasticsearch service. It is used by the [instance-level scenario](https://gitlab.com/gitlab-org/gitlab-qa#definitions) [`Test::Integration::Elasticsearch`](https://gitlab.com/gitlab-org/gitlab/-/blob/72b62b51bdf513e2936301cb6c7c91ec27c35b4d/qa/qa/ee/scenario/test/integration/elasticsearch.rb) to include only tests that require Elasticsearch. | | `:kubernetes` | The test includes a GitLab instance that is configured to be run behind an SSH tunnel, allowing a TLS-accessible GitLab. This test will also include provisioning of at least one Kubernetes cluster to test against. *This tag is often be paired with `:orchestrated`.* | -| `:orchestrated` | The GitLab instance under test may be [configured by `gitlab-qa`](https://gitlab.com/gitlab-org/gitlab-qa/-/blob/master/docs/what_tests_can_be_run.md#orchestrated-tests) to be different to the default GitLab configuration, or `gitlab-qa` may launch additional services in separate docker containers, or both. Tests tagged with `:orchestrated` are excluded when testing environments where we can't dynamically modify GitLab's configuration (for example, Staging). | +| `:orchestrated` | The GitLab instance under test may be [configured by `gitlab-qa`](https://gitlab.com/gitlab-org/gitlab-qa/-/blob/master/docs/what_tests_can_be_run.md#orchestrated-tests) to be different to the default GitLab configuration, or `gitlab-qa` may launch additional services in separate Docker containers, or both. Tests tagged with `:orchestrated` are excluded when testing environments where we can't dynamically modify GitLab's configuration (for example, Staging). | | `:quarantine` | The test has been [quarantined](https://about.gitlab.com/handbook/engineering/quality/guidelines/debugging-qa-test-failures/#quarantining-tests), will run in a separate job that only includes quarantined tests, and is allowed to fail. The test will be skipped in its regular job so that if it fails it will not hold up the pipeline. | | `:reliable` | The test has been [promoted to a reliable test](https://about.gitlab.com/handbook/engineering/quality/guidelines/reliable-tests/#promoting-an-existing-test-to-reliable) meaning it passes consistently in all pipelines, including merge requests. | | `:requires_admin` | The test requires an admin account. Tests with the tag are excluded when run against Canary and Production environments. | | `:runner` | The test depends on and will set up a GitLab Runner instance, typically to run a pipeline. | | `:gitaly_ha` | The test will run against a GitLab instance where repositories are stored on redundant Gitaly nodes behind a Praefect node. All nodes are [separate containers](../../../administration/gitaly/praefect.md#requirements-for-configuring-a-gitaly-cluster). Tests that use this tag have a longer setup time since there are three additional containers that need to be started. | | `:skip_live_env` | The test will be excluded when run against live deployed environments such as Staging, Canary, and Production. | +| `:jira` | The test requires a Jira Server. [GitLab-QA](https://gitlab.com/gitlab-org/gitlab-qa) will provision the Jira Server in a Docker container when the `Test::Integration::Jira` test scenario is run. diff --git a/doc/development/testing_guide/end_to_end/running_tests_that_require_special_setup.md b/doc/development/testing_guide/end_to_end/running_tests_that_require_special_setup.md index f360226d922..77d820e1686 100644 --- a/doc/development/testing_guide/end_to_end/running_tests_that_require_special_setup.md +++ b/doc/development/testing_guide/end_to_end/running_tests_that_require_special_setup.md @@ -2,13 +2,13 @@ ## Jenkins spec -The [`jenkins_build_status_spec`](https://gitlab.com/gitlab-org/gitlab/blob/163c8a8c814db26d11e104d1cb2dcf02eb567dbe/qa/qa/specs/features/ee/browser_ui/3_create/jenkins/jenkins_build_status_spec.rb) spins up a Jenkins instance in a docker container based on an image stored in the [GitLab-QA container registry](https://gitlab.com/gitlab-org/gitlab-qa/container_registry). -The docker image it uses is preconfigured with some base data and plugins. +The [`jenkins_build_status_spec`](https://gitlab.com/gitlab-org/gitlab/blob/163c8a8c814db26d11e104d1cb2dcf02eb567dbe/qa/qa/specs/features/ee/browser_ui/3_create/jenkins/jenkins_build_status_spec.rb) spins up a Jenkins instance in a Docker container based on an image stored in the [GitLab-QA container registry](https://gitlab.com/gitlab-org/gitlab-qa/container_registry). +The Docker image it uses is preconfigured with some base data and plugins. The test then configures the GitLab plugin in Jenkins with a URL of the GitLab instance that will be used to run the tests. Unfortunately, the GitLab Jenkins plugin does not accept ports so `http://localhost:3000` would -not be accepted. Therefore, this requires us to run GitLab on port 80 or inside a docker container. +not be accepted. Therefore, this requires us to run GitLab on port 80 or inside a Docker container. -To start a docker container for GitLab based on the nightly image: +To start a Docker container for GitLab based on the nightly image: ```shell docker run \ @@ -24,7 +24,7 @@ To run the tests from the `/qa` directory: CHROME_HEADLESS=false bin/qa Test::Instance::All http://localhost -- qa/specs/features/ee/browser_ui/3_create/jenkins/jenkins_build_status_spec.rb ``` -The test will automatically spinup a docker container for Jenkins and tear down once the test completes. +The test will automatically spin up a Docker container for Jenkins and tear down once the test completes. However, if you need to run Jenkins manually outside of the tests, use this command: @@ -46,5 +46,5 @@ only to prevent it from running in the pipelines for live environments such as S ### Troubleshooting -If Jenkins docker container exits without providing any information in the logs, try increasing the memory used by +If Jenkins Docker container exits without providing any information in the logs, try increasing the memory used by the Docker Engine. diff --git a/doc/development/testing_guide/end_to_end/style_guide.md b/doc/development/testing_guide/end_to_end/style_guide.md index 9c02af12d5d..7e9f097f624 100644 --- a/doc/development/testing_guide/end_to_end/style_guide.md +++ b/doc/development/testing_guide/end_to_end/style_guide.md @@ -49,7 +49,7 @@ Notice that in the above example, before clicking the `:operations_environments_ When adding new elements to a page, it's important that we have a uniform element naming convention. -We follow a simple formula roughly based on hungarian notation. +We follow a simple formula roughly based on Hungarian notation. *Formula*: `element :<descriptor>_<type>` @@ -109,7 +109,7 @@ we use the name of the page object in [snake_case](https://en.wikipedia.org/wiki (all lowercase, with words separated by an underscore). See good and bad examples below. While we prefer to follow the standard in most cases, it is also acceptable to -use common abbreviations (e.g., mr) or other alternatives, as long as +use common abbreviations (e.g., `mr`) or other alternatives, as long as the name is not ambiguous. This can include appending `_page` if it helps to avoid confusion or make the code more readable. For example, if a page object is named `New`, it could be confusing to name the block argument `new` because that @@ -123,7 +123,7 @@ Capybara DSL, potentially leading to confusion and bugs. **Good** ```ruby -Page::Project::Settings::Members.perform do |members| +Page::Project::Members.perform do |members| members.do_something end ``` @@ -149,7 +149,7 @@ end **Bad** ```ruby -Page::Project::Settings::Members.perform do |project_settings_members_page| +Page::Project::Members.perform do |project_settings_members_page| project_settings_members_page.do_something end ``` |