diff options
Diffstat (limited to 'doc/development/testing_guide')
| -rw-r--r-- | doc/development/testing_guide/best_practices.md | 14 | ||||
| -rw-r--r-- | doc/development/testing_guide/ci.md | 4 | ||||
| -rw-r--r-- | doc/development/testing_guide/frontend_testing.md | 60 | ||||
| -rw-r--r-- | doc/development/testing_guide/review_apps.md | 8 | ||||
| -rw-r--r-- | doc/development/testing_guide/testing_levels.md | 4 |
5 files changed, 75 insertions, 15 deletions
diff --git a/doc/development/testing_guide/best_practices.md b/doc/development/testing_guide/best_practices.md index 2c8d488877b..cfe0e6f70fc 100644 --- a/doc/development/testing_guide/best_practices.md +++ b/doc/development/testing_guide/best_practices.md @@ -40,7 +40,7 @@ bundle exec rspec spec/[path]/[to]/[spec].rb to separate phases. - Use `Gitlab.config.gitlab.host` rather than hard coding `'localhost'` - Don't assert against the absolute value of a sequence-generated attribute (see - [Gotchas](../gotchas.md#dont-assert-against-the-absolute-value-of-a-sequence-generated-attribute)). + [Gotchas](../gotchas.md#do-not-assert-against-the-absolute-value-of-a-sequence-generated-attribute)). - Don't supply the `:each` argument to hooks since it's the default. - On `before` and `after` hooks, prefer it scoped to `:context` over `:all` - When using `evaluate_script("$('.js-foo').testSomething()")` (or `execute_script`) which acts on a given element, @@ -168,12 +168,13 @@ instead of 30+ seconds in case of a regular `spec_helper`. ### `let` variables -GitLab's RSpec suite has made extensive use of `let` variables to reduce -duplication. However, this sometimes [comes at the cost of clarity][lets-not], +GitLab's RSpec suite has made extensive use of `let`(along with it strict, non-lazy +version `let!`) variables to reduce duplication. However, this sometimes [comes at the cost of clarity][lets-not], so we need to set some guidelines for their use going forward: -- `let` variables are preferable to instance variables. Local variables are - preferable to `let` variables. +- `let!` variables are preferable to instance variables. `let` variables + are preferable to `let!` variables. Local variables are preferable to + `let` variables. - Use `let` to reduce duplication throughout an entire spec file. - Don't use `let` to define variables used by a single test; define them as local variables inside the test's `it` block. @@ -183,6 +184,9 @@ so we need to set some guidelines for their use going forward: - Try to avoid overriding the definition of one `let` variable with another. - Don't define a `let` variable that's only used by the definition of another. Use a helper method instead. +- `let!` variables should be used only in case if strict evaluation with defined + order is required, otherwise `let` will suffice. Remember that `let` is lazy and won't + be evaluated until it is referenced. [lets-not]: https://robots.thoughtbot.com/lets-not diff --git a/doc/development/testing_guide/ci.md b/doc/development/testing_guide/ci.md index 5aa668290b4..7a7fca46534 100644 --- a/doc/development/testing_guide/ci.md +++ b/doc/development/testing_guide/ci.md @@ -31,7 +31,11 @@ After that, the next pipeline will use the up-to-date The GitLab test suite is [monitored] for the `master` branch, and any branch that includes `rspec-profile` in their name. +A [public dashboard] is available for everyone to see. Feel free to look at the +slowest test files and try to improve them. + [monitored]: ../performance.md#rspec-profiling +[public dashboard]: https://redash.gitlab.com/public/dashboards/l1WhHXaxrCWM5Ai9D7YDqHKehq6OU3bx5gssaiWe?org_slug=default ## CI setup diff --git a/doc/development/testing_guide/frontend_testing.md b/doc/development/testing_guide/frontend_testing.md index 0470a071d39..5b66e513a76 100644 --- a/doc/development/testing_guide/frontend_testing.md +++ b/doc/development/testing_guide/frontend_testing.md @@ -13,6 +13,42 @@ in the future. See the [Testing Standards and Style Guidelines](index.md) page for more information on general testing practices at GitLab. +## Jest + +GitLab has started to migrate tests to the (Jest)[https://jestjs.io] +testing framework. You can read a [detailed evaluation](https://gitlab.com/gitlab-org/gitlab-ce/issues/49171) +of Jest compared to our use of Karma and Jasmine. In summary, it will allow us +to improve the performance and consistency of our frontend tests. + +Jest tests can be found in `/spec/frontend` and `/ee/spec/frontend` in EE. + +It is not yet a requirement to use Jest. You can view the +[epic](https://gitlab.com/groups/gitlab-org/-/epics/873) of issues +we need to solve before being able to use Jest for all our needs. + +### Timeout error + +The default timeout for Jest is set in +[`/spec/frontend/test_setup.js`](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/spec/frontend/test_setup.js). + +If your test exceeds that time, it will fail. + +If you cannot improve the performance of the tests, you can increase the timeout +for a specific test using +[`jest.setTimeout`](https://jestjs.io/docs/en/jest-object#jestsettimeouttimeout). + +```javascript +beforeAll(() => { + jest.setTimeout(500); +}); + +describe('Component', () => { + // ... +}); +``` + +Remember that the performance of each test depends on the environment. + ## Karma test suite GitLab uses the [Karma][karma] test runner with [Jasmine] as its test @@ -134,7 +170,7 @@ placeholders, and recalling when they are called and the arguments that are passed to them. These tools should be used liberally, to test for expected behavior, to mock responses, and to block unwanted side effects (such as a method that would generate a network request or alter `window.location`). The -documentation for these methods can be found in the [jasmine introduction page](https://jasmine.github.io/2.0/introduction.html#section-Spies). +documentation for these methods can be found in the [Jasmine introduction page](https://jasmine.github.io/2.0/introduction.html#section-Spies). Sometimes you may need to spy on a method that is directly imported by another module. GitLab has a custom `spyOnDependency` method which utilizes @@ -168,7 +204,7 @@ export of a module who's import you want to stub, rather than an object which contains a method you wish to stub (if the module does not have a default export, one is be generated by the babel plugin). The second parameter is the name of the import you wish to change. The result of the function is a Spy -object which can be treated like any other jasmine spy object. +object which can be treated like any other Jasmine spy object. Further documentation on the babel rewire pluign API can be found on [its repository Readme doc](https://github.com/speedskater/babel-plugin-rewire#babel-plugin-rewire). @@ -177,6 +213,14 @@ Further documentation on the babel rewire pluign API can be found on If you cannot avoid using [`setTimeout`](https://developer.mozilla.org/en-US/docs/Web/API/WindowOrWorkerGlobalScope/setTimeout) in tests, please use the [Jasmine mock clock](https://jasmine.github.io/api/2.9/Clock.html). +#### Migrating flaky Karma tests to Jest + +Some of our Karma tests are flaky because they access the properties of a shared scope. +This also means that they are not easily parallelized. + +Migrating flaky Karma tests to Jest will help significantly as each test is executed +in an isolated scope, improving performance and predictability. + ### Vue.js unit tests See this [section][vue-test]. @@ -194,21 +238,21 @@ is sufficient (and saves you some time). ### Live testing and focused testing -While developing locally, it may be helpful to keep karma running so that you +While developing locally, it may be helpful to keep Karma running so that you can get instant feedback on as you write tests and modify code. To do this -you can start karma with `yarn run karma-start`. It will compile the javascript +you can start Karma with `yarn run karma-start`. It will compile the javascript assets and run a server at `http://localhost:9876/` where it will automatically run the tests on any browser which connects to it. You can enter that url on multiple browsers at once to have it run the tests on each in parallel. -While karma is running, any changes you make will instantly trigger a recompile +While Karma is running, any changes you make will instantly trigger a recompile and retest of the entire test suite, so you can see instantly if you've broken -a test with your changes. You can use [jasmine focused][jasmine-focus] or -excluded tests (with `fdescribe` or `xdescribe`) to get karma to run only the +a test with your changes. You can use [Jasmine focused][jasmine-focus] or +excluded tests (with `fdescribe` or `xdescribe`) to get Karma to run only the tests you want while you're working on a specific feature, but make sure to remove these directives when you commit your code. -It is also possible to only run karma on specific folders or files by filtering +It is also possible to only run Karma on specific folders or files by filtering the run tests via the argument `--filter-spec` or short `-f`: ```bash diff --git a/doc/development/testing_guide/review_apps.md b/doc/development/testing_guide/review_apps.md index 703e342fc13..fda3ff57316 100644 --- a/doc/development/testing_guide/review_apps.md +++ b/doc/development/testing_guide/review_apps.md @@ -95,6 +95,14 @@ You can also manually start the `review-qa-all`: it runs the full QA suite. Note that both jobs first wait for the `review-deploy` job to be finished. +## Performance Metrics + +On every [pipeline][gitlab-pipeline] during the `test` stage, the +`review-performance` job is automatically started: this job does basic +browser performance testing using [Sitespeed.io Container](https://docs.gitlab.com/ee/user/project/merge_requests/browser_performance_testing.html) . + +This job waits for the `review-deploy` job to be finished. + ## How to? ### Log into my Review App? diff --git a/doc/development/testing_guide/testing_levels.md b/doc/development/testing_guide/testing_levels.md index a7a3459719b..5d46833a1e2 100644 --- a/doc/development/testing_guide/testing_levels.md +++ b/doc/development/testing_guide/testing_levels.md @@ -46,7 +46,7 @@ They're useful to test permissions, redirections, what view is rendered etc. | `app/mailers/` | `spec/mailers/` | RSpec | | | `lib/api/` | `spec/requests/api/` | RSpec | | | `lib/ci/api/` | `spec/requests/ci/api/` | RSpec | | -| `app/assets/javascripts/` | `spec/javascripts/` | Karma | More details in the [JavaScript](#javascript) section. | +| `app/assets/javascripts/` | `spec/javascripts/` | Karma | More details in the [Karma JavaScript test suite](frontend_testing.md#karma-test-suite) section. | ### About controller tests @@ -210,7 +210,7 @@ trade-off: - Integration tests are a bit more expensive, but don't abuse them. A system test is often better than an integration test that is stubbing a lot of internals. - System tests are expensive (compared to unit tests), even more if they require - a JavaScript driver. Make sure to follow the guidelines in the [Speed](#test-speed) + a JavaScript driver. Make sure to follow the guidelines in the [Speed](best_practices.md#test-speed) section. Another way to see it is to think about the "cost of tests", this is well |