diff options
Diffstat (limited to 'doc/ci/junit_test_reports.md')
| -rw-r--r-- | doc/ci/junit_test_reports.md | 280 |
1 files changed, 2 insertions, 278 deletions
diff --git a/doc/ci/junit_test_reports.md b/doc/ci/junit_test_reports.md index 8bc55a6e4f3..449f9bf5fcd 100644 --- a/doc/ci/junit_test_reports.md +++ b/doc/ci/junit_test_reports.md @@ -1,281 +1,5 @@ --- -stage: Verify -group: Testing -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers -type: reference +redirect_to: 'unit_test_reports.md' --- -# JUnit test reports - -> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/45318) in GitLab 11.2. Requires GitLab Runner 11.2 and above. - -## Overview - -It is very common that a [CI/CD pipeline](pipelines/index.md) contains a -test job that will verify your code. -If the tests fail, the pipeline fails and users get notified. The person that -works on the merge request will have to check the job logs and see where the -tests failed so that they can fix them. - -You can configure your job to use JUnit test reports, and GitLab will display a -report on the merge request so that it's easier and faster to identify the -failure without having to check the entire log. - -If you don't use Merge Requests but still want to see the JUnit output without searching through job logs, the full [JUnit test reports](#viewing-junit-test-reports-on-gitlab) are available in the pipeline detail view. - -## Use cases - -Consider the following workflow: - -1. Your `master` branch is rock solid, your project is using GitLab CI/CD and - your pipelines indicate that there isn't anything broken. -1. Someone from your team submits a merge request, a test fails and the pipeline - gets the known red icon. To investigate more, you have to go through the job - logs to figure out the cause of the failed test, which usually contain - thousands of lines. -1. You configure the JUnit test reports and immediately GitLab collects and - exposes them in the merge request. No more searching in the job logs. -1. Your development and debugging workflow becomes easier, faster and efficient. - -## How it works - -First, GitLab Runner uploads all JUnit XML files as artifacts to GitLab. Then, -when you visit a merge request, GitLab starts comparing the head and base branch's -JUnit test reports, where: - -- The base branch is the target branch (usually `master`). -- The head branch is the source branch (the latest pipeline in each merge request). - -The reports panel has a summary showing how many tests failed, how many had errors -and how many were fixed. If no comparison can be done because data for the base branch -is not available, the panel will just show the list of failed tests for head. - -There are four types of results: - -1. **Newly failed tests:** Test cases which passed on base branch and failed on head branch -1. **Newly encountered errors:** Test cases which passed on base branch and failed due to a - test error on head branch -1. **Existing failures:** Test cases which failed on base branch and failed on head branch -1. **Resolved failures:** Test cases which failed on base branch and passed on head branch - -Each entry in the panel will show the test name and its type from the list -above. Clicking on the test name will open a modal window with details of its -execution time and the error output. - - - -## How to set it up - -To enable the JUnit reports in merge requests, you need to add -[`artifacts:reports:junit`](pipelines/job_artifacts.md#artifactsreportsjunit) -in `.gitlab-ci.yml`, and specify the path(s) of the generated test reports. -The reports must be `.xml` files, otherwise [GitLab returns an Error 500](https://gitlab.com/gitlab-org/gitlab/-/issues/216575). - -In the following examples, the job in the `test` stage runs and GitLab -collects the JUnit test report from each job. After each job is executed, the -XML reports are stored in GitLab as artifacts and their results are shown in the -merge request widget. - -To make the JUnit output files browsable, include them with the -[`artifacts:paths`](yaml/README.md#artifactspaths) keyword as well, as shown in the [Ruby example](#ruby-example). - -NOTE: **Note:** -You cannot have multiple tests with the same name and class in your JUnit report. - -### Ruby example - -Use the following job in `.gitlab-ci.yml`. This includes the `artifacts:paths` keyword to provide a link to the JUnit output file. - -```yaml -## Use https://github.com/sj26/rspec_junit_formatter to generate a JUnit report with rspec -ruby: - stage: test - script: - - bundle install - - bundle exec rspec --format progress --format RspecJunitFormatter --out rspec.xml - artifacts: - paths: - - rspec.xml - reports: - junit: rspec.xml -``` - -### Go example - -Use the following job in `.gitlab-ci.yml`, and ensure you use `-set-exit-code`, -otherwise the pipeline will be marked successful, even if the tests fail: - -```yaml -## Use https://github.com/jstemmer/go-junit-report to generate a JUnit report with go -golang: - stage: test - script: - - go get -u github.com/jstemmer/go-junit-report - - go test -v 2>&1 | go-junit-report -set-exit-code > report.xml - artifacts: - reports: - junit: report.xml -``` - -### Java examples - -There are a few tools that can produce JUnit reports in Java. - -#### Gradle - -In the following example, `gradle` is used to generate the test reports. -If there are multiple test tasks defined, `gradle` will generate multiple -directories under `build/test-results/`. In that case, you can leverage glob -matching by defining the following path: `build/test-results/test/**/TEST-*.xml`: - -```yaml -java: - stage: test - script: - - gradle test - artifacts: - reports: - junit: build/test-results/test/**/TEST-*.xml -``` - -NOTE: **Note:** -Support for `**` was added in [GitLab Runner 13.0](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/2620). - -#### Maven - -For parsing [Surefire](https://maven.apache.org/surefire/maven-surefire-plugin/) -and [Failsafe](https://maven.apache.org/surefire/maven-failsafe-plugin/) test -reports, use the following job in `.gitlab-ci.yml`: - -```yaml -java: - stage: test - script: - - mvn verify - artifacts: - reports: - junit: - - target/surefire-reports/TEST-*.xml - - target/failsafe-reports/TEST-*.xml -``` - -### Python example - -This example uses pytest with the `--junitxml=report.xml` flag to format the output -for JUnit: - -```yaml -pytest: - stage: test - script: - - pytest --junitxml=report.xml - artifacts: - reports: - junit: report.xml -``` - -### C/C++ example - -There are a few tools that can produce JUnit reports in C/C++. - -#### GoogleTest - -In the following example, `gtest` is used to generate the test reports. -If there are multiple gtest executables created for different architectures (`x86`, `x64` or `arm`), -you will be required to run each test providing a unique filename. The results -will then be aggregated together. - -```yaml -cpp: - stage: test - script: - - gtest.exe --gtest_output="xml:report.xml" - artifacts: - reports: - junit: report.xml -``` - -#### CUnit - -[CUnit](https://cunity.gitlab.io/cunit/) can be made to produce [JUnit XML reports](https://cunity.gitlab.io/cunit/group__CI.html) automatically when run using its `CUnitCI.h` macros: - -```yaml -cunit: - stage: test - script: - - ./my-cunit-test - artifacts: - reports: - junit: ./my-cunit-test.xml -``` - -### .NET example - -The [JunitXML.TestLogger](https://www.nuget.org/packages/JunitXml.TestLogger/) NuGet -package can generate test reports for .Net Framework and .Net Core applications. The following -example expects a solution in the root folder of the repository, with one or more -project files in sub-folders. One result file is produced per test project, and each file -is placed in a new artifacts folder. This example includes optional formatting arguments, which -improve the readability of test data in the test widget. A full .Net Core -[example is available](https://gitlab.com/Siphonophora/dot-net-cicd-test-logging-demo). - -```yaml -## Source code and documentation are here: https://github.com/spekt/junit.testlogger/ - -Test: - stage: test - script: - - 'dotnet test --test-adapter-path:. --logger:"junit;LogFilePath=..\artifacts\{assembly}-test-result.xml;MethodFormat=Class;FailureBodyFormat=Verbose"' - artifacts: - when: always - paths: - - ./**/*test-result.xml - reports: - junit: - - ./**/*test-result.xml -``` - -## Viewing JUnit test reports on GitLab - -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/24792) in GitLab 12.5 behind a feature flag (`junit_pipeline_view`), disabled by default. -> - The feature flag was removed and the feature was [made generally available](https://gitlab.com/gitlab-org/gitlab/-/issues/216478) in GitLab 13.3. - -If JUnit XML files are generated and uploaded as part of a pipeline, these reports -can be viewed inside the pipelines details page. The **Tests** tab on this page will -display a list of test suites and cases reported from the XML file. - - - -You can view all the known test suites and click on each of these to see further -details, including the cases that make up the suite. - -You can also retrieve the reports via the [GitLab API](../api/pipelines.md#get-a-pipelines-test-report). - -## Viewing JUnit screenshots on GitLab - -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/202114) in GitLab 13.0. -> - It's deployed behind a feature flag, disabled by default. -> - To use it in GitLab self-managed instances, ask a GitLab administrator to [enable it](#enabling-the-junit-screenshots-feature-core-only). **(CORE ONLY)** - -If JUnit XML files contain an `attachment` tag, GitLab parses the attachment. - -Upload your screenshots as [artifacts](pipelines/job_artifacts.md#artifactsreportsjunit) to GitLab. The `attachment` tag **must** contain the absolute path to the screenshots you uploaded. - -```xml -<testcase time="1.00" name="Test"> - <system-out>[[ATTACHMENT|/absolute/path/to/some/file]]</system-out> -</testcase> -``` - -When [this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/6061) is complete, the attached file will be visible on the pipeline details page. - -### Enabling the JUnit screenshots feature **(CORE ONLY)** - -This feature comes with the `:junit_pipeline_screenshots_view` feature flag disabled by default. - -To enable this feature, ask a GitLab administrator with [Rails console access](../administration/feature_flags.md#how-to-enable-and-disable-features-behind-flags) to run the -following command: - -```ruby -Feature.enable(:junit_pipeline_screenshots_view) -``` +This document was moved to [unit_test_reports](unit_test_reports.md). |