1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
|
# Code Coverage
Provides an overview of how to use code coverage tools when running the unit
tests in the EC codebase.
[TOC]
## Test Coverage Requirements
All changes to the EC code base require you include tests covering at least 80%
of any new or changed lines of code. Refer to the [ChromeOS EC Firmware Test
Requirements] for details.
## Availability
Code coverage is only available for host-based unit tests, as opposed to manual
tests that run on target hardware.
## Building for code coverage
To build host-based unit tests for code coverage, invoke `make` with the
`coverage` target, as follows:
`make coverage -j`
This target will compile and link the unit tests with `--coverage` flag (which
pulls in the `gcov` libraries), run the tests, and then process the profiling
data into a code coverage report using the `lcov` and `genhtml` tools.
The coverage report top-level page is `build/coverage/coverage_rpt/index.html`.
To get a report for one specific board's coverage run these commands:
```
BOARD=eldrid
make -j$(nproc) build/coverage/initial-${BOARD}.info test-coverage
# Merge board coverage and test coverage
lcov -o build/coverage/${BOARD}_merged.info --rc lcov_branch_coverage=1 \
-a build/coverage/initial-${BOARD}.info -a build/coverage/lcov.info
# Filter out some unhelpful paths
lcov -o build/coverage/${BOARD}_filtered.info --rc lcov_branch_coverage=1 \
-r build/coverage/${BOARD}_merged.info ${PWD}'/third_party/**' \
${PWD}'/build/**' '/usr/include/**' '/usr/lib/**' '${PWD}/test/**' \
${PWD}'/private/fingerprint/*/mcutest/**'
# Restrict to only files used by the board
grep "SF:" "build/coverage/initial-${BOARD}.info" | sort -u | \
sed -e 's|^SF:||' | xargs lcov --rc lcov_branch_coverage=1 \
-o build/coverage/${BOARD}_final.info \
-e build/coverage/${BOARD}_filtered.info
# Generate HTML
genhtml --branch-coverage -q -o build/coverage/${BOARD}_rpt \
-t "${BOARD} coverage" -s build/coverage/${BOARD}_final.info
```
### Noise in the build output
When building for code coverage, you may see multiple warnings of the form
`geninfo: WARNING: no data found for
/mnt/host/source/src/platform/ec/core/host/cpu.h` and `genhtml: WARNING:
function data mismatch at
/mnt/host/source/src/platform/ec/common/math_util.c:134`
These warnings can be ignored. (FYI, the "function data mismatch" warnings
appear to be caused in part by using relative paths instead of absolute paths.)
## Zephyr ztest code coverage
To build the Zephyr unit tests for code coverage run:
`./twister -v -i --coverage -p native_posix -p unit_testing`
The coverage report top-level page is
`twister-out/coverage/index.html`.
However you probably want to merge that with a single board's coverage report
also, so that you can include code that is not part of any test as well.
```
zmake build --coverage herobrine
./twister -v -i --coverage -p native_posix -p unit_testing
genhtml -q -s --branch-coverage -o build/zephyr/coverage_rpt/ \
twister-out/coverage.info build/zephyr/herobrine/output/zephyr.info
```
The coverage report top-level page is
`build/zephyr/coverage_rpt/index.html`.
For coverage report for a single test you can run:
`./twister -v -i --coverage -p native_posix -p unit_testing -s external/platform/ec/zephyr/test/<testDir>/<testName>`
Example of running test tasks.default from zephyr/test/tasks/testcase.yaml:
`./twister -v -i --coverage -p native_posix -p unit_testing -s external/platform/ec/zephyr/test/tasks/tasks.default`
## Code Coverage in CQ
There are several ways to see the code coverage without running the tests
locally, depending on what information you want to see. Many of the links
below are only available to Googlers or TVCs with google.com accounts.
### Code search
To see the coverage of each directory, visit
http://cs/chromeos_public/src/platform/ec/ and turn on the "Directory Coverage"
layer. The denominator for the percentage covered is not clear, so these
numbers are really only useful if you are looking in very general terms. I.e.
zephyr is covered better than common. Don't get too fixated on the specific
percent shown. The results are also the last 7 days of builds combined, so there
may be some odd results if the code has changed greatly in the last week.
The coverage of files is much more useful. If you are about to write a test
and not sure what to focus on, you can look at the uncovered lines in code
search. Visit [a file](http://cs/chromeos_public/src/platform/ec/common/mkbp_event.c)
in code search and make sure the "File Coverage" layer is enabled. Lines that
are not covered by any test are in red, tested lines are in green, and uncolored
lines were not built at all in any board or test.
### Presubmit
Every gerrit cl, if you did a dry-run or full run of the CQ will have coverage
results. They are slightly difficult to get to, but are very useful.
On the "Checks" tab, find the build "firmware-zephyr-cov-cq" and open it.
On the LUCI page, expand the "test firmware" step and click on "response". That
will show you a very basic summary of the coverage.
For a detailed report, you can download the coverage report. Expand "try to
upload artifacts", then "upload artifacts", and click on "gs upload dir".
From there, click on the download icon for the html.tbz2 file, and untar it
locally. Open lcov_rpt/index.html to view your results.
### Post-submit
If you are interested in the state of the world, not a specific CL, and the
coverage info in Code Search is not sufficient, you can download the coverage
report from the post-submit CQ build.
Visit https://ci.chromium.org/p/chromeos/builders/postsubmit/firmware-zephyr-cov-postsubmit
and click on the latest successful build.
From there, it is exactly the same steps as above to get to the artifacts.
[ChromeOS EC Firmware Test Requirements]: ./chromeos-ec-firmware-test-requirements.md
|
