1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
|
## Test262 Authoring Guidelines
### Test Case Names
These are guidelines. There is a substantial amount of variation in existing test names.
Test cases are often named by the portion of the spec or draft spec they reference. A test for ECMAScript 5.1 section [8.7.2 PutValue](http://www.ecma-international.org/ecma-262/5.1/#sec-8.7.2) should be placed in the directory `test262/suite/ch08/8.7/8.7.2/` and have a filename which contains the substring `8.7.2`. A test for ECMAScript 6.0 draft section [25.4.4.1 Promise.all](https://people.mozilla.org/~jorendorff/es6-draft.html#sec-promise.all) should be placed in directory `test262/suite/es6/ch25/25.4/25.4.4/25.4.4.1` and have a filename that contains the substring `25.4.4.1`.
One common convention is to break down a section into a number of testable assertions (A) and to test each assertion with one or more test cases (T). For example, section 25.4.4.1 of ES6 could be broken down like this:
* A1 Promise.all is a function of one argument
- A1.1 Promise.all is callable
- A1.2 Promise.all expects 1 argument
* A2 Promise.all([]) is a Promise immediately resolved with []
- A2.1 Promise.all([]) returns a Promise
- A2.2 Promise.all([]) is resolved immediately
- A2.3 Promise.all([]) is resolved with a new empty array
with test case `S25.4.4.1_A1.1_T1.js` being the first test for assertion A1.1, and `S25.4.4.1_A2.3_T3.js` being the third test for assertion A2.3.
### Test Case Style
A test file has three sections: Copyright, Frontmatter, and Body. A test looks roughly like this:
```javascript
// Copyright (C) 2014 [Contributor Name]. All rights reserved.
// This code is governed by the BSD license found in the LICENSE file.
/*---
info: >
verbose test description, multiple lines OK.
description: brief description
---*/
[Test Code]
```
#### Copyright
The copyright block must be the first section of the test. The copyright block must use `//` style comments.
#### Frontmatter
The Test262 frontmatter is a string of YAML enclosed by the comment start tag `/*---` and end tag `---*/`. There must be exactly one Frontmatter per test.
Test262 supports the following tags: **info**, **description**, **negative**, **es5id**, **includes**, and **author**
##### info: long-desc
This allows a long, free-form comment.
Eg: Single line comments can contain Line Terminator at the end of line
##### description: short-desc
This should be a short, one-line description of the purpose of this testcase. This is the string displayed by the browser runnner.
Eg: Insert <LS> between chunks of one string
##### negative: [regex]
This means the test is expected to throw an error of the given type. If no error is thrown, a test failure is reported.
If an error is thrown, it is implicitly converted to a string. The second parameter is a regular expression that will be matched against this string. If the match fails, a test failure is reported. Thus the regular expression can match either the error name, or the message contents, or both.
For best practices on how to use the negative tag please see Handling Errors and Negative Test Cases, below.
##### author: [author-name]
This tag is used to identify the author of a test case. It's optional.
##### es5id: [es5-test-id]
This tag identifies the portion of the ECMAScript 5.1 standard that is tested by this test. It was automatically generated for tests that were originally written for the ES5 version of the test suite and are now part of the ES6 version.
When writing a new test for ES6, it is only necessary to include this tag when the test covers a part of the ES5 spec that is incorporated into ES6.
##### includes: [file-list]
This tag is a list of helper files that should be included in the test environment prior to running the test. Filenames **must** include the `.js` extension.
The helper files are found in `test/harness/`. The packaging script will ensure that files from `test/harness` will be copied to `website/harness` when it prepares the `website/` directory for publishing.
You can compactly include a single file like this: `includes: [helperFile.js]`, or use the full YAML list syntax
```
includes:
- helperOne.js
- helperTwo.js
```
##### flags: [list]
This tag is for boolean properties associated with the test.
Some flags are used only by the Python runner: **flags** **onlyStrict**, **noStrict**
##### flags: [onlyStrict]
Will only run the test in strict mode
##### flags: [noStrict]
Will only run the test in non-strict mode
#### Obsolete Tags
##### path
This tag is obsolete. Do not manually enter this tag.
##### flags: [negative]
This is an old-style way of documenting a negative test. New tests should use the **negative: [errortype]** style documented above.
### Test Environment
Each test case is run in a fresh JavaScript environment; in a browser, this will be a new `IFRAME`; for a console runner, this will be a new process. The test harness code is loaded before the test is run. The test harness defines the following helper functions:
Function | Purpose
---------|--------
Test262Error(message) | constructor for an error object that indicates a test failure
$ERROR(message) | helper function: construct a Test262Error object and throw it
$DONE(arg) | helper function for asynchronous tests; see Writing Asynchronous Tests, below
NotEarlyError | preconstructed error object used for testing syntax and other early errors; see Syntax Error & Eearly Error, below
```
/// error class
function Test262Error(message) {
//[omitted body]
}
/// helper function that throws
function $ERROR(message) {
throw new Test262Error(message);
}
/// helper function for asynchronous tests
function $DONE(arg) {
//[omitted body]
}
var NotEarlyError = new Error(...);
```
### Custom Helpers
When some code is used repeatedly across a group of tests, a new helper function (or group of helpers) can be defined. To define new helpers, create a file in `test/harness/` with extension `.js`.
To use a custom helper file, name it in the `includes` directive of the Frontmatter, e.g.,
```
/*---
includes: [helper.js]
---*/
```
**Note:** Helper inclusion used to be performed by the `$INCLUDE` function, which is now obsolete. New tests should use the YAML `includes`.
### Handling Errors and Negative Test Cases
The following patterns are considered the best practice:
#### Runtime Error:
```javascript
/*---
negative: ReferenceError
---*/
[test code]
```
The example uses ReferenceError however it's also possible to use any of the following errors here:
- EvalError
- RangeError
- ReferenceError
- TypeError
- URIError
#### Syntax Error & Early Error:
```javascript
/*
* @negative ^((?!NotEarlyError).)*$
*/
throw NotEarlyError;
[test code]
```
There are *very* few cases where a syntax error is **not** an early error. In those cases use the Runtime Error pattern but wrap the test code in an eval statement. Be careful, eval code is not global code!
### Writing Asynchronous Tests
An asynchronous test is any test that includes the string `$DONE` anywhere in the test file. The test runner checks for the presence of this string; if it is found, the runner expects that the `$DONE()` function will be called to signal test completion.
* If the argument to `$DONE` is omitted, is `undefined`, or is any other falsy value, the test is considered to have passed.
* If the argument to `$DONE` is a truthy value, the test is considered to have failed and the argument is displayed as the failure reason.
A common idiom when writing asynchronous tests is the following:
```js
var p = new Promise(function () { /* some test code */ });
p.then(function checkAssertions(arg) {
if (!expected_condition) {
$ERROR("failure message");
}
}).then($DONE, $DONE);
```
Function `checkAssertions` implicitly returns `undefined` if the expected condition is observed. The return value of function `checkAssertions` is then used to asynchronously invoke the first function of the final `then` call, resulting in a call to `$DONE(undefined)`, which signals a passing test.
If the expected condition is not observed, function `checkAssertions` throws a `Test262Error` via function $ERROR. This is caught by the Promise and then used to asynchronously invoke the second function in the call -- which is also `$DONE` -- resulting in a call to `$DONE(error_object)`, which signals a failing test.
#### Checking Exception Type and Message in Asynchronous Tests
This idiom can be extended to check for specific exception types or messages:
```js
p.then(function () {
// some code that is expected to throw a TypeError
return "Expected exception to be thrown";
}).then($DONE, function (e) {
if (!e instanceof TypeError) {
$ERROR("Expected TypeError but got " + e);
}
if (!/expected message/.test(e.message)) {
$ERROR("Expected message to contain 'expected message' but found " + e.message);
}
}).then($DONE, $DONE);
```
### Other Features
#### $INCLUDE(fileName) method
This method is obsolete. Use the `includes:` frontmatter.
|
