[stable-2.14] validate-modules: don't fail on invalid YAML (#79682)

* validate-modules: don't fail on invalid YAML

When validate-modules encounters invalid YAML (e.g. in the EXAMPLES
section), it tries to reformat the exception to include the line number
in the Python file instead of the line number of the embedded YAML
document. However, PyYAML doesn't allow modification of the Mark object
(anymore) which leads to a new exception being raised, instead of
reporting the original exception.

As the original exception is not needed in other places anymore, we
don't have to modify it at all and can just compute the right line
number when reporting the error via ansible-test.

Fixes: #75837

* Add test for invalid module doc YAML syntax.

Co-authored-by: Matt Clay <matt@mystile.com>
(cherry picked from commit a7111c4dbb91e912aeb799d9728ce739f2ae83d3)

Co-authored-by: Evgeni Golov <evgeni@golov.de>

5 files changed, 38 insertions, 4 deletions

diff --git a/changelogs/fragments/75837-validate-modules-invalid-yaml.yml b/changelogs/fragments/75837-validate-modules-invalid-yaml.yml
new file mode 100644
index 0000000000..f2c57756bb
--- /dev/null
+++ b/changelogs/fragments/75837-validate-modules-invalid-yaml.yml
@@ -0,0 +1,2 @@
+bugfixes:
+  - ansible-test sanity - correctly report invalid YAML in validate-modules (https://github.com/ansible/ansible/issues/75837).
diff --git a/test/integration/targets/ansible-test-sanity-validate-modules/ansible_collections/ns/col/plugins/modules/invalid_yaml_syntax.py b/test/integration/targets/ansible-test-sanity-validate-modules/ansible_collections/ns/col/plugins/modules/invalid_yaml_syntax.py
new file mode 100644
index 0000000000..5dd753f75a
--- /dev/null
+++ b/test/integration/targets/ansible-test-sanity-validate-modules/ansible_collections/ns/col/plugins/modules/invalid_yaml_syntax.py
@@ -0,0 +1,27 @@
+#!/usr/bin/python
+# GNU General Public License v3.0+ (see COPYING or https://www.gnu.org/licenses/gpl-3.0.txt)
+
+from __future__ import absolute_import, division, print_function
+__metaclass__ = type
+
+DOCUMENTATION = '''
+- key: "value"wrong
+'''
+
+EXAMPLES = '''
+- key: "value"wrong
+'''
+
+RETURN = '''
+- key: "value"wrong
+'''
+
+from ansible.module_utils.basic import AnsibleModule
+
+
+def main():
+    AnsibleModule(argument_spec=dict())
+
+
+if __name__ == '__main__':
+    main()
diff --git a/test/integration/targets/ansible-test-sanity-validate-modules/expected.txt b/test/integration/targets/ansible-test-sanity-validate-modules/expected.txt
new file mode 100644
index 0000000000..95f12f390a
--- /dev/null
+++ b/test/integration/targets/ansible-test-sanity-validate-modules/expected.txt
@@ -0,0 +1,5 @@
+plugins/modules/invalid_yaml_syntax.py:0:0: deprecation-mismatch: "meta/runtime.yml" and DOCUMENTATION.deprecation do not agree.
+plugins/modules/invalid_yaml_syntax.py:0:0: missing-documentation: No DOCUMENTATION provided
+plugins/modules/invalid_yaml_syntax.py:8:15: documentation-syntax-error: DOCUMENTATION is not valid YAML
+plugins/modules/invalid_yaml_syntax.py:12:15: invalid-examples: EXAMPLES is not valid YAML
+plugins/modules/invalid_yaml_syntax.py:16:15: return-syntax-error: RETURN is not valid YAML
diff --git a/test/integration/targets/ansible-test-sanity-validate-modules/runme.sh b/test/integration/targets/ansible-test-sanity-validate-modules/runme.sh
index 1b051b3aca..1821e5a8bd 100755
--- a/test/integration/targets/ansible-test-sanity-validate-modules/runme.sh
+++ b/test/integration/targets/ansible-test-sanity-validate-modules/runme.sh
@@ -4,7 +4,9 @@ source ../collection/setup.sh
 
 set -eux
 
-ansible-test sanity --test validate-modules --color --truncate 0 "${@}"
+ansible-test sanity --test validate-modules --color --truncate 0 --failure-ok --lint "${@}" 1> actual-stdout.txt 2> actual-stderr.txt
+diff -u "${TEST_DIR}/expected.txt" actual-stdout.txt
+grep -f "${TEST_DIR}/expected.txt" actual-stderr.txt
 
 cd ../ps_only
 
diff --git a/test/lib/ansible_test/_util/controller/sanity/validate-modules/validate_modules/utils.py b/test/lib/ansible_test/_util/controller/sanity/validate-modules/validate_modules/utils.py
index 5b20db8da9..88d5b01af1 100644
--- a/test/lib/ansible_test/_util/controller/sanity/validate-modules/validate_modules/utils.py
+++ b/test/lib/ansible_test/_util/controller/sanity/validate-modules/validate_modules/utils.py
@@ -154,11 +154,9 @@ def parse_yaml(value, lineno, module, name, load_all=False, ansible_loader=False
         if load_all:
             data = list(data)
     except yaml.MarkedYAMLError as e:
-        e.problem_mark.line += lineno - 1
-        e.problem_mark.name = '%s.%s' % (module, name)
         errors.append({
             'msg': '%s is not valid YAML' % name,
-            'line': e.problem_mark.line + 1,
+            'line': e.problem_mark.line + lineno,
             'column': e.problem_mark.column + 1
         })
         traces.append(e)