Add more cases that emit bad-plugin-value (#7284)

* Add x-failing test for issue 7264 case 3

This is the case where a plugin can be imported only after the init-hook
is run, and the init hook is specified in a pylintrc file
We would expect the module to not load in any case, and cause the
emission of a bad-plugin-value message.

* _dynamic_plugins is a dict not a set

This is in preparation for caching the loaded modules, and for storing
other information that will help in identifying times loading is
dependent on init-hook magic.

* Use cached module objects for load_configuration

This fixes case 3 in #7264, and is technically a breaking change, in
that it now emits a message for what would have previously been a silent
failure.

* Interim review comment fixes

1. Remove the xfail that snuck back in after the rebases.
2. Now that fake_home yields a str, update the type hint.

* Add test for bad plugin with duplicated load

This is case 6 in issue #7264, and represents the other silent failure
that is being made non-silent by this change.

* Apply review feedback

- Add an ordering test for CLI arguments
- Add clarifying comments on affected functions
- Tidy up test assertions to be more pythonic

Co-authored-by: Daniël van Noord <13665637+DanielNoord@users.noreply.github.com>
Co-authored-by: Pierre Sassoulas <pierre.sassoulas@gmail.com>

1 files changed, 8 insertions, 0 deletions

diff --git a/doc/whatsnew/fragments/7264.bugfix b/doc/whatsnew/fragments/7264.bugfix
new file mode 100644
index 000000000..dc2aa5d40
--- /dev/null
+++ b/doc/whatsnew/fragments/7264.bugfix
@@ -0,0 +1,8 @@
+Fixed a case where custom plugins specified by command line could silently fail.
+
+Specifically, if a plugin relies on the ``init-hook`` option changing ``sys.path`` before
+it can be imported, this will now emit a ``bad-plugin-value`` message. Before this
+change, it would silently fail to register the plugin for use, but would load
+any configuration, which could have unintended effects.
+
+Fixes part of #7264.