Switch from pkg_resources to importlib.metadata (#2155)

* Switch from pkg_resources to importlib.metadata

importlib.metadata is a standard library replacement for
pkg_resources, available starting with Python 3.8. It is faster than
pkg_resources. This lets the plugin interface use it when available,
i.e. on Python 3.8 and later. On earlier Python, it uses the
importlib_metadata PyPI backport if available, and finally falls back
on pkg_resources. setup.cfg gains an extra called plugins, which can
allow a project to install "pygments[plugins]" in order to ensure that
plugins support is present even if installed with older Python
versions.

Timings for lexing an empty file with a lexer from a random plugin:

Before:

real	0m0,238s
user	0m0,210s
sys	0m0,029s

After:

real	0m0,141s
user	0m0,125s
sys	0m0,017s

Fixes #2116, #2126

3 files changed, 65 insertions, 13 deletions

diff --git a/doc/docs/plugins.rst b/doc/docs/plugins.rst
index 1008013a..67388607 100644
--- a/doc/docs/plugins.rst
+++ b/doc/docs/plugins.rst
@@ -1,6 +1,6 @@
-================
-Register Plugins
-================
+=======
+Plugins
+=======
 
 If you want to extend Pygments without hacking the sources, but want to
 use the lexer/formatter/style/filter lookup functions (`lexers.get_lexer_by_name`
@@ -13,8 +13,37 @@ That means you can use your highlighter modules with the `pygmentize` script,
 which relies on the mentioned functions.
 
 
-Entrypoints
-===========
+Plugin discovery
+================
+
+At runtime, discovering plugins is preferentially done using Python's
+standard library module `importlib.metadata`_, available in Python 3.8
+and higher. In earlier Python versions, Pygments attempts to use the
+`importlib_metadata`_ backport, if available. If not available, a
+fallback is attempted on the older `pkg_resources`_ module. Finally, if
+``pkg_resources`` is not available, no plugins will be loaded at
+all. Note that ``pkg_resources`` is distributed with `setuptools`_, and
+thus available on most Python environments. However, ``pkg_resources``
+is considerably slower than ``importlib.metadata`` or its
+``importlib_metadata`` backport. For this reason, if you run Pygments
+under Python older than 3.8, it is recommended to install
+``importlib-metadata``. Pygments defines a ``plugins`` packaging extra,
+so you can ensure it is installed with best plugin support (i.e., that
+``importlib-metadata`` is also installed in case you are running Python
+earlier than 3.8) by specifying ``pygments[plugins]`` as the
+requirement, for example, with ``pip``:
+
+.. sourcecode:: shell
+
+   $ python -m pip install --user pygments[plugins]
+
+.. _importlib.metadata: https://docs.python.org/3.10/library/importlib.metadata.html
+.. _importlib_metadata: https://pypi.org/project/importlib-metadata
+.. _pkg_resources: https://setuptools.pypa.io/en/latest/pkg_resources.html
+
+
+Defining plugins through entrypoints
+====================================
 
 Here is a list of setuptools entrypoints that Pygments understands:
 
diff --git a/pygments/plugin.py b/pygments/plugin.py
index 247d5083..0ffef47e 100644
--- a/pygments/plugin.py
+++ b/pygments/plugin.py
@@ -2,9 +2,12 @@
     pygments.plugin
     ~~~~~~~~~~~~~~~
 
-    Pygments setuptools plugin interface. The methods defined
-    here also work if setuptools isn't installed but they just
-    return nothing.
+    Pygments plugin interface. By default, this tries to use
+    ``importlib.metadata``, which is in the Python standard
+    library since Python 3.8, or its ``importlib_metadata``
+    backport for earlier versions of Python. It falls back on
+    ``pkg_resources`` if not found. Finally, if ``pkg_resources``
+    is not found either, no plugins are loaded at all.
 
     lexer plugins::
 
@@ -34,6 +37,7 @@
     :copyright: Copyright 2006-2022 by the Pygments team, see AUTHORS.
     :license: BSD, see LICENSE for details.
 """
+
 LEXER_ENTRY_POINT = 'pygments.lexers'
 FORMATTER_ENTRY_POINT = 'pygments.formatters'
 STYLE_ENTRY_POINT = 'pygments.styles'
@@ -42,11 +46,26 @@ FILTER_ENTRY_POINT = 'pygments.filters'
 
 def iter_entry_points(group_name):
     try:
-        import pkg_resources
-    except (ImportError, OSError):
-        return []
-
-    return pkg_resources.iter_entry_points(group_name)
+        from importlib.metadata import entry_points
+    except ImportError:
+        try:
+            from importlib_metadata import entry_points
+        except ImportError:
+            try:
+                from pkg_resources import iter_entry_points
+            except (ImportError, OSError):
+                return []
+            else:
+                return iter_entry_points(group_name)
+    groups = entry_points()
+    if hasattr(groups, 'select'):
+        # New interface in Python 3.10 and newer versions of the
+        # importlib_metadata backport.
+        return groups.select(group=group_name)
+    else:
+        # Older interface, deprecated in Python 3.10 and recent
+        # importlib_metadata, but we need it in Python 3.8 and 3.9.
+        return groups.get(group_name, [])
 
 
 def find_plugin_lexers():
diff --git a/setup.cfg b/setup.cfg
index 661914c2..a5908226 100644
--- a/setup.cfg
+++ b/setup.cfg
@@ -48,6 +48,10 @@ include =
 console_scripts =
     pygmentize = pygments.cmdline:main
 
+[options.extras_require]
+plugins =
+    importlib-metadata;python_version<'3.8'
+
 [aliases]
 release = egg_info -Db ''
 upload = upload --sign --identity=36580288