Merge PEP 660 implementation (#3488)

5 files changed, 291 insertions, 53 deletions

diff --git a/docs/userguide/development_mode.rst b/docs/userguide/development_mode.rst
index 4130ab73..e7c755ad 100644
--- a/docs/userguide/development_mode.rst
+++ b/docs/userguide/development_mode.rst
@@ -1,34 +1,260 @@
-Development Mode
-================
-
-Under normal circumstances, the ``setuptools`` assume that you are going to
-build a distribution of your project, not use it in its "raw" or "unbuilt"
-form.  However, if you were to use the ``setuptools`` to build a distribution,
-you would have to rebuild and reinstall your project every time you made a
-change to it during development.
-
-Another problem that sometimes comes is that you may
-need to do development on two related projects at the same time.  You may need
-to put both projects' packages in the same directory to run them, but need to
-keep them separate for revision control purposes.  How can you do this?
-
-Setuptools allows you to deploy your projects for use in a common directory or
-staging area, but without copying any files.  Thus, you can edit each project's
-code in its checkout directory, and only need to run build commands when you
-change files that need to be compiled or the provided metadata and setuptools configuration.
-
-You can perform a ``pip`` installation passing the ``-e/--editable``
-flag (e.g., ``pip install -e .``). It works very similarly to
-``pip install .``, except that it doesn't actually install anything.
-Instead, it creates a special ``.egg-link`` file in the target directory
-(usually ``site-packages``) that links to your project's source code.
-It may also update an existing ``easy-install.pth`` file
-to include your project's source code, thereby making
-it available on ``sys.path`` for all programs using that Python installation.
-
-You can deploy the same project to multiple staging areas, e.g., if you have
-multiple projects on the same machine that are sharing the same project you're
-doing development work.
+Development Mode (a.k.a. "Editable Installs")
+=============================================
+
+When creating a Python project, developers usually want to implement and test
+changes iteratively, before cutting a release and preparing a distribution archive.
+
+In normal circumstances this can be quite cumbersome and require the developers
+to manipulate the ``PATHONPATH`` environment variable or to continuous re-build
+and re-install the project.
+
+To facilitate iterative exploration and experimentation, setuptools allows
+users to instruct the Python interpreter and its import machinery to load the
+code under development directly from the project folder without having to
+copy the files to a different location in the disk.
+This means that changes in the Python source code can immediately take place
+without requiring a new installation.
+
+You can enter this "development mode" by performing an :doc:`editable installation
+<pip:topics/local-project-installs>` inside of a :term:`virtual environment`,
+using :doc:`pip's <pip:cli/pip_install>` ``-e/--editable`` flag, as shown bellow:
+
+.. code-block:: bash
+
+   $ cd your-python-project
+   $ python -m venv .venv
+   # Activate your environemt with:
+   #      `source .venv/bin/activate` on Unix/macOS
+   # or   `.venv\Scripts\activate` on Windows
+
+   $ pip install --editable .
+
+   # Now you have access to your package
+   # as if it was installed in .venv
+   $ python -c "import your_python_project"
+
+
+An "editable installation" works very similarly to a regular install with
+``pip install .``, except that it only installs your package dependencies,
+metadata and wrappers for :ref:`console and GUI scripts <console-scripts>`.
+Under the hood, setuptools will try to create a special :mod:`.pth file <site>`
+in the target directory (usually ``site-packages``) that extends the
+``PYTHONPATH`` or install a custom :doc:`import hook <python:reference/import>`.
 
 When you're done with a given development task, you can simply uninstall your
 package (as you would normally do with ``pip uninstall <package name>``).
+
+Please note that, by default an editable install will expose at least all the
+files that would be available in a regular installation. However, depending on
+the file and directory organization in your project, it might also expose
+as a side effect files that would not be normally available.
+This is allowed so you can iteratively create new Python modules.
+Please have a look on the following section if you are looking for a different behaviour.
+
+.. admonition:: Virtual Environments
+
+   You can think about virtual environments as "isolated Python runtime deployments"
+   that allow users to install different sets of libraries and tools without
+   messing with the global behaviour of the system.
+
+   They are a safe way of testing new projects and can be created easily
+   with the :mod:`venv` module from the standard library.
+
+   Please note however that depending on your operating system or distribution,
+   ``venv`` might not come installed by default with Python. For those cases,
+   you might need to use the OS package manager to install it.
+   For example, in Debian/Ubuntu-based systems you can obtain it via:
+
+   .. code-block:: bash
+
+       sudo apt install python3-venv
+
+   Alternatively, you can also try installing :pypi:`virtualenᴠ`.
+   More information is available on the Python Packaging User Guide on
+   :doc:`PyPUG:guides/installing-using-pip-and-virtual-environments`.
+
+.. note::
+    .. versionchanged:: v64.0.0
+       Editable installation hooks implemented according to :pep:`660`.
+       Support for :pep:`namespace packages <420>` is still **EXPERIMENTAL**.
+
+
+"Strict" editable installs
+--------------------------
+
+When thinking about editable installations, users might have the following
+expectations:
+
+1. It should allow developers to add new files (or split/rename existing ones)
+   and have them automatically exposed.
+2. It should behave as close as possible to a regular installation and help
+   users to detect problems (e.g. new files not being included in the distribution).
+
+Unfortunately these expectations are in conflict with each other.
+To solve this problem ``setuptools`` allows developers to choose a more
+*"strict"* mode for the editable installation. This can be done by passing
+a special *configuration setting* via :pypi:`pip`, as indicated bellow:
+
+.. code-block:: bash
+
+    pip install -e . --config-settings editable_mode=strict
+
+In this mode, new files **won't** be exposed and the editable installs will
+try to mimic as much as possible the behavior of a regular install.
+Under the hood, ``setuptools`` will create a tree of file links in an auxiliary
+directory (``$your_project_dir/build``) and add it to ``PYTHONPATH`` via a
+:mod:`.pth file <site>`. (Please be careful to not delete this repository
+by mistake otherwise your files may stop being accessible).
+
+.. warning::
+   Strict editable installs require auxiliary files to be placed in a
+   ``build/__editable__.*`` directory (relative to your project root).
+
+   Please be careful to not remove this directory while testing your project,
+   otherwise your editable installation may be compromised.
+
+   You can remove the ``build/__editable__.*`` directory after uninstalling.
+
+
+.. note::
+    .. versionadded:: v64.0.0
+       Added new *strict* mode for editable installations.
+       The exact details of how this mode is implemented may vary.
+
+
+Limitations
+-----------
+
+- The *editable* term is used to refer only to Python modules
+  inside the package directories. Non-Python files, external (data) files,
+  executable script files, binary extensions, headers and metadata may be
+  exposed as a *snapshot* of the version they were at the moment of the
+  installation.
+- Adding new dependencies, entry-points or changing your project's metadata
+  require a fresh "editable" re-installation.
+- Console scripts and GUI scripts **MUST** be specified via :doc:`entry-points
+  </userguide/entry_point>` to work properly.
+- *Strict* editable installs require the file system to support
+  either :wiki:`symbolic <symbolic link>` or :wiki:`hard links <hard link>`.
+  This installation mode might also generate auxiliary files under the project directory.
+- There is *no guarantee* that the editable installation will be performed
+  using a specific technique. Depending on each project, ``setuptools`` may
+  select a different approach to ensure the package is importable at runtime.
+- There is *no guarantee* that files outside the top-level package directory
+  will be accessible after an editable install.
+- There is *no guarantee* that attributes like ``__path__`` or ``__file__``
+  will correspond to the exact location of the original files (e.g.,
+  ``setuptools`` might employ file links to perform the editable installation).
+  Users are encouraged to use tools like :mod:`importlib.resources` or
+  :mod:`importlib.metadata` when trying to access package files directly.
+- Editable installations may not work with
+  :doc:`namespaces created with pkgutil or pkg_resouces
+  <PyPUG:guides/packaging-namespace-packages>`.
+  Please use :pep:`420`-style implicit namespaces [#namespaces]_.
+- Support for :pep:`420`-style implicit namespace packages for
+  projects structured using :ref:`flat-layout` is still **experimental**.
+  If you experience problems, you can try converting your package structure
+  to the :ref:`src-layout`.
+
+.. attention::
+   Editable installs are **not a perfect replacement for regular installs**
+   in a test environment. When in doubt, please test your projects as
+   installed via a regular wheel. There are tools in the Python ecosystem,
+   like :pypi:`tox` or :pypi:`nox`, that can help you with that
+   (when used with appropriate configuration).
+
+
+Legacy Behavior
+---------------
+
+If your project is not compatible with the new "editable installs" or you wish
+to replicate the legacy behavior, for the time being you can also perform the
+installation in the ``compat`` mode:
+
+.. code-block:: bash
+
+    pip install -e . --config-settings editable_mode=compat
+
+This installation mode will try to emulate how ``python setup.py develop``
+works (still within the context of :pep:`660`).
+
+.. warning::
+   The ``compat`` mode is *transitional* and will be removed in
+   future versions of ``setuptools``, it exists only to help during the
+   migration period.
+   Also note that support for this mode is limited:
+   it is safe to assume that the ``compat`` mode is offered "as is", and
+   improvements are unlikely to be implemented.
+   Users are encouraged to try out the new editable installation techniques
+   and make the necessary adaptations.
+
+If the ``compat`` mode does not work for you, you can also disable the
+:pep:`editable install <660>` hooks in ``setuptools`` by setting an environment
+variable:
+
+.. code-block::
+
+   SETUPTOOLS_USE_FEATURE="legacy-editable"
+
+This *may* cause the installer (e.g. ``pip``) to effectively run the "legacy"
+installation command: ``python setup.py develop`` [#installer]_.
+
+
+How editable installations work?
+--------------------------------
+
+*Advanced topic*
+
+There are many techniques that can be used to expose packages under development
+in such a way that they are available as if they were installed.
+Depending on the project file structure and the selected mode, ``setuptools``
+will choose one of these approaches for the editable installation [#criteria]_.
+
+A non-exhaustive list of implementation mechanisms is presented below.
+More information is available on the text of :pep:`PEP 660 <660#what-to-put-in-the-wheel>`.
+
+- A static ``.pth`` file [#static_pth]_ can be added to one of the directories
+  listed in :func:`site.getsitepackages` or :func:`site.getusersitepackages` to
+  extend :obj:`sys.path`.
+- A directory containing a *farm of file links* that mimic the
+  project structure and point to the original files can be employed.
+  This directory can then be added to :obj:`sys.path` using a static ``.pth`` file.
+- A dynamic ``.pth`` file [#dynamic_pth]_ can also be used to install an
+  "import :term:`finder`" (:obj:`~importlib.abc.MetaPathFinder` or
+  :obj:`~importlib.abc.PathEntryFinder`) that will hook into Python's
+  :doc:`import system <python:reference/import>` machinery.
+
+.. attention::
+   ``Setuptools`` offers **no guarantee** of which technique will be used to
+   perform an editable installation. This will vary from project to project
+   and may change depending on the specific version of ``setuptools`` being
+   used.
+
+
+----
+
+.. rubric:: Notes
+
+.. [#namespaces]
+   You *may* be able to use *strict* editable installations with namespace
+   packages created with ``pkgutil`` or ``pkg_namespaces``, however this is not
+   officially supported.
+
+.. [#installer]
+   For this workaround to work, the installer tool needs to support legacy
+   editable installations. (Future versions of ``pip``, for example, may drop
+   support for this feature).
+
+.. [#criteria]
+   ``setuptools`` strives to find a balance between allowing the user to see
+   the effects of project files being edited while still trying to keep the
+   editable installation as similar as possible to a regular installation.
+
+.. [#static_pth]
+   i.e., a ``.pth`` file where each line correspond to a path that should be
+   added to :obj:`sys.path`. See :mod:`Site-specific configuration hook <site>`.
+
+.. [#dynamic_pth]
+   i.e., a ``.pth`` file that starts where each line starts with an ``import``
+   statement and executes arbitrary Python code. See :mod:`Site-specific
+   configuration hook <site>`.
diff --git a/docs/userguide/entry_point.rst b/docs/userguide/entry_point.rst
index eff20cf0..6ba00287 100644
--- a/docs/userguide/entry_point.rst
+++ b/docs/userguide/entry_point.rst
@@ -21,6 +21,8 @@ highlighting tool :pypi:`pygments` allows specifying additional styles
 using the entry point ``pygments.styles``.
 
 
+.. _console-scripts:
+
 Console Scripts
 ===============
 
diff --git a/docs/userguide/extension.rst b/docs/userguide/extension.rst
index 31d32d61..b49816b0 100644
--- a/docs/userguide/extension.rst
+++ b/docs/userguide/extension.rst
@@ -56,8 +56,8 @@ a ``foo`` command, you might add something like this to your project:
     distutils.commands =
          foo = mypackage.some_module:foo
 
-(Assuming, of course, that the ``foo`` class in ``mypackage.some_module`` is
-a ``setuptools.Command`` subclass.)
+Assuming, of course, that the ``foo`` class in ``mypackage.some_module`` is
+a ``setuptools.Command`` subclass (documented bellow).
 
 Once a project containing such entry points has been activated on ``sys.path``,
 (e.g. by running ``pip install``) the command(s) will be available to any
@@ -72,9 +72,22 @@ Custom commands should try to replicate the same overall behavior as the
 original classes, and when possible, even inherit from them.
 
 You should also consider handling exceptions such as ``CompileError``,
-``LinkError``, ``LibError``, among others.  These exceptions are available in
+``LinkError``, ``LibError``, among others. These exceptions are available in
 the ``setuptools.errors`` module.
 
+.. autoclass:: setuptools.Command
+   :members:
+
+
+Supporting sdists and editable installs in ``build`` sub-commands
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+``build`` sub-commands (like ``build_py`` and ``build_ext``)
+are encouraged to implement the following protocol:
+
+.. autoclass:: setuptools.command.build.SubCommand
+   :members:
+
 
 Adding Arguments
 ----------------
diff --git a/docs/userguide/pyproject_config.rst b/docs/userguide/pyproject_config.rst
index 709bf919..b1d4a4e3 100644
--- a/docs/userguide/pyproject_config.rst
+++ b/docs/userguide/pyproject_config.rst
@@ -7,10 +7,10 @@ Configuring setuptools using ``pyproject.toml`` files
 .. note:: New in 61.0.0
 
 .. important::
-   For the time being [#pep660-status]_, ``pip`` still might require a ``setup.py`` file
-   to support :doc:`editable installs <pip:cli/pip_install>` [#setupcfg-caveats]_.
-
-   A simple script will suffice, for example:
+   If compatibility with legacy builds or versions of tools that don't support
+   certain packaging standards (e.g. :pep:`517` or :pep:`660`), a simple ``setup.py``
+   script can be added to your project [#setupcfg-caveats]_
+   (while keeping the configuration in ``pyproject.toml``):
 
    .. code-block:: python
 
@@ -214,12 +214,10 @@ however please keep in mind that all non-comment lines must conform with :pep:`5
 
 .. rubric:: Notes
 
-.. [#pep660-status] Editable install without ``setup.py`` will be supported in
-   future versions of ``setuptools``. Check https://github.com/pypa/setuptools/issues/2816 for detail.
-
 .. [#setupcfg-caveats] ``pip`` may allow editable install only with ``pyproject.toml``
-   and ``setup.cfg``. However, this behavior may not be consistent over various build
-   tools. Having a ``setup.py`` is still recommended if you rely on one of these tools.
+   and ``setup.cfg``. However, this behavior may not be consistent over various ``pip``
+   versions and other packaging-related tools
+   (``setup.py`` is more reliable on those scenarios).
 
 .. [#entry-points] Dynamic ``scripts`` and ``gui-scripts`` are a special case.
    When resolving these metadata keys, ``setuptools`` will look for
diff --git a/docs/userguide/quickstart.rst b/docs/userguide/quickstart.rst
index 24c71b8e..bf76f2c8 100644
--- a/docs/userguide/quickstart.rst
+++ b/docs/userguide/quickstart.rst
@@ -117,8 +117,7 @@ distributing into something that looks like the following
 (optional files marked with ``#``)::
 
     mypackage
-    ├── pyproject.toml
-    |   # setup.cfg or setup.py (depending on the configuration method)
+    ├── pyproject.toml  # and/or setup.cfg/setup.py (depending on the configuration method)
     |   # README.rst or README.md (a nice description of your package)
     |   # LICENCE (properly chosen license information, e.g. MIT, BSD-3, GPL-3, MPL-2, etc...)
     └── mypackage
@@ -378,19 +377,18 @@ Here's how to do it::
 
     pip install --editable .
 
-This creates a link file in your interpreter site package directory which
-associate with your source code. For more information, see :doc:`development_mode`.
+See :doc:`development_mode` for more information.
 
 .. tip::
 
     Prior to :ref:`pip v21.1 <pip:v21-1>`, a ``setup.py`` script was
     required to be compatible with development mode. With late
-    versions of pip, ``setup.cfg``-only projects may be installed in this mode.
+    versions of pip, projects without ``setup.py`` may be installed in this mode.
 
-    If you are experimenting with :doc:`configuration using pyproject.toml <pyproject_config>`,
-    or have version of ``pip`` older than v21.1, you might need to keep a
+    If you have a version of ``pip`` older than v21.1 or is using a different
+    packaging-related tool that does not support :pep:`660`, you might need to keep a
     ``setup.py`` file in file in your repository if you want to use editable
-    installs (for the time being).
+    installs.
 
     A simple script will suffice, for example:
 
@@ -400,8 +398,9 @@ associate with your source code. For more information, see :doc:`development_mod
 
         setup()
 
-    You can still keep all the configuration in :doc:`setup.cfg </userguide/declarative_config>`
-    (or :doc:`pyproject.toml </userguide/pyproject_config>`).
+    You can still keep all the configuration in
+    :doc:`pyproject.toml </userguide/pyproject_config>` and/or
+    :doc:`setup.cfg </userguide/declarative_config>`
 
 
 Uploading your package to PyPI