_project.py and docs: Move defaults into the defaults yaml file

Over time, the _project.py module has regressed into expressing
some defaults only hard coded into the python file instead of
properly exposing their default in the base configuration file
in data/projectconfig.yaml, where the default values can be observed
by users.

This patch rectifies that, and also restructures the relevant
surrounding documentation a bit.

4 files changed, 229 insertions, 193 deletions

diff --git a/buildstream/_project.py b/buildstream/_project.py
index db0b7aee9..6c71f03d3 100644
--- a/buildstream/_project.py
+++ b/buildstream/_project.py
@@ -174,7 +174,7 @@ class Project():
         self.name = _yaml.node_get(config, str, 'name')
         self.element_path = os.path.join(
             self.directory,
-            _yaml.node_get(config, str, 'element-path', default_value='.')
+            _yaml.node_get(config, str, 'element-path')
         )
 
         # Load project options
@@ -215,7 +215,7 @@ class Project():
         self._ensure_workspace_config_format()
 
         # Assert project version
-        format_version = _yaml.node_get(config, int, 'format-version', default_value=0)
+        format_version = _yaml.node_get(config, int, 'format-version')
         if BST_FORMAT_VERSION < format_version:
             major, minor = utils.get_bst_version()
             raise LoadError(
@@ -293,14 +293,12 @@ class Project():
         self._splits = _yaml.node_get(config, Mapping, 'split-rules')
 
         # Fail on overlap
-        self._fail_on_overlap = _yaml.node_get(config, bool, 'fail-on-overlap',
-                                               default_value=False)
+        self._fail_on_overlap = _yaml.node_get(config, bool, 'fail-on-overlap')
 
         # Parse shell options
-        shell_options = _yaml.node_get(config, Mapping, 'shell', default_value={})
+        shell_options = _yaml.node_get(config, Mapping, 'shell')
         _yaml.node_validate(shell_options, ['command', 'environment', 'host-files'])
-        self._shell_command = _yaml.node_get(shell_options, list, 'command',
-                                             default_value=['sh', '-i'])
+        self._shell_command = _yaml.node_get(shell_options, list, 'command')
 
         # Perform environment expansion right away
         shell_environment = _yaml.node_get(shell_options, Mapping, 'environment', default_value={})
diff --git a/buildstream/data/projectconfig.yaml b/buildstream/data/projectconfig.yaml
index f2ac83397..7bbcf69e0 100644
--- a/buildstream/data/projectconfig.yaml
+++ b/buildstream/data/projectconfig.yaml
@@ -1,6 +1,19 @@
 # Default BuildStream project configuration.
 
 
+# General configuration defaults
+#
+
+# Require format version 0
+format-version: 0
+
+# Overlaps are just warnings
+fail-on-overlap: False
+
+# Elements are found at the project root
+element-path: .
+
+
 # Variable Configuration
 #
 variables:
@@ -172,3 +185,12 @@ split-rules:
     %{datadir}/zoneinfo
   - |
     %{datadir}/zoneinfo/**
+
+
+# Default behavior for `bst shell`
+#
+shell:
+
+  # Command to run when `bst shell` does not provide a command
+  #
+  command: [ 'sh', '-i' ]
diff --git a/doc/source/formatintro.rst b/doc/source/formatintro.rst
index 35cf980bc..a1f4e8f26 100644
--- a/doc/source/formatintro.rst
+++ b/doc/source/formatintro.rst
@@ -62,23 +62,27 @@ Below are the various sources of configuration which go into an element or sourc
 order in which they are applied. Configurations which are applied later have a higher
 priority and override configurations which precede them.
 
-1. BuildStream Default Project Configuration
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
-The :ref:`builtin defaults <project_builtin_defaults>` provide a set of default values for *variables*
-and the *environment*.
+1. Builtin Defaults
+~~~~~~~~~~~~~~~~~~~
+The :ref:`builtin defaults <project_builtin_defaults>` provide a set of builtin
+default default values for ``project.conf``.
+
+The project wide defaults defined in the builtin project configuration, such as the
+*variables* or *environment* sections, form the base configuration of all elements.
 
 
 2. Project Configuration
 ~~~~~~~~~~~~~~~~~~~~~~~~
+The :ref:`project wide defaults <project_defaults>` specified in your
+``project.conf`` are now applied on top of builtin defaults.
 
-The project wide defaults in ``project.conf`` are now applied on top
-of builtin defaults.  If you specify anything in the *variables* or
-*environment* sections in your ``project.conf`` then it will override
-the builtin defaults.
+Defaults such as the :ref:`variables <project_defaults_variables>` or
+:ref:`environment <project_defaults_environment>` which are specified in
+your ``project.conf`` override the builtin defaults for elements.
 
-Note that plugin-specific configuration in ``project.conf`` is not applied
-until later.
+Note that :ref:`plugin type specific configuration <project_overrides>`
+in ``project.conf`` is not applied until later.
 
 
 3. Plugin Defaults
@@ -100,14 +104,17 @@ Source plugins do not have a ``.yaml`` file, and do not have *variables* or
 
 4. Project Configuration Overrides
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-The ``project.conf`` now gives you another opportunity to override *variables*, *environment*
-and *config* sections on a per plugin basis.
+The ``project.conf`` now gives you :ref:`another opportunity <project_overrides>` to
+override configuration on a per plugin basis.
+
+Configurations specified in the :ref:`elements <project_element_overrides>` or
+:ref:`sources <project_source_overrides>` sections of the ``project.conf``
+will override the given plugin's defaults.
 
-Configurations specified in the *elements* or *sources* sections of the ``project.conf``
-will override the given plugin's default.
+In this phase, it is possible to override any configurations of a given plugin,
+including configuration in element specific *config* sections.
 
-See also :ref:`Source Overrides<project_source_overrides>` and
-:ref:`Element Overrides<project_element_overrides>`
+See also :ref:`project_overrides`
 
 
 5. Plugin Declarations
diff --git a/doc/source/projectconf.rst b/doc/source/projectconf.rst
index c09288098..0db06fad8 100644
--- a/doc/source/projectconf.rst
+++ b/doc/source/projectconf.rst
@@ -126,168 +126,6 @@ and the order that the elements were overlapped.
   fail-on-overlap: true
 
 
-.. _project_shell:
-
-Customizing the shell
----------------------
-Since BuildStream cannot know intimate details about your host or about
-the nature of the runtime and software that you are building, the shell
-environment for debugging and testing applications may need some help.
-
-The ``shell`` section allows some customization of the shell environment.
-
-.. note::
-
-   The ``shell`` section is available since :ref:`format version 1 <project_format_version>`
-
-
-Interactive Shell Command
-~~~~~~~~~~~~~~~~~~~~~~~~~
-By default, BuildStream will use ``sh -i`` when running an interactive
-shell, unless a specific command is given to the ``bst shell`` command.
-
-BuildStream will automatically set a convenient prompt via the ``PS1``
-environment variable for interactive shells; which might be overwritten
-depending on the shell you use in your runtime.
-
-If you are using ``bash``, we recommend the following configuration to
-ensure that the customized prompt is not overwritten:
-
-.. code:: yaml
-
-   shell:
-
-     # Specify the command to run by default for interactive shells
-     command: [ 'bash', '--noprofile', '--norc', '-i' ]
-
-
-Environment Assignments
-~~~~~~~~~~~~~~~~~~~~~~~
-In order to cooperate with your host environment, a debugging shell
-sometimes needs to be configured with some extra knowledge inheriting
-from your host environment.
-
-This can be achieved by setting up the shell ``environment`` configuration,
-which is expressed as a dictionary very similar to the
-:ref:`default environment <project_defaults_environment>`, except that it
-supports host side environment variable expansion in values.
-
-.. note::
-
-   The ``environment`` configuration is available since :ref:`format version 4 <project_format_version>`
-
-For example, to share your host ``DISPLAY`` and ``DBUS_SESSION_BUS_ADDRESS``
-environments with debugging shells for your project, specify the following:
-
-.. code:: yaml
-
-   shell:
-
-     # Share some environment variables from the host environment
-     environment:
-       DISPLAY: '$DISPLAY'
-       DBUS_SESSION_BUS_ADDRESS: '$DBUS_SESSION_BUS_ADDRESS'
-
-Or, a more complex example is how one might share the host pulseaudio
-server with a ``bst shell`` environment:
-
-.. code:: yaml
-
-   shell:
-
-     # Set some environment variables explicitly
-     environment:
-       PULSE_SERVER: 'unix:${XDG_RUNTIME_DIR}/pulse/native'
-
-
-Host Files
-~~~~~~~~~~
-It can be useful to share some files on the host with a shell so that
-it can integrate better with the host environment.
-
-The ``host-files`` configuration allows one to specify files and
-directories on the host to be bind mounted into the sandbox.
-
-.. note::
-
-   The ``host-files`` configuration is available since :ref:`format version 4 <project_format_version>`
-
-.. warning::
-
-   One should never mount directories where one expects to
-   find data and files which belong to the user, such as ``/home``
-   on POSIX platforms.
-
-   This is because the unsuspecting user may corrupt their own
-   files accidentally as a result. Instead users can use the
-   ``--mount`` option of ``bst shell`` to mount data into the shell.
-
-
-The ``host-files`` configuration is an ordered list of *mount specifications*.
-
-Members of the list can be *fully specified* as a dictionary, or a simple
-string can be used if only the defaults are required.
-
-The fully specified dictionary has the following members:
-
-* ``path``
-
-  The path inside the sandbox. This is the only mandatory
-  member of the mount specification.
-
-* ``host_path``
-
-  The host path to mount at ``path`` in the sandbox. This
-  will default to ``path`` if left unspecified.
-
-* ``optional``
-
-  Whether the mount should be considered optional. This
-  is ``False`` by default.
-
-
-Here is an example of a *fully specified mount specification*:
-
-.. code:: yaml
-
-   shell:
-
-     # Mount an arbitrary resolv.conf from the host to
-     # /etc/resolv.conf in the sandbox, and avoid any
-     # warnings if the host resolv.conf doesnt exist.
-     host-files:
-     - host_path: '/usr/local/work/etc/resolv.conf'
-       path: '/etc/resolv.conf'
-       optional: True
-
-Here is an example of using *shorthand mount specifications*:
-
-.. code:: yaml
-
-   shell:
-
-     # Specify a list of files to mount in the sandbox
-     # directory from the host.
-     #
-     # If these do not exist on the host, a warning will
-     # be issued but the shell will still be launched.
-     host-files:
-     - '/etc/passwd'
-     - '/etc/group'
-     - '/etc/resolv.conf'
-
-Host side environment variable expansion is also supported:
-
-.. code:: yaml
-
-   shell:
-
-     # Mount a host side pulseaudio server socket into
-     # the shell environment at the same location.
-     host-files:
-     - '${XDG_RUNTIME_DIR}/pulse/native'
-
-
 .. _project_plugins:
 
 External Plugins
@@ -605,8 +443,8 @@ same syntax as other Flag options.
 
 .. _project_defaults:
 
-Specifying Defaults
---------------------
+Element Default Configuration
+-----------------------------
 The ``project.conf`` plays a role in defining elements by
 providing default values and also by overriding values declared
 by plugins on a plugin wide basis.
@@ -615,6 +453,8 @@ See the :ref:`composition <format_composition>` documentation for
 more detail on how elements are composed.
 
 
+.. _project_defaults_variables:
+
 Variables
 ~~~~~~~~~
 The defaults for :ref:`Variables <format_variables>` used in your
@@ -671,13 +511,21 @@ be specified here.
        %{libdir}/lib*.la
 
 
+.. _project_overrides:
+
+Overriding Plugin Defaults
+--------------------------
+Base attributes declared by element and source plugins can be overridden
+on a project wide basis. This section explains how to make project wide
+statements which augment the configuration of an element or source plugin.
+
+
 .. _project_element_overrides:
 
 Element Overrides
 ~~~~~~~~~~~~~~~~~
-Base attributes declared by element default yaml files can be overridden
-on a project wide basis. The elements dictionary can be used to override
-variables, environments or plugin specific configuration data as shown below.
+The elements dictionary can be used to override variables, environments
+or plugin specific configuration data as shown below.
 
 
 .. code:: yaml
@@ -701,9 +549,8 @@ variables, environments or plugin specific configuration data as shown below.
 
 Source Overrides
 ~~~~~~~~~~~~~~~~
-Default values (overriding built-in defaults) can be set on a project
-wide basis. The sources dictionary can be used to override plugin specific
-configuration data as shown below.
+The sources dictionary can be used to override source plugin
+specific configuration data as shown below.
 
 
 .. code:: yaml
@@ -721,6 +568,168 @@ configuration data as shown below.
    The ``sources`` override is available since :ref:`format version 1 <project_format_version>`
 
 
+.. _project_shell:
+
+Customizing the shell
+---------------------
+Since BuildStream cannot know intimate details about your host or about
+the nature of the runtime and software that you are building, the shell
+environment for debugging and testing applications may need some help.
+
+The ``shell`` section allows some customization of the shell environment.
+
+.. note::
+
+   The ``shell`` section is available since :ref:`format version 1 <project_format_version>`
+
+
+Interactive Shell Command
+~~~~~~~~~~~~~~~~~~~~~~~~~
+By default, BuildStream will use ``sh -i`` when running an interactive
+shell, unless a specific command is given to the ``bst shell`` command.
+
+BuildStream will automatically set a convenient prompt via the ``PS1``
+environment variable for interactive shells; which might be overwritten
+depending on the shell you use in your runtime.
+
+If you are using ``bash``, we recommend the following configuration to
+ensure that the customized prompt is not overwritten:
+
+.. code:: yaml
+
+   shell:
+
+     # Specify the command to run by default for interactive shells
+     command: [ 'bash', '--noprofile', '--norc', '-i' ]
+
+
+Environment Assignments
+~~~~~~~~~~~~~~~~~~~~~~~
+In order to cooperate with your host environment, a debugging shell
+sometimes needs to be configured with some extra knowledge inheriting
+from your host environment.
+
+This can be achieved by setting up the shell ``environment`` configuration,
+which is expressed as a dictionary very similar to the
+:ref:`default environment <project_defaults_environment>`, except that it
+supports host side environment variable expansion in values.
+
+.. note::
+
+   The ``environment`` configuration is available since :ref:`format version 4 <project_format_version>`
+
+For example, to share your host ``DISPLAY`` and ``DBUS_SESSION_BUS_ADDRESS``
+environments with debugging shells for your project, specify the following:
+
+.. code:: yaml
+
+   shell:
+
+     # Share some environment variables from the host environment
+     environment:
+       DISPLAY: '$DISPLAY'
+       DBUS_SESSION_BUS_ADDRESS: '$DBUS_SESSION_BUS_ADDRESS'
+
+Or, a more complex example is how one might share the host pulseaudio
+server with a ``bst shell`` environment:
+
+.. code:: yaml
+
+   shell:
+
+     # Set some environment variables explicitly
+     environment:
+       PULSE_SERVER: 'unix:${XDG_RUNTIME_DIR}/pulse/native'
+
+
+Host Files
+~~~~~~~~~~
+It can be useful to share some files on the host with a shell so that
+it can integrate better with the host environment.
+
+The ``host-files`` configuration allows one to specify files and
+directories on the host to be bind mounted into the sandbox.
+
+.. note::
+
+   The ``host-files`` configuration is available since :ref:`format version 4 <project_format_version>`
+
+.. warning::
+
+   One should never mount directories where one expects to
+   find data and files which belong to the user, such as ``/home``
+   on POSIX platforms.
+
+   This is because the unsuspecting user may corrupt their own
+   files accidentally as a result. Instead users can use the
+   ``--mount`` option of ``bst shell`` to mount data into the shell.
+
+
+The ``host-files`` configuration is an ordered list of *mount specifications*.
+
+Members of the list can be *fully specified* as a dictionary, or a simple
+string can be used if only the defaults are required.
+
+The fully specified dictionary has the following members:
+
+* ``path``
+
+  The path inside the sandbox. This is the only mandatory
+  member of the mount specification.
+
+* ``host_path``
+
+  The host path to mount at ``path`` in the sandbox. This
+  will default to ``path`` if left unspecified.
+
+* ``optional``
+
+  Whether the mount should be considered optional. This
+  is ``False`` by default.
+
+
+Here is an example of a *fully specified mount specification*:
+
+.. code:: yaml
+
+   shell:
+
+     # Mount an arbitrary resolv.conf from the host to
+     # /etc/resolv.conf in the sandbox, and avoid any
+     # warnings if the host resolv.conf doesnt exist.
+     host-files:
+     - host_path: '/usr/local/work/etc/resolv.conf'
+       path: '/etc/resolv.conf'
+       optional: True
+
+Here is an example of using *shorthand mount specifications*:
+
+.. code:: yaml
+
+   shell:
+
+     # Specify a list of files to mount in the sandbox
+     # directory from the host.
+     #
+     # If these do not exist on the host, a warning will
+     # be issued but the shell will still be launched.
+     host-files:
+     - '/etc/passwd'
+     - '/etc/group'
+     - '/etc/resolv.conf'
+
+Host side environment variable expansion is also supported:
+
+.. code:: yaml
+
+   shell:
+
+     # Mount a host side pulseaudio server socket into
+     # the shell environment at the same location.
+     host-files:
+     - '${XDG_RUNTIME_DIR}/pulse/native'
+
+
 .. _project_builtin_defaults:
 
 Builtin Defaults