From d88c4aa38a2cb119c36c0648baa722b35369f786 Mon Sep 17 00:00:00 2001 From: Raoul Hidalgo Charman Date: Thu, 30 May 2019 17:23:41 +0100 Subject: docs: Update configuring cache server Now that we have both artifact and source caches the documentation is updated to reflect that. Some sections headings/links etc. have been expanded and changed. Part of #1025 --- doc/source/format_project.rst | 4 +- doc/source/main_using.rst | 2 +- doc/source/using_config.rst | 6 +- doc/source/using_configuring_artifact_server.rst | 252 ---------------------- doc/source/using_configuring_cache_server.rst | 257 +++++++++++++++++++++++ 5 files changed, 264 insertions(+), 257 deletions(-) delete mode 100644 doc/source/using_configuring_artifact_server.rst create mode 100644 doc/source/using_configuring_cache_server.rst diff --git a/doc/source/format_project.rst b/doc/source/format_project.rst index 214b6c616..ac6786c79 100644 --- a/doc/source/format_project.rst +++ b/doc/source/format_project.rst @@ -189,7 +189,7 @@ for more detail. Artifact server ~~~~~~~~~~~~~~~ -If you have setup an :ref:`artifact server ` for your +If you have setup an :ref:`artifact server ` for your project then it is convenient to configure the following in your ``project.conf`` so that users need not have any additional configuration to communicate with an artifact share. @@ -274,7 +274,7 @@ using the `remote-execution` option: instance-name: development-emea-1 storage-service specifies a remote CAS store and the parameters are the -same as those used to specify an :ref:`artifact server `. +same as those used to specify an :ref:`artifact server `. The action-cache-service specifies where built actions are cached, allowing buildstream to check whether an action has already been executed and download it diff --git a/doc/source/main_using.rst b/doc/source/main_using.rst index d56023e74..48553087c 100644 --- a/doc/source/main_using.rst +++ b/doc/source/main_using.rst @@ -17,4 +17,4 @@ guides and information on user preferences and configuration. using_examples using_config using_commands - using_configuring_artifact_server + using_configuring_cache_server diff --git a/doc/source/using_config.rst b/doc/source/using_config.rst index 2b74b2de5..2582e711f 100644 --- a/doc/source/using_config.rst +++ b/doc/source/using_config.rst @@ -32,8 +32,8 @@ the supported configurations on a project wide basis are listed here. Artifact server ~~~~~~~~~~~~~~~ -Although project's often specify a :ref:`remote artifact cache ` in -their ``project.conf``, you may also want to specify extra caches. +Although project's often specify a :ref:`remote artifact cache ` +in their ``project.conf``, you may also want to specify extra caches. Assuming that your host/server is reachable on the internet as ``artifacts.com`` (for example), there are two ways to declare remote caches in your user @@ -100,6 +100,8 @@ pull only access and push/pull access. For information regarding this and the server/client certificates and keys, please see: :ref:`Key pair for the server `. +.. _config_sources: + Source cache server ~~~~~~~~~~~~~~~~~~~ Similarly global and project specific source caches servers can be specified in diff --git a/doc/source/using_configuring_artifact_server.rst b/doc/source/using_configuring_artifact_server.rst deleted file mode 100644 index 6eb64113c..000000000 --- a/doc/source/using_configuring_artifact_server.rst +++ /dev/null @@ -1,252 +0,0 @@ - - -.. _artifacts: - -Configuring Artifact Server -=========================== -BuildStream caches the results of builds in a local artifact cache, and will -avoid building an element if there is a suitable build already present in the -local artifact cache. - -In addition to the local artifact cache, you can configure one or more remote -artifact caches and BuildStream will then try to pull a suitable build from one -of the remotes, falling back to a local build if needed. - -Configuring BuildStream to use remote caches --------------------------------------------- -A project will often set up continuous build infrastructure that pushes -built artifacts to a shared cache, so developers working on the project can -make use of these pre-built artifacts instead of having to each build the whole -project locally. The project can declare this cache in its -:ref:`project configuration file `. - -Users can declare additional remote caches in the :ref:`user configuration -`. There are several use cases for this: your project may not -define its own cache, it may be useful to have a local mirror of its cache, or -you may have a reason to share artifacts privately. - -Remote artifact caches are identified by their URL. There are currently two -supported protocols: - -* ``http``: Pull and push access, without transport-layer security -* ``https``: Pull and push access, with transport-layer security - -BuildStream allows you to configure as many caches as you like, and will query -them in a specific order: - -1. Project-specific overrides in the user config -2. Project configuration -3. User configuration - -When an artifact is built locally, BuildStream will try to push it to all the -caches which have the ``push: true`` flag set. You can also manually push -artifacts to a specific cache using the :ref:`bst artifact push command `. - -Artifacts are identified using the element's :ref:`cache key ` so -the builds provided by a cache should be interchangable with those provided -by any other cache. - - -Setting up a remote artifact cache ----------------------------------- -The rest of this page outlines how to set up a shared artifact cache. - -Setting up the user -~~~~~~~~~~~~~~~~~~~ -A specific user is not needed, however, a dedicated user to own the -artifact cache is recommended. - -.. code:: bash - - useradd artifacts - -The recommended approach is to run two instances on different ports. -One instance has push disabled and doesn't require client authentication. -The other instance has push enabled and requires client authentication. - -Alternatively, you can set up a reverse proxy and handle authentication -and authorization there. - - -Installing the server -~~~~~~~~~~~~~~~~~~~~~ -You will also need to install BuildStream on the artifact server in order -to receive uploaded artifacts over ssh. Follow the instructions for installing -BuildStream `here `_. - -When installing BuildStream on the artifact server, it must be installed -in a system wide location, with ``pip3 install .`` in the BuildStream -checkout directory. - -Otherwise, some tinkering is required to ensure BuildStream is available -in ``PATH`` when its companion ``bst-artifact-server`` program is run -remotely. - -You can install only the artifact server companion program without -requiring BuildStream's more exigent dependencies by setting the -``BST_ARTIFACTS_ONLY`` environment variable at install time, like so: - -.. code:: - - BST_ARTIFACTS_ONLY=1 pip3 install . - - -.. _artifact_command_reference: - -Command reference -~~~~~~~~~~~~~~~~~ - -.. click:: buildstream._cas.casserver:server_main - :prog: bst-artifact-server - - -.. _server_authentication: - -Key pair for the server -~~~~~~~~~~~~~~~~~~~~~~~ - -For TLS you need a key pair for the server. The following example creates -a self-signed key, which requires clients to have a copy of the server certificate -(e.g., in the project directory). -You can also use a key pair obtained from a trusted certificate authority instead. - -.. code:: bash - - openssl req -new -newkey rsa:4096 -x509 -sha256 -days 3650 -nodes -batch -subj "/CN=artifacts.com" -out server.crt -keyout server.key - -.. note:: - - Note that in the ``-subj "/CN="`` argument, ``/CN`` is the *certificate common name*, - and as such ```` should be the public hostname of the server. IP addresses will - **not** provide you with working authentication. - - In addition to this, ensure that the host server is recognised by the client. - You may need to add the line: ```` ```` to - your ``/etc/hosts`` file. - -Authenticating users -~~~~~~~~~~~~~~~~~~~~ -In order to give permission to a given user to upload -artifacts, create a TLS key pair on the client. - -.. code:: bash - - openssl req -new -newkey rsa:4096 -x509 -sha256 -days 3650 -nodes -batch -subj "/CN=client" -out client.crt -keyout client.key - -Copy the public client certificate ``client.crt`` to the server and then add it -to the authorized keys, like so: - -.. code:: bash - - cat client.crt >> /home/artifacts/authorized.crt - - -Serve the cache over https -~~~~~~~~~~~~~~~~~~~~~~~~~~ - -Public instance without push: - -.. code:: bash - - bst-artifact-server --port 11001 --server-key server.key --server-cert server.crt /home/artifacts/artifacts - -Instance with push and requiring client authentication: - -.. code:: bash - - bst-artifact-server --port 11002 --server-key server.key --server-cert server.crt --client-certs authorized.crt --enable-push /home/artifacts/artifacts - -Managing the cache with systemd -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -We recommend running the cache as a systemd service, especially if it is running -on a dedicated server, as this will allow systemd to manage the cache, in case -the server encounters any issues. - -Below are two examples of how to run the cache server as a systemd service. The -first, is for pull only and the other is configured for push & pull. Notice that -the two configurations use different ports. - -``bst-artifact-serve.service``: - -.. code:: ini - - # - # Pull - # - [Unit] - Description=Buildstream Artifact pull server - After=remote-fs.target network-online.target - - [Service] - Environment="LC_ALL=C.UTF-8" - ExecStart=/usr/local/bin/bst-artifact-server --port 11001 --server-key {{certs_path}}/server.key --server-cert {{certs_path}}/server.crt {{artifacts_path}} - User=artifacts - - [Install] - WantedBy=multi-user.target - - -``bst-artifact-serve-receive.service``: - -.. code:: ini - - # - # Pull/Push - # - [Unit] - Description=Buildstream Artifact pull/push server - After=remote-fs.target network-online.target - - [Service] - Environment="LC_ALL=C.UTF-8" - ExecStart=/usr/local/bin/bst-artifact-server --port 11002 --server-key {{certs_path}}/server.key --server-cert {{certs_path}}/server.crt --client-certs {{certs_path}}/authorized.crt --enable-push {{artifacts_path}} - User=artifacts - - [Install] - WantedBy=multi-user.target - - -Here we define when systemd should start the service, which is after the networking -stack has been started, we then define how to run the cache with the desired -configuration, under the artifacts user. The {{ }} are there to denote where you -should change these files to point to your desired locations. - -.. note:: - - You may need to run some of the following commands as the superuser. - -These files should be copied to ``/etc/systemd/system/``. We can then start these services -with: - -.. code:: bash - - systemctl enable bst-artifact-serve.service - systemctl enable bst-artifact-serve-receive.service - -Then, to start these services: - -.. code:: bash - - systemctl start bst-artifact-serve.service - systemctl start bst-artifact-serve-receive.service - -We can then check if the services are successfully running with: - -.. code:: bash - - journalctl -u bst-artifact-serve.service - journalctl -u bst-artifact-serve-receive.service - -For more information on systemd services see: -`Creating Systemd Service Files `_. - -Declaring remote artifact caches -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -Remote artifact caches can be declared within either: - -1. The :ref:`project configuration `, or -2. The :ref:`user configuration `. - -Please follow the above links to see examples showing how we declare remote -caches in both the project configuration and the user configuration, respectively. diff --git a/doc/source/using_configuring_cache_server.rst b/doc/source/using_configuring_cache_server.rst new file mode 100644 index 000000000..856046f35 --- /dev/null +++ b/doc/source/using_configuring_cache_server.rst @@ -0,0 +1,257 @@ + + +.. _cache_servers: + +Configuring Cache Servers +========================= +BuildStream caches the results of builds in a local artifact cache, and will +avoid building an element if there is a suitable build already present in the +local artifact cache. Similarly it will cache sources and avoid pulling them if +present in the local cache. See :ref:`caches ` for more details. + +In addition to the local caches, you can configure one or more remote caches and +BuildStream will then try to pull a suitable object from one of the remotes, +falling back to performing a local build or fetching a source if needed. + +Configuring BuildStream to use remote caches +-------------------------------------------- +A project will often set up continuous build infrastructure that pushes +cached objects to a shared cache, so developers working on the project can +make use of these pre-made objects instead of having to each build the whole +project locally. The project can declare this cache in its +project configuration file for :ref:`artifacts ` +and :ref:`sources `. + +Users can declare additional remote caches in the :ref:`user configuration +`. There are several use cases for this: your project may not +define its own cache, it may be useful to have a local mirror of its cache, or +you may have a reason to share artifacts privately. + +Remote caches are identified by their URL. There are currently two supported +protocols: + +* ``http``: Pull and push access, without transport-layer security +* ``https``: Pull and push access, with transport-layer security + +BuildStream allows you to configure as many caches as you like, and will query +them in a specific order: + +1. Project-specific overrides in the user config +2. Project configuration +3. User configuration + +When an an object is created locally, BuildStream will try to push it to all the +caches which have the ``push: true`` flag set. You can also manually push +artifacts to a specific cache using the :ref:`bst artifact push command +`. + +Objects are identified using the element or sources :ref:`cache key ` +so the objects provided by a cache should be interchangable with those provided +by any other cache. + + +Setting up a remote cache +------------------------- +The rest of this page outlines how to set up a shared cache. + +Setting up the user +~~~~~~~~~~~~~~~~~~~ +A specific user is not needed, however, a dedicated user to own the cache is +recommended. + +.. code:: bash + + useradd artifacts + +The recommended approach is to run two instances on different ports. +One instance has push disabled and doesn't require client authentication. +The other instance has push enabled and requires client authentication. + +Alternatively, you can set up a reverse proxy and handle authentication +and authorization there. + + +Installing the server +~~~~~~~~~~~~~~~~~~~~~ +You will also need to install BuildStream on the cache server in order +to receive uploaded artifacts over ssh. Follow the instructions for installing +BuildStream `here `_. + +When installing BuildStream on the cache server, it must be installed +in a system wide location, with ``pip3 install .`` in the BuildStream +checkout directory. + +Otherwise, some tinkering is required to ensure BuildStream is available +in ``PATH`` when its companion ``bst-artifact-server`` program is run +remotely. + +You can install only the artifact server companion program without +requiring BuildStream's more exigent dependencies by setting the +``BST_ARTIFACTS_ONLY`` environment variable at install time, like so: + +.. code:: + + BST_ARTIFACTS_ONLY=1 pip3 install . + + +.. _artifact_command_reference: + +Command reference +~~~~~~~~~~~~~~~~~ + +.. click:: buildstream._cas.casserver:server_main + :prog: bst-artifact-server + + +.. _server_authentication: + +Key pair for the server +~~~~~~~~~~~~~~~~~~~~~~~ + +For TLS you need a key pair for the server. The following example creates +a self-signed key, which requires clients to have a copy of the server certificate +(e.g., in the project directory). +You can also use a key pair obtained from a trusted certificate authority instead. + +.. code:: bash + + openssl req -new -newkey rsa:4096 -x509 -sha256 -days 3650 -nodes -batch -subj "/CN=artifacts.com" -out server.crt -keyout server.key + +.. note:: + + Note that in the ``-subj "/CN="`` argument, ``/CN`` is the *certificate common name*, + and as such ```` should be the public hostname of the server. IP addresses will + **not** provide you with working authentication. + + In addition to this, ensure that the host server is recognised by the client. + You may need to add the line: ```` ```` to + your ``/etc/hosts`` file. + +Authenticating users +~~~~~~~~~~~~~~~~~~~~ +In order to give permission to a given user to upload +artifacts, create a TLS key pair on the client. + +.. code:: bash + + openssl req -new -newkey rsa:4096 -x509 -sha256 -days 3650 -nodes -batch -subj "/CN=client" -out client.crt -keyout client.key + +Copy the public client certificate ``client.crt`` to the server and then add it +to the authorized keys, like so: + +.. code:: bash + + cat client.crt >> /home/artifacts/authorized.crt + + +Serve the cache over https +~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Public instance without push: + +.. code:: bash + + bst-artifact-server --port 11001 --server-key server.key --server-cert server.crt /home/artifacts/artifacts + +Instance with push and requiring client authentication: + +.. code:: bash + + bst-artifact-server --port 11002 --server-key server.key --server-cert server.crt --client-certs authorized.crt --enable-push /home/artifacts/artifacts + +Managing the cache with systemd +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +We recommend running the cache as a systemd service, especially if it is running +on a dedicated server, as this will allow systemd to manage the cache, in case +the server encounters any issues. + +Below are two examples of how to run the cache server as a systemd service. The +first, is for pull only and the other is configured for push & pull. Notice that +the two configurations use different ports. + +``bst-artifact-serve.service``: + +.. code:: ini + + # + # Pull + # + [Unit] + Description=Buildstream Artifact pull server + After=remote-fs.target network-online.target + + [Service] + Environment="LC_ALL=C.UTF-8" + ExecStart=/usr/local/bin/bst-artifact-server --port 11001 --server-key {{certs_path}}/server.key --server-cert {{certs_path}}/server.crt {{artifacts_path}} + User=artifacts + + [Install] + WantedBy=multi-user.target + + +``bst-artifact-serve-receive.service``: + +.. code:: ini + + # + # Pull/Push + # + [Unit] + Description=Buildstream Artifact pull/push server + After=remote-fs.target network-online.target + + [Service] + Environment="LC_ALL=C.UTF-8" + ExecStart=/usr/local/bin/bst-artifact-server --port 11002 --server-key {{certs_path}}/server.key --server-cert {{certs_path}}/server.crt --client-certs {{certs_path}}/authorized.crt --enable-push {{artifacts_path}} + User=artifacts + + [Install] + WantedBy=multi-user.target + + +Here we define when systemd should start the service, which is after the networking +stack has been started, we then define how to run the cache with the desired +configuration, under the artifacts user. The {{ }} are there to denote where you +should change these files to point to your desired locations. + +.. note:: + + You may need to run some of the following commands as the superuser. + +These files should be copied to ``/etc/systemd/system/``. We can then start these services +with: + +.. code:: bash + + systemctl enable bst-artifact-serve.service + systemctl enable bst-artifact-serve-receive.service + +Then, to start these services: + +.. code:: bash + + systemctl start bst-artifact-serve.service + systemctl start bst-artifact-serve-receive.service + +We can then check if the services are successfully running with: + +.. code:: bash + + journalctl -u bst-artifact-serve.service + journalctl -u bst-artifact-serve-receive.service + +For more information on systemd services see: +`Creating Systemd Service Files `_. + +Declaring remote caches +~~~~~~~~~~~~~~~~~~~~~~~ +Remote caches can be declared within either: + +1. The project configuration for :ref:`artifact ` + and :ref:`sources `, or +2. The user configuration for :ref:`artifacts ` and + :ref:`sources `. + +Please follow the above links to see examples showing how we declare remote +caches in both the project configuration and the user configuration, respectively. -- cgit v1.2.1