From f8c5383016228c77d1eab28c9b2a080777e6b1f8 Mon Sep 17 00:00:00 2001 From: Sam Thursfield Date: Thu, 25 Aug 2022 11:58:19 +0200 Subject: [PATCH 1/8] doc/source/main_install.rst: Clarify install-container section Note that special container config is needed (particularly relevant when working with CI providers). --- doc/source/main_install.rst | 24 +++++++++++++++++------- 1 file changed, 17 insertions(+), 7 deletions(-) diff --git a/doc/source/main_install.rst b/doc/source/main_install.rst index 5be8b7e7e..d759744fe 100644 --- a/doc/source/main_install.rst +++ b/doc/source/main_install.rst @@ -285,13 +285,23 @@ A regular way to do this is to add the following line to the end of your ``~/.ba .. _install-container: - Buildstream Inside a Container ------------------------------- -If your system cannot provide the base requirements, it is possible to run -BuildStream within a container. This gives you an easy way to get started -using BuildStream on any Unix-like platform where containers are available, -including macOS. -For details, see the `Buildstream Docker documentation -`_ +It is possible to run BuildStream in an OCI container tool such as Docker. +This gives you an easy way to get started using BuildStream on any Unix-like +platform where containers are available, including macOS. + +Prebuilt images are available, see the documentation +`here `_ + +You can also produce your own container images, either by adapting the +`buildstream-docker-images project `_, +or by following the full installation instructions above. + +Note that some special configuration is often needed to run BuildStream in a container: + + * User namespaces are used to isolate and control builds. This requires the + Docker ``--privileged`` mode. + * FUSE should be available in the container, achieved via the Docker + ``--device /dev/fuse`` option. From f547923dc30d4ad5ecd4e71737b848f40e319104 Mon Sep 17 00:00:00 2001 From: Sam Thursfield Date: Thu, 25 Aug 2022 12:58:10 +0200 Subject: [PATCH 2/8] doc/source/main_install.rst: Refresh intro * Recommend distro packages where possible, and link to repology * Install BuildStream before BuildBox, as the binary wheel packages of BuildStream already contain BuildBox so it's not always necessary to install BuildBox separately --- doc/source/main_install.rst | 13 ++++++++----- 1 file changed, 8 insertions(+), 5 deletions(-) diff --git a/doc/source/main_install.rst b/doc/source/main_install.rst index d759744fe..2db321d2a 100644 --- a/doc/source/main_install.rst +++ b/doc/source/main_install.rst @@ -1,16 +1,19 @@ Installing ========== -Until BuildStream is available in your distro, you may need to install -it yourself from source. The necessary steps are: +BuildStream is packaged in some Linux distributions. If your distro has an +up-to-date package we recommend using that. The table at +`repology.org `_ may be +useful. + +For full install instructions, read on: * :ref:`Install dependencies` -* :ref:`Install BuildBox` * :ref:`Install BuildStream` (from a git checkout, or from PyPi) +* :ref:`Install BuildBox` (if needed) * :ref:`Update PATH` -Alternatively, BuildStream can be run in :ref:`a container`. - +BuildStream can also be run in :ref:`a container`. .. _install-dependencies: From 92c1b904ce4afd7fd92486b4adf8b3b1a7643fc2 Mon Sep 17 00:00:00 2001 From: Sam Thursfield Date: Thu, 25 Aug 2022 13:00:01 +0200 Subject: [PATCH 3/8] doc/source/main_install.rst: Refresh 'Installing Dependencies' Provide a 'recommended' list of host tools required by source plugins - almost all real-world projects are using buildstream-plugins and the `git` source, so we should mention this up-front. Mention host tools required by BuildBox: * If buildbox-run-bwrap isn't working, builds won't work. * If buildbox-fuse cannot provide the CAS, then the BuildStream sandbox is more limited in functionality and not all projects can be built. (Specifically the buildbox-casd HardLinkStager backend stages files in the source tree as read-only, but some projects expect to be able to overwrite files in the source tree during build commands). Mention the install-time only dependencies required by Pip, if prebuilt binary packages are not available for BuildStream and the various dependencies (grpcio, protobuf, ujson, pyroaming) that contain Cython and/or bundled C++ code. --- doc/source/main_install.rst | 75 +++++++++++++++++++++++++++---------- 1 file changed, 56 insertions(+), 19 deletions(-) diff --git a/doc/source/main_install.rst b/doc/source/main_install.rst index 2db321d2a..d976f93e2 100644 --- a/doc/source/main_install.rst +++ b/doc/source/main_install.rst @@ -20,39 +20,65 @@ BuildStream can also be run in :ref:`a container`. Installing Dependencies ----------------------- -Before installing BuildStream from source, it is necessary to first install -the system dependencies. Below are some linux distribution specific instructions -for installing these dependencies. - -BuildStream requires the following base system requirements: +BuildStream has the following base system requirements: - python3 >= 3.7 -- pip -- lzip (optional, for ``.tar.lz`` support) -- :ref:`buildbox-casd` +- pip (during installation only) + +Some :mod:`Source ` plugins require additional host tools. +Here is a useful subset based on the :ref:`core source plugins ` +and `buildstream-plugins `_. + +- git (for ``git`` sources) +- lzip (for ``.tar.lz`` support in ``tar`` sources) +- patch (for ``patch`` sources) + +Some BuildBox tools used by BuildStream require additional host tools +for full functionality and performance: + +- bubblewrap (for ``buildbox-run-bubblewrap``) +- fusermount3 (for ``buildbox-fuse``) + +Additional Python dependencies will be installed via Pip in the next stage. +Several of these include some Cython code and/or bundled C++ source code. +Prebuilt binary "wheel" packages are provided for some platforms and in this +case there are no extra requirements on the host. -BuildStream also depends on the host tools for the :mod:`Source ` plugins. -Refer to the respective :ref:`source plugin ` documentation for host tool -requirements of specific plugins. +In the case that no binary package is available, Pip will try to install from +source. This implies extra install-time requirements: +- C and C++ toolchain +- Python development headers Arch Linux ~~~~~~~~~~ -Install the dependencies with:: + +Install the recommended dependencies with:: + + + sudo pacman -S bubblewrap fuse3 git lzip patch python python-pip + + +If needed, get the additional install-time dependencies with:: - sudo pacman -S python python-pip + sudo pacman -S gcc Debian ~~~~~~ -Install the dependencies with:: +Install the recommended dependencies with:: sudo apt-get install \ - python3 python3-pip python3-dev + git lzip patch bubblewrap fuse3 python3-pip python3 +If needed, get the additional install-time dependencies with:: + + + sudo apt-get install g++ python3-dev + Fedora ~~~~~~ For recent fedora systems, the following line should get you the system @@ -60,16 +86,27 @@ requirements you need:: dnf install -y \ - python3 python3-pip python3-devel + python3 python3-pip bubblewrap fuse3 lzip git patch + + +If needed, get the additional install-time dependencies with:: + + dnf install -y gcc-c++ python3-devel Ubuntu ~~~~~~ -Install the dependencies with:: +Install the recommended dependencies with:: + + + sudo apt-get install \ + git lzip patch bubblewrap fuse3 python3-pip python3 + + +If needed, get the additional install-time dependencies with:: - sudo apt install \ - python3 python3-pip python3-dev + sudo apt-get install g++ python3-dev .. _install-buildbox: From 4cce53a9615cdb60b534c72624699b5be92df309 Mon Sep 17 00:00:00 2001 From: Sam Thursfield Date: Thu, 25 Aug 2022 15:52:43 +0200 Subject: [PATCH 4/8] doc/source/main_install.rst: Refresh "Installing BuildStream" section Move this before "Installing BuildBox", as the latter may not be needed. Mention that BuildBox is bundled in the binary packages. Don't list the extra host dependencies here, refer back to previous section. BuildStream 2 is now "in beta", this sounds more complete than "dev snapshot". Clarify that pip `--editable` is optional, link to its documentation. Avoid referring to "pure Python" dependencies, at least 4 of BuildStream's Python dependencies contain Cython bundled and/or bundled C++. Avoid "simply", nothing in tech is ever simple :) --- doc/source/main_install.rst | 154 ++++++++++++++++++------------------ 1 file changed, 75 insertions(+), 79 deletions(-) diff --git a/doc/source/main_install.rst b/doc/source/main_install.rst index d976f93e2..4bf6cbe59 100644 --- a/doc/source/main_install.rst +++ b/doc/source/main_install.rst @@ -109,60 +109,6 @@ If needed, get the additional install-time dependencies with:: sudo apt-get install g++ python3-dev -.. _install-buildbox: - -Installing BuildBox -------------------- - -BuildStream master now depends on buildbox-casd to manage the local CAS cache -and communicate with CAS servers. buildbox-run is used for sandboxing. BuildBox -components are still in development and there are no stable releases yet. -Thus, they're not available yet in Linux distros and they have to be manually -installed. - -These components can be installed from binaries, or built from source. - -Install binaries -~~~~~~~~~~~~~~~~ -Linux x86-64 users can download the `latest statically linked binaries here -`_, -or browse the `release history of static binaries here -`_. - -The contents of the ``binaries.tgz`` tarball should be extracted into a directory -in ``PATH``, e.g., ``~/.local/bin``. - - -Build from source -~~~~~~~~~~~~~~~~~ - -Each of the 4 buildbox components can be installed separately from their -respective git repositiories, and each respository has individual install -instructions. Make sure that you're installing the correct version of -each component. - -| **Buildbox-common:** See the installation section in: -| https://gitlab.com/BuildGrid/buildbox/buildbox-common/-/blob/0.0.38/README.rst -| (Be sure to install from the 0.0.38 tag.) - -| **Buildbox-casd:** See the installation section in: -| https://gitlab.com/BuildGrid/buildbox/buildbox-casd/-/blob/0.0.38/README.rst \ -| (Be sure to install from the 0.0.38 tag.) - -| **Buildbox-fuse:** See -| https://gitlab.com/BuildGrid/buildbox/buildbox-fuse/-/blob/0.0.14/INSTALL.rst -| (Be sure to install from the 0.0.14 tag.) - -| **Buildbox-run-bublewrap:** See the installation section in: -| https://gitlab.com/BuildGrid/buildbox/buildbox-run-bubblewrap/-/blob/master/README.rst -| (Be sure to install from the 0.0.8 tag.) - -Finally, configure buildbox-run-bubblewrap as the default buildbox-run -implementation:: - - ln -sv buildbox-run-bubblewrap /usr/local/bin/buildbox-run - - .. _install-buildstream: Installing BuildStream @@ -170,28 +116,23 @@ Installing BuildStream Installing from PyPI ~~~~~~~~~~~~~~~~~~~~ -Once you have the base system dependencies, you can install the BuildStream -python package as a regular user. - -To install from PyPI, you will additionally require: - - - pip for python3 (only required for setup) - - Python 3 development libraries and headers - -For the latest dev snapshot of BuildStream 2, simply run the following command:: +For the latest pre-release of BuildStream 2, including the necessary Python +dependencies and BuildBox tools, run the following command:: pip3 install --user --pre BuildStream -This will install latest dev snapshot of BuildStream and its pure python -dependencies into your user's homedir in ``~/.local``. +This will install BuildStream and its dependencies into your user's homedir in +``~/.local``. Pip will use binary "wheel" packages from PyPI where these are +available for your platform. Otherwise it will build bundled C++ and Cython +code from source, which requires the additional install-time only dependencies +documented in the previous section. .. note:: - At time of writing, BuildStream 2 is only available as dev snapshots; this - is why the ``--pre`` option is required. Running - ``pip3 install --user BuildStream`` (without the ``--pre`` option) - will install Buildsteam 1. + At time of writing, BuildStream 2 is in beta; this is why the ``--pre`` + option is required. Running ``pip3 install --user BuildStream`` (without + the ``--pre`` option) will install Buildsteam 1. You can also install a specific dev snapshot of Buildstream by specifying the version. eg ``pip3 install --user BuildStream==1.93.2.dev0``. @@ -215,11 +156,8 @@ to the latest dev snapshot like so:: Installing from a git checkout ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ To install directly from the `git repository `_ -using python's ``pip`` package manager, you will additionally require: - -- pip for python3 (only required for setup) -- Python 3 development libraries and headers -- git (to checkout BuildStream) +using python's ``pip`` package manager, you will require the extra install-time +dependencies documented above. Before installing, please check the existing tags in the git repository and determine which version you want to install. @@ -232,9 +170,12 @@ Run the following commands:: git checkout pip3 install --user . -This will install BuildStream's pure python dependencies into -your user's homedir in ``~/.local`` and will run BuildStream directly -from the git checkout directory. +This will install BuildStream into your user's homedir in ``~/.local``, along +with neccessary Python dependencies fetched from PyPI. + +You can optionally use Pip's +`editable mode `_ +(the ``-e`` flag) in this case. Keep following the instructions below to ensure that the ``bst`` command is in your ``PATH`` and to enable bash completions for it. @@ -242,8 +183,9 @@ command is in your ``PATH`` and to enable bash completions for it. Upgrading from a git checkout +++++++++++++++++++++++++++++ -If you installed BuildStream from a local git checkout using ``-e`` option, all -you need to do to upgrade BuildStream is to update your local git checkout:: +If you installed BuildStream from a local git checkout using the ``-e`` +option, all you need to do to upgrade BuildStream is to update your local git +checkout:: cd /path/to/buildstream git pull --rebase @@ -299,6 +241,60 @@ If you do not want to manage your virtual environments manually, you can consider using `pipx `_. +.. _install-buildbox: + +Installing BuildBox +------------------- + +BuildStream master now depends on buildbox-casd to manage the local CAS cache +and communicate with CAS servers. buildbox-run is used for sandboxing. BuildBox +components are still in development and there are no stable releases yet. +Thus, they're not available yet in Linux distros and they have to be manually +installed. + +These components can be installed from binaries, or built from source. + +Install binaries +~~~~~~~~~~~~~~~~ +Linux x86-64 users can download the `latest statically linked binaries here +`_, +or browse the `release history of static binaries here +`_. + +The contents of the ``binaries.tgz`` tarball should be extracted into a directory +in ``PATH``, e.g., ``~/.local/bin``. + + +Build from source +~~~~~~~~~~~~~~~~~ + +Each of the 4 buildbox components can be installed separately from their +respective git repositiories, and each respository has individual install +instructions. Make sure that you're installing the correct version of +each component. + +| **Buildbox-common:** See the installation section in: +| https://gitlab.com/BuildGrid/buildbox/buildbox-common/-/blob/0.0.38/README.rst +| (Be sure to install from the 0.0.38 tag.) + +| **Buildbox-casd:** See the installation section in: +| https://gitlab.com/BuildGrid/buildbox/buildbox-casd/-/blob/0.0.38/README.rst \ +| (Be sure to install from the 0.0.38 tag.) + +| **Buildbox-fuse:** See +| https://gitlab.com/BuildGrid/buildbox/buildbox-fuse/-/blob/0.0.14/INSTALL.rst +| (Be sure to install from the 0.0.14 tag.) + +| **Buildbox-run-bublewrap:** See the installation section in: +| https://gitlab.com/BuildGrid/buildbox/buildbox-run-bubblewrap/-/blob/master/README.rst +| (Be sure to install from the 0.0.8 tag.) + +Finally, configure buildbox-run-bubblewrap as the default buildbox-run +implementation:: + + ln -sv buildbox-run-bubblewrap /usr/local/bin/buildbox-run + + .. _post-install: Post-install setup From d7fa2e0ff2e5ad041e6ff0e50f70baef91f0604a Mon Sep 17 00:00:00 2001 From: Sam Thursfield Date: Thu, 25 Aug 2022 16:44:37 +0200 Subject: [PATCH 5/8] doc/source/main_install.rst: Refresh "Installing BuildBox section" Reword intro, remove "now" and add info about the bundled binaries in wheel packages. Update buildbox-integration permalink info. Recommend use of latest stable tag, not some out of date versions. --- doc/source/main_install.rst | 47 ++++++++++++++++++++++--------------- 1 file changed, 28 insertions(+), 19 deletions(-) diff --git a/doc/source/main_install.rst b/doc/source/main_install.rst index 4bf6cbe59..577fafe23 100644 --- a/doc/source/main_install.rst +++ b/doc/source/main_install.rst @@ -246,23 +246,32 @@ consider using `pipx `_. Installing BuildBox ------------------- -BuildStream master now depends on buildbox-casd to manage the local CAS cache -and communicate with CAS servers. buildbox-run is used for sandboxing. BuildBox -components are still in development and there are no stable releases yet. -Thus, they're not available yet in Linux distros and they have to be manually -installed. +The BuildStream binary packages from PyPI contain working BuildBox binaries. +If these are installed on your system, the following command will tell you:: + + pip3 show --files buildstream | grep subprojects/buildbox + +If you see no output here, you will need to follow the below instructions to +obtain BuildBox. + +BuildStream depends on the following tools from +`BuildBox `_: + + * ``buildbox-casd`` (to manage local and remote content-addressed storage) + * ``buildbox-fuse`` (to check out content from the local CAS) + * ``buildbox-run-bubblewrap`` (to run element commands in a controlled sandbox) These components can be installed from binaries, or built from source. Install binaries ~~~~~~~~~~~~~~~~ -Linux x86-64 users can download the `latest statically linked binaries here -`_, -or browse the `release history of static binaries here +Browse the `release history of static binaries here `_. -The contents of the ``binaries.tgz`` tarball should be extracted into a directory -in ``PATH``, e.g., ``~/.local/bin``. +Linux x86-64 users can download the `latest statically linked binaries here +`_, +The contents of the tarball should be extracted into a directory in ``PATH``, +e.g., ``~/.local/bin``. Build from source @@ -270,24 +279,24 @@ Build from source Each of the 4 buildbox components can be installed separately from their respective git repositiories, and each respository has individual install -instructions. Make sure that you're installing the correct version of -each component. +instructions. We recommend installing the latest release tag of each +component. | **Buildbox-common:** See the installation section in: -| https://gitlab.com/BuildGrid/buildbox/buildbox-common/-/blob/0.0.38/README.rst -| (Be sure to install from the 0.0.38 tag.) +| https://gitlab.com/BuildGrid/buildbox/buildbox-common/-/blob/master/README.rst +| (Be sure to install from the latest stable release tag.) | **Buildbox-casd:** See the installation section in: -| https://gitlab.com/BuildGrid/buildbox/buildbox-casd/-/blob/0.0.38/README.rst \ -| (Be sure to install from the 0.0.38 tag.) +| https://gitlab.com/BuildGrid/buildbox/buildbox-casd/-/blob/master/README.rst \ +| (Be sure to install from the latest stable release tag.) | **Buildbox-fuse:** See -| https://gitlab.com/BuildGrid/buildbox/buildbox-fuse/-/blob/0.0.14/INSTALL.rst -| (Be sure to install from the 0.0.14 tag.) +| https://gitlab.com/BuildGrid/buildbox/buildbox-fuse/-/blob/master/INSTALL.rst +| (Be sure to install from the latest stable release tag.) | **Buildbox-run-bublewrap:** See the installation section in: | https://gitlab.com/BuildGrid/buildbox/buildbox-run-bubblewrap/-/blob/master/README.rst -| (Be sure to install from the 0.0.8 tag.) +| (Be sure to install from the latest stable release tag.) Finally, configure buildbox-run-bubblewrap as the default buildbox-run implementation:: From 2c1b5e6e79a431cfdffd97f0577d40c1692844c5 Mon Sep 17 00:00:00 2001 From: Sam Thursfield Date: Thu, 25 Aug 2022 16:46:16 +0200 Subject: [PATCH 6/8] doc/source/main_install.rst: Refresh "Post-install setup" Virtualenv users don't need to modify PATH, so change wording slightly. --- doc/source/main_install.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/doc/source/main_install.rst b/doc/source/main_install.rst index 577fafe23..9f8bd483b 100644 --- a/doc/source/main_install.rst +++ b/doc/source/main_install.rst @@ -310,13 +310,13 @@ Post-install setup ------------------ After having installed from source using any of the above methods, some -setup will be required to use BuildStream. +setup may be required to use BuildStream. Adjust ``PATH`` ~~~~~~~~~~~~~~~ -Since BuildStream is now installed under your local user's install directories, +If BuildStream is now installed under your local user's install directories, you need to ensure that ``PATH`` is adjusted. A regular way to do this is to add the following line to the end of your ``~/.bashrc``:: From 98a55f94d1d62e2eb07387c586fafe7134c52e80 Mon Sep 17 00:00:00 2001 From: Sam Thursfield Date: Thu, 25 Aug 2022 16:52:55 +0200 Subject: [PATCH 7/8] doc/source/core_plugins.rst: Include buildstream-plugins repo in list --- doc/source/core_plugins.rst | 1 + 1 file changed, 1 insertion(+) diff --git a/doc/source/core_plugins.rst b/doc/source/core_plugins.rst index d0eaf725f..4786ec4a4 100644 --- a/doc/source/core_plugins.rst +++ b/doc/source/core_plugins.rst @@ -55,5 +55,6 @@ running BuildStream and loaded using the :ref:`pip method ` Here is a list of BuildStream plugin projects known to us at this time: +* `buildstream-plugins `_ * `bst-plugins-experimental `_ * `bst-plugins-container `_ From e26e1077dbbc17bdd973fd6f377ad47e4b91b6c3 Mon Sep 17 00:00:00 2001 From: Sam Thursfield Date: Thu, 25 Aug 2022 18:01:59 +0200 Subject: [PATCH 8/8] doc/source/main_install.rst: Update for BuildStream 2.0 release --- doc/source/main_install.rst | 36 ++++++++++++++---------------------- 1 file changed, 14 insertions(+), 22 deletions(-) diff --git a/doc/source/main_install.rst b/doc/source/main_install.rst index 9f8bd483b..c415d6f28 100644 --- a/doc/source/main_install.rst +++ b/doc/source/main_install.rst @@ -117,10 +117,10 @@ Installing BuildStream Installing from PyPI ~~~~~~~~~~~~~~~~~~~~ -For the latest pre-release of BuildStream 2, including the necessary Python +For the latest release of BuildStream 2, including the necessary Python dependencies and BuildBox tools, run the following command:: - pip3 install --user --pre BuildStream + pip3 install --user BuildStream This will install BuildStream and its dependencies into your user's homedir in ``~/.local``. Pip will use binary "wheel" packages from PyPI where these are @@ -128,12 +128,6 @@ available for your platform. Otherwise it will build bundled C++ and Cython code from source, which requires the additional install-time only dependencies documented in the previous section. -.. note:: - - At time of writing, BuildStream 2 is in beta; this is why the ``--pre`` - option is required. Running ``pip3 install --user BuildStream`` (without - the ``--pre`` option) will install Buildsteam 1. - You can also install a specific dev snapshot of Buildstream by specifying the version. eg ``pip3 install --user BuildStream==1.93.2.dev0``. Available versions can be found on the BuildStream history page `on PyPi @@ -146,10 +140,10 @@ command is in your ``PATH``. Upgrading from PyPI +++++++++++++++++++ Once you have already installed BuildStream from PyPI, you can later update -to the latest dev snapshot like so:: +to the latest release like so:: - pip3 install --user --upgrade --pre BuildStream + pip3 install --user --upgrade BuildStream @@ -210,28 +204,26 @@ Here is how to install BuildStream stable and development snapshot releases in virtual environments of their own:: - # Install BuildStream stable in an environment called "venv-bst-stable" - # (At time of writing, this will be BuildStream 1) - python3 -m venv venv-bst-stable + # Install BuildStream 2.0 in an environment called "venv-bst2" + python3 -m venv venv-bst2 venv-bst-stable/bin/pip install BuildStream - # Install BuildStream latest development snapshot in an environment - # called "venv-bst-latest" - # (At time of writing, this will be Buildstream 2) - python3 -m venv venv-bst-latest - venv-bst-latest/bin/pip install --pre BuildStream + # Install BuildStream 1.0 in an environment called "venv-bst1" + # (At time of writing, this will be Buildstream 1) + python3 -m venv venv-bst1 + venv-bst-latest/bin/pip install BuildStream==1 To start using BuildStream from the desired environment, you will need to activate it first. Activating it will automatically add ``bst`` to your ``PATH`` and set up other necessary environment variables:: - # Use BuildStream stable from venv-bst-stable - source venv-bst-stable/bin/activate + # Use BuildStream 2.0 from venv-bst2 + source venv-bst2/bin/activate bst --version - # Use BuildStream latest from venv-bst-latest - source venv-bst-latest/bin/activate + # Use BuildStream 1.0 from venv-bst1 + source venv-bst1/bin/activate bst --version # Once you are done, remember to deactivate the virtual environment