From cc9951c1bce0325f81637e52e597f937fd130905 Mon Sep 17 00:00:00 2001 From: Sam Thursfield Date: Tue, 30 Aug 2022 13:49:39 +0200 Subject: [PATCH 1/5] doc/source/core_plugins.rst: List buildstream-plugins --- doc/source/core_plugins.rst | 1 + 1 file changed, 1 insertion(+) diff --git a/doc/source/core_plugins.rst b/doc/source/core_plugins.rst index d0eaf725f..4786ec4a4 100644 --- a/doc/source/core_plugins.rst +++ b/doc/source/core_plugins.rst @@ -55,5 +55,6 @@ running BuildStream and loaded using the :ref:`pip method ` Here is a list of BuildStream plugin projects known to us at this time: +* `buildstream-plugins `_ * `bst-plugins-experimental `_ * `bst-plugins-container `_ From ef44df8b61bb9cd23cb6570c03fc2e92ea596309 Mon Sep 17 00:00:00 2001 From: Sam Thursfield Date: Tue, 30 Aug 2022 13:49:35 +0200 Subject: [PATCH 2/5] doc/source/main_install.rst: Rework as "Install from source" guide For rationale, see: https://github.com/apache/buildstream/pull/1741#issuecomment-1227820050 Changes include: * provide a list of host tools required by source plugins and by BuildBox - almost no real-world projects can be built unless these are installed. * clarify which dependencies are needed only at install time * recommend use of latest BuildBox tag, not specific outdated tags * remove "install from PyPI" instructions - this belongs on the website * remove container install docs - these belong on website * add info about Bash completions * BuildStream 2 is now "in beta", this sounds more complete than "dev snapshot". * avoid "simply", nothing in tech is ever simple :) --- doc/source/main_install.rst | 284 +++++++++++++++--------------------- 1 file changed, 114 insertions(+), 170 deletions(-) diff --git a/doc/source/main_install.rst b/doc/source/main_install.rst index 5be8b7e7e..7337a94ba 100644 --- a/doc/source/main_install.rst +++ b/doc/source/main_install.rst @@ -1,96 +1,104 @@ -Installing -========== +Installing from Source +====================== -Until BuildStream is available in your distro, you may need to install -it yourself from source. The necessary steps are: +This page explains how to build and install this version of BuildStream from +source. For general purpose installation instructions consult the +`website `_. + +For full install instructions, read on: * :ref:`Install dependencies` * :ref:`Install BuildBox` -* :ref:`Install BuildStream` (from a git checkout, or from PyPi) -* :ref:`Update PATH` - -Alternatively, BuildStream can be run in :ref:`a container`. - +* :ref:`Install BuildStream` +* :ref:`Install completions` .. _install-dependencies: Installing Dependencies ----------------------- -Before installing BuildStream from source, it is necessary to first install -the system dependencies. Below are some linux distribution specific instructions -for installing these dependencies. +Runtime requirements +~~~~~~~~~~~~~~~~~~~~ -BuildStream requires the following base system requirements: +BuildStream requires the following Python environment to run: - python3 >= 3.7 -- pip -- lzip (optional, for ``.tar.lz`` support) -- :ref:`buildbox-casd` - -BuildStream also depends on the host tools for the :mod:`Source ` plugins. -Refer to the respective :ref:`source plugin ` documentation for host tool -requirements of specific plugins. - - -Arch Linux -~~~~~~~~~~ -Install the dependencies with:: +- PyPI packages as specified in + `requirements.txt `_. +Some :mod:`Source ` plugins require specific tools installed +on the host. Here is a commonly used subset based on the +:ref:`core source plugins ` +and `buildstream-plugins `_. - sudo pacman -S python python-pip +- git (for ``git`` sources) +- lzip (for ``.tar.lz`` support in ``tar`` sources) +- patch (for ``patch`` sources) +Some BuildBox tools used by BuildStream require additional host tools: -Debian -~~~~~~ -Install the dependencies with:: +- bubblewrap (for ``buildbox-run-bubblewrap``) +- fusermount3 (for ``buildbox-fuse``) +Install-time requirements +~~~~~~~~~~~~~~~~~~~~~~~~~ - sudo apt-get install \ - python3 python3-pip python3-dev +BuildStream contains Cython code which implies the following extra +dependencies at install-time only: +- C and C++ toolchain +- Python development headers -Fedora -~~~~~~ -For recent fedora systems, the following line should get you the system -requirements you need:: +These instructions use ``pip3`` to install necessary PyPI packages. +Packagers and integrators may use a different tool and can ignore +the `pip` dependency below. +Distribution-specific guides +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - dnf install -y \ - python3 python3-pip python3-devel +This table gives you a list of packages for specific distros: +.. list-table:: -Ubuntu -~~~~~~ -Install the dependencies with:: - - - sudo apt install \ - python3 python3-pip python3-dev - + * - **Distribution** + - **Runtime requires** + - **Install requires** + * - Arch Linux + - bubblewrap fuse3 git lzip patch python + - gcc python-pip + * - Debian + - bubblewrap fuse3 git lzip patch python3 + - g++ python3-dev python3-pip + * - Fedora + - bubblewrap fuse3 git lzip patch python3 + - gcc-c++ python3-devel python3-pip + * - Ubuntu + - bubblewrap fuse3 git lzip patch python3 + - g++ python3-dev python3-pip .. _install-buildbox: Installing BuildBox ------------------- -BuildStream master now depends on buildbox-casd to manage the local CAS cache -and communicate with CAS servers. buildbox-run is used for sandboxing. BuildBox -components are still in development and there are no stable releases yet. -Thus, they're not available yet in Linux distros and they have to be manually -installed. +BuildStream depends on the following tools from +`BuildBox `_: + + * ``buildbox-casd`` (to manage local and remote content-addressed storage) + * ``buildbox-fuse`` (to check out content from the local CAS) + * ``buildbox-run-bubblewrap`` (to run element commands in a controlled sandbox) These components can be installed from binaries, or built from source. Install binaries ~~~~~~~~~~~~~~~~ -Linux x86-64 users can download the `latest statically linked binaries here -`_, -or browse the `release history of static binaries here +Browse the `release history of static binaries here `_. -The contents of the ``binaries.tgz`` tarball should be extracted into a directory -in ``PATH``, e.g., ``~/.local/bin``. +Linux x86-64 users can download the `latest statically linked binaries here +`_, +The contents of the tarball should be extracted into a directory in ``PATH``, +e.g., ``~/.local/bin``. Build from source @@ -98,24 +106,24 @@ Build from source Each of the 4 buildbox components can be installed separately from their respective git repositiories, and each respository has individual install -instructions. Make sure that you're installing the correct version of -each component. +instructions. We recommend installing the latest release tag of each +component. | **Buildbox-common:** See the installation section in: -| https://gitlab.com/BuildGrid/buildbox/buildbox-common/-/blob/0.0.38/README.rst -| (Be sure to install from the 0.0.38 tag.) +| https://gitlab.com/BuildGrid/buildbox/buildbox-common/-/blob/master/README.rst +| (Be sure to install from the latest stable release tag.) | **Buildbox-casd:** See the installation section in: -| https://gitlab.com/BuildGrid/buildbox/buildbox-casd/-/blob/0.0.38/README.rst \ -| (Be sure to install from the 0.0.38 tag.) +| https://gitlab.com/BuildGrid/buildbox/buildbox-casd/-/blob/master/README.rst \ +| (Be sure to install from the latest stable release tag.) | **Buildbox-fuse:** See -| https://gitlab.com/BuildGrid/buildbox/buildbox-fuse/-/blob/0.0.14/INSTALL.rst -| (Be sure to install from the 0.0.14 tag.) +| https://gitlab.com/BuildGrid/buildbox/buildbox-fuse/-/blob/master/INSTALL.rst +| (Be sure to install from the latest stable release tag.) | **Buildbox-run-bublewrap:** See the installation section in: | https://gitlab.com/BuildGrid/buildbox/buildbox-run-bubblewrap/-/blob/master/README.rst -| (Be sure to install from the 0.0.8 tag.) +| (Be sure to install from the latest stable release tag.) Finally, configure buildbox-run-bubblewrap as the default buildbox-run implementation:: @@ -123,99 +131,52 @@ implementation:: ln -sv buildbox-run-bubblewrap /usr/local/bin/buildbox-run -.. _install-buildstream: - -Installing BuildStream ----------------------- - -Installing from PyPI -~~~~~~~~~~~~~~~~~~~~ -Once you have the base system dependencies, you can install the BuildStream -python package as a regular user. - -To install from PyPI, you will additionally require: - - - pip for python3 (only required for setup) - - Python 3 development libraries and headers - - -For the latest dev snapshot of BuildStream 2, simply run the following command:: - - pip3 install --user --pre BuildStream - -This will install latest dev snapshot of BuildStream and its pure python -dependencies into your user's homedir in ``~/.local``. - -.. note:: - At time of writing, BuildStream 2 is only available as dev snapshots; this - is why the ``--pre`` option is required. Running - ``pip3 install --user BuildStream`` (without the ``--pre`` option) - will install Buildsteam 1. - -You can also install a specific dev snapshot of Buildstream by specifying the -version. eg ``pip3 install --user BuildStream==1.93.2.dev0``. -Available versions can be found on the BuildStream history page `on PyPi -`_. -Note that some of the oldest versions are not available on PyPI. - -Keep following the :ref:`instructions below` to ensure that the ``bst`` -command is in your ``PATH``. - -Upgrading from PyPI -+++++++++++++++++++ -Once you have already installed BuildStream from PyPI, you can later update -to the latest dev snapshot like so:: - - - pip3 install --user --upgrade --pre BuildStream - - - -Installing from a git checkout -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -To install directly from the `git repository `_ -using python's ``pip`` package manager, you will additionally require: - -- pip for python3 (only required for setup) -- Python 3 development libraries and headers -- git (to checkout BuildStream) +.. _install-buildstream: -Before installing, please check the existing tags in the git repository -and determine which version you want to install. +Installing BuildStream from a git checkout +------------------------------------------ -Run the following commands:: +First, clone the repository. Please check the existing tags in the git +repository and determine which version you want to install:: git clone https://github.com/apache/buildstream.git cd buildstream git checkout + +We recommend ``pip`` as a frontend to the underlying ``setuptools`` build +system. The following command will build and install BuildStream into your +user's homedir in ``~/.local``, and will attempt to fetch and install any +required PyPI dependencies from the internet at the same time:: + + pip3 install --user . -This will install BuildStream's pure python dependencies into -your user's homedir in ``~/.local`` and will run BuildStream directly -from the git checkout directory. +You can optionally use Pip's +`editable mode `_ +(the ``-e`` flag) as a developer convenience. -Keep following the instructions below to ensure that the ``bst`` -command is in your ``PATH`` and to enable bash completions for it. +If you want to stop Pip from fetching missing dependencies, use the +``--no-index`` and ``--no-deps`` options. +Finally, check that the ``PATH`` variable contains the ``~/.local/bin`` directory. +If it doesn't, you could add this to the end of your Bash configuration ``~/.bashrc`` +and restart Bash:: -Upgrading from a git checkout -+++++++++++++++++++++++++++++ -If you installed BuildStream from a local git checkout using ``-e`` option, all -you need to do to upgrade BuildStream is to update your local git checkout:: + export PATH="${PATH}:${HOME}/.local/bin" - cd /path/to/buildstream - git pull --rebase +Note for packagers +~~~~~~~~~~~~~~~~~~ -If you did not specify the ``-e`` option at install time, you will -need to cleanly reinstall BuildStream:: +Distro packaging standards may recommend a specific installation method +for Python packages. BuildStream can be installed with any build frontend that +supports the `PEP517 standard `_. You are +also welcome to use the underlying +`setuptools `_ build backend directly. - pip3 uninstall buildstream - cd /path/to/buildstream - git pull --rebase - pip3 install --user . +.. _install-virtual-environment: Installing in virtual environments ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -258,40 +219,23 @@ and set up other necessary environment variables:: If you do not want to manage your virtual environments manually, you can consider using `pipx `_. +.. _install-completions: -.. _post-install: - -Post-install setup ------------------- - -After having installed from source using any of the above methods, some -setup will be required to use BuildStream. - - - -Adjust ``PATH`` -~~~~~~~~~~~~~~~ -Since BuildStream is now installed under your local user's install directories, -you need to ensure that ``PATH`` is adjusted. - -A regular way to do this is to add the following line to the end of your ``~/.bashrc``:: - - export PATH="${PATH}:${HOME}/.local/bin" - -.. note:: +Installing completions +---------------------- - You will have to restart your terminal in order for these changes to take effect. +BuildStream integrates with Bash to provide helpful tab-completion. These +are provided by the script ``src/buildstream/data/bst`` which requires +separate installation. +To install for the current user, copy it into +``share/bash-completion/completions/`` inside the user's ``$XDG_DATA_HOME`` +directory (the path is usually: ``$HOME/.local/share/bash-completion/completions``). -.. _install-container: +To install completions system-wide, get the installation path as follows:: -Buildstream Inside a Container -------------------------------- -If your system cannot provide the base requirements, it is possible to run -BuildStream within a container. This gives you an easy way to get started -using BuildStream on any Unix-like platform where containers are available, -including macOS. + pkg-config --variable=completionsdir bash-completion -For details, see the `Buildstream Docker documentation -`_ +See the `bash-completion FAQ `_ +for more information. From 2c41d13e16e3057adb3fca69a92951dc215265de Mon Sep 17 00:00:00 2001 From: Sam Thursfield Date: Tue, 20 Sep 2022 14:46:50 +0200 Subject: [PATCH 3/5] doc/source/main_install.rst: Dis-recommend `pip install -e` --- doc/source/main_install.rst | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/doc/source/main_install.rst b/doc/source/main_install.rst index 7337a94ba..9bbea46ad 100644 --- a/doc/source/main_install.rst +++ b/doc/source/main_install.rst @@ -153,9 +153,8 @@ required PyPI dependencies from the internet at the same time:: pip3 install --user . -You can optionally use Pip's -`editable mode `_ -(the ``-e`` flag) as a developer convenience. +We do not recommend using Pip's `editable mode `_ +(the ``-e`` flag). See `this issue `_ for discussion. If you want to stop Pip from fetching missing dependencies, use the ``--no-index`` and ``--no-deps`` options. From 7925769d95b9b4965b6380d11f3f0afa83426f87 Mon Sep 17 00:00:00 2001 From: Sam Thursfield Date: Tue, 20 Sep 2022 14:47:10 +0200 Subject: [PATCH 4/5] doc/source/main_install.rst: Rework completions instructions --- doc/source/main_install.rst | 15 ++++++++------- 1 file changed, 8 insertions(+), 7 deletions(-) diff --git a/doc/source/main_install.rst b/doc/source/main_install.rst index 9bbea46ad..35043545c 100644 --- a/doc/source/main_install.rst +++ b/doc/source/main_install.rst @@ -224,15 +224,16 @@ Installing completions ---------------------- BuildStream integrates with Bash to provide helpful tab-completion. These -are provided by the script ``src/buildstream/data/bst`` which requires -separate installation. +are provided by the `bst` completion script, available online +`here `_ +and in your local Git clone at ``src/buildstream/data/bst``. The completion +script requires manaul installation. -To install for the current user, copy it into -``share/bash-completion/completions/`` inside the user's ``$XDG_DATA_HOME`` -directory (the path is usually: ``$HOME/.local/share/bash-completion/completions``). - -To install completions system-wide, get the installation path as follows:: +To install for the current user, paste the contents of the completion script +into the file ``~/.bash_completion``. +To install completions system-wide, copy the completion script to the system-wide +bash-completion installation path, which you can discover as follows:: pkg-config --variable=completionsdir bash-completion From ec28d17ae871350824f4dc719443651aa8048b16 Mon Sep 17 00:00:00 2001 From: Tristan Van Berkom Date: Tue, 27 Sep 2022 11:02:06 -0400 Subject: [PATCH 5/5] main_install.rst: specify dependencies from requirements.in The requirements.txt file specifies the frozen dependencies used in CI, the requirements.in are the actual requirements checked by setup.py --- doc/source/main_install.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/source/main_install.rst b/doc/source/main_install.rst index 35043545c..a95d8f30e 100644 --- a/doc/source/main_install.rst +++ b/doc/source/main_install.rst @@ -24,7 +24,7 @@ BuildStream requires the following Python environment to run: - python3 >= 3.7 - PyPI packages as specified in - `requirements.txt `_. + `requirements.in `_. Some :mod:`Source ` plugins require specific tools installed on the host. Here is a commonly used subset based on the