diff --git a/doc/source/core_plugins.rst b/doc/source/core_plugins.rst index d0eaf725f..4786ec4a4 100644 --- a/doc/source/core_plugins.rst +++ b/doc/source/core_plugins.rst @@ -55,5 +55,6 @@ running BuildStream and loaded using the :ref:`pip method ` Here is a list of BuildStream plugin projects known to us at this time: +* `buildstream-plugins `_ * `bst-plugins-experimental `_ * `bst-plugins-container `_ diff --git a/doc/source/main_install.rst b/doc/source/main_install.rst index 5be8b7e7e..c415d6f28 100644 --- a/doc/source/main_install.rst +++ b/doc/source/main_install.rst @@ -1,126 +1,112 @@ Installing ========== -Until BuildStream is available in your distro, you may need to install -it yourself from source. The necessary steps are: +BuildStream is packaged in some Linux distributions. If your distro has an +up-to-date package we recommend using that. The table at +`repology.org `_ may be +useful. + +For full install instructions, read on: * :ref:`Install dependencies` -* :ref:`Install BuildBox` * :ref:`Install BuildStream` (from a git checkout, or from PyPi) +* :ref:`Install BuildBox` (if needed) * :ref:`Update PATH` -Alternatively, BuildStream can be run in :ref:`a container`. - +BuildStream can also be run in :ref:`a container`. .. _install-dependencies: Installing Dependencies ----------------------- -Before installing BuildStream from source, it is necessary to first install -the system dependencies. Below are some linux distribution specific instructions -for installing these dependencies. - -BuildStream requires the following base system requirements: +BuildStream has the following base system requirements: - python3 >= 3.7 -- pip -- lzip (optional, for ``.tar.lz`` support) -- :ref:`buildbox-casd` +- pip (during installation only) -BuildStream also depends on the host tools for the :mod:`Source ` plugins. -Refer to the respective :ref:`source plugin ` documentation for host tool -requirements of specific plugins. +Some :mod:`Source ` plugins require additional host tools. +Here is a useful subset based on the :ref:`core source plugins ` +and `buildstream-plugins `_. +- git (for ``git`` sources) +- lzip (for ``.tar.lz`` support in ``tar`` sources) +- patch (for ``patch`` sources) -Arch Linux -~~~~~~~~~~ -Install the dependencies with:: +Some BuildBox tools used by BuildStream require additional host tools +for full functionality and performance: +- bubblewrap (for ``buildbox-run-bubblewrap``) +- fusermount3 (for ``buildbox-fuse``) - sudo pacman -S python python-pip +Additional Python dependencies will be installed via Pip in the next stage. +Several of these include some Cython code and/or bundled C++ source code. +Prebuilt binary "wheel" packages are provided for some platforms and in this +case there are no extra requirements on the host. +In the case that no binary package is available, Pip will try to install from +source. This implies extra install-time requirements: -Debian -~~~~~~ -Install the dependencies with:: +- C and C++ toolchain +- Python development headers +Arch Linux +~~~~~~~~~~ - sudo apt-get install \ - python3 python3-pip python3-dev +Install the recommended dependencies with:: -Fedora -~~~~~~ -For recent fedora systems, the following line should get you the system -requirements you need:: + sudo pacman -S bubblewrap fuse3 git lzip patch python python-pip - dnf install -y \ - python3 python3-pip python3-devel +If needed, get the additional install-time dependencies with:: -Ubuntu + sudo pacman -S gcc + + +Debian ~~~~~~ -Install the dependencies with:: +Install the recommended dependencies with:: - sudo apt install \ - python3 python3-pip python3-dev + sudo apt-get install \ + git lzip patch bubblewrap fuse3 python3-pip python3 -.. _install-buildbox: +If needed, get the additional install-time dependencies with:: -Installing BuildBox -------------------- -BuildStream master now depends on buildbox-casd to manage the local CAS cache -and communicate with CAS servers. buildbox-run is used for sandboxing. BuildBox -components are still in development and there are no stable releases yet. -Thus, they're not available yet in Linux distros and they have to be manually -installed. + sudo apt-get install g++ python3-dev -These components can be installed from binaries, or built from source. +Fedora +~~~~~~ +For recent fedora systems, the following line should get you the system +requirements you need:: -Install binaries -~~~~~~~~~~~~~~~~ -Linux x86-64 users can download the `latest statically linked binaries here -`_, -or browse the `release history of static binaries here -`_. -The contents of the ``binaries.tgz`` tarball should be extracted into a directory -in ``PATH``, e.g., ``~/.local/bin``. + dnf install -y \ + python3 python3-pip bubblewrap fuse3 lzip git patch -Build from source -~~~~~~~~~~~~~~~~~ +If needed, get the additional install-time dependencies with:: -Each of the 4 buildbox components can be installed separately from their -respective git repositiories, and each respository has individual install -instructions. Make sure that you're installing the correct version of -each component. + dnf install -y gcc-c++ python3-devel -| **Buildbox-common:** See the installation section in: -| https://gitlab.com/BuildGrid/buildbox/buildbox-common/-/blob/0.0.38/README.rst -| (Be sure to install from the 0.0.38 tag.) -| **Buildbox-casd:** See the installation section in: -| https://gitlab.com/BuildGrid/buildbox/buildbox-casd/-/blob/0.0.38/README.rst \ -| (Be sure to install from the 0.0.38 tag.) +Ubuntu +~~~~~~ +Install the recommended dependencies with:: -| **Buildbox-fuse:** See -| https://gitlab.com/BuildGrid/buildbox/buildbox-fuse/-/blob/0.0.14/INSTALL.rst -| (Be sure to install from the 0.0.14 tag.) -| **Buildbox-run-bublewrap:** See the installation section in: -| https://gitlab.com/BuildGrid/buildbox/buildbox-run-bubblewrap/-/blob/master/README.rst -| (Be sure to install from the 0.0.8 tag.) + sudo apt-get install \ + git lzip patch bubblewrap fuse3 python3-pip python3 -Finally, configure buildbox-run-bubblewrap as the default buildbox-run -implementation:: - ln -sv buildbox-run-bubblewrap /usr/local/bin/buildbox-run +If needed, get the additional install-time dependencies with:: + + + sudo apt-get install g++ python3-dev .. _install-buildstream: @@ -130,28 +116,17 @@ Installing BuildStream Installing from PyPI ~~~~~~~~~~~~~~~~~~~~ -Once you have the base system dependencies, you can install the BuildStream -python package as a regular user. - -To install from PyPI, you will additionally require: - - - pip for python3 (only required for setup) - - Python 3 development libraries and headers +For the latest release of BuildStream 2, including the necessary Python +dependencies and BuildBox tools, run the following command:: -For the latest dev snapshot of BuildStream 2, simply run the following command:: + pip3 install --user BuildStream - pip3 install --user --pre BuildStream - -This will install latest dev snapshot of BuildStream and its pure python -dependencies into your user's homedir in ``~/.local``. - -.. note:: - - At time of writing, BuildStream 2 is only available as dev snapshots; this - is why the ``--pre`` option is required. Running - ``pip3 install --user BuildStream`` (without the ``--pre`` option) - will install Buildsteam 1. +This will install BuildStream and its dependencies into your user's homedir in +``~/.local``. Pip will use binary "wheel" packages from PyPI where these are +available for your platform. Otherwise it will build bundled C++ and Cython +code from source, which requires the additional install-time only dependencies +documented in the previous section. You can also install a specific dev snapshot of Buildstream by specifying the version. eg ``pip3 install --user BuildStream==1.93.2.dev0``. @@ -165,21 +140,18 @@ command is in your ``PATH``. Upgrading from PyPI +++++++++++++++++++ Once you have already installed BuildStream from PyPI, you can later update -to the latest dev snapshot like so:: +to the latest release like so:: - pip3 install --user --upgrade --pre BuildStream + pip3 install --user --upgrade BuildStream Installing from a git checkout ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ To install directly from the `git repository `_ -using python's ``pip`` package manager, you will additionally require: - -- pip for python3 (only required for setup) -- Python 3 development libraries and headers -- git (to checkout BuildStream) +using python's ``pip`` package manager, you will require the extra install-time +dependencies documented above. Before installing, please check the existing tags in the git repository and determine which version you want to install. @@ -192,9 +164,12 @@ Run the following commands:: git checkout pip3 install --user . -This will install BuildStream's pure python dependencies into -your user's homedir in ``~/.local`` and will run BuildStream directly -from the git checkout directory. +This will install BuildStream into your user's homedir in ``~/.local``, along +with neccessary Python dependencies fetched from PyPI. + +You can optionally use Pip's +`editable mode `_ +(the ``-e`` flag) in this case. Keep following the instructions below to ensure that the ``bst`` command is in your ``PATH`` and to enable bash completions for it. @@ -202,8 +177,9 @@ command is in your ``PATH`` and to enable bash completions for it. Upgrading from a git checkout +++++++++++++++++++++++++++++ -If you installed BuildStream from a local git checkout using ``-e`` option, all -you need to do to upgrade BuildStream is to update your local git checkout:: +If you installed BuildStream from a local git checkout using the ``-e`` +option, all you need to do to upgrade BuildStream is to update your local git +checkout:: cd /path/to/buildstream git pull --rebase @@ -228,28 +204,26 @@ Here is how to install BuildStream stable and development snapshot releases in virtual environments of their own:: - # Install BuildStream stable in an environment called "venv-bst-stable" - # (At time of writing, this will be BuildStream 1) - python3 -m venv venv-bst-stable + # Install BuildStream 2.0 in an environment called "venv-bst2" + python3 -m venv venv-bst2 venv-bst-stable/bin/pip install BuildStream - # Install BuildStream latest development snapshot in an environment - # called "venv-bst-latest" - # (At time of writing, this will be Buildstream 2) - python3 -m venv venv-bst-latest - venv-bst-latest/bin/pip install --pre BuildStream + # Install BuildStream 1.0 in an environment called "venv-bst1" + # (At time of writing, this will be Buildstream 1) + python3 -m venv venv-bst1 + venv-bst-latest/bin/pip install BuildStream==1 To start using BuildStream from the desired environment, you will need to activate it first. Activating it will automatically add ``bst`` to your ``PATH`` and set up other necessary environment variables:: - # Use BuildStream stable from venv-bst-stable - source venv-bst-stable/bin/activate + # Use BuildStream 2.0 from venv-bst2 + source venv-bst2/bin/activate bst --version - # Use BuildStream latest from venv-bst-latest - source venv-bst-latest/bin/activate + # Use BuildStream 1.0 from venv-bst1 + source venv-bst1/bin/activate bst --version # Once you are done, remember to deactivate the virtual environment @@ -259,19 +233,82 @@ If you do not want to manage your virtual environments manually, you can consider using `pipx `_. +.. _install-buildbox: + +Installing BuildBox +------------------- + +The BuildStream binary packages from PyPI contain working BuildBox binaries. +If these are installed on your system, the following command will tell you:: + + pip3 show --files buildstream | grep subprojects/buildbox + +If you see no output here, you will need to follow the below instructions to +obtain BuildBox. + +BuildStream depends on the following tools from +`BuildBox `_: + + * ``buildbox-casd`` (to manage local and remote content-addressed storage) + * ``buildbox-fuse`` (to check out content from the local CAS) + * ``buildbox-run-bubblewrap`` (to run element commands in a controlled sandbox) + +These components can be installed from binaries, or built from source. + +Install binaries +~~~~~~~~~~~~~~~~ +Browse the `release history of static binaries here +`_. + +Linux x86-64 users can download the `latest statically linked binaries here +`_, +The contents of the tarball should be extracted into a directory in ``PATH``, +e.g., ``~/.local/bin``. + + +Build from source +~~~~~~~~~~~~~~~~~ + +Each of the 4 buildbox components can be installed separately from their +respective git repositiories, and each respository has individual install +instructions. We recommend installing the latest release tag of each +component. + +| **Buildbox-common:** See the installation section in: +| https://gitlab.com/BuildGrid/buildbox/buildbox-common/-/blob/master/README.rst +| (Be sure to install from the latest stable release tag.) + +| **Buildbox-casd:** See the installation section in: +| https://gitlab.com/BuildGrid/buildbox/buildbox-casd/-/blob/master/README.rst \ +| (Be sure to install from the latest stable release tag.) + +| **Buildbox-fuse:** See +| https://gitlab.com/BuildGrid/buildbox/buildbox-fuse/-/blob/master/INSTALL.rst +| (Be sure to install from the latest stable release tag.) + +| **Buildbox-run-bublewrap:** See the installation section in: +| https://gitlab.com/BuildGrid/buildbox/buildbox-run-bubblewrap/-/blob/master/README.rst +| (Be sure to install from the latest stable release tag.) + +Finally, configure buildbox-run-bubblewrap as the default buildbox-run +implementation:: + + ln -sv buildbox-run-bubblewrap /usr/local/bin/buildbox-run + + .. _post-install: Post-install setup ------------------ After having installed from source using any of the above methods, some -setup will be required to use BuildStream. +setup may be required to use BuildStream. Adjust ``PATH`` ~~~~~~~~~~~~~~~ -Since BuildStream is now installed under your local user's install directories, +If BuildStream is now installed under your local user's install directories, you need to ensure that ``PATH`` is adjusted. A regular way to do this is to add the following line to the end of your ``~/.bashrc``:: @@ -285,13 +322,23 @@ A regular way to do this is to add the following line to the end of your ``~/.ba .. _install-container: - Buildstream Inside a Container ------------------------------- -If your system cannot provide the base requirements, it is possible to run -BuildStream within a container. This gives you an easy way to get started -using BuildStream on any Unix-like platform where containers are available, -including macOS. -For details, see the `Buildstream Docker documentation -`_ +It is possible to run BuildStream in an OCI container tool such as Docker. +This gives you an easy way to get started using BuildStream on any Unix-like +platform where containers are available, including macOS. + +Prebuilt images are available, see the documentation +`here `_ + +You can also produce your own container images, either by adapting the +`buildstream-docker-images project `_, +or by following the full installation instructions above. + +Note that some special configuration is often needed to run BuildStream in a container: + + * User namespaces are used to isolate and control builds. This requires the + Docker ``--privileged`` mode. + * FUSE should be available in the container, achieved via the Docker + ``--device /dev/fuse`` option.