Skip to content

Developer Quickstart

Jump to bottom
Yoann Valeri edited this page Mar 15, 2023 · 22 revisions

Using Phobos

The basic installation process and usage is described in Installing Phobos.

Compiling

These instructions describe how to compile Phobos.

Dependencies

An always up-to-date dependency list can be retrieved from the Phobos specfile.

Build dependencies:

  • postgresql94-devel for CentOS 7 or postgresql-devel for CentOS 8 (from EPEL or any version compatible with Postgres 9.4)
  • glib2-devel >= 2.28
  • python3-devel for CentOS 7 or python36-devel for CentOS 8
  • jansson-devel >= 2.5
  • libini_config-devel
  • openssl-devel
  • libattr-devel
  • sg3_utils-devel
  • protobuf-c-devel
  • systemd
  • autoconf
  • automake
  • libtool
  • make

Runtime dependencies:

  • postgresql94-server for CentOS 7 or postgresql-server for CentOS 8 (from EPEL or any version compatible with Postgres 9.4), although Postgres does not have to run on the same node as phobos.
  • postgresql94-contrib for CentOS 7 or postgresql-contrib for CentOS 8
  • glib2 >= 2.28
  • jansson
  • libini_config
  • openssl
  • attr
  • libattr
  • python3 for CentOS 7 or python36 for CentOS 8
  • python-argparse if you are on CentOS 7, not needed for CentOS 8
  • python3-PyYAML for CentOS 7 or python36-PyYAML for CentOS 8
  • python3-clustershell for CentOS 7 or python36-clustershell for CentOS 8
  • python3-psycopg2 for CentOS 7 or python36-psycopg2 for CentOS 8
  • python3-tabulate for CentOS 7 or python36-tabulate for CentOS 8
  • protobuf
  • protobuf-c

Test dependencies:

  • libcmocka
  • libcmocka-devel
  • python3-pylint for CentOS 7 or python36-pylint for CentOS 8
  • git
  • bc
  • mtx for tape testing
  • kernel-devel for tape testing
  • mdadm for tape testing
  • librados2 for rados testing

Compilation

$ ./autogen.sh
$ ./configure
$ make

Testing

Setting up the database

Phobos tests use a PostgreSQL database configured with a name defined in the tests/phobos.conf file (currently, phobos_test), a user named phobos with phobos as a password. This configuration can be manually done or automatically done by invoking the following line (after the project has been successfully compiled):

$ phobos_conf=tests/phobos.conf                                                   
$ test_db="$(grep "dbname" "$phobos_conf" | awk -F 'dbname=' '{print $2}' | cut -d ' ' -f1)"
$ sudo -u postgres ./scripts/phobos_db_local setup_db -s -d "$test_db" -p phobos

Depending on your umask, you may have to make some files readable and executable for the postgres user first:

chmod a+rx ./scripts/phobos_db{,_local}

Note : For production use, manual experimentation or the integration test script acceptance.sh, the database name, user and password can be set to arbitrary values (either manually or via phobos_db_local, see ./scripts/phobos_db_local setup_db -h). In this case, you can export the following environment variable:

$ export PHOBOS_DSS_connect_string="dbname='<your db name>' user='<your user>' password='<your password>' > host=<postgresql instance hostname>"

However, tests as they are written necessitate the hardcoded credentials (except for acceptance.sh).

Running the test suite

$ make check

By default, only tests that use directories will be done. To execute tests with the rados pools. To execute tests with the tapes, you must either have a tape library available on your test node, or setup a virtual tape library, as explained below.

Setting up Ceph

If you have all the Ceph related packages installed, you can use the provided script to properly setup Ceph and RADOS for the tests:

$ sudo ci/setup_ceph.sh

Then, to execute the tests, you must first recompile the project and add --enable-rados to the ./configure call, as such:

$ ./autogen.sh
$ ./configure --enable-rados
$ make

Then, you can run the tests, including those specific for RADOS:

$ make check

Setting up a Virtual Tape Library (VTL)

To test all the tape, LTFS and SCSI related code, Phobos needs to have access to a tape library. By default, it will look for the /dev/changer device to gain access to a tape library (can be changed in configuration).

Having a real tape library to test Phobos is not very handy, this is why we tend to use a VTL (Virtual Tape Library), emulating a real tape library. As Phobos uses LTFS, the VTL needs to support physical partitioning of tapes, this is why we use Quadstorvtl instead of the opensource mhvtl.

The project was tested with QuadstorVTL 3.0.36, the RPM can be downloaded here https://www.quadstor.com/vtl-extended-edition-downloads.html.

Once QuadstorVTL is installed, run ci/setup_quadstor.sh to set up the appropriate drives and tapes for the Phobos test suite:

$ sudo ci/setup_quadstor.sh

The test suite can then be run as root (for /dev/changer access), and this time all tape related tests (test_scsi.sh, and a subpart of test_store_retry.sh and acceptance.sh) will be executed:

$ sudo make check

CI scripts

Most of the setup described in this page is summed up by the ci/run-ci.sh script.

Clone this wiki locally