Skip to content

Commit 914ab63

Browse files
committed
Update documentation to reflect the new installation process
1 parent 17d3ed3 commit 914ab63

File tree

4 files changed

+129
-145
lines changed

4 files changed

+129
-145
lines changed

docs/Docker image.rst

Lines changed: 21 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -5,17 +5,17 @@ We provide a docker image (defined in :gh_blob:`Dockerfile`) that can be used to
55
easily get an instance of CMS running locally with all the necessary
66
dependencies. We also provide:
77

8-
* :gh_blob:`cms-test.sh`: This file uses :gh_blob:`docker-compose.test.yml` to
8+
* :gh_blob:`docker/cms-test.sh`: This file uses :gh_blob:`docker/docker-compose.test.yml` to
99
spawn a volatile database (not persisted on disk) as well as a CMS instance
1010
that automatically runs all unit tests and functional tests.
1111

12-
* :gh_blob:`cms-stresstest.sh`: Similar to `cms_test.sh` but runs the stress
12+
* :gh_blob:`docker/cms-stresstest.sh`: Similar to `cms_test.sh` but runs the stress
1313
tests instead of the unit tests. The stress test consists of: creating a
1414
database, populating it with some sample tasks, and then simulating some
1515
users logging in via ContestWebServer and repeatedly performing actions
1616
such as download task statements and submitting solutions.
1717

18-
* :gh_blob:`cms-dev.sh`: This file uses :gh_blob:`docker-compose.dev.yml` to
18+
* :gh_blob:`docker/cms-dev.sh`: This file uses :gh_blob:`docker/docker-compose.dev.yml` to
1919
spawn a database (**persisted** in the local ``.dev/postgres-data`` folder
2020
within the repository) as well as a CMS container that only runs `bash`,
2121
leaving you with a shell from where you can start cms services. Changes
@@ -24,7 +24,7 @@ dependencies. We also provide:
2424
ports are also automatically forwarded on the host machine (respectively to
2525
``15432`` for the database, and ``8888-8890`` for CMS), which allows you to
2626
access the CMS web server from your host machine, but that also means you
27-
can only use `./cms-dev.sh` for one git branch at a time, as the ports are
27+
can only use `docker/cms-dev.sh` for one git branch at a time, as the ports are
2828
already in use.
2929

3030
Make sure that you have a recent version of Docker installed, as well as Docker
@@ -39,14 +39,14 @@ You can simply run:
3939

4040
.. sourcecode:: bash
4141

42-
./cms-test.sh
42+
docker/cms-test.sh
4343

44-
Or, you can issue the full command (that is defined in ``cms-test.sh``) which
44+
Or, you can issue the full command (that is defined in ``docker/cms-test.sh``) which
4545
is similar to:
4646

4747
.. sourcecode:: bash
4848

49-
docker compose -p someprojectname -f docker-compose.test.yml run --build --rm testcms
49+
docker compose -p someprojectname -f docker/docker-compose.test.yml run --build --rm testcms
5050

5151
.. note::
5252

@@ -56,14 +56,14 @@ is similar to:
5656

5757
The ``-p`` flag is used as a namespace for the containers that will be
5858
created. When you're running tests on a separate branch, it can be useful to
59-
include the branch name there, to avoid any conflict. The ``cms-test.sh``
59+
include the branch name there, to avoid any conflict. The ``docker/cms-test.sh``
6060
script uses the **name of the current git branch** and passes it to ``-p``.
6161

6262
Note also that if you are not part of the ``docker`` group then you'll need
63-
to run every docker command with ``sudo``, including ``sudo ./cms-test.sh``.
63+
to run every docker command with ``sudo``, including ``sudo docker//cms-test.sh``.
6464
We recommend adding yourself to the ``docker`` group.
6565

66-
What the ``cms-test.sh`` command does is: first build a fresh CMS image when
66+
What the ``docker/cms-test.sh`` command does is: first build a fresh CMS image when
6767
necessary, and then create (assuming you are on the ``main`` git branch) a
6868
``main-testdb-1`` container for the database, and a
6969
``main-testcms-run-<random_string>`` container for CMS.
@@ -85,25 +85,27 @@ To run a local development instance of CMS, you can simply run:
8585

8686
.. sourcecode:: bash
8787

88-
./cms-dev.sh
88+
docker/cms-dev.sh
8989

90-
Or, you can issue the full command (that is defined in ``cms-dev.sh``) which is
90+
Or, you can issue the full command (that is defined in ``docker/cms-dev.sh``) which is
9191
similar to:
9292

9393
.. sourcecode:: bash
9494

95-
docker compose -p someprojectname -f docker-compose.dev.yml run --build --rm --service-ports devcms
95+
docker compose -p someprojectname -f docker/docker-compose.dev.yml run --build --rm --service-ports devcms
9696

9797
The command will build a fresh CMS image when necessary, and drop you into a
98-
bash prompt where the repository is mounted on ``~/cms`` for ease of
98+
bash prompt where the repository is mounted on ``~/src`` for ease of
9999
development. You can edit the code from the host (i.e. outside the container)
100-
and then reinstall CMS (``pip install .``) directly from inside the
101-
container, without having to rebuild the image every time.
100+
and then reinstall CMS (``./install.py cms``) directly from inside the
101+
container, without having to rebuild the image every time. Alternatively,
102+
you can use ``./install.py cms --editable`` to get an editable installation
103+
symlinked to the source tree.
102104

103-
Upon running ``cms-dev.sh`` for the first time, the database will initially be
105+
Upon running ``docker/cms-dev.sh`` for the first time, the database will initially be
104106
empty. You need to initialize it (notice that the following commands are
105107
indicated with a ``>>>`` prompt because they are meant to be executed **inside**
106-
the container, from the prompt that you get to after running ``cms-dev.sh``)
108+
the container, from the prompt that you get to after running ``docker/cms-dev.sh``)
107109
like so:
108110

109111
.. sourcecode:: bash
@@ -134,7 +136,7 @@ When the server is finally running, you can check (from the host machine) that
134136
the server is reachable at http://localhost:8888/
135137

136138
You can also verify that upon exiting the container's bash shell and reentering
137-
it (by running ``cms-dev.sh`` again) you won't need to re-import the contest, as
139+
it (by running ``docker/cms-dev.sh`` again) you won't need to re-import the contest, as
138140
the database is persisted on disk on the host machine. Even manually destroying
139141
and recreating the database container will retain the same data. If for some
140142
reason you need to reset the database, we recommend using the ``dropdb -h devdb

docs/Installation.rst

Lines changed: 75 additions & 120 deletions
Original file line numberDiff line numberDiff line change
@@ -1,15 +1,26 @@
11
Installation
22
************
33

4+
Overview
5+
========
6+
7+
CMS runs on Linux. We test it on Debian and Ubuntu, but any modern
8+
distribution should work, too.
9+
10+
You can run CMS as a Docker container. If you want to do so, please
11+
continue to the :doc:`container installation instructions <Docker image>`.
12+
13+
Otherwise, please follow this chapter, which explains how to install CMS
14+
and its dependencies.
15+
416
.. _installation_dependencies:
517

18+
619
Dependencies and available compilers
720
====================================
821

922
These are our requirements (in particular we highlight those that are not usually installed by default) - previous versions may or may not work:
1023

11-
* build environment for the programming languages allowed in the competition;
12-
1324
* `PostgreSQL <http://www.postgresql.org/>`_ >= 9.4;
1425

1526
.. We need 9.4 because of the JSONB data type.
@@ -18,13 +29,14 @@ These are our requirements (in particular we highlight those that are not usuall
1829

1930
* `Python <http://www.python.org/>`_ >= 3.11;
2031

21-
* `libcg <http://libcg.sourceforge.net/>`_;
32+
* `Isolate <https://github.com/ioi/isolate/>`_ >= 2.0;
2233

2334
* `TeX Live <https://www.tug.org/texlive/>`_ (only for printing);
2435

2536
* `a2ps <https://www.gnu.org/software/a2ps/>`_ (only for printing).
2637

2738
You will also require a Linux kernel with support for `cgroupv2 <https://docs.kernel.org/admin-guide/cgroup-v2.html>`_.
39+
Most Linux distributions provide such kernels by default.
2840

2941
Then you require the compilation and execution environments for the languages you will use in your contest:
3042

@@ -49,180 +61,123 @@ All dependencies can be installed automatically on most Linux distributions.
4961
Ubuntu
5062
------
5163

52-
On Ubuntu 24.04, one will need to run the following script to satisfy all dependencies:
64+
On Ubuntu 24.04, one will need to run the following script as root to satisfy all dependencies:
5365

5466
.. sourcecode:: bash
5567

5668
# Feel free to change OpenJDK packages with your preferred JDK.
57-
sudo apt-get install build-essential openjdk-11-jdk-headless fp-compiler \
58-
postgresql postgresql-client python3.12 cppreference-doc-en-html \
59-
cgroup-lite libcap-dev zip
69+
apt install build-essential openjdk-11-jdk-headless fp-compiler \
70+
postgresql postgresql-client \
71+
python3.12 python3.12-dev python3-pip python3-venv \
72+
libpq-dev libcups2-dev libyaml-dev libffi-dev \
73+
cppreference-doc-en-html zip curl
6074

61-
# Only if you are going to use pip/venv to install python dependencies
62-
sudo apt-get install python3.12-dev libpq-dev libcups2-dev libyaml-dev \
63-
libffi-dev python3-pip
75+
# Isolate from upstream package repository
76+
echo 'deb [arch=amd64 signed-by=/etc/apt/keyrings/isolate.asc] http://www.ucw.cz/isolate/debian/ noble-isolate main' >/etc/apt/sources.list.d/isolate.list
77+
curl https://www.ucw.cz/isolate/debian/signing-key.asc >/etc/apt/keyrings/isolate.asc
78+
apt update && apt install isolate
6479

6580
# Optional
66-
sudo apt-get install nginx-full php-cli texlive-latex-base \
81+
apt install nginx-full php-cli texlive-latex-base \
6782
a2ps ghc rustc mono-mcs pypy3
6883

6984
The above commands provide a very essential Pascal environment. Consider installing the following packages for additional units: `fp-units-base`, `fp-units-fcl`, `fp-units-misc`, `fp-units-math` and `fp-units-rtl`.
7085

7186
Arch Linux
7287
----------
7388

74-
On Arch Linux, the following command will install almost all dependencies (some
75-
of them can be found in the AUR):
89+
On Arch Linux, run the following commands as root to install almost all dependencies
90+
(some of them can be found in the AUR):
7691

7792
.. sourcecode:: bash
7893

79-
sudo pacman -S base-devel jdk8-openjdk fpc postgresql postgresql-client \
80-
python libcap
94+
pacman -S base-devel jdk8-openjdk fpc postgresql postgresql-client \
95+
python python-pip postgresql-libs libcups libyaml
8196

8297
# Install the following from AUR.
83-
# https://aur.archlinux.org/packages/libcgroup/
8498
# https://aur.archlinux.org/packages/cppreference/
85-
86-
# Only if you are going to use pip/venv to install python dependencies
87-
sudo pacman -S --needed postgresql-libs libcups libyaml python-pip
99+
# https://aur.archlinux.org/packages/isolate
88100

89101
# Optional
90-
sudo pacman -S --needed nginx php php-fpm phppgadmin texlive-core \
102+
pacman -S --needed nginx php php-fpm phppgadmin texlive-core \
91103
a2ps ghc rust mono pypy3
92104

105+
93106
Preparation steps
94107
=================
95108

96-
Download :gh_download:`CMS` |release| from GitHub as an archive, then extract it on your filesystem. You should then access the ``cms`` folder using a terminal.
97-
98-
.. warning::
99-
100-
If you decided to ``git clone`` the repository instead of downloading the archive, and you didn't use the ``--recursive`` option when cloning, then **you need** to issue the following command to fetch the source code of the sandbox:
101-
102-
.. sourcecode:: bash
103-
104-
git submodule update --init
105-
106-
In order to run CMS there are some preparation steps to run (like installing the sandbox, compiling localization files, creating the ``cmsuser``, and so on). You can either do all these steps by hand or you can run the following command:
107-
108-
.. sourcecode:: bash
109-
110-
sudo python3 prerequisites.py install
111-
112-
.. FIXME -- The following part probably does not need to be mentioned. Moreover, it would be better if isolate was just a dependency (like postgresql) to be installed separately, with its own group (e.g. 'isolate' instead of 'cmsuser'). The 'cmsuser' group could just become deprected, at that point.
113-
114-
This script will add you to the ``cmsuser`` group if you answer ``Y`` when asked. If you want to handle your groups by yourself, answer ``N`` and then run:
109+
We recommend to create a new user account for CMS, usually called ``cmsuser``:
115110

116111
.. sourcecode:: bash
117112

118-
sudo usermod -a -G cmsuser <your user>
113+
sudo useradd --user-group --create-home --comment CMS cmsuser
119114

120-
You can verify to be in the group by issuing the command:
115+
If you are using a packaged version of Isolate, you need to add ``cmsuser``
116+
to the ``isolate`` group:
121117

122118
.. sourcecode:: bash
123119

124-
groups
120+
sudo usermod -a -G isolate cmsuser
125121

126-
Remember to logout, to make the change effective.
127122

128-
.. warning::
123+
Installing CMS
124+
==============
129125

130-
Users in the group ``cmsuser`` will be able to launch the ``isolate`` program with root permission. They may exploit this to gain root privileges. It is then imperative that no untrusted user is allowed in the group ``cmsuser``.
126+
The installation of CMS should be performed as the ``cmsuser``.
131127

132-
.. _installation_updatingcms:
128+
First obtain the source code of CMS. Download :gh_download:`CMS release`
129+
|release| from GitHub as an archive, extract it and start a shell inside.
130+
Alternatively, if you like living at the bleeding edge, check out the CMS
131+
`Git repository <https://github.com/cms-dev/cms>`_ instead.
133132

133+
The preferred method of installation is using :samp:`./install.py --dir={target} cms`,
134+
which does the following:
134135

135-
Installing CMS and its Python dependencies
136-
==========================================
136+
* Creates a *target directory* of the given name. It contains a Python
137+
virtual environment and subdirectories where CMS stores its data, logs, and caches.
138+
If you omit the ``--dir`` option, CMS is installed to ``~/cms`` (``cms`` in the
139+
home directory of the current user). Make sure that it is different from the
140+
source directory.
137141

138-
There are a number of ways to install CMS and its Python dependencies:
142+
* Populates the virtual environment with CMS and Python packages on which CMS depends.
139143

140-
Method 1: Virtual environment
141-
-----------------------------
144+
* Checks that Isolate is available.
142145

143-
The recommended method to install CMS is via a `virtual environment
144-
<https://virtualenv.pypa.io/en/latest/>`_, which is an isolated Python
145-
environment that you can put wherever you like and that can be
146-
activated/deactivated at will.
146+
* Installs the sample configuration files to :samp:`{target}/etc/cms.toml`
147+
and :samp:`{target}/etc/cms_ranking.toml`.
147148

148-
You will need to create a virtual environment somewhere in your filesystem. For
149-
example, let's assume that you decided to create it under your home directory
150-
(as ``~/cms_venv``):
149+
Now you can run CMS commands from the shell directly as :samp:`{target}/bin/{command}`.
150+
It is usually more convenient to activate the virtual environment, which adds
151+
:samp:`{target}/bin` to your ``$PATH``. This can be done by adding the following line
152+
to your ``~/.profile``:
151153

152154
.. sourcecode:: bash
153155

154-
python3 -m venv ~/cms_venv
156+
source $TARGET/bin/activate
155157

156-
To activate it:
158+
(with ``$TARGET`` replaced by the path to your target directory).
157159

158-
.. sourcecode:: bash
159160

160-
source ~/cms_venv/bin/activate
161+
Development installs
162+
--------------------
161163

162-
After the activation, the ``pip`` command will *always* be available (even if it
163-
was not available globally, e.g. because you did not install it). In general,
164-
every python command (python, pip) will refer to their corresponding virtual
165-
version. So, you can install python dependencies by issuing:
166-
167-
.. sourcecode:: bash
168-
169-
pip install -r requirements.txt
170-
pip install .
171-
172-
.. note::
173-
174-
Once you finished using CMS, you can deactivate the virtual environment by
175-
issuing:
176-
177-
.. sourcecode:: bash
178-
179-
deactivate
180-
181-
Method 2: Using the Docker image
182-
--------------------------------
183-
184-
See :doc:`here <Docker image>` for more information. This method is the
185-
recommended way for running tests locally and for local development. It hasn't
186-
been tested in production yet, so use it at your own risk.
187-
188-
Method 3: Global installation with pip
189-
--------------------------------------
190-
191-
There are good reasons to install CMS and its Python dependencies via pip
192-
instead of your package manager (e.g. apt-get). For example: two different Linux
193-
distro (or two different versions of the same distro) may offer two different
194-
versions of ``python-sqlalchemy``. When using pip, you can choose to install a
195-
*specific version* of ``sqlalchemy`` that is known to work correctly with CMS.
196-
197-
Assuming you have ``pip`` installed, you can do this:
198-
199-
.. sourcecode:: bash
200-
201-
sudo pip3 install -r requirements.txt
202-
sudo pip3 install .
203-
204-
This command installs python dependencies globally. Note that on some distros, like Arch Linux, this might interfere with the system package manager. If you want to perform the installation in your home folder instead, then you can do this instead:
205-
206-
.. sourcecode:: bash
207-
208-
pip3 install --user -r requirements.txt
209-
pip3 install --user .
210-
211-
Method 4: Using your distribution's system packages
212-
---------------------------------------------------
213-
214-
You might be able to install compatible versions of the dependencies in `requirements.txt`
215-
through your distribution's packages; this method is not supported.
164+
If you want to develop CMS, you can use :samp:`./install.py --dir={target} cms --devel --editable`.
165+
This includes development dependencies. It also makes the installation linked to the
166+
source directory, so you don't need to reinstall if you edit the source.
216167

217168

218169
Configuring the worker machines
219170
===============================
220171

221-
Worker machines need to be carefully set up in order to ensure that evaluation results are valid and consistent. Just running the evaluations under isolate does not achieve this: for example, if the machine has an active swap partition, memory limit will not be honored.
222-
223-
Apart from validity, there are many possible tweaks to reduce the variability in resource usage of an evaluation.
172+
Worker machines need to be carefully set up in order to ensure that evaluation
173+
results are valid and consistent. Just running the evaluations under Isolate
174+
does not achieve this: for example, if the machine has CPU power management
175+
configured, it might affect execution time in an unpredictable way.
176+
Having an active swap partition may allow programs to evade memory limits.
224177

225-
We suggest following isolate's `guidelines <https://github.com/ioi/isolate/blob/c679ae936d8e8d64e5dab553bdf1b22261324315/isolate.1.txt#L292>`_ for reproducible results.
178+
We suggest following Isolate's `guidelines <https://www.ucw.cz/isolate/isolate.1.html#_reproducibility>`_ for reproducible results
179+
and running the ``isolate-check-environment`` command which checks your system
180+
for common issues.
226181

227182

228183
Updating CMS

0 commit comments

Comments
 (0)