Update documentation

Fixes #69
Fixes #50

Author: alvarolopez

doc/source/conf.py

@@ -16,19 +16,20 @@
 import sys
 
 sys.path.insert(0, os.path.abspath('../..'))
+sys.path.insert(0, os.path.abspath('../'))
+sys.path.insert(0, os.path.abspath('./'))
 # -- General configuration ----------------------------------------------------
 
 # Add any Sphinx extension module names here, as strings. They can be
 # extensions coming with Sphinx (named 'sphinx.ext.*') or your custom ones.
 extensions = [
     'sphinx.ext.autodoc',
-    #'sphinx.ext.intersphinx',
-    #'oslosphinx'
+    'oslo_config.sphinxconfiggen',
+    'oslo_config.sphinxext',
 ]
 
-# autodoc generation is a bit aggressive and a nuisance when doing heavy
-# text edit cycles.
-# execute "export SPHINX_DEBUG=1" in your terminal to disable
+config_generator_config_file = '../../etc/caso/caso-config-generator.conf'
+sample_config_basename = 'static/caso'
 
 # The suffix of source filenames.
 source_suffix = '.rst'

doc/source/configuration-file.rst

@@ -0,0 +1,9 @@
+cASO configuration file
+=======================
+
+.. show-options::
+   :split-namespaces:
+
+   caso
+   oslo.config
+   oslo.log

doc/source/configuration.rst

@@ -19,8 +19,49 @@ Configuration
 OpenStack Configuration
 =======================
 
-Publishing benchmark information
---------------------------------
+Apart from configuring cASO, several actions need to be performed in your
+OpenStack installation in order to be able to extract accounting records.
+
+User credentials (required)
+---------------------------
+
+In the next section you will configure an OpenStack Keystone credentials in
+order to extract the records. The cASO user has to be a member of each of the
+projects (another option is to convert that user in an administrator, but the
+former option is a safer approach) for which it is extracting the accounting.
+Otherwise, ``cASO`` will not be able to get the usages and will fail.
+
+In order to do so, we are going to setup a new role ``accounting`` a new user
+``accounting``, adding it to each of the projects with that role::
+
+    openstack role create accounting
+    openstack user create --password <password> accounting
+    # For each of the projects, add the user with the accounting role
+    openstack role add --user accounting --project <project> accounting
+
+Moreover, this user needs access to Keystone so as to extract the users
+information. In this case, we can can grant the user just the rights for
+listing the users adding the appropriate rules in your
+``/etc/keystone/policy.json`` as follows. Replace the line::
+
+      "identity:list_users": "rule:admin_required",
+
+with::
+
+      "identity:list_users": "rule:admin_required or role:accounting",
+
+Recent Keystone versions leverage a ``/etc/keystone/policy-yaml`` file, if this
+is your case, substitute the line::
+
+   "identity:list_users": "rule:admin_required"
+
+with::
+
+   "identity:list_users": "rule:admin_required or role:accounting"
+
+
+Publishing benchmark information for OpenStack flavors (optional)
+-----------------------------------------------------------------
 
 Starting with the V0.4 of the accounting record it is possible to publish
 benchmark information. In order to do so, you need to add this information to
@@ -30,13 +71,13 @@ are two different values that need to be added to the flavor:
 * The benchmark name, indicated with the ``accounting:benchmark_name`` flavor property.
 * The benchmark value, indicated with the ``accounting:benchmark_value`` flavor property.
 
-So, if you are using HEPSPEC06 and the benchmark value is ``99`` for the flavor
-``m1.foo`` you should set this as follows::
+For example, if you are using HEPSPEC06 and the benchmark value is ``99`` for
+the flavor ``m1.foo``, the benchmark information is configured as follows::
 
-    openstack flavor set --property benchmark_name="HEPSPEC06" --property accounting:benchmark_value=99 m1.foo
+    openstack flavor set --property accounting:benchmark_name="HEPSPEC06" --property accounting:benchmark_value=99 m1.foo
 
-Using different keys
-~~~~~~~~~~~~~~~~~~~~
+Using different keys to specify bechmark information
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
 If you do not want to use cASO's default flavor properties ``accounting:benchmark_name`` and
 ``accounting:benchmark_value`` (for example because you are using different benchmark types
@@ -52,48 +93,12 @@ the ``benchmark_name_key`` ``benchkark_value_key`` in the configuration file.
     properties in cASO's configuration file, please include the complete name
     (i.e. ``scope:property``).
 
-
-User credentials
-----------------
-
-The user configured in the previous section has to be a member of each of the
-project (another option is to convert that user in an administrator, but the
-former option is a safer approach) for which it is extracting the accounting.
-Otherwise, ``cASO`` will not be able to get the usages and will fail::
-
-    openstack role create accounting
-    openstack user create --password <password> accounting
-    # For each of the projects, add the user with the accounting role
-    openstack role add --user accounting --project <project> accounting
-
-Also, this user needs access to Keystone so as to extract the users
-information.
-
-* If you are using the V2 identity API, you have to give admin rights to the
-  ``accounting`` user, editing the ``/etc/keystone/policy.json`` file and
-  replacing the line::
-
-      "admin_required": "role:admin or is_admin:1 or",
-
-  with::
-
-      "admin_required": "role:admin or is_admin:1 or role:accounting",
-
-* If you are using the V3 identity API you can grant the user just the rights
-  for listing the users adding the appropriate rules in the
-  ``/etc/keystone/policy.json`` as follows. Replace the line::
-
-      "identity:list_users": "rule:admin_required",
-
-  with::
-
-      "identity:list_users": "rule:admin_required or role:accounting",
-
 cASO configuration
 ==================
 
 ``cASO`` uses a config file (default at ``/etc/caso/caso.conf``) with several
-sections. A sample file is available at ``etc/caso/caso.conf.sample``.
+sections. A sample file is available at
+:download:`etc/caso/caso.conf.sample <static/caso.conf.sample>`.
 
 ``[DEFAULT]`` section
 ---------------------
@@ -114,19 +119,23 @@ of every option. You should check at least the following options:
 * ``messengers`` (list, default: ``noop``). List of the messengers to publish
   data to. Records will be pushed to all these messengers, in order. Valid
   messengers shipped with cASO are:
-    * ``ssm`` for publishing APEL V0.4 records.
-    * ``logstash`` for publishing to Logstash.
-    * ``noop`` do nothing at all.
+
+      * ``ssm`` for publishing APEL V0.4 records.
+      * ``logstash`` for publishing to Logstash.
+      * ``noop`` do nothing at all.
+
   Note that there might be other messengers available in the system if they are
   registered into the ``caso.messenger`` entry point namespace.
 * ``mapping_file`` (default: ``/etc/caso/voms.json``). File containing the
   mapping from VOs to local projects as configured in Keystone-VOMS, in the
   form::
+
     {
         "VO": {
             "projects": ["foo", "bar"],
         }
     }
+
 * ``benchmark_name_key`` and ``benchmark_value_key``. These two configuration
   options are used by ``cASO`` to retrieve the benchmark information form the
   OpenStack flavors.
@@ -141,7 +150,6 @@ authentication, so that it is possible to use any authentication plugin that is
 available there (so starting on version 1.0 of cASO it is possible to use the
 Keystone V3 API).
 
-
 .. important::
    You need to specify the ``auth_type`` that you want to use (normally
    ``v3password`` is a good choice.
@@ -168,3 +176,16 @@ messenger. Available options:
 
 * ``host`` (default: ``localhost``), host of Logstash server.
 * ``port`` (default: ``5000``), Logstash server port.
+
+Other cASO configuration options
+--------------------------------
+
+For an exhaustive list of the defined options, please check the following page:
+
+.. toctree::
+   :maxdepth: 1
+   :titlesonly:
+
+   configuration-file
+
+

doc/source/index.rst

@@ -33,4 +33,5 @@ Contents:
 
    installation
    configuration
+   configuration-file
    usage

doc/source/installation.rst

@@ -24,8 +24,9 @@ Pre-requisites
 
 If you are planning to use ``cASO`` for generating accounting records for EGI,
 you will need a valid APEL/SSM configuration. Follow the documentation
-available at the `EGI FedCloud wiki
-<https://wiki.egi.eu/wiki/Fedcloud-tf:WorkGroups:Scenario4#Publishing_Records>`_
+available at the `EGI.eu Federated Cloud documentation
+<https://egi-federated-cloud.readthedocs.io/en/latest/federation.html#apel-and-accounting-portal>`_
+in order to set it up.
 
 Installation
 ------------
@@ -35,53 +36,17 @@ repositories and packages provided in the EGI AppDB:
 
     https://appdb.egi.eu/store/software/caso
 
-
 Manual installation
 *******************
 
-At the command line::
-
-    $ pip install caso
-
-Or, if you have virtualenvwrapper installed::
-
-    $ mkvirtualenv caso
-    $ pip install caso
-
-CentOS 6
-^^^^^^^^
-
-On CentOS 6, you can use Software Collections to install Python 2.7::
-
-    $ yum -y install centos-release-SCL
-    $ yum -y install python27
-
-There are also some dependencies of the packages used by ``cASO`` that need to
-be installed (``gcc``, ``libffi-devel`` and ``openssl-devel``)::
-
-    $ yum -y install gcc libffi-devel openssl-devel
-
-You can then install ``pip`` for that version of Python and use that to install
-``cASO``::
-
-    $ scl enable python27 bash
-    $ easy_install-2.7 pip
-    $ pip install caso
-    $ exit    # this terminates bash with the SCL python2.7
-
-In this case you can later on use ``caso-extract`` with the following command
-line::
-
-    $ scl enable python27 caso-extract
-
-Alternatively, if you want to use a virtualenv::
+Even the reccomended method is to use a package from the EGI AppDB, it is
+possible to install it from the `Python Pacakge Index
+<https://pypi.org/project/caso/>`_ as follows::
 
-    $ scl enable python27 bash
-    $ virtualenv caso
-    $ . caso/bin/activate
     $ pip install caso
-    $ exit    # this terminates bash with the SCL python2.7
 
-Running from the virtualenv::
+Or you can install it on a virtualenv::
 
-    $ scl enable python27 caso/bin/caso-extract
+    $ virtualenv --python python3 caso
+    $ source caso/bin/activate
+    (caso) $ pip install caso

doc/source/usage.rst

@@ -43,6 +43,9 @@ of time.  If not time zone is specified, UTC will be used.
 Available options
 =================
 
+Apart from other options, the following ones are the ones that specify how to
+extract accountig records:
+
 .. option:: --config-dir DIR
 
   Path to a config directory to pull `*.conf` files from. This file set is
@@ -61,6 +64,7 @@ Available options
 
   If set to true, the logging level will be set to DEBUG
                         instead of the default INFO level.
+
 .. option:: --dry-run, --dry_run
 
   Extract records but do not push records to SSM. This will not update the last
@@ -90,21 +94,9 @@ Available options
 
    List of projects to extract accounting records from.
 
-
 Running as a cron job
 ---------------------
 
 The best way of running ``cASO`` is via a cron job like the following::
 
     10 * * * * caso-extract
-
-Migration from OSSSM
---------------------
-
-If you had a previous installation of osssm, you can migrate to ``cASO``
-following these steps:
-
-#. Remove the previous osssm installation (e.g. remove ``apel-ssm-openstack`` rpm).
-#. Remove any cron jobs related to ``osssm.extract`` or ``osssm.push``, a single
-   cron job as described above is enough. You should keep the cron job that executes
-   ``ssmsend``, this is still needed to send the records to the accounting database.

etc/caso/caso-config-generator.conf

@@ -2,5 +2,5 @@
 output_file = etc/caso/caso.conf.sample
 wrap_width = 79
 namespace = caso
-namespace = caso.config
 namespace = oslo.log
+namespace = oslo.config