Merge pull request #2734 from cta-observatory/data_format

Docs: Rename data model -> data format, update

Author: maxnoe

docs/api-reference/io/index.rst

@@ -133,11 +133,13 @@ Writing Output Files
 ====================
 
 The `DataWriter` Component allows one to write a series of events (stored in
-`ctapipe.containers.ArrayEventContainer`) to a standardized HDF5 format file
-following the data model (see :ref:`datamodels`). This includes all related datasets
-such as the instrument and simulation configuration information, simulated
-shower and image information, observed images and parameters and reconstruction
-information. It can be used in an event loop like:
+`ctapipe.containers.ArrayEventContainer`) to an HDF5 format file
+following the data format described in :ref:`dataformat`.
+This includes all related datasets such as the instrument and simulation
+configuration information, simulated shower and image information, observed
+images and parameters and reconstruction information.
+
+It can be used in an event loop like:
 
 .. code-block:: python
 

docs/changes/2738.bugfix.rst

@@ -0,0 +1 @@
+Fix inverted sign of ``DispReconstructor`` prediction.

docs/user-guide/data_models/index.rst docs/user-guide/data_format/index.rstdocs/user-guide/data_models/index.rst renamed to docs/user-guide/data_format/index.rst

@@ -1,7 +1,7 @@
-.. _datamodels:
+.. _dataformat:
 
 ***********
-Data Models
+Data Format
 ***********
 
 The ``ctapipe`` output files are using the HDF5 format files,
@@ -13,8 +13,16 @@ generated by ctapipe (e.g. the ``ctapipe-process`` tool which uses the
 
 The tables are written with `pytables <https://www.pytables.org>`_ (not ``h5py``).
 
-DL1 Data Model
-==============
+The in-memory data format is defined by the `~ctapipe.containers` structure.
+
+To read this data, there are two high-level APIs available:
+
+- Bulk-reading into astropy `~astropy.table.Table` objects using `~ctapipe.io.TableLoader`
+- Event-wise using the `~ctapipe.io.HDF5EventSource`
+
+
+DL1 Data Format
+===============
 
 This describes data that change per-event.
 The following datasets will be written to the group ``/dl1/event/`` in the  output file:
@@ -34,22 +42,22 @@ The following datasets will be written to the group ``/dl1/event/`` in the  outp
     * - ``telescope/``
       - Per-telescope Per-event information
       - (group)
-    * - ``telescope/parameters/tel_{TEL_ID:03d}``
-      - tables of image parameters (one per telescope)
-      - :py:class:`~ctapipe.containers.TelEventIndexContainer`, :py:class:`~ctapipe.containers.ImageParametersContainer`
     * - ``telescope/images/tel_{TEL_ID:03d}``
       - tables of telescope images (one per telescope)
       - :py:class:`~ctapipe.containers.TelEventIndexContainer`, :py:class:`~ctapipe.containers.DL1CameraContainer`
+    * - ``telescope/parameters/tel_{TEL_ID:03d}``
+      - tables of image parameters (one per telescope)
+      - :py:class:`~ctapipe.containers.TelEventIndexContainer`, :py:class:`~ctapipe.containers.ImageParametersContainer`
 
 
-DL2 Data Model
-==============
+DL2 Data Format
+===============
 
 This describes data that change per-event. The following datasets will be
-written to the group ``/dl2/event/stereo/<algorithm>/`` and or
-``/dl2/event/mono/<algorithm>/``, one for each reconstruction algorithm in the
-output file, where ``<algorithm>`` is the identifier of the algorithm (e.g.
-"Hillas"):
+written to the group ``/dl2/event/subarray/<algorithm>/`` and or
+``/dl2/event/telescope/<algorithm>/``, one for each reconstruction algorithm in the
+output file, where ``<algorithm>`` is the identifier of the algorithm
+(e.g. "HillasReconstructor"):
 
 .. list-table::
     :widths: 25 50 25
@@ -69,8 +77,8 @@ output file, where ``<algorithm>`` is the identifier of the algorithm (e.g.
       - :py:class:`~ctapipe.containers.EventIndexContainer`, :py:class:`~ctapipe.containers.ParticleClassificationContainer`
 
 
-Simulation Data Model
-=====================
+Simulation Data Format
+======================
 
 .. list-table::
     :header-rows: 1
@@ -92,8 +100,8 @@ Simulation Data Model
       - :py:class:`~ctapipe.containers.SimulatedShowerDistribution`
 
 
-Configuration Data Model
-========================
+Configuration Data Format
+=========================
 
 The output file should also contain serializations of the instrument, observation (if
 applicable), simulation (if applicable) configuration information, written to the

docs/user-guide/index.rst

@@ -9,6 +9,6 @@ User Guide
 
    getting-started
    tools/index
-   data_models/index
+   data_format/index
    examples_tutorials
    FAQ

src/ctapipe/reco/sklearn.py

@@ -703,7 +703,7 @@ def _predict(self, key, table):
             else:
                 prediction[valid] = valid_norms
 
-            sign_proba = self._models[key][1].predict_proba(X)[:, 0]
+            sign_proba = self._models[key][1].predict_proba(X)[:, 1]
             # proba is [0 and 1] where 0 => very certain -1, 1 => very certain 1
             # and 0.5 means random guessing either. So we transform to a score
             # where 0 means "guessing" and 1 means "very certain"