Dim slicer (#120)

* Initial dim slicing: WIP no groups handling, Slicer untested as yet.

* Start of indexer testing (WIP: incomplete).

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Small improvements.

* Generalise testing + extend to Slicer tests.

* More tests; small fixes; more docs and api-docs.

whoops

tweak

* Add whatsnew.

* Add whastnew for the new utilities page.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Define .slicer for Ncdata; support full == checking on datasets and variables.

* Simplify usage modes and API of indexing utilities.

* Fix tests for simplified indexing features.

* Add tests for new core object methods.

* Fix docstrings + doctests.

* Replace dataset difference with equality tests in examples.

* Document relation between equality testing and difference utilities.

* Added whatsnew for dataset/variable equality support.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Don't keep fragments when building html.

* Improved indexing whatsnew.

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

Author: pp-mo

docs/Makefile

@@ -19,7 +19,7 @@ allapi:
 	sphinx-apidoc -Mfe -o ./details/api ../lib/ncdata
 
 towncrier:
-	towncrier build --keep
+	towncrier build --yes
 
 
 # Tweaked "make html", which restores the changelog state after docs build.

docs/changelog_fragments/161.doc.rst

@@ -0,0 +1 @@
+Added a `userguide page <userdocs/user_guide/utilities.html>`_ summarising all the utility features in :mod:`ncdata.utils`.

docs/changelog_fragments/166.feat.rst

@@ -0,0 +1,3 @@
+Provide exact == and !=  for datasets and variables, by just calling the difference utilities.
+This can be inefficient, but is simple to understand and generally useful.
+See: :ref:`equality_testing`

docs/changelog_fragments/68.feat.rst

@@ -0,0 +1,6 @@
+Added the ability to extract a sub-region by indexing/slicing over dimensions.
+The :class:`ncdata.NcData` objects can be indexed with the ``[]`` operation, or over
+specifed dimensions with the :meth:`~ncdata.NcData.slicer` method.
+This is based on the new :meth:`~ncdata.utils.index_by_dimensions()` utility method
+and :class:`~ncdata.utils.Slicer` class.
+See: :ref:`indexing_overview`

docs/userdocs/user_guide/common_operations.rst

@@ -73,6 +73,8 @@ Example :
     The utility function :func:`~ncdata.utils.rename_dimension` is provided for this.
     See : :ref:`howto_rename_dimension`.
 
+.. _copy_notes:
+
 Copying
 -------
 All core objects support a ``.copy()`` method.  See for instance
@@ -115,23 +117,24 @@ For real data, this is just ``var.data = var.data.copy()``.
 There is also a utility function :func:`ncdata.utils.ncdata_copy` :  This is
 effectively the same thing as the NcData object :meth:`~ncdata.NcData.copy` method.
 
+.. _equality_testing:
+
+Equality Testing
+----------------
+We implement equality operations ``==`` / ``!=`` for all the core data objects.
 
-Equality Checking
------------------
-We provide a simple, comprehensive  ``==`` check for :mod:`~ncdata.NcDimension` and
-:mod:`~ncdata.NcAttribute` objects, but not at present :mod:`~ncdata.NcVariable` or
-:mod:`~ncdata.NcData`.
+However, simple equality testing on :class:`@ncdata.NcData` and :class:`@ncdata.NcVariable`
+objects can be very costly if it requires comparing large data arrays.
 
-So, using ``==`` on :mod:`~ncdata.NcVariable` or :mod:`~ncdata.NcData` objects
-will only do an identity check -- that is, it tests ``id(A) == id(B)``, or ``A is B``.
+If you need to avoid comparing large (and possibly lazy) arrays then you can use the
+:func:`ncdata.utils.dataset_differences` and
+:func:`ncdata.utils.variable_differences` utility functions.
+These functions also provide multiple options to enable more tolerant comparison,
+such as allowing variables to have a different ordering.
 
-However, these objects **can** be properly compared with the dataset comparison
-utilities, :func:`ncdata.utils.dataset_differences` and
-:func:`ncdata.utils.variable_differences`.  By default, these operations are very
-comprehensive and may be very costly for instance comparing large data arrays, but they
-also allow more nuanced and controllable checking, e.g. to skip data array comparisons
-or ignore variable ordering.
+See: :ref:`utils_equality`
 
+.. _object_creation:
 
 Object Creation
 ---------------
@@ -184,8 +187,7 @@ The result is the same:
 
 .. doctest:: python
 
-    >>> from ncdata.utils import dataset_differences
-    >>> print(dataset_differences(data1, data2))
-    []
+    >>> data1 == data2
+    True
 
 

docs/userdocs/user_guide/howtos.rst

@@ -288,7 +288,7 @@ attribute already exists or not.
 .. Note::
 
     Assigning attributes when *creating* a dataset, variable or group is somewhat
-    simpler, discussed :ref:`here <todo>`.
+    simpler, discussed :ref:`here <object_creation>`.
 
 
 .. _howto_create_variable:
@@ -356,6 +356,66 @@ It can be freely overwritten by the user.
     valid dimensions, and that ``.data`` arrays match the dimensions.
 
 
+.. _howto_copy:
+
+Make a copy of data
+-------------------
+Use the :meth:`ncdata.NcData.copy` method to make a copy.
+
+.. doctest::
+
+    >>> data2 = data.copy()
+    >>> data == data2
+    True
+
+Note that this creates all-new independent ncdata objects, but all variable data arrays
+will be linked to the originals (to avoid making copies).
+
+See: :ref:`copy_notes`
+
+.. _howto_slice:
+
+Extract a subsection by indexing
+--------------------------------
+The nicest way is usually just to use the :meth:`~ncdata.Ncdata.slicer` method to specify
+dimensions to index, and then index the result.
+
+.. testsetup::
+
+    >>> from ncdata import NcData, NcDimension
+    >>> from ncdata.utils import Slicer
+    >>> full_data = NcData(dimensions=[NcDimension("x", 7), NcDimension("y", 6)])
+    >>> for nn, dim in full_data.dimensions.items():
+    ...    full_data.variables.add(NcVariable(nn, dimensions=[nn], data=np.arange(dim.size)))
+
+.. doctest::
+
+    >>> for dimname in full_data.dimensions:
+    ...     print(dimname, ':', full_data.variables[dimname].data)
+    x : [0 1 2 3 4 5 6]
+    y : [0 1 2 3 4 5]
+
+.. doctest::
+
+    >>> data_region = full_data.slicer("y", "x")[3, 1::2]
+
+.. doctest::
+
+    >>> for dimname in data_region.dimensions:
+    ...     print(dimname, ':', data_region.variables[dimname].data)
+    x : [1 3 5]
+
+You can also slice data directly, which simply acts on the dimensions in order:
+
+.. doctest::
+
+    >>> data_region_2 = full_data[1::2, 3]
+    >>> data_region_2 == data_region
+    True
+
+See: :ref:`indexing_overview`
+
+
 Read data from a NetCDF file
 ----------------------------
 Use the :func:`ncdata.netcdf4.from_nc4` function to load a dataset from a netCDF file.
@@ -658,8 +718,7 @@ In fact, there should be NO difference between these two.
 
 .. doctest:: python
 
-    >>> from ncdata.utils import dataset_differences
-    >>> print(dataset_differences(data, data2) == [])
+    >>> data == data2
     True
 
 

docs/userdocs/user_guide/user_guide.rst

@@ -9,5 +9,6 @@ Detailed explanations, beyond the basic tutorial-style introductions
     design_principles
     data_objects
     common_operations
+    utilities
     general_topics
     howtos