Merge pull request #93 from aaxelb/9697-granularity

[ENG-9697] index granularity

Author: aaxelb

CONTRIBUTING.md

@@ -1,48 +1,70 @@
 # Contributing
-(TODO: update)
+(TODO: expand)
 
-## Code conventions
+## code conventions
 
+### naming
 
-## Setting up local devloop
+### docstrings
 
-### 0. get the code
+### type annotations
+
+### ...
+
+## setting up local devloop
+
+### get a copy of the code
 for example:
 ```
 git clone https://github.com/CenterForOpenScience/django-elasticsearch-metrics.git djelme
 cd djelme
 ```
 
-### 1. put environment together
+### be in a fitting environment
 create and activate a [python virtual environment](https://docs.python.org/3.14/tutorial/venv.html)
-with python version 3.10+ -- there are many tools for this; you may already have your own way.
+with python version 3.10+ and [`poetry` for python package management](https://python-poetry.org) -- there are many ways to do this (including with `poetry` itself); you may already have your own way.
+
+(if you'd rather work with containers and `docker-compose`-likes, see `using docker-style container tools`, below)
 
 here's an example using `venv` (in python's standard lib) to create the virtual environment in a `.venv` directory, with bash:
 ```
 python -m venv .venv && source .venv/bin/activate
 ```
 
-install dependencies (listed in `pyproject.toml`), including the `dev` extra (and maybe `anydjango`) -- here's an example using `pip`:
+install [poetry](https://python-poetry.org):
 ```
-pip install -e '.[dev,anydjango]'
+pip install poetry
 ```
 
-### Run tests and checks
+install the current project (in editable mode) and dependencies (with the `dev` dependency group and the extras `elastic6`, `elastic8`, `anydjango`):
+```
+poetry install --with=dev --extras=elastic6 --extras=elastic8 --extras=anydjango
+```
+
+### run tests and checks
 
 these expect elasticsearches to be running and configured in `elasticsearch_metrics/tests/settings.py` (or set environment variables `ELASTICSEARCH6_URL` and `ELASTICSEARCH8_URL`) -- see `using docker-style container tools`, below, for one way to do that
 
 running the python module `elasticsearch_metrics.tests` will run tests and linting checks
--- should always pass before merging to `main`
+-- any code merged to `main` should pass for all supported python and django combinations
 
 ```
 python -m elasticsearch_metrics.tests
 ```
+without args, equivalent to `--test --lint`
+
 optional args:
-- `--devloop`: adds `--failfast --pdb` to test args
-- `--coverage`: print code coverage report
-- `--test`: run only tests (not linting)
-- `--lint`: run only linting checks (not tests)
+- `--test`: run tests and display code coverage report
+- `--lint`: run linting checks
+- `--autofix`: autofix linting errors; autoformat code
+- `--no-coverage`: skip gathering and reporting code coverage
+
+any additional args are passed thru to [django-admin test](https://docs.djangoproject.com/en/5.2/ref/django-admin/#test)
 
+example devloop:
+```
+python -m elasticsearch_metrics.tests --failfast --pdb
+```
 see `elasticsearch_metrics/tests/__main__.py` for more details
 
 ### Run the shell with:
@@ -67,7 +89,7 @@ tox
 ```
 
 ### (optional) using docker-style container tools
-see `testbox.Containerfile` and `docker-compose.yml` -- running `testbox` runs `python -m elasticsearch_metrics.tests`
+see `testbox.Containerfile` and `docker-compose.yml` for a local setup -- running `testbox` runs `python -m elasticsearch_metrics.tests`
 
 (note: examples use `pc` aliased to a `docker-compose` equivalent)
 
@@ -81,7 +103,7 @@ run tests and lint
 pc up testbox
 ```
 
-example devloop -- use current code, debugger on error, stop on failure
+example devloop -- build with current code, lint and run tests with debugger on error, stop on failure
 ```
 pc run --build --rm --no-deps testbox poetry run python -m elasticsearch_metrics.tests --devloop
 ```

README.md

@@ -14,7 +14,7 @@ python importables:
 
 * Python >=3.10
 * Django 4.2, 5.1, or 5.2
-* Elasticsearch 6 or 8
+* Elasticsearch 8 (or 6, for deprecated back-compat)
 
 ## Install
 
@@ -69,7 +69,6 @@ DJELME_AUTOSETUP = True
 
 ...or be sure to run the `djelme_backend_setup` management command before trying to store anything.
 ```shell
-# This will 
 # This will create an index template for usagerecord timeseries indexes
 python manage.py djelme_backend_setup
 ```
@@ -94,8 +93,8 @@ UsageRecord.search()
 
 ## Timeseries indexes
 
-By default, behind the scenes, a new index is created for each record type for each month
-in which a record is saved (using UTC timezone). You can change the per-index timespan by
+By default, behind the scenes, a new elasticsearch index is created for each record type for each month
+in which a record is saved (using UTC timezone). You can set a default change the per-index timespan by
 setting `Meta.timedepth` on the record type.
 
 - index per day, '...YYYY_MM_DD...': `timedepth = 3`
@@ -185,7 +184,7 @@ def test_something():
 
 ## Configuration
 
-* `DJELME_TIMESERIES_BACKENDS`: Named backends for storing or searching records from your django app
+* `DJELME_BACKENDS`: Named backends for storing or searching records from your django app
   -- nested mapping from backend name (any string, your choice) to python-importable paths
   for modules that (like `"elasticsearch_metrics.imps.elastic8"`)
   to "imp kwargs" config dictionaries given to the imp module's `djelme_backend` constructor
@@ -202,12 +201,18 @@ def test_something():
   }
   ```
 
-* `DJELME_AUTOSETUP`: Optional feature, default `False` -- set `True` for backend setup
-  (like creating index templates in elasticsearch) to run automatically when your django app starts.
-
-* `ELASTICSEARCH_METRICS_DATE_FORMAT`: Date format to use when creating
-    indexes. Default: `%Y.%m.%d` (same date format Elasticsearch uses for
-    [date math](https://www.elastic.co/guide/en/elasticsearch/reference/current/date-math-index-names.html))
+* `DJELME_AUTOSETUP`: Optional feature, default `False` --
+    set `True` to run backend setup automatically when your django app starts
+    (like creating index templates in elasticsearch, if they don't already exist)
+
+* `DJELME_DEFAULT_TIMEDEPTH`: Set the granularity of timeseries indexes by the number of "time parts" in index names
+    ```
+    DJELME_DEFAULT_TIMEDEPTH = 1  # yearly indexes; YYYY
+    DJELME_DEFAULT_TIMEDEPTH = 2  # monthly indexes; YYYY_MM
+    DJELME_DEFAULT_TIMEDEPTH = 3  # daily indexes; YYYY_MM_DD (this is the default)
+    DJELME_DEFAULT_TIMEDEPTH = 4  # hourly indexes; YYYY_MM_DD_HH
+    ```
+    you can also set `Meta.timedepth` on a specific record type; this will take precedence
 
 ## Management commands
 

elasticsearch_metrics/__init__.py

@@ -1 +1,10 @@
-__version__ = "2022.0.6"
+__all__ = (
+    "apps",
+    "exceptions",
+    "protocols",
+    "registry",
+    "signals",
+    "imps",
+    "management",
+    "util",
+)

elasticsearch_metrics/factory.py

@@ -3,6 +3,7 @@
 consider this code frozen/deprecated -- will be removed once no longer needed
 """
 
+from __future__ import annotations
 import collections
 from collections.abc import Iterator
 import dataclasses
@@ -13,11 +14,13 @@
 from django.conf import settings
 from django.utils import timezone
 from elasticsearch6.exceptions import NotFoundError
-import elasticsearch6_dsl
 from elasticsearch6_dsl import Document, connections, Date
 from elasticsearch6_dsl.document import IndexMeta, MetaField
 from elasticsearch6_dsl.index import Index
 
+# re-export all fields, for back-compat convenience
+from elasticsearch6_dsl.field import *  # noqa: F40
+
 from elasticsearch_metrics import signals
 from elasticsearch_metrics import exceptions
 from elasticsearch_metrics.protocols import ProtoDjelmeBackend
@@ -27,8 +30,6 @@
 
 logger = logging.getLogger(__name__)
 
-fields = elasticsearch6_dsl.field
-
 
 class ReadonlyAttrMap:
     def __init__(self, inner_obj):
@@ -49,7 +50,12 @@ def __setitem__(self, key, value):
 
 def _get_default_using():
     """get the elasticsearch-dsl connection name to use"""
-    (_backend_name,) = djelme_registry.each_backend_name(imp_module_name=__name__)
+    _available_backends = djelme_registry.each_backend_name(imp_module_name=__name__)
+    try:
+        (_backend_name,) = _available_backends
+    except ValueError:
+        logger.warning(f"no djelme backends configured using imp module {__name__!r}!")
+        return None
     return _backend_name
 
 
@@ -294,6 +300,10 @@ def save(self, using=None, index=None, validate=True, **kwargs):
         signals.post_save.send(cls, instance=self, using=using, index=index)
         return ret
 
+    def djelme_index_name(self) -> str:  # for ProtoDjelmeRecord
+        assert self.timestamp is not None
+        return self.get_index_name(self.timestamp)
+
     @classmethod
     def _default_index(cls, index=None):
         """Overrides Document._default_index so that .search, .get, etc.
@@ -332,6 +342,12 @@ def elastic6_client(self):
         # assumes `connections.configure` was already called
         return connections.get_connection(self.backend_name)
 
+    def djelme_backend_name(self) -> str:  # for ProtoDjelmeBackend
+        return self.backend_name
+
+    def djelme_imp_kwargs(self) -> dict[str, str]:  # for ProtoDjelmeBackend
+        return self.imp_kwargs
+
     def djelme_setup(self, recordtypes: collections.abc.Iterable[type]) -> None:
         for _metric_type in recordtypes:
             assert issubclass(_metric_type, Metric)
@@ -358,5 +374,8 @@ def djelme_when_ready(  # for ProtoDjelmeImp
     backends: Iterator[ProtoDjelmeBackend],
 ) -> None:
     connections.configure(
-        **{_backend.backend_name: _backend.imp_kwargs for _backend in backends}
+        **{
+            _backend.djelme_backend_name(): _backend.djelme_imp_kwargs()
+            for _backend in backends
+        }
     )