You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
1. the class `Active`, which is a shimmy to NetCDF4 (and HDF5) storage via kerchunk metadata and the zarr indexer. It does not however, use zarr for the actual read.
38
+
1. the class `Active`, which is a shimmy to NetCDF4 (and HDF5) via a [`Pyfive.File`](https://github.com/NCAS-CMS/pyfive) file object
39
39
2. The actual reads are done in the methods of `storage.py` or `reductionist.py`, which are called from within an `Active.__getitem__`.
40
40
41
-
Example usage is in the file `tests/test_harness.py`, but it's basically this simple:
41
+
Example usage is in the test files, depending on the case:
active = Active(file.Path | Pyfive.Dataset, ncvar="some_var")
51
+
active._version=2
52
+
result = active.mean[0:2, 4:6, 7:9]
47
53
```
48
54
49
-
where `result` will be the mean of the appropriate slice of the hyperslab in `var`.
55
+
where `result` will be the mean of the appropriate slice of the hyperslab in `some_var` variable data.
50
56
51
57
There are some (relatively obsolete) documents from our exploration of zarr internals in the docs4understanding, but they are not germane to the usage of the Active class.
52
58
53
59
## Storage types
54
60
55
61
PyActiveStorage is designed to interact with various storage backends.
56
-
The storage backend is specified using the `storage_type` argument to `Active` constructor.
62
+
The storage backend is automatically detected, but can still be specified using the `storage_type` argument to the`Active` constructor.
57
63
There are two main integration points for a storage backend:
58
64
59
65
#. Load netCDF metadata
@@ -77,6 +83,10 @@ To load metadata, netCDF files are opened using `s3fs`, with `h5netcdf` used to
77
83
Chunk reductions are implemented in `activestorage.reductionist`, with each operation resulting in an API request to the Reductionist server.
78
84
From there on, `Active` works as per normal.
79
85
86
+
### HTTPS-compatible on an NGINX server
87
+
88
+
The same infrastructure as for S3, but the file is passed in as an `https` URI.
89
+
80
90
## Testing overview
81
91
82
92
We have written unit and integration tests, and employ a coverage measurement tool - Codecov, see PyActiveStorage [test coverage](https://app.codecov.io/gh/NCAS-CMS/PyActiveStorage) with current coverage of 87%; our Continuous Integration (CI) testing is deployed on [Github Actions](https://github.com/NCAS-CMS/PyActiveStorage/actions), and we have nightly tests that run the entire testing suite, to be able to detect any issues introduced by updated versions of our dependencies. Github Actions (GA) tests also test the integration of various storage types we currently support; as such, we have dedicated tests that test Active Storage with S3 storage (by creating and running a MinIO client from within the test, and deploying and testing PyActiveStorage with data shipped to the S3 client).
0 commit comments