2.0.1

What's Changed

• switching to tiledb vfs usage instead of boto3 by @kylemann16 in #151

Full Changelog: 2.0.0...2.0.1

2.0.0: Dense arrays, large dataset optimization, attribute/metric filters

Functionality Changes

• SilviMetric is now writing to a Dense TileDB array. Dense arrays allow us to take more advantage of the benefits that TileDB offers without many drawbacks.
• Attribute and Metric Filters. We're now writing Attribute and Metric data with TileDB's ZstdFilter, with the level set to 7. Variable length arrays will now take advantage of the PositiveOffsetFilter. These changes will provide size reduction for output data.
• Storage config now requires a xsize and ysize variable to indicate how big the extents of tiledb tiles should be. This was in response to memory problems from tiledb when it was unspecified.
• Updated info call:
  • added a metrics option to the cli
  • fixed history output
  • removed necessary info from the concise output.
• Updated extract call:
  • handle_overlaps speedup
  • removed extent indexing, was too slow and could get them in other ways
• adding start_datetime and end_datetime to tiledb attributes being written, in similar fashion to count
  • This will allow users to query by start and end time

Behind the Scenes

• SM no longer writes to a specific timestamp for a write, this turns out to be a TileDB anti-pattern. We now write to the current timestamp and write a start and end timestamp attribute for collection dates of data. These attributes can be queried with normal tiledb operations.
• Deletions will now be overwrites. TileDB dense arrays don't support deletion operations, so we'll instead be writing new data at the current timestamp over the old data.
• In order to operate better on larger datasets, SilviMetric will now operate in chunks the size of the TileDB x and y sizes (see note in Functionality Changes about StorageConfig changes). This means there is very little need to consolidate commits to the array, and should increase speed and memory performance.
• updated metrics:
  • added nan_value member variable
  • added nan_policy member variable
  • added logic to handle bad return values and bad dependency values depending on nan_policy
• Updated storage config to adjust a relative path to absolute for tdb dir
• Adjusted aad metrics to use variables that were already created
• Added nan handling to several metrics in which it was possible

What's Changed

• 1.4.1 by @kylemann16 in #123
• Adjustments for large datasets by @kylemann16 in #124
• fixed naming mismatch tdb, tifs by @gannon-guess in #131
• tutorial metric import module fix by @gannon-guess in #133
• Issue 126 by @gannon-guess in #135
• Finish sentence in About document by @danielrode in #137
• protect users from going oob with tile sizes by @kylemann16 in #140
• pin pandas to <3.0.0 by @kylemann16 in #148
• 141 - added offset filter and level by @gannon-guess in #142
• Issue 129 by @gannon-guess in #134
• bumping to 2.0.0 by @kylemann16 in #149

New Contributors

• @gannon-guess made their first contribution in #131 🎉

Full Changelog: 1.4.0...2.0.0

1.4.0

What's Changed

• Add linting to keep consistency by @kylemann16 in #120
• Update bounds.py align_to_cell_lines() to match FUSION alignment by @bmcgaughey1 in #112
• Added grid_metrics suite of metrics that match FUSION
• Added tests to confirm that metrics match FUSION as closely as possible
• 1.4.0 by @kylemann16 in #122

Full Changelog: 1.3.1...1.4.0

1.3.1

What's Changed

• Kmann/task_graph by @kylemann16 in #107
• Convert docs to jupyterbook by @hobu in #108
• docs and peripherals fixes by @kylemann16 in #109

Full Changelog: 1.3.0...1.3.1

1.3.0

What's Changed

• Added FUSION metrics by @bmcgaughey1 in #70

• Update quickstart.rst: correct missing flags by @danielrode in #100

• Stability update by @kylemann16 in #104

• Kmann/dep fix by @kylemann16 in #106

• Dask distributed is disabled for now *

New Contributors

• @danielrode made their first contribution in #100

Full Changelog: 1.2.3...1.3.0

1.2.3

What's Changed

• Fix data type handling bug by @kylemann16 in #94

Full Changelog: 1.2.2...1.2.3

1.2.2

What's Changed

• Add colab link to main page of docs by @kylemann16 in #88
• Avert RTD Upcoming breakage by @j9ac9k in #89
• Make use of Pandas Extensions and TileDB from_pandas method by @kylemann16 in #86
• version bump by @kylemann16 in #93

Full Changelog: 1.2.1...1.2.2

1.2.1

What's Changed

• Fixing context handling for python 3.11+ by @kylemann16 in #85

Full Changelog: 1.2.0...1.2.1

1.2.0

What's Changed

• Switch to DataFrames and add Metrics Filters by @kylemann16 in #84

• We've changed from using mainly numpy ndarrays to pandas DataFrames. This change makes data much easier to manipulate, filter, and pass around efficiently.

• You can now add Filters to a Metric that allow you to manipulate point data before it arrives at the main Metric Method. Eg. You want to remove any point data that doesn't have a NumberOfReturns less than 2. Only point data with NumberOfReturns >= 2 will be passed to the Metric.

• You can now specify Attribute dependencies for a Metric, which indicates to SilviMetric that this Metric requires this Attribute to operate. The database makes sure that all Attribute dependencies are satisfied at creation, and Shatter will now raise an exception if the data you are trying to ingest doesn't have the required Attribute.

• Extract has been simplified, thanks in large part to the switch to DataFrames.

• Full Changelog: 1.1.1...1.2.0

1.1.1: Kmann/version bump (#82)

What's Changed

• Kmann/manage by @kylemann16 in #78
• Kmann/quickstart by @kylemann16 in #81
• Version bump to 1.1.1

Full Changelog: 1.0.0...1.1.1