FileSize of .tdb

[Prakash-Basnet]

Shattering and extracting forest metrics as a product is the best part, but after shattering, the .tdb file size becomes equal to the size of the input point clouds. This might not be a problem for some small-scale work (e.g., 4-5 GB of data), but at the large regional or national scale with 100s or 1000s of GB of data, processing to get products requires the same amount of storage. Literally, it needs double the space.

I found that inside the .tdb file, "_fragments" and "a1_var.tdb" exist. Under this file, perhaps the input data is stored.

Can this file be excluded, and only the output in the specified raster be returned?

[kylemann16]

Yeah that totally makes sense that this is a problem for large-scale projects, and it's definitely something I'm looking at. I think that we could possibly add an option to exclude the point data from the TileDB output in future versions.

I think you're right that a1_var.tdb is likely the input point data, corroborated by the tiledb docs.

From those docs:

         |      |_ a1.tdb                   # var-sized attribute (offsets)
         |      |_ a1_var.tdb               # var-sized attribute (values)

So yes, as a workaround for now, you could conceivably remove the a1_var.tdb and the associated a1.tdb file that should house the offsets, according to the docs.

I would say that I haven't explored doing this myself, so I would be very cautious about deleting anything that you might want back.

[kylemann16]

@Prakash-Basnet are you installing from source, or from a package manager? Thanks!

[Prakash-Basnet]

Hi Kyle,
I installed SilviMetric via conda inside a LiDAR-specific environment using: conda install -c conda-forge silvimetric
I’m not building from source at the moment.
Thanks for looking into the possibility of excluding point data from the TileDB output, that would be very helpful for large-scale projects.

[Prakash-Basnet]

Yeah that totally makes sense that this is a problem for large-scale projects, and it's definitely something I'm looking at. I think that we could possibly add an option to exclude the point data from the TileDB output in future versions.

I think you're right that a1_var.tdb is likely the input point data, corroborated by the tiledb docs.

From those docs:

         |      |_ a1.tdb                   # var-sized attribute (offsets)
         |      |_ a1_var.tdb               # var-sized attribute (values)

So yes, as a workaround for now, you could conceivably remove the a1_var.tdb and the associated a1.tdb file that should house the offsets, according to the docs.

I would say that I haven't explored doing this myself, so I would be very cautious about deleting anything that you might want back.

Once, I deleted all "a1_var.tdb" files, of-course it significatly reduce tiledb file size (acceptable), but while extracting it didn't work; See
"tiledb.libtiledb.TileDBError: [TileDB::Task] Error: Caught std::exception: IO Error: Cannot get file size of '/home/tile_db.tdb/__fragments/__3_3_36231d210bb1c7e90a736cf30e87e1d2_22/a1_var.tdb'; No such file or directory"