Could someone can explain what puffin file use case is?

[JunseoChoJJ]

I've read this
https://www.dremio.com/blog/puffins-and-icebergs-additional-stats-for-apache-iceberg-tables/

1. still don't understand any usecase is?
2. when do you normally use nvd in dataset? (on the article when you join table)
3. where does puffin file is? is it in the manifest list? is it in the metadatafile?
4. since puffin is also file format like parquet should we need to make it and store in S3 like parquet file

[findepi]

Puffin files are stored in s3, along with Iceberg table metadata.
NDV is important input for Trino CBO algorithms.
See https://trino.io/blog/2019/07/04/cbo-introduction.html

[JunseoChoJJ]

thanks for telling me @findepi

is it .stats file is puffin file ?
and how do you open this?

i know on metadata.json file, there are statistics part and I can see snapshot-id, statistics-path, etc.

I saw that there is a fields part and ndv part. ndv has number on it. it means that there is same values on number?
and I want to figure out what field is, it assume there is a string inside it, but assume I need to get into .stats path to get that information.

ex)
"type" : "apache-datasketches-theta-v1",
"snapshot-id" : ,
"sequence-number" : 2,
"fields" : [ 5 ],
"properties" : {
"ndv" : "128"
}

[findepi]

is it .stats file is puffin file ?

yes

and how do you open this?

https://github.com/apache/iceberg/blob/c07f2aabc0a1d02f068ecf1514d2479c0fbdd3b0/core/src/main/java/org/apache/iceberg/puffin/Puffin.java#L82

i am not aware of a CLI command for puffin files

[JunseoChoJJ]

thanks a lot 👍