Just curious, how is going on for this project

[itismewxg]

NULL as Title

[sadikovi]

I have not done much to improve it, mainly because Spark 2.3+ performance is already really good, and we have implemented similar functionality at work. I will be spending some time in the future to play with different features to speed up queries on Parquet, e.g. using index pages and introduce a proper metastore.

Any suggestions are welcome!

[itismewxg]

I totally agree with your vision. for me, I hope expand parquet from storage format, semi-storage engine to full-featured storage engine (with lots of versioned parquet files)

1. index pages. with short-index definition
2. rollup based on index definition
3. define schema-hash for rollups
4. MVCC as accepting loading data as mini-batch. and compact with designated policy
5. Metastore, for full-table scan or conditional scan, pickup good rollups and versioned parquet files.

[sadikovi]

Would it be something similar to Iceberg (https://github.com/Netflix/iceberg)?

[sadikovi]

I was actually thinking if it would be better to implement something based on Arrow and following paper on Albis file format https://www.usenix.org/system/files/conference/atc18/atc18-trivedi.pdf

[itismewxg]

Thank you so much for the refer. they are interesting projects in the way~.

[Aaaaaaron]

Parquet now has its own page index(base on statistics, a little weak, I think), maybe we can do something based on it.

[sadikovi]

Yes, you are right - we should. I will try reevaluating the project.

[Aaaaaaron]

Let's open a new issue to follow this 😄. I am very willing to be involved.