Support for newer reader and writer versions

[ruskipitbull]

Is there a plan to support reader and writer versions above what is currently supported. Particularly problematic for my use case is that the library cannot parse V2 checkpoints (_last_checkpoint file) since 15.2 due to strong typing during JSON parsing.

[rtyler]

@ruskipitbull would you be able to share an error message that you're seeing? Or perhaps any steps that might create a table which is causing similar problems to what you're seeing?

I have a hunch of what you might be experiencing, but would benefit from a more clear demonstration case

[ruskipitbull]

Hi @rtyler ,

The error I receive is

Traceback (most recent call last): File "[REDACTED]Desktop/code/playground/deltatable.py", line 14, in <module> table = DeltaTable(location, without_files=False) File "[REDACTED]/lib/python3.10/site-packages/deltalake/table.py", line 412, in __init__ self._table = RawDeltaTable( _internal.DeltaProtocolError: Invalid JSON in file stats: invalid type: map, expected a string at line 1 column 83

Here are the contents of the _last_checkpoint file:
{"version":10,"size":30,"sizeInBytes":24333,"numOfAddFiles":16,"checkpointSchema":[{"name":"txn","dataType":[{"name":"appId","dataType":{"ordering":{}},"nullable":true,"metadata":{"map":{}}},{"name":"version","dataType":{"numeric":{},"integral":{},"ordering":{},"exactNumeric":{}},"nullable":true,"metadata":{"map":{}}},{"name":"lastUpdated","dataType":{"numeric":{},"integral":{},"ordering":{},"exactNumeric":{}},"nullable":true,"metadata":{"map":{}}}],"nullable":true,"metadata":{"map":{}}},{"name":"add","dataType":[{"name":"path","dataType":{"ordering":{}},"nullable":true,"metadata":{"map":{}}},{"name":"partitionValues","dataType":{"keyType":{"ordering":{}},"valueType":{"ordering":{}},"valueContainsNull":true},"nullable":true,"metadata":{"map":{}}},{"name":"size","dataType":{"numeric":{},"integral":{},"ordering":{},"exactNumeric":{}},"nullable":true,"metadata":{"map":{}}},{"name":"modificationTime","dataType":{"numeric":{},"integral":{},"ordering":{},"exactNumeric":{}},"nullable":true,"metadata":{"map":{}}},{"name":"dataChange","dataType":{"ordering":{}},"nullable":true,"metadata":{"map":{}}},{"name":"tags","dataType":{"keyType":{"ordering":{}},"valueType":{"ordering":{}},"valueContainsNull":true},"nullable":true,"metadata":{"map":{}}},{"name":"deletionVector","dataType":[{"name":"storageType","dataType":{"ordering":{}},"nullable":true,"metadata":{"map":{}}},{"name":"pathOrInlineDv","dataType":{"ordering":{}},"nullable":true,"metadata":{"map":{}}},{"name":"offset","dataType":{"numeric":{},"integral":{},"ordering":{},"exactNumeric":{}},"nullable":true,"metadata":{"map":{}}},{"name":"sizeInBytes","dataType":{"numeric":{},"integral":{},"ordering":{},"exactNumeric":{}},"nullable":true,"metadata":{"map":{}}},{"name":"cardinality","dataType":{"numeric":{},"integral":{},"ordering":{},"exactNumeric":{}},"nullable":true,"metadata":{"map":{}}},{"name":"maxRowIndex","dataType":{"numeric":{},"integral":{},"ordering":{},"exactNumeric":{}},"nullable":true,"metadata":{"map":{}}}],"nullable":true,"metadata":{"map":{}}},{"name":"stats","dataType":{"ordering":{}},"nullable":true,"metadata":{"map":{}}},{"name":"partitionValues_parsed","dataType":[{"name":"record_type","dataType":{"ordering":{}},"nullable":true,"metadata":{"map":{}}},{"name":"record_date","dataType":{"ordering":{}},"nullable":true,"metadata":{"map":{}}}],"nullable":true,"metadata":{"map":{}}}],"nullable":true,"metadata":{"map":{}}},{"name":"remove","dataType":[{"name":"path","dataType":{"ordering":{}},"nullable":true,"metadata":{"map":{}}},{"name":"deletionTimestamp","dataType":{"numeric":{},"integral":{},"ordering":{},"exactNumeric":{}},"nullable":true,"metadata":{"map":{}}},{"name":"dataChange","dataType":{"ordering":{}},"nullable":true,"metadata":{"map":{}}},{"name":"extendedFileMetadata","dataType":{"ordering":{}},"nullable":true,"metadata":{"map":{}}},{"name":"partitionValues","dataType":{"keyType":{"ordering":{}},"valueType":{"ordering":{}},"valueContainsNull":true},"nullable":true,"metadata":{"map":{}}},{"name":"size","dataType":{"numeric":{},"integral":{},"ordering":{},"exactNumeric":{}},"nullable":true,"metadata":{"map":{}}},{"name":"deletionVector","dataType":[{"name":"storageType","dataType":{"ordering":{}},"nullable":true,"metadata":{"map":{}}},{"name":"pathOrInlineDv","dataType":{"ordering":{}},"nullable":true,"metadata":{"map":{}}},{"name":"offset","dataType":{"numeric":{},"integral":{},"ordering":{},"exactNumeric":{}},"nullable":true,"metadata":{"map":{}}},{"name":"sizeInBytes","dataType":{"numeric":{},"integral":{},"ordering":{},"exactNumeric":{}},"nullable":true,"metadata":{"map":{}}},{"name":"cardinality","dataType":{"numeric":{},"integral":{},"ordering":{},"exactNumeric":{}},"nullable":true,"metadata":{"map":{}}},{"name":"maxRowIndex","dataType":{"numeric":{},"integral":{},"ordering":{},"exactNumeric":{}},"nullable":true,"metadata":{"map":{}}}],"nullable":true,"metadata":{"map":{}}}],"nullable":true,"metadata":{"map":{}}},{"name":"metaData","dataType":[{"name":"id","dataType":{"ordering":{}},"nullable":true,"metadata":{"map":{}}},{"name":"name","dataType":{"ordering":{}},"nullable":true,"metadata":{"map":{}}},{"name":"description","dataType":{"ordering":{}},"nullable":true,"metadata":{"map":{}}},{"name":"format","dataType":[{"name":"provider","dataType":{"ordering":{}},"nullable":true,"metadata":{"map":{}}},{"name":"options","dataType":{"keyType":{"ordering":{}},"valueType":{"ordering":{}},"valueContainsNull":true},"nullable":true,"metadata":{"map":{}}}],"nullable":true,"metadata":{"map":{}}},{"name":"schemaString","dataType":{"ordering":{}},"nullable":true,"metadata":{"map":{}}},{"name":"partitionColumns","dataType":{"elementType":{"ordering":{}},"containsNull":true},"nullable":true,"metadata":{"map":{}}},{"name":"configuration","dataType":{"keyType":{"ordering":{}},"valueType":{"ordering":{}},"valueContainsNull":true},"nullable":true,"metadata":{"map":{}}},{"name":"createdTime","dataType":{"numeric":{},"integral":{},"ordering":{},"exactNumeric":{}},"nullable":true,"metadata":{"map":{}}}],"nullable":true,"metadata":{"map":{}}},{"name":"protocol","dataType":[{"name":"minReaderVersion","dataType":{"numeric":{},"integral":{},"ordering":{},"exactNumeric":{}},"nullable":true,"metadata":{"map":{}}},{"name":"minWriterVersion","dataType":{"numeric":{},"integral":{},"ordering":{},"exactNumeric":{}},"nullable":true,"metadata":{"map":{}}},{"name":"readerFeatures","dataType":{"elementType":{"ordering":{}},"containsNull":true},"nullable":true,"metadata":{"map":{}}},{"name":"writerFeatures","dataType":{"elementType":{"ordering":{}},"containsNull":true},"nullable":true,"metadata":{"map":{}}}],"nullable":true,"metadata":{"map":{}}}],"checksum":"c1e19829bcc2d0c34905c2711c807d18"}

[REDACTED] % pip show deltalake Name: deltalake Version: 0.20.2 Summary: Native Delta Lake Python binding based on delta-rs with Pandas integration Home-page: https://github.com/delta-io/delta-rs Author: Qingping Hou <[email protected]>, Will Jones <[email protected]> Author-email: Qingping Hou <[email protected]>, Will Jones <[email protected]> License: Apache-2.0 Location: /Users/ric961/Desktop/code/accelerate-campaign-execution-validator/lib/python3.10/site-packages Requires: pyarrow Required-by:

Let me know if this is sufficient or if you need additional info.

[rtyler]

This is useful thank you, can you share what engine (Spark, DBR, delta-rs, etc) created the table to begin with?

[rtyler]

I have done some digging through the code in delta-rs and delta-kernel-rs. I am not yet certain the checkpoint is the actual problem. There may be file stats in the actual checkpoint file which in this case would be named something like 0000010.checkpoint.parquet.

The original error could also mean that a normal transaction contains unreadable file stats, which is also where the engine becomes a potential factor here 🤔

Depending on your timezone, I may be able to help diagnose the error more directly via screen share in our Delta Users Slack in the #delta-rs channel

[ruskipitbull]

Hi @rtyler , there are two patterns used, and it is either Spark or Databricks. I'll have to track down with the right team the answer. As far as timezones, I'm on US east time, but can work around your schedule if needed.