Delta create_checkpoint() breaks Synapse SQL OPENROWSET #1611
Replies: 1 comment
-
@Trebla2010 unfortunately this seems like it might be the Synapse SQL delta integration having a bug in how it's handling the Delta protocol The
Sorry if this isn't the answer you were hoping for but our princess is in another castle |
Beta Was this translation helpful? Give feedback.
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
-
After running create_checkpoint(), I'm unable to run OPENROWSET. Getting below error.
I'm running this on Azure Synapse Serverless SQL.
SELECT
TOP 100 *
FROM
OPENROWSET(
BULK 'https://devstoragereporting.dfs.core.windows.net/enriched/TestData/',
FORMAT = 'DELTA'
) AS [result]
Resolving Delta logs on path 'ADLSProcessing/staging/TestData/' failed with error: Cannot parse field field 'parts' in json object.
I found that the file "_last_checkpoint" that gets created into the delta_log folder is what's breaking Synapse SQL. Fortunately, deleting this file restores functionality to my delta table and I keep the performance gain.
What is the purpose of the _last_checkpoint file? Is there a way to prevent this file from being generated?
Below is the content inside the _last_checkpoint file.
{"num_of_add_files":null,"parts":null,"size":132,"size_in_bytes":77024,"version":89}
Beta Was this translation helpful? Give feedback.
All reactions