import delta_sharing
profile_file = r"/Workspace/Users/user@domain.com/sharing/its_config.share"
client = delta_sharing.SharingClient(profile_file)
table_url = profile_file + "#on-prem-delta-sharing.dmsa_in_nrt.shp_shipment"
print(delta_sharing.load_as_pandas(table_url, limit=10))
HTTPError: 400 Client Error: Bad Request for url: https://southeastasia.azuredatabricks.net/api/2.0/delta-sharing/metastores/nnnnnnnnnnnnnnnnnnnnnnnnnnnn/shares/on-prem-delta-sharing/schemas/dmsa_in_nrt/tables/shp_shipment/metadata
Response from server:
{ 'error_code': 'BAD_REQUEST',
'message': 'Invalid table found at abfss://dmsa-prod@storageaccountname.dfs.core.windows.net/datalake/db/shp_nrt/shp_shipment: Missing checkpoint at version 853180[Trace Id: 421d04a04d38ec7ab2c44dd10859876f]'}
But I checked source delta table history
the latest version is only 853178. and I checked transaction log under the ADLS gen2 delta table's delta_log folder, it also shows the latest version is only 853178.
Spark: 3.5
Just using pip install delta-sharing to install this lib.
Note: On Azure databricks, it's normal to directly query this delta table no matter in Spark SQL or other way
Would like to know why it could happen.
import delta_sharing
profile_file = r"/Workspace/Users/user@domain.com/sharing/its_config.share"
client = delta_sharing.SharingClient(profile_file)
table_url = profile_file + "#on-prem-delta-sharing.dmsa_in_nrt.shp_shipment"
print(delta_sharing.load_as_pandas(table_url, limit=10))
HTTPError: 400 Client Error: Bad Request for url: https://southeastasia.azuredatabricks.net/api/2.0/delta-sharing/metastores/nnnnnnnnnnnnnnnnnnnnnnnnnnnn/shares/on-prem-delta-sharing/schemas/dmsa_in_nrt/tables/shp_shipment/metadata
Response from server:
{ 'error_code': 'BAD_REQUEST',
'message': 'Invalid table found at
abfss://dmsa-prod@storageaccountname.dfs.core.windows.net/datalake/db/shp_nrt/shp_shipment:Missing checkpoint at version 853180[Trace Id: 421d04a04d38ec7ab2c44dd10859876f]'}But I checked source delta table history
the latest version is only 853178. and I checked transaction log under the ADLS gen2 delta table's delta_log folder, it also shows the latest version is only 853178.
Spark: 3.5
Just using pip install delta-sharing to install this lib.
Note: On Azure databricks, it's normal to directly query this delta table no matter in Spark SQL or other way
Would like to know why it could happen.