Add Directory Based Access To Protocol#845
Conversation
PROTOCOL.md
Outdated
|
|
||
| Note: the `shareId` field is optional. If `shareId` is populated for a table, its value should be unique across the sharing server and immutable through the table's lifecycle. | ||
|
|
||
| Note: `location` should point to the root directory of the table. |
There was a problem hiding this comment.
This filed if present must point to the root directory of the table where the delta log exists. If the server supports dir based access for the table, this field must be present (see accessModes).
| Note: `location` should point to the root directory of the table. | ||
|
|
||
| Note: `auxiliaryLocations` is an optional field which represents any auxiliary storage locations for the table. These should be supported in the `auxiliaryLocation` field of the Generate Temporary Table Credential request body | ||
|
|
There was a problem hiding this comment.
These are only needed if the table's data files are stored in multiple locations. Most tables are expected to have all the files in the root directory. Some tables may have subset of data files in these additional auxiliary locations (usually no more than 1). In such cases, the delta log of the table in the root directory will contain absolute paths to data files in these auxiliary locations.
If the client does not support reading from one or more auxiliary locations, they should either fall back to url based access if the server supports it, or fail the request.
PROTOCOL.md
Outdated
|
|
||
| Note: `auxiliaryLocations` is an optional field which represents any auxiliary storage locations for the table. These should be supported in the `auxiliaryLocation` field of the Generate Temporary Table Credential request body | ||
|
|
||
| Note: `accessModes` represents the supported access modes for the table. This can be `url`, `dir`, or both. If `url` is present, the [QueryTable](#read-data-from-a-table) endpoint should be implemented for the table. If `dir` is present, the [GenerateTemporaryTableCredential](#generate-temporary-table-credential) endpoint should be implemented for the table. |
There was a problem hiding this comment.
For backward compatibility, If this field is not present, the client will assume that the server only supports url based access.
| "s3://secondary/deltasharing/vaccine_share/acme_vaccine_ingredient_data/vaccine_ingredients", | ||
| "s3://tertiary2/delta_project/alt_share/alternate_schema/alternate_table", | ||
| ], | ||
| "accessModes": ["url","dir"] |
There was a problem hiding this comment.
In this example, let's only use location.
PROTOCOL.md
Outdated
| "location": "s3://my-bucket/tables/customer", | ||
| "auxiliaryLocations": [ | ||
| "s3://secondary-bucket/tables/customer", | ||
| "s3://other-bucket/tables/customer" |
There was a problem hiding this comment.
Let's only use 1 aux location since we don't really support multiple.
PROTOCOL.md
Outdated
| <td>Request Body</td> | ||
| <td> | ||
|
|
||
| The `location` field is optional and specifies the location URL path to generate temporary credentials for. If this field is not provided, the response should contain credentials for the table's main location. If the main location is specified the server should still respond with the credential. |
There was a problem hiding this comment.
Perhaps add more details:
This api should be called for the root location as well as all the aux locations. If a table has auxiliary locations and a client does not support reading from multiple locations, they should either fall back to url based access via QueryTable api or throw an error.
|
|
||
| Note: `location` if present must point to the root directory of the table where the delta log exists. If the server supports `dir` based access for the table, this field must be present (see `accessModes`). | ||
|
|
||
| Note: `auxiliaryLocations` is optional and lists extra storage locations for table files (usually no more than one). These should be supported in the `auxiliaryLocation` field of the [GenerateTemporaryTableCredential](#generate-temporary-table-credential) request body. Most tables use only the root directory, but if some files are stored elsewhere, the delta log in the root will include absolute paths to them. If a client can't read from an auxiliary location, it should fall back to URL access (if available) or fail the request. |
There was a problem hiding this comment.
Is there an auxiliaryLocation field in GTTC? (it is just location).
PROTOCOL.md
Outdated
|
|
||
| Note: `auxiliaryLocations` is optional and lists extra storage locations for table files (usually no more than one). These should be supported in the `auxiliaryLocation` field of the [GenerateTemporaryTableCredential](#generate-temporary-table-credential) request body. Most tables use only the root directory, but if some files are stored elsewhere, the delta log in the root will include absolute paths to them. If a client can't read from an auxiliary location, it should fall back to URL access (if available) or fail the request. | ||
|
|
||
| If the client does not support reading from one or more auxiliary locations, they should either fall back to url based access if the server supports it, or fail the request. |
There was a problem hiding this comment.
This block seems to be missing from the earlier comments.
Is there a way to dedup these comments?
linzhou-db
left a comment
linzhou-db
left a comment
There was a problem hiding this comment.
should there be a Compatibility section like this: https://github.com/delta-io/delta-sharing/blob/main/PROTOCOL.md#includeendstreamaction
I remember we had it somewhere?
3 participants
Update protocol to include details for GenerateTemporaryTableCredential and required metadata updates. Closes #782