[#9622] improvement(docs): Add guide for Lance REST integration with Spark and Ray

[yuqi1129]

What changes were proposed in this pull request?

This pull request adds a new documentation page that explains how to integrate the Lance REST service with Spark and Ray using their respective connectors. The guide covers prerequisites, compatibility, example usage for both Spark and Ray, troubleshooting tips, and a compatibility matrix for versions.

Why are the changes needed?

For a better user experience
Fix: #9622

Does this PR introduce any user-facing change?

N/A.

How was this patch tested?

N/A.

[Copilot]

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

[yuqi1129]

This PR will be blocked by several bugs and improvement, the followings are:

• [#9520] improvement(lance): Refine the concept of createEmptyTable in Lance REST #9589
• Can't write data to ray with exmaple data lance-format/lance-ray#68
• Could not read the data in MinIO by lance-ray lance-format/lance-ray#70

[yuqi1129]

This PR will be blocked by several bugs and improvement, the followings are:

• [#9520] improvement(lance): Refine the concept of createEmptyTable in Lance REST #9589
• Can't write data to ray with exmaple data lance-format/lance-ray#68
• Could not read the data in MinIO by lance-ray lance-format/lance-ray#70

The first two have been resolved.

[yuqi1129]

This PR will be blocked by several bugs and improvement, the followings are:

• [#9520] improvement(lance): Refine the concept of createEmptyTable in Lance REST #9589
• Can't write data to ray with exmaple data lance-format/lance-ray#68
• Could not read the data in MinIO by lance-ray lance-format/lance-ray#70

The first two have been resolved.

All resolved and ready to review.

[Copilot]

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 8 comments.

[Copilot]

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 10 comments.

[jerryshao]

Why does this chapter belong to local storage? Also, as I remembered, external table must specify the location, am I right?

[yuqi1129]

Also, as I remembered, external table must specify the location, am I right?

I'm afraid not. According to the code base, whether it's an external or not, the calculation method is the same: If no location is specified in table level, it will try to use the location of its parent, like the schema or catalog. It seems that we only use the property external when droping or purging the table.

gravitino/catalogs/catalog-lakehouse-generic/src/main/java/org/apache/gravitino/catalog/lakehouse/generic/GenericCatalogOperations.java

Lines 224 to 237 in c206c07

	public Table createTable(
	NameIdentifier ident,
	Column[] columns,
	String comment,
	Map<String, String> properties,
	Transform[] partitions,
	Distribution distribution,
	SortOrder[] sortOrders,
	Index[] indexes)
	throws NoSuchSchemaException, TableAlreadyExistsException {
	Schema schema = loadSchema(NameIdentifier.of(ident.namespace().levels()));
	String tableLocation = calculateTableLocation(schema, ident, properties);
	String format = properties.getOrDefault(Table.PROPERTY_TABLE_FORMAT, null);

external table must specify the location

Do you mean users need to specify the location in the table explicitly, or do they set it on any of the catalog, schema, or table?

[Copilot]

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 5 comments.

[Copilot]

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 4 comments.