[PROTOCOL Change Request] Return latestTableVersion in the CommitStore.getCommits API (delta-io#2712)

## Protocol Change Request

### Description of the protocol change
This change puts added responsibility on CommitStore to return the
latestTableVersion in the CommitStore.getCommits API along with the list
of Commits.

Protocol RFC issue: delta-io#2598

### Willingness to contribute

The Delta Lake Community encourages protocol innovations. Would you or
another member of your organization be willing to contribute this
feature to the Delta Lake code base?

- [x] Yes. I can contribute.
- [ ] Yes. I would be willing to contribute with guidance from the Delta
Lake community.
- [ ] No. I cannot contribute at this time.

Author: sumeet-db

protocol_rfcs/managed-commits.md

@@ -143,7 +143,7 @@ are responsible to define the commit atomicity and backfill protocols which the
 
 At a high level, the `commit-owner` needs to provide:
 - API to atomically commit a version `x` with given set of `actions`. This is explained in detail in the [commit protocol](#commit-protocol) section.
-- API to retrieve information about the recent commits on the table. This is explained in detail in the [getting un-backfilled commits from commit-owner](#getting-un-backfilled-commits-from-commit-owner) section.
+- API to retrieve information about the recent commits and the latest ratified version on the table. This is explained in detail in the [getting un-backfilled commits from commit-owner](#getting-un-backfilled-commits-from-commit-owner) section.
 
 ### Commit Protocol
 
@@ -161,7 +161,14 @@ Even after a commit succeeds, Delta clients can only discover the commit through
 have no way to determine which file in `_delta_log/_commits` directory corresponds to the actual commit `v`.
 
 The commit-owner is responsible to implement an API (defined by the Delta client) that Delta clients can use to retrieve information about un-backfilled commits maintained
-by the commit-owner. Delta clients who are unaware of the commit-owner (or unwilling to talk to it), may not see recent un-backfilled commits and thus may encounter stale reads.
+by the commit-owner. The API must also return the latest version of the table ratified by the commit-owner (if any).
+Providing the latest ratified table version helps address potential race conditions between listing commits and contacting the commit-owner.
+For example, if a client performs a listing before a recently ratified commit is backfilled, and then contacts the commit-owner after the backfill completes,
+the commit-owner may return an empty list of un-backfilled commits. Without knowing the latest ratified version, the client might incorrectly assume their listing was complete
+and read a stale snapshot.
+
+Delta clients who are unaware of the commit-owner (or unwilling to talk to it), may not see recent un-backfilled commits and thus may encounter stale reads.
+
 
 ## Sample Commit Owner API
 
@@ -176,7 +183,7 @@ interface CommitStore {
      * @param version The version we want to commit.
      * @param actions Actions that need to be committed.
      *
-     * returns CommitResponse which has details around the new committed delta file.
+     * @return CommitResponse which has details around the new committed delta file.
      */
     def commit(
         version: Long,
@@ -191,13 +198,16 @@ interface CommitStore {
      * Note that the first version returned by this API may not be equal to the `startVersion`. This
      * happens when few versions starting from `startVersion` are already backfilled and so
      * CommitStore may have stopped tracking them.
+     * The returned latestTableVersion is the maximum commit version ratified by the Commit-Owner.
+     * Note that returning latestTableVersion as -1 is acceptable only if the commit-owner never
+     * ratified any version i.e. it never accepted any un-backfilled commit.
      *
-     * @return a list of `Commit` which are tracked by commit-owner.
-     *
+     * @return GetCommitsResponse which contains a list of `Commit`s and the latestTableVersion
+     *         tracked by the commit-owner.
      */
     def getCommits(
         startVersion: Long,
-        endVersion: Long): Seq[Commit]
+        endVersion: Long): GetCommitsResponse
 
     /**
      * API to ask the commit-owner to backfill all commits <= given `version`.