[Enhancement] Avoid duplicate Hive metadata loading when estimating HMS external table row count

[foxtail463]

Search before asking

• I had searched in the issues and found no similar issues.

Description

For HMS external tables, Doris may estimate table row count by listing Hive files when HMS table parameters do not contain row count and enable_get_row_count_from_file_list is enabled.

Currently, this row-count estimation path may read Hive partition and file metadata without filling Doris' Hive external metadata cache. In a normal query planning flow, the scan planning phase still needs the same partition and file metadata later, so Doris can read the same HMS/file metadata twice in one query planning process.

This is inefficient for Hive tables with many partitions or files, especially when HMS access is expensive.

Expected behavior:

• Query planning should be able to reuse Hive metadata fetched during row-count estimation.
• Non-query metadata display paths, such as SHOW TABLE STATUS, SHOW STATS, or information_schema.tables, should still avoid filling heavy Hive metadata caches just for displaying cached row count.

Solution

Introduce a separate row-count loading mode for query planning and metadata display paths.

• ExternalTable.getRowCount() should load row count in a query-planning mode that may fill external metadata cache.
• ExternalTable.getCachedRowCount() and display-oriented paths should keep the lightweight behavior and avoid filling heavy metadata cache.
• HMSExternalTable should choose cached or non-cached Hive metadata APIs when estimating row count from file list.

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[zclllyybb]

Breakwater-GitHub-Analysis-Slot: slot_75b5f7a31eb3

Initial triage: this looks like a valid performance/caching enhancement for the HMS external table planning path, not a correctness bug.

I checked current origin/master locally at 4483daf9f03. The relevant flow matches the report:

• ExternalTable.getRowCount() initializes the table and loads the row-count cache through ExternalRowCountCache.getCachedRowCount(...).
• ExternalRowCountCache.RowCountCacheLoader calls table.fetchRowCount().
• HMSExternalTable.fetchRowCount() first tries HMS table parameters, then falls back to getRowCountFromFileList() when the row count is unknown and enable_get_row_count_from_file_list is enabled.
• In that fallback, getFilesForPartitions(...) explicitly uses cache.getAllPartitionsWithoutCache(...) and then cache.getFilesByPartitions(..., withCache=false, ...).
• Later scan planning for the same HMS table uses the normal cached path: HiveScanNode.getPartitions() calls getAllPartitionsWithCache(...), and file split generation calls getFilesByPartitions(...) with withCache = Config.max_external_file_cache_num > 0.

So the same Hive partition/file metadata can indeed be fetched once for row-count estimation and then fetched again for scan planning in one planning process. The issue description is consistent with the current code.

Suggested implementation direction:

• Add an explicit row-count loading mode, for example QUERY_PLANNING vs DISPLAY_ONLY, rather than inferring intent inside HMSExternalTable.
• Use the planning mode from ExternalTable.getRowCount() / optimizer stats paths, and allow the HMS file-list row-count fallback to populate or reuse the normal Hive partition/file caches in that mode.
• Keep display-oriented paths such as getCachedRowCount(), SHOW TABLE STATUS, SHOW STATS, and information_schema.tables lightweight. If the intent is strictly "cached value only", consider wiring them to ExternalRowCountCache.getCachedRowCountIfPresent(...) or equivalent non-loading behavior; the method already exists but is not currently used.
• For HMS row-count estimation, make the partition and file metadata APIs mode-dependent: cached APIs for query planning, non-cached APIs for display/background-only loading.

Recommended validation:

• Add an FE unit test or mock-cache test proving that planning-time row-count estimation for an HMS table uses the cached partition/file path.
• Add a companion test proving that display-only calls do not populate heavy Hive partition/file caches just to render row count.
• If possible, include a simple before/after planning profile or FE log snippet for a partitioned HMS table without numRows in HMS parameters and with enable_get_row_count_from_file_list=true, showing that partition/file listing happens once after the change.

No additional user logs are required to accept this as an enhancement, but a concrete reproduction table shape would help measure impact: Doris commit/version, catalog properties, partition count, file count, hive_stats_partition_sample_size, max_external_file_cache_num, fetch_hive_row_count_sync, and a representative EXPLAIN or planning profile.