InMemoryFIleIndex 缓存filelist信息，怎么解决两个sparksession 长期运行同步问题？

[melin]

例如两个SparkSession A和B，在A中先查询表T1，接着在B中往表T1写入数据，之后在A中查询不到B 写入的数据。目前有两种做法：
1，关闭SparkSession A，重启动
2、调用 refresh table command。
这两种做法对于业务用户不知道什么时候去操作，不具可操作性。频繁刷新refresh，影响交互性能。

最理想做法是支持redis 等集中式缓存方案

[pan3793]

It's by design in Hive tables and Spark built-in file format tables, the datalake table formats like delta and iceberg could resolve your problems.

[yaooqinn]

I have a PR(maybe not solve your problem yet, but going the right direction with follow-up might achieve that goal).

Sadly, it haven't get much attention from spark community yet, I do see a chance to merge it and do followups

[cloud-fan]

How is apache/spark#37355 related to this issue? This is about the file listing cache, not the table data cache.

[yaooqinn]

Not directly related, just my maybe-followups for other cache stuff skipping