Data source name format

[abhineet13]

Background [Optional]

Hi, the data source tab lists data source name as the last string in storage path, for example
spark.read.parquet("s3://bucket/folder/partition=2020-01-01") will have a data source name of "partition=2020-01-01" and
val df= spark.read.parquet("s3://bucket/folder")
df.createOrReplaceTempView("df")
spark.sql("select * from df where partition=2020-01-01") will create a data source name of "folder" in Spline UI.

Question

Is it possible to keep data source naming consistent at folder level, otherwise for daily jobs Spline UI will show multiple data source names for each partition.

Thanks

[wajda]

Unfortunately, it all depends on what the agent provides. The UI representation is as good as the metadata that Spline server receives.
In your example Spark actually sees two different datasource URIs according to what is provided to the Read operation. The UI simply shown the last portion of the URI as a short name as it simply doesn't have any more precise information.
We expect this problem to be at least partially solved with the help of Data source management feature that we plan to implement in the future (see #689)