Can we completely replace the standard Hive Metastore with Open Metadata?

[akashrn5]

Can we completely replace the standard Hive Metastore with Open Metadata? If So, how to do it, lets say i want to use open metadata with spark and want to replace hive metastore.

[harshach]

@akashrn5 OpenMetadata is not a technical metadata store and doesn't implement hive metastore's thrift interface which is required for all the hive clients to work.

[RandyAbernethy]

OpenMetadata support for the Hive Metastore would be a widely useful feature. Ingesting metadata to populate the Hive Metastore and then doing it again to populate OpenMetadata is not optimal. Layering OpenMetadata on top of Hive using the OpenMetadata Hive connector is also less than perfect, leaving users exposed to duplicate efforts and or layering where Hive ingestion acts as a choke, limiting the rich features of OpenMetadata.

Using OpenMetadata as the backend for the Hive Metastore has many advantages, in particular access to all the powerful ingestors and pipeline tooling offered by OpenMetadata. Replacing the Hive Metastore would be like the removal of Kafka as a universal element of the metadata ingestion process, greatly simplifying systems by eliminating redundant efforts.
This could be as easy as running the actual Hive Metastore (which is now released independently from the rest of Hive) and pointing it at the OpenMetadata MySQL db. An ingestor checkbox could inform ingestors that they need to populate the Hive compatibility tables in addition to the normal OpenMetadata tables.

I understand the value of keeping OpenMetadata prescriptive and simple but my intuition says the user-centric platform simplicity this would create would greatly outweigh the added maintenance of the necessary Hive Metastore tables. One ingestion setup with OpenMetadata and you get Hive, Presto, SparkSQL, etc., for free.

If the OpenMetadata maintainers think this would be useful I would be interested in contributing (I am a Thrift maintainer, though I don’t think that is a required skill).