Securely storing credentials in Gravitino

[vishnu-chalil]

Hi Gravitino Community,

I had tested filesets in gravitino and is working pretty well. But the credentials I configured for s3 in the catalog is visible whenever I view the catalog details in UI. Can somebody explain to how credentials are stored in gravitino? Is there any plans for integrating external secure stores for credentials.

[roryqi]

It is stored as the property now. We don't have secure store for credentials yet. If you have interest, you can improve it.

[vishnu-chalil]

I would like to implement it. Is there someone who can explain the development process

[roryqi]

You can finish a design document about this feature and start a
discussion in our dev mail list (dev@gravitino.apache.org) to collect
others suggestion. After design document, we can split the feature
into some small jobs. For every job, we create a subtask issue for it.
The whole job, we create an epic issue. We create pull requests for
the issues.

[vishnu-chalil]

@jerqi I’ve been exploring the implementation of a feature in Apache Gravitino to securely store credentials in HashiCorp Vault. During my investigation, a few design-related questions arose that I’d like to clarify.

Catalog Creation & Test Connection
I noticed that when creating a catalog, there doesn’t appear to be a test connection performed using the provided credentials. For example, Hadoop catalogs simply return null without validation. Is this an intentional design choice, or am I overlooking something?

Ideally, should the credentials be extracted and validated at the CatalogManager layer before persisting them (along with other properties) to the database? Similarly, when creating schemas or entities (tables, filesets, etc.), should these configurations be fetched separately and verified?

Externally Managed Entities & Secure Storage
For externally managed entities, I’m considering extracting sensitive credentials and storing them securely in Vault (using its key-value store), while keeping the remaining properties in the database. Does this approach align with Gravitino’s design principles? Are there existing patterns or constraints I should be aware of?

I’d appreciate any insights or guidance on these points, especially regarding the intended behavior for credential validation and the preferred integration strategy with HashiCorp Vault.

Thanks in advance for your help!

[roryqi]

cc @mchades Could hep answer the question about test connections.

2. It's good for me. It's better to define interfaces first for it. It will make it easy to adapt to other credential storage system.

[mchades]

Catalog Creation & Test Connection
I noticed that when creating a catalog, there doesn’t appear to be a test connection performed using the provided credentials. For example, Hadoop catalogs simply return null without validation. Is this an intentional design choice, or am I overlooking something?

yes, it's an intentional design. plz see the code comment:

gravitino/catalogs/catalog-hadoop/src/main/java/org/apache/gravitino/catalog/hadoop/HadoopCatalogOperations.java

Lines 901 to 912 in c7484b1

	/**
	* Since the Hadoop catalog was completely managed by Gravitino, we don't need to test the
	* connection
	*
	* @param catalogIdent the name of the catalog.
	* @param type the type of the catalog.
	* @param provider the provider of the catalog.
	* @param comment the comment of the catalog.
	* @param properties the properties of the catalog.
	*/
	@Override
	public void testConnection(

For externally managed entities, I’m considering extracting sensitive credentials and storing them securely in Vault (using its key-value store), while keeping the remaining properties in the database.

How does it integrate with the GVFS and engine connectors (such as Spark and Trino)?

[vishnu-chalil]

How does it integrate with the GVFS and engine connectors (such as Spark and

@mchades Thanks for asking this. I haven't given much thought about this. WhatI thought was that the credential extraction points in the code are properly covered it would automatically cover the rest of the scenarios. Please let me know if you see any challenges ahead with it.

[roryqi]

@mchades Gently ping.

[mchades]

I still don't understand how to resolve the issue you described: "The credentials I configured for S3 in the catalog are visible whenever I view the catalog details in the UI." Have I missed anything when using HashiCorp Vault? could you plz give some examples with your design?