feat: ability to delete evaluators from the eval library (LLM-as-a-judge)

[AkioNuernberger]

Describe the feature or potential improvement

I have heard multiple times that users want to delete evaluators not only from the main eval page (shipped in Apr-25), but specifically from the eval library. Would be curious to hear if there are other people interested in this and what scenario drives this need.

Additional information

No response

[masonk]

My main use-case is to reduce clutter. Early on, we experimented with custom evaluators from the UI. We never use any of the evaluators in that library at all - neither our custom evaluators (which are all deprecated) nor the built-ins (which are too simplistic).

The biggest issue for me is how badly they clutter up the dataset view. 53 columns in my dataset UI view, and most of the columns are empty scores. There's no way to deeplink to a cleaner view of a dataset or alter the view defaults for a dataset.

[rmaceissoft]

My use case for this feature:
I initially created a custom evaluator and set up a running instance for it. Later, I needed to add a human annotation score config, but I ran into a naming issue:

1. The human annotation score config has a 35-character limit for names
2. I wanted both the evaluator and the score config to follow our naming convention (using matching names with different prefixes)
3. I could rename the running evaluator instance (which was possible), but I couldn't rename the custom evaluator in the library itself (which is impossible currently)
4. This forced me to create a completely new evaluator with the correct name, leaving the old one cluttering my evaluator library with no way to remove it

Suggested alternative/priority:
If allowing deletion of evaluators from the library requires significant development time, could you consider prioritizing the ability to rename custom evaluators? This would prevent situations like mine where users create duplicates simply due to naming constraints, which then contributes to the clutter problem mentioned by others in this thread.

Being able to rename would be a lighter-weight solution that addresses a key pain point that leads to the need for deletion in the first place.

[ronancummins]

Experimenting with evaluators can lead to a number of similar ones being created so being able to rename or relabel them would be helpful so that they are not confused with ones that you want being run.

[fjaguero]

In my case we were learning to use the custom evaluators and now we have many tests there that are misleading. Renaming would help us solve the problem, while the ability to delete them would be a nice to have.

[ronancummins]

Agreed, or even archived/retired so that they are removed from the view of in-use evaluators.