how to use with custom provider

[da1z]

hey @mattpocock . very great library. how to please explain or maybe add docs on how to use autoeval with custom provider. for example azure. especially theirs llm scorers. thanks!

[da1z]

turns out there is small bug in autoeval that does not pass through azure option. created PR for it. meanwhile i am having hard time creating ts helper function to wrap autoeval scorer to make it work the evalite. could you please help?
what i have so far:

 scorers: [
    (options) =>
      Factuality({
        ...options,
        azureOpenAi: {
          apiKey: process.env.AZURE_API_KEY!,
          endpoint: `https://${process.env.AZURE_RESOURCE_NAME}.openai.azure.com/`,
          apiVersion: process.env.AZURE_API_VERSION!,
        },
        model: 'gpt-4o-mini',
      }),
    Levenshtein,
    containsParis,
  ],

helper function:

const azureScorer = <Output, Extra>(
  autoevalScorer: ScorerWithPartial<Output, Extra>
) => {
  return createScorer<string>({
    name: autoevalScorer.name,
    scorer: async (options) => {
      return autoevalScorer({
        ...options,
        azureOpenAi: {
          apiKey: process.env.AZURE_API_KEY!,
          endpoint: `https://${process.env.AZURE_RESOURCE_NAME}.openai.azure.com/`,
          apiVersion: process.env.AZURE_API_VERSION!,
        },
        model: 'gpt-4o-mini',
      });
    },
  });
};

having issue making it do perfect types

usage example:

const Factuality = azureScorer(FactualityAutoeval);

[mattpocock]

The current best way is to reimplement Factuality with a custom scorer. Have added a guide on how to do that here:

https://www.evalite.dev/guides/scorers