feat: add a script for running an evaluated app locally #29
Conversation
crisbeto
force-pushed
the
run-script
branch
2 times, most recently
from
9db6609 to
d494e7a
Compare
|
|
||
| async function resolveConfig(options: Options) { | ||
| if (!options.environment) { | ||
| throw new UserFacingError( |
There was a problem hiding this comment.
We aren't using Yargs demandOption here because we want this helpful error, right?
There was a problem hiding this comment.
Yep, the Yargs error isn't super readable.
runner/run-cli.ts
Outdated
| .option('prompt', { | ||
| type: 'string', | ||
| default: '', | ||
| description: 'Prompt within the environment that should be run', |
There was a problem hiding this comment.
I think the description is a bit ambiguous/confusing. Is this a path to a prompt, or a basename? Maybe it should be a path?
There was a problem hiding this comment.
It's actually the ID within the llm-output/<environment.id>. I'll update the message.
runner/run-cli.ts
Outdated
| console.error( | ||
| chalk.red('An error occurred during the assessment process:') | ||
| ); | ||
| console.error(chalk.red(error)); |
There was a problem hiding this comment.
should we print the stack in those cases? (if available)
There was a problem hiding this comment.
For the UserFacingError we don't print the stack trace, because it can be noisy. It's meant for more readable errors that we produce (e.g. the environment path is wrong).
There was a problem hiding this comment.
Right, but we could print if it's not the user facing error?
There was a problem hiding this comment.
Ah yeah in that case we should just throw. I copied this over from the eval-cli 😅
crisbeto
force-pushed
the
run-script
branch
from
d494e7a to
3d2ee12
Compare
Adds the `web-codegen-scorer run` script that allows users to run an evaluated app in their browser. It spins up a server using the local LLM output and the existing environment config.
crisbeto
force-pushed
the
run-script
branch
from
3d2ee12 to
4380765
Compare
2 participants
Adds the
web-codegen-scorer runscript that allows users to run an evaluated app in their browser. It spins up a server using the local LLM output and the existing environment config.