Add 'EvalTask'

[william-price01]

I want to be able to give my structure an Evalutation Task to make sure my output is correct.

Like this:

from griptape.structures import Pipeline
from griptape.tasks import EvalTask, PromptTask

pipeline = Pipeline(
tasks=[PromptTask("Write me a haiku, with lines from the matrix")])

pipeline.add_task(
EvalTask(
criteria="Evaluate the parent output {{parent_output}} against the rules provided.",
),
rules=[Rule("Make sure that the output includes lines from the matrix")]
)

pipeline.run()

[shhlife]

Would you expect the previous task to run again if the evaluation wasn't good enough?

[collindutter]

I had asked @william-price01 to create this issue but I'm going to convert to discussion since there are more unknowns than I originally thought of. For instance:

• What does the EvalTask return? The Engine's score, reason?
• What would you do with these results? @shhlife mentioned re-running previous Tasks, but that is a whole can of worms. What if you wanted to re-run a grandparent? What if you didn't want to loop at all?

There are clearly some unsolved use-cases here, but I'd like to think them through a bit more before beginning work.

[SavagePencil]

We've run into this on a separate project for quiz generation. We used a State Machine because fixing problems may not be as clear-cut. We need to know what was broken and then give guidance for how to course-correct those specific aspects.

[collindutter]

Exactly. This seems hard to generalize with something as simple as an EvalTask. I'm exploring some patterns for generating Tasks on the fly that might work here.