-
Notifications
You must be signed in to change notification settings - Fork 49
feat: add requirements and preconditions to gen slots #226
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from all commits
9a7bcb0
d9be154
5776225
6d471d8
04161de
feeb795
c92375e
ebbfbb6
238fdb4
4cc52ad
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,70 @@ | ||
| from typing import Literal | ||
|
|
||
| from mellea import generative, start_session | ||
| from mellea.stdlib.genslot import PreconditionException | ||
| from mellea.stdlib.requirement import Requirement, simple_validate | ||
| from mellea.stdlib.sampling.base import RejectionSamplingStrategy | ||
|
|
||
|
|
||
| @generative | ||
| def classify_sentiment(text: str) -> Literal["positive", "negative", "unknown"]: | ||
| """Classify the sentiment of the text.""" | ||
| ... | ||
|
|
||
|
|
||
| if __name__ == "__main__": | ||
| m = start_session() | ||
|
|
||
| # Add preconditions and requirements. | ||
| sentiment_component = classify_sentiment( | ||
| m, | ||
| text="I love this!", | ||
| # Preconditions are only checked with basic validation. Don't use the strategy. | ||
| precondition_requirements=["the text arg should be less than 100 words"], | ||
| # Reqs to use with the strategy. You could also just remove "unknown" from the structured output for this. | ||
| requirements=["avoid classifying the sentiment as unknown"], | ||
| strategy=RejectionSamplingStrategy(), # Must specify a strategy for gen slots | ||
| ) | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I was thinking of these parameters as parameters to the decorator. Was. there a reason to add it to the function call?
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Yes, there were some python type hinting limitations that could not be easily overcome: #190 (comment) |
||
|
|
||
| print( | ||
| f"Prompt to the model looked like:\n```\n{m.last_prompt()[0]['content']}\n```" | ||
| ) # type: ignore | ||
| # Prompt to the model looked like: | ||
| # ``` | ||
| # Your task is to imitate the output of the following function for the given arguments. | ||
| # Reply Nothing else but the output of the function. | ||
|
|
||
| # Function: | ||
| # def classify_sentiment(text: str) -> Literal['positive', 'negative', 'unknown']: | ||
| # """Classify the sentiment of the text. | ||
|
|
||
| # Postconditions: | ||
| # - avoid classifying the sentiment as unknown | ||
| # """ | ||
|
|
||
| # Arguments: | ||
| # - text: "I love this!" (type: <class 'str'>) | ||
| # ``` | ||
|
|
||
| print("\nOutput sentiment is:", sentiment_component) | ||
|
|
||
| # We can also force a precondition failure. | ||
| try: | ||
| sentiment_component = classify_sentiment( | ||
| m, | ||
| text="I hate this!", | ||
| # Requirement always fails to validate given the lambda. | ||
| precondition_requirements=[ | ||
| Requirement( | ||
| "the text arg should be only one word", | ||
| validation_fn=simple_validate(lambda x: (False, "Forced to fail!")), | ||
| ) | ||
| ], | ||
| ) | ||
| except PreconditionException as e: | ||
| print(f"exception: {str(e)}") | ||
|
|
||
| # Look at why the precondition validation failed. | ||
| print("Failure reasons:") | ||
| for val_result in e.validation: | ||
| print("-", val_result.reason) | ||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
That's cool. Is there a scenario where a precondition_check would be helpful for
m.instructas well?Uh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think most of the functionality that we add to a function like
@generativeorm.instruct, can / should be exposed at the level ofm.actat least. However, I think we would need to figure out all the hooks / processing / validation steps we want before doing that.