Skip to content

Commit fde7e52

Browse files
dbczumardbczumar
authored
Initial generated test cases for complex input structures and markdown inputs (#1808)
* generated Signed-off-by: dbczumar <[email protected]> * fix Signed-off-by: dbczumar <[email protected]> * fix Signed-off-by: dbczumar <[email protected]> * fix Signed-off-by: dbczumar <[email protected]> --------- Signed-off-by: dbczumar <[email protected]>
1 parent 83f24f9 commit fde7e52

File tree

15 files changed

+926
-0
lines changed

15 files changed

+926
-0
lines changed

‎tests/reliability/complex_types/generated/test_many_types_1/inputs/input1.json‎

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,26 @@
1+
{
2+
"assertions": [
3+
"The 'processedTupleField' should be a tuple containing a string and a number. Note that 'processedNestedObjectField.tupleField' should NOT actually be a tuple.",
4+
"The 'processedEnumField' should be one of the allowed enum values: 'option1', 'option2', or 'option3'.",
5+
"The 'processedDatetimeField' should be a date-time",
6+
"The 'processedLiteralField' should be exactly 'literalValue'.",
7+
"The 'processedObjectField' should contain 'subField1' (string), 'subField2' (number), and an additional boolean field 'additionalField'.",
8+
"The 'processedNestedObjectField' should contain 'tupleField' (which is actually a list with a string and a number - the name is misleading), 'enumField' (one of the allowed enum values), 'datetimeField' (string formatted as date-time), 'literalField' (exactly 'literalValue'), and an additional boolean field 'additionalField'."
9+
],
10+
"input": {
11+
"datetimeField": "2023-10-12T07:20:50.52Z",
12+
"enumField": "option1",
13+
"literalField": "literalValue",
14+
"nestedObjectField": {
15+
"datetimeField": "2023-10-12T07:20:50.52Z",
16+
"enumField": "option2",
17+
"literalField": "literalValue",
18+
"tupleField": ["nestedString", 789]
19+
},
20+
"objectField": {
21+
"subField1": "example",
22+
"subField2": 456
23+
},
24+
"tupleField": ["string1", 123]
25+
}
26+
}

‎tests/reliability/complex_types/generated/test_many_types_1/inputs/input2.json‎

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,26 @@
1+
{
2+
"assertions": [
3+
"The 'processedTupleField' should be an tuple with exactly two elements: the first element being a string and the second element being a number. Note that 'processedNestedObjectField.tupleField' should NOT actually be a tuple",
4+
"The 'processedEnumField' should be one of the predefined options: 'option1', 'option2', or 'option3'.",
5+
"The 'processedDatetimeField' should be a date-time",
6+
"The 'processedLiteralField' should be the enum 'literalValue'.",
7+
"The 'processedObjectField' should be an object containing 'subField1' as a string, 'subField2' as a number, and an 'additionalField' as a boolean.",
8+
"The 'processedNestedObjectField' should be an object containing 'tupleField' as a list (NOT a tuple) with exactly two elements (a string and a number), 'enumField' as one of the predefined options (option1, option2, or option3), 'datetimeField' as a 'date-time' object, 'literalField' as the string 'literalValue', and an 'additionalField' as a boolean."
9+
],
10+
"input": {
11+
"datetimeField": "2023-10-01T12:00:00Z",
12+
"enumField": "option1",
13+
"literalField": "literalValue",
14+
"nestedObjectField": {
15+
"datetimeField": "2023-11-01T12:00:00Z",
16+
"enumField": "option2",
17+
"literalField": "literalValue",
18+
"tupleField": ["nestedString", 789]
19+
},
20+
"objectField": {
21+
"subField1": "Patriotism is a feeling of love, devotion, and sense of attachment to one's country. This attachment can be a combination of many different feelings relating to one's homeland, including ethnic, cultural, political or historical aspects. It encompasses a set of concepts closely related to those of nationalism. In the context of patriotism, people may express their feelings in a variety of ways, including supporting their country's interests and policies, celebrating national holidays, and participating in civic activities. Patriotism often involves a sense of pride in one's country and a willingness to defend it against any threats. It can also include a commitment to improving the country and making it a better place for future generations. The concept of patriotism is often linked with the idea of national identity, which is the sense of a nation as a cohesive whole, as represented by distinctive traditions, culture, language, and politics. Patriots may feel a strong sense of loyalty and duty to their country, and they may take actions to support and protect it. However, it is important to note that patriotism can also be a complex and sometimes controversial concept. While it can inspire positive actions and a sense of community, it can also lead to exclusionary or aggressive behaviors if taken to an extreme. In some cases, excessive patriotism can result in nationalism, which can lead to conflicts with other nations or groups. Despite these potential issues, many people view patriotism as a positive force that can unite people and inspire them to work together for the common good. It can foster a sense of belonging and purpose, and it can motivate individuals to contribute to the well-being of their country. Overall, patriotism is a multifaceted and deeply personal sentiment that can manifest in many different ways, depending on an individual's experiences, beliefs, and values.",
22+
"subField2": 456
23+
},
24+
"tupleField": ["exampleString", 123]
25+
}
26+
}

‎tests/reliability/complex_types/generated/test_many_types_1/program.py‎

Lines changed: 120 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,120 @@
1+
### Input models ###
2+
3+
4+
from datetime import datetime
5+
from enum import Enum
6+
from typing import List, Tuple
7+
8+
from pydantic import BaseModel, Field
9+
10+
11+
class EnumField(Enum):
12+
option1 = "option1"
13+
option2 = "option2"
14+
option3 = "option3"
15+
16+
17+
class LiteralField(Enum):
18+
literalValue = "literalValue"
19+
20+
21+
class ObjectField(BaseModel):
22+
subField1: str
23+
subField2: float
24+
25+
26+
class NestedObjectField(BaseModel):
27+
tupleField: Tuple[str, float]
28+
enumField: EnumField
29+
datetimeField: datetime
30+
literalField: LiteralField
31+
32+
33+
class ProgramInputs(BaseModel):
34+
tupleField: Tuple[str, float]
35+
enumField: EnumField
36+
datetimeField: datetime
37+
literalField: LiteralField
38+
objectField: ObjectField
39+
nestedObjectField: NestedObjectField
40+
41+
42+
### Output models ###
43+
44+
45+
from datetime import datetime
46+
from enum import Enum
47+
from typing import List, Tuple, Union
48+
49+
from pydantic import BaseModel, Field
50+
51+
52+
class ProcessedEnumField(Enum):
53+
option1 = "option1"
54+
option2 = "option2"
55+
option3 = "option3"
56+
57+
58+
class ProcessedLiteralField(Enum):
59+
literalValue = "literalValue"
60+
61+
62+
class ProcessedObjectField(BaseModel):
63+
subField1: str
64+
subField2: float
65+
additionalField: bool
66+
67+
68+
class EnumField(Enum):
69+
option1 = "option1"
70+
option2 = "option2"
71+
option3 = "option3"
72+
73+
74+
class LiteralField(Enum):
75+
literalValue = "literalValue"
76+
77+
78+
class ProcessedNestedObjectField(BaseModel):
79+
tupleField: List[Union[str, float]] = Field(..., max_items=2, min_items=2)
80+
enumField: EnumField
81+
datetimeField: datetime
82+
literalField: LiteralField
83+
additionalField: bool
84+
85+
86+
class ProgramOutputs(BaseModel):
87+
processedTupleField: Tuple[str, float]
88+
processedEnumField: ProcessedEnumField
89+
processedDatetimeField: datetime
90+
processedLiteralField: ProcessedLiteralField
91+
processedObjectField: ProcessedObjectField
92+
processedNestedObjectField: ProcessedNestedObjectField
93+
94+
95+
### Program definition ###
96+
97+
import dspy
98+
99+
100+
class BaseSignature(dspy.Signature):
101+
"""
102+
The program is designed to process various data types including tuples, enums, datetime values, literals, objects, and nested objects containing these types. The program will accept inputs of these types, perform specified operations on them, and return the results. The operations could include validation, transformation, and extraction of information from these inputs.
103+
"""
104+
105+
106+
program_signature = BaseSignature
107+
for input_field_name, input_field in ProgramInputs.model_fields.items():
108+
program_signature = program_signature.append(
109+
name=input_field_name,
110+
field=dspy.InputField(description=input_field.description),
111+
type_=input_field.annotation,
112+
)
113+
for output_field_name, output_field in ProgramOutputs.model_fields.items():
114+
program_signature = program_signature.append(
115+
name=output_field_name,
116+
field=dspy.OutputField(description=input_field.description),
117+
type_=output_field.annotation,
118+
)
119+
120+
program = dspy.Predict(program_signature)

‎tests/reliability/complex_types/generated/test_many_types_1/schema.json‎

Lines changed: 184 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,184 @@
1+
{
2+
"description": "The program is designed to process various data types including tuples, enums, datetime values, literals, objects, and nested objects containing these types. The program will accept inputs of these types, perform specified operations on them, and return the results. The operations could include validation, transformation, and extraction of information from these inputs.",
3+
"properties": {
4+
"datetimeField": {
5+
"desc": null,
6+
"format": "date-time",
7+
"prefix": "Datetime Field:",
8+
"type": "string"
9+
},
10+
"enumField": {
11+
"enum": ["option1", "option2", "option3"],
12+
"type": "string"
13+
},
14+
"literalField": {
15+
"const": "literalValue",
16+
"enum": ["literalValue"],
17+
"type": "string"
18+
},
19+
"nestedObjectField": {
20+
"properties": {
21+
"datetimeField": {
22+
"format": "date-time",
23+
"type": "string"
24+
},
25+
"enumField": {
26+
"enum": ["option1", "option2", "option3"],
27+
"type": "string"
28+
},
29+
"literalField": {
30+
"const": "literalValue",
31+
"enum": ["literalValue"],
32+
"type": "string"
33+
},
34+
"tupleField": {
35+
"items": {
36+
"anyOf": [
37+
{
38+
"type": "string"
39+
},
40+
{
41+
"type": "number"
42+
}
43+
]
44+
},
45+
"maxItems": 2,
46+
"minItems": 2,
47+
"type": "array"
48+
}
49+
},
50+
"required": ["tupleField", "enumField", "datetimeField", "literalField"],
51+
"type": "object"
52+
},
53+
"objectField": {
54+
"properties": {
55+
"subField1": {
56+
"type": "string"
57+
},
58+
"subField2": {
59+
"type": "number"
60+
}
61+
},
62+
"required": ["subField1", "subField2"],
63+
"type": "object"
64+
},
65+
"processedDatetimeField": {
66+
"desc": null,
67+
"format": "date-time",
68+
"prefix": "Processed Datetime Field:",
69+
"type": "string"
70+
},
71+
"processedEnumField": {
72+
"enum": ["option1", "option2", "option3"],
73+
"type": "string"
74+
},
75+
"processedLiteralField": {
76+
"const": "literalValue",
77+
"enum": ["literalValue"],
78+
"type": "string"
79+
},
80+
"processedNestedObjectField": {
81+
"properties": {
82+
"additionalField": {
83+
"type": "boolean"
84+
},
85+
"datetimeField": {
86+
"format": "date-time",
87+
"type": "string"
88+
},
89+
"enumField": {
90+
"enum": ["option1", "option2", "option3"],
91+
"type": "string"
92+
},
93+
"literalField": {
94+
"const": "literalValue",
95+
"enum": ["literalValue"],
96+
"type": "string"
97+
},
98+
"tupleField": {
99+
"items": {
100+
"anyOf": [
101+
{
102+
"type": "string"
103+
},
104+
{
105+
"type": "number"
106+
}
107+
]
108+
},
109+
"maxItems": 2,
110+
"minItems": 2,
111+
"type": "array"
112+
}
113+
},
114+
"required": [
115+
"tupleField",
116+
"enumField",
117+
"datetimeField",
118+
"literalField",
119+
"additionalField"
120+
],
121+
"type": "object"
122+
},
123+
"processedObjectField": {
124+
"properties": {
125+
"additionalField": {
126+
"type": "boolean"
127+
},
128+
"subField1": {
129+
"type": "string"
130+
},
131+
"subField2": {
132+
"type": "number"
133+
}
134+
},
135+
"required": ["subField1", "subField2", "additionalField"],
136+
"type": "object"
137+
},
138+
"processedTupleField": {
139+
"desc": null,
140+
"items": {
141+
"anyOf": [
142+
{
143+
"type": "string"
144+
},
145+
{
146+
"type": "number"
147+
}
148+
]
149+
},
150+
"prefix": "Processed Tuple Field:",
151+
"type": "array"
152+
},
153+
"tupleField": {
154+
"desc": null,
155+
"items": {
156+
"anyOf": [
157+
{
158+
"type": "string"
159+
},
160+
{
161+
"type": "number"
162+
}
163+
]
164+
},
165+
"prefix": "Tuple Field:",
166+
"type": "array"
167+
}
168+
},
169+
"required": [
170+
"tupleField",
171+
"enumField",
172+
"datetimeField",
173+
"literalField",
174+
"objectField",
175+
"nestedObjectField",
176+
"processedTupleField",
177+
"processedEnumField",
178+
"processedDatetimeField",
179+
"processedLiteralField",
180+
"processedObjectField",
181+
"processedNestedObjectField"
182+
],
183+
"type": "object"
184+
}

‎tests/reliability/complex_types/generated/test_nesting_1/inputs/input1.json‎

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,27 @@
1+
{
2+
"assertions": [
3+
"The top-level output should contain the key 'resultLevel1'.",
4+
"'resultLevel1' should contain the key 'resultLevel2'.",
5+
"'resultLevel2' should contain the key 'resultLevel3'.",
6+
"'resultLevel3' should contain the key 'resultLevel4'.",
7+
"'resultLevel4' should contain the key 'resultLevel5'.",
8+
"'resultLevel5' should contain the key 'outputField1' which should be of type boolean.",
9+
"'resultLevel5' should contain the key 'outputField2' which should be an array of strings.",
10+
"'outputField1' should indicate success or failure with a boolean value.",
11+
"'outputField2' should contain messages represented as strings."
12+
],
13+
"input": {
14+
"level1": {
15+
"level2": {
16+
"level3": {
17+
"level4": {
18+
"level5": {
19+
"field1": "test_string",
20+
"field2": 42
21+
}
22+
}
23+
}
24+
}
25+
}
26+
}
27+
}

0 commit comments

Comments
 (0)