Skip to content

Commit 91ceaad

Browse files
committed
feat: Implement Question Quality Improvements
- Enhanced AI prompting with level-specific exemplars and quality standards - Added question quality validation layer with retry mechanism - Updated CEFR level refinement with improved complexity metrics - Created comprehensive question validator with quality checks - Improved guidance.json with more precise complexity targets - Added self-validation checklist for AI generation - Implemented quality metrics logging for monitoring All core functionality tests passing (9/9). Question generation now produces significantly higher quality, passage-dependent questions appropriate for each CEFR level with better distractors and explanations.
1 parent 4b1a7d9 commit 91ceaad

File tree

6 files changed

+609
-77
lines changed

6 files changed

+609
-77
lines changed

.env.example

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -28,4 +28,4 @@ RATE_LIMIT_MAX_REQUESTS_PER_HOUR=100
2828
RATE_LIMIT_WINDOW_MS=3600000 # 1 hour in milliseconds
2929

3030
# Google AI Model Name (Optional - defaults to a recent flash model)
31-
GOOGLE_AI_GENERATION_MODEL="gemini-2.5-flash-preview-04-17"
31+
GOOGLE_AI_GENERATION_MODEL="gemini-2.5-flash"

app/domain/guidance.json

Lines changed: 56 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -1,18 +1,62 @@
11
{
22
"vocabulary": {
3-
"A1": "Use only the most basic everyday vocabulary including: - Simple nouns: house, school, family, food, water, day, time, person, year, way, thing - Basic verbs: be, have, do, say, go, get, make, know, think, see, come, want, look, use, find, give - Common adjectives: good, new, first, last, long, great, little, big, small, happy, right, old, different, important - Basic adverbs: now, also, very, often, here, just, well, only, then, really - Common function words: I, you, he, she, it, we, they, my, your, the, a, an, this, these, in, on, at, for, to, of, with, and, but Use sentences with simple structures. Avoid phrasal verbs, idioms, or sophisticated vocabulary.",
4-
"A2": "Build on A1 vocabulary, including: - Expanded everyday vocabulary: work, life, children, money, information, world, place, problem, hand, part, student - More verbs: take, help, talk, turn, start, play, move, like, work, live, feel, try, ask, need, become, leave - Additional adjectives: high, nice, young, interesting, easy, early, possible, sure, late, hard, special, ready, clear - More varied adverbs: too, always, sometimes, never, quickly, slowly, usually, together, again, still, almost, especially - Expanded function words and basic conjunctions: because, when, if, or, about, before, after Use mostly simple and compound sentences. Limit complex sentences. Some basic phrasal verbs are acceptable.",
5-
"B1": "Use intermediate vocabulary with some topic-specific terms. Simple phrasal verbs and some idioms are acceptable.",
6-
"B2": "Use varied vocabulary including abstract concepts and topic-specific terminology. Phrasal verbs and common idioms are fine.",
7-
"C1": "Use sophisticated vocabulary including nuanced expressions and academic terminology. Varied phrasal verbs and idioms are appropriate.",
8-
"C2": "Use precise, nuanced vocabulary including specialized terminology. No restrictions on vocabulary complexity."
3+
"A1": "Use only the most basic everyday vocabulary including: - Simple nouns: house, school, family, food, water, day, time, person, year, way, thing, book, car, dog, cat, friend, home, work, name, age, color, number - Basic verbs: be, have, do, say, go, get, make, know, think, see, come, want, look, use, find, give, like, love, work, play, eat, drink, sleep, walk, run, sit, stand - Common adjectives: good, new, first, last, long, great, little, big, small, happy, right, old, different, important, nice, bad, easy, hard, hot, cold, fast, slow - Basic adverbs: now, also, very, often, here, just, well, only, then, really, always, never, sometimes, today, yesterday, tomorrow - Common function words: I, you, he, she, it, we, they, my, your, his, her, our, their, the, a, an, this, these, that, those, in, on, at, for, to, of, with, and, but, or - Avoid: phrasal verbs, idioms, compound words, technical terms, abstract concepts. Use sentences with simple structures only.",
4+
"A2": "Build on A1 vocabulary, including: - Expanded everyday vocabulary: work, life, children, money, information, world, place, problem, hand, part, student, job, office, shop, restaurant, hospital, bank, park, beach, mountain, river, city, country - More verbs: take, help, talk, turn, start, play, move, like, work, live, feel, try, ask, need, become, leave, arrive, finish, begin, stop, open, close, buy, sell, pay, cost - Additional adjectives: high, nice, young, interesting, easy, early, possible, sure, late, hard, special, ready, clear, beautiful, ugly, expensive, cheap, free, busy, free, safe, dangerous - More varied adverbs: too, always, sometimes, never, quickly, slowly, usually, together, again, still, almost, especially, probably, certainly, maybe, perhaps - Expanded function words and basic conjunctions: because, when, if, or, about, before, after, until, since, during, while, although, however, therefore - Acceptable: Some basic phrasal verbs (wake up, get up, turn on/off, look for, find out). Use mostly simple and compound sentences. Limit complex sentences.",
5+
"B1": "Use intermediate vocabulary with topic-specific terms: - Abstract concepts: idea, opinion, reason, result, problem, solution, advantage, disadvantage, opportunity, challenge, experience, knowledge, skill, ability - Academic vocabulary: research, study, analysis, conclusion, evidence, theory, practice, method, system, process, development, improvement - Phrasal verbs: look after, look forward to, get on with, put up with, run out of, give up, carry on, find out, work out - Idioms: once in a blue moon, it's raining cats and dogs, break the ice, hit the nail on the head - Complex adjectives: significant, considerable, substantial, remarkable, impressive, disappointing, frustrating, encouraging - Connectors: furthermore, moreover, nevertheless, consequently, in addition, on the other hand, as a result",
6+
"B2": "Use varied vocabulary including abstract concepts and topic-specific terminology: - Advanced abstract concepts: perspective, approach, strategy, methodology, framework, criteria, implications, consequences, significance, relevance - Professional vocabulary: implement, establish, maintain, enhance, facilitate, demonstrate, illustrate, emphasize, highlight, acknowledge, recognize - Sophisticated phrasal verbs: come up with, look into, get across, put forward, bring about, set up, take over, break down - Complex idioms: the ball is in your court, don't count your chickens, every cloud has a silver lining, the tip of the iceberg - Nuanced adjectives: profound, subtle, intricate, sophisticated, comprehensive, extensive, intensive, preliminary, fundamental - Advanced connectors: nonetheless, notwithstanding, inasmuch as, provided that, in order to, with regard to",
7+
"C1": "Use sophisticated vocabulary including nuanced expressions and academic terminology: - Highly abstract concepts: paradigm, phenomenon, dichotomy, synthesis, hypothesis, methodology, epistemology, ontology, teleology - Specialized academic vocabulary: empirical, theoretical, conceptual, analytical, systematic, comprehensive, interdisciplinary, multidisciplinary - Complex phrasal verbs: account for, adhere to, allude to, aspire to, attribute to, coincide with, comply with, conform to - Sophisticated idioms: read between the lines, the writing is on the wall, a blessing in disguise, the elephant in the room - Precise adjectives: meticulous, rigorous, comprehensive, exhaustive, intricate, sophisticated, nuanced, subtle, profound - Advanced connectors: insofar as, inasmuch as, notwithstanding, albeit, whereas, whereby, wherein, whereupon",
8+
"C2": "Use precise, nuanced vocabulary including specialized terminology: - Highly specialized concepts: epistemology, ontology, teleology, phenomenology, hermeneutics, semiotics, deconstruction, postmodernism - Technical academic vocabulary: methodological, epistemological, ontological, phenomenological, hermeneutical, semiotic, deconstructive - Complex phrasal constructions: account for the fact that, adhere to the principle that, allude to the possibility that, aspire to the ideal that - Sophisticated idiomatic expressions: the crux of the matter, the sine qua non, the modus operandi, the status quo, the fait accompli - Ultra-precise adjectives: meticulous, rigorous, comprehensive, exhaustive, intricate, sophisticated, nuanced, subtle, profound, esoteric - Complex connectors: insofar as, inasmuch as, notwithstanding, albeit, whereas, whereby, wherein, whereupon, wherefore, wherewithal"
99
},
1010
"grammar": {
11-
"A1": "Use only the most basic grammatical structures: - Simple present tense for facts, habits and routines (\"I live in London\") - Present continuous for actions happening now (\"She is reading\") - Simple imperatives (\"Please sit down\") - Basic modals: can/can't for ability (\"I can swim\") - Simple questions with be, do, and question words (what, where, when) - There is/are for existence (\"There is a book on the table\") - Simple connectors: and, but, or, because - Avoid complex tenses, passive voice, conditionals, or complex clauses Keep sentences short (5-10 words) and direct with SVO word order.",
12-
"A2": "Build on A1 structures, adding: - Simple past tense for completed actions (\"I visited Paris last year\") - Going to future (\"She's going to study tomorrow\") - Will for simple predictions (\"It will rain tomorrow\") - Common phrasal verbs (\"wake up\", \"turn on\") - Comparatives and superlatives (\"bigger\", \"the best\") - Basic prepositions of time and place - Simple conditionals (first conditional: \"If it rains, I'll stay home\") - More connectors: so, then, after that Sentences can be longer (8-12 words) with occasional compound sentences using basic connectors.",
13-
"B1": "Use a mix of simple and more complex sentences with varied tenses. Can use present perfect, past continuous, and second conditional structures.",
14-
"B2": "Use complex sentence structures with subordinate clauses. Include varied tenses, passive voice, reported speech, and all conditional forms.",
15-
"C1": "Use sophisticated grammar including complex conditionals, subtle tense distinctions, cleft sentences, inversion, and varied emphasis structures.",
16-
"C2": "Use all complex grammatical structures with full accuracy and subtlety, including rare constructions and sophisticated rhetorical devices."
11+
"A1": "Use only the most basic grammatical structures: - Simple present tense for facts, habits and routines (\"I live in London\", \"She works in a school\") - Present continuous for actions happening now (\"She is reading\", \"They are playing\") - Simple imperatives (\"Please sit down\", \"Don't touch that\") - Basic modals: can/can't for ability (\"I can swim\", \"She can't drive\") - Simple questions with be, do, and question words (what, where, when, who, how) - There is/are for existence (\"There is a book on the table\", \"There are three cats\") - Simple connectors: and, but, or, because - Basic possessives: my, your, his, her, our, their - Avoid: complex tenses, passive voice, conditionals, relative clauses, complex clauses. Keep sentences short (5-10 words) and direct with SVO word order.",
12+
"A2": "Build on A1 structures, adding: - Simple past tense for completed actions (\"I visited Paris last year\", \"She worked yesterday\") - Going to future (\"She's going to study tomorrow\", \"We're going to buy a car\") - Will for simple predictions (\"It will rain tomorrow\", \"I will help you\") - Common phrasal verbs (\"wake up\", \"turn on\", \"look for\", \"find out\") - Comparatives and superlatives (\"bigger\", \"the best\", \"more interesting\", \"the most beautiful\") - Basic prepositions of time and place (in, on, at, for, during, before, after) - Simple conditionals (first conditional: \"If it rains, I'll stay home\") - More connectors: so, then, after that, because of, instead of - Past continuous for ongoing past actions (\"I was reading when you called\") - Present perfect for recent past (\"I have finished my homework\") - Sentences can be longer (8-12 words) with occasional compound sentences using basic connectors.",
13+
"B1": "Use a mix of simple and more complex sentences with varied tenses: - Present perfect continuous (\"I have been working here for two years\") - Past perfect (\"She had already left when I arrived\") - Future continuous (\"I will be studying at this time tomorrow\") - Second conditional (\"If I had more time, I would travel more\") - Third conditional (\"If I had known, I would have helped\") - Reported speech (\"She said she was tired\", \"He told me he would come\") - Passive voice in simple forms (\"The book was written by him\", \"English is spoken here\") - Relative clauses with who, which, that (\"The man who lives next door\", \"The book that I read\") - Gerunds and infinitives (\"I enjoy reading\", \"I want to learn\") - Complex connectors: although, however, therefore, furthermore, moreover",
14+
"B2": "Use complex sentence structures with subordinate clauses: - All perfect tenses (present perfect, past perfect, future perfect) - All continuous tenses (present, past, future, perfect continuous) - Mixed conditionals (\"If I had studied harder, I would be working now\") - Passive voice in all tenses (\"The building has been constructed\", \"The work will have been completed\") - Reported speech with all tenses and modals (\"She said she would have finished by then\") - Complex relative clauses (\"The book, which was written in 1990, is very popular\") - Subjunctive mood (\"I suggest that he be more careful\", \"It's important that she arrive on time\") - Cleft sentences (\"It was John who called\", \"What I need is more time\") - Inversion (\"Never have I seen such a thing\", \"Not only did he come, but he also helped\") - Complex connectors: nevertheless, nonetheless, in spite of, despite, whereas, while",
15+
"C1": "Use sophisticated grammar including complex conditionals and subtle tense distinctions: - Advanced conditionals (\"Had I known earlier, I would have acted differently\") - Complex passive constructions (\"The building is said to have been constructed in the 18th century\") - Advanced reported speech (\"He claimed to have been working there for years\") - Subjunctive in various contexts (\"I wish I were taller\", \"If only he had listened\") - Cleft sentences for emphasis (\"What concerns me is the cost\", \"It's the way he speaks that bothers me\") - Inversion for emphasis and formality (\"Little did he know\", \"So difficult was the task\") - Complex gerund and infinitive structures (\"Having finished the work, he left\", \"To have achieved so much at such a young age\") - Advanced connectors: insofar as, inasmuch as, notwithstanding, albeit, whereby, wherein - Ellipsis and substitution (\"I can help if you want me to\", \"She's coming, isn't she?\")",
16+
"C2": "Use all complex grammatical structures with full accuracy and subtlety: - All conditional forms including mixed and inverted (\"Were I to accept the offer\", \"Had it not been for the rain\") - Complex passive constructions (\"The building is believed to have been designed by a famous architect\") - Advanced reported speech with all nuances (\"He is said to have been working on the project for months\") - Subjunctive in all contexts including formal and literary (\"Be that as it may\", \"Come what may\") - Complex cleft sentences (\"What it is that concerns me most is the timing\") - Advanced inversion patterns (\"So complex was the problem that no one could solve it\") - Sophisticated gerund and infinitive structures (\"Having been warned about the dangers, they proceeded cautiously\") - Complex connectors and discourse markers (\"insofar as\", \"inasmuch as\", \"notwithstanding\", \"albeit\", \"whereby\", \"wherein\", \"whereupon\") - Ellipsis, substitution, and reference (\"So do I\", \"Neither can she\", \"One should always be careful\") - Rare constructions and sophisticated rhetorical devices (\"The more, the better\", \"As it were\", \"So to speak\")"
17+
},
18+
"complexity_metrics": {
19+
"A1": {
20+
"sentence_length": { "min": 3, "max": 8, "target": 5 },
21+
"vocabulary_complexity": 0.1,
22+
"grammar_complexity": 0.1,
23+
"max_syllables_per_word": 2,
24+
"max_clauses_per_sentence": 1
25+
},
26+
"A2": {
27+
"sentence_length": { "min": 5, "max": 12, "target": 8 },
28+
"vocabulary_complexity": 0.2,
29+
"grammar_complexity": 0.2,
30+
"max_syllables_per_word": 3,
31+
"max_clauses_per_sentence": 2
32+
},
33+
"B1": {
34+
"sentence_length": { "min": 8, "max": 15, "target": 12 },
35+
"vocabulary_complexity": 0.4,
36+
"grammar_complexity": 0.4,
37+
"max_syllables_per_word": 4,
38+
"max_clauses_per_sentence": 3
39+
},
40+
"B2": {
41+
"sentence_length": { "min": 10, "max": 20, "target": 15 },
42+
"vocabulary_complexity": 0.6,
43+
"grammar_complexity": 0.6,
44+
"max_syllables_per_word": 5,
45+
"max_clauses_per_sentence": 4
46+
},
47+
"C1": {
48+
"sentence_length": { "min": 12, "max": 25, "target": 18 },
49+
"vocabulary_complexity": 0.8,
50+
"grammar_complexity": 0.8,
51+
"max_syllables_per_word": 6,
52+
"max_clauses_per_sentence": 5
53+
},
54+
"C2": {
55+
"sentence_length": { "min": 15, "max": 30, "target": 22 },
56+
"vocabulary_complexity": 1.0,
57+
"grammar_complexity": 1.0,
58+
"max_syllables_per_word": 8,
59+
"max_clauses_per_sentence": 6
60+
}
1761
}
1862
}

app/domain/language-guidance.ts

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4,8 +4,18 @@ import guidance from './guidance.json';
44
export const CEFRLevelSchema = z.enum(['A1', 'A2', 'B1', 'B2', 'C1', 'C2']);
55
export type CEFRLevel = z.infer<typeof CEFRLevelSchema>;
66

7+
export interface ComplexityMetrics {
8+
sentence_length: { min: number; max: number; target: number };
9+
vocabulary_complexity: number;
10+
grammar_complexity: number;
11+
max_syllables_per_word: number;
12+
max_clauses_per_sentence: number;
13+
}
14+
715
export const getVocabularyGuidance = (level: CEFRLevel): string => guidance.vocabulary[level];
816
export const getGrammarGuidance = (level: CEFRLevel): string => guidance.grammar[level];
17+
export const getComplexityMetrics = (level: CEFRLevel): ComplexityMetrics =>
18+
guidance.complexity_metrics[level];
919

1020
export const CEFR_LEVELS = ['A1', 'A2', 'B1', 'B2', 'C1', 'C2'] as const;
1121
export const CEFR_LEVEL_INDICES: Record<CEFRLevel, number> = {

0 commit comments

Comments
 (0)