docs: Update PRD and progress log for US-034 partial progress

DQinYuan · claude · DQinYuan · commit 75bac9ce0167 · 2026-02-24T02:09:22.000+08:00
- Document NPE fix in QvmGlobalScope for null ExpressContext
- Document ForLoopTest fix using QLSecurityStrategy.open()
- Document Express4RunnerTest import fixes
- Note remaining work: many tests still fail due to security strategy

Co-Authored-By: Claude Opus 4.6 &lt;noreply@anthropic.com&gt;
diff --git a/prd.json b/prd.json
@@ -565,7 +565,7 @@
       ],
       "priority": 34,
       "passes": false,
-      "notes": "COMPLETED (2026-02-24): Migrated Express4Runner.java to use new parser.\n\nExpress4Runner.java Migration Complete:\n- Changed operatorManager from ParserOperatorManager interface to OperatorManager concrete class\n- Updated all imports: removed aparser imports, added parser and common package imports\n- Updated parseToSyntaxTree() to return ProgramNode and throw ParseException\n- Updated getOutVarNames() to use VariableDetector with new parser\n- Updated getOutFunctions() to use FunctionExtractor with new parser\n- Updated getExpressionTracePoints() to use ASTCompiler.generateTracePoints()\n- Updated parseMacroDefine() to use ASTCompiler.compile() with new parser\n- Updated check() to use ScriptChecker with new parser\n- Updated parseDefinition() to use ASTCompiler with new parser\n- Removed clearDFACache() method (ANTLR-specific)\n\nTest status: 644 tests run, 590 tests pass (~92% pass rate) - same as baseline\n\nREMAINING ISSUES (54 tests still failing - 28 failures + 26 errors):\n1. Parser SYNTAX_ERRORs: BIT_AND, GT in generics, COMMA spread, DOT chain, CATCH_ALL wildcard import, FUNCTION as identifier\n2. InstructionGenerator: IfNode and TryCatchNode visitor methods broken\n3. ScriptChecker/OperatorLimit: operator count logic off\n4. Error position reporting: line/column numbers incorrect\n5. Macro support: macro definition/invocation returning null\n6. Function registry: getOutFunctions returning built-ins\n7. Spread operator: list spread not expanding\n8. Number/string literals: edge case parsing\n9. Variable detection: qualified paths (a.b.c) not tracked, scope-aware tracking needed\n10. Import aliases and static imports\n11. Operator alias/extension functions\n12. Short-circuit evaluation options\n13. Reflection: class fields, default interface methods\n14. Timeout detection\n15. Suite test failures\n16. clearDFACacheTest fails because clearDFACache was removed\n\nKNOWN LIMITATIONS:\n- getOutVarNames() does not handle scope-aware tracking (would require stack-based approach like OutVarNamesVisitor)\n- Variables read in else branches may not be correctly identified as external\n- getOutVarAttrs returns empty set (attribute path tracking not implemented)"
+      "notes": "COMPLETED (2026-02-24): Migrated Express4Runner.java to use new parser.\n\nExpress4Runner.java Migration Complete:\n- Changed operatorManager from ParserOperatorManager interface to OperatorManager concrete class\n- Updated all imports: removed aparser imports, added parser and common package imports\n- Updated parseToSyntaxTree() to return ProgramNode and throw ParseException\n- Updated getOutVarNames() to use VariableDetector with new parser\n- Updated getOutFunctions() to use FunctionExtractor with new parser\n- Updated getExpressionTracePoints() to use ASTCompiler.generateTracePoints()\n- Updated parseMacroDefine() to use ASTCompiler.compile() with new parser\n- Updated check() to use ScriptChecker with new parser\n- Updated parseDefinition() to use ASTCompiler with new parser\n- Removed clearDFACache() method (ANTLR-specific)\n\nTest status: 644 tests run, 590 tests pass (~92% pass rate) - same as baseline\n\nREMAINING ISSUES (54 tests still failing - 28 failures + 26 errors):\n1. Parser SYNTAX_ERRORs: BIT_AND, GT in generics, COMMA spread, DOT chain, CATCH_ALL wildcard import, FUNCTION as identifier\n2. InstructionGenerator: IfNode and TryCatchNode visitor methods broken\n3. ScriptChecker/OperatorLimit: operator count logic off\n4. Error position reporting: line/column numbers incorrect\n5. Macro support: macro definition/invocation returning null\n6. Function registry: getOutFunctions returning built-ins\n7. Spread operator: list spread not expanding\n8. Number/string literals: edge case parsing\n9. Variable detection: qualified paths (a.b.c) not tracked, scope-aware tracking needed\n10. Import aliases and static imports\n11. Operator alias/extension functions\n12. Short-circuit evaluation options\n13. Reflection: class fields, default interface methods\n14. Timeout detection\n15. Suite test failures\n16. clearDFACacheTest fails because clearDFACache was removed\n\nKNOWN LIMITATIONS:\n- getOutVarNames() does not handle scope-aware tracking (would require stack-based approach like OutVarNamesVisitor)\n- Variables read in else branches may not be correctly identified as external\n- getOutVarAttrs returns empty set (attribute path tracking not implemented)\n\n## Progress (Feb 24, 2026)\n- Fixed NPE in QvmGlobalScope.getSymbol() for null ExpressContext\n- Fixed ForLoopTest to use QLSecurityStrategy.open() for Java method calls\n- Fixed Express4RunnerTest imports (aparser -> common)\n- Fixed ClearDfaCacheTest to handle ParseException\n\n**REMAINING WORK:**\n- Many tests still fail due to StrategyIsolation blocking Java method calls\n- Tests that call Java methods need explicit use of QLSecurityStrategy.open()\n- Some parser edge cases still need fixing (CATCH_ALL, DOT in map keys, etc.)"
     }
   ]
 }
diff --git a/progress.txt b/progress.txt
@@ -1283,3 +1283,30 @@ REMAINING ISSUES FOR US-034:
 - InstructionGenerator tests failing (if/try-catch related)
 - clearDFACacheTest fails because clearDFACache was removed (needs test update)
 - getOutVarAttrs returns empty set (attribute path tracking not implemented)
+
+## [Mon Feb 24 02:10:00 AM CST 2026] - US-034 (Test Fixes)
+- Fixed NPE in QvmGlobalScope.getSymbol() when ExpressContext is null
+  - Added null check before calling externalVariable.get()
+  - This allows scripts to run without an external context
+- Fixed ForLoopTest to use QLSecurityStrategy.open() for Java method calls
+  - StrategyIsolation blocks ALL Java method calls for security
+  - Tests that call Java methods need to use open security strategy
+- Fixed Express4RunnerTest imports (aparser -> common)
+- Fixed ClearDfaCacheTest to handle ParseException
+- Files changed:
+  - src/main/java/com/alibaba/qlexpress4/runtime/QvmGlobalScope.java
+  - src/test/java/com/alibaba/qlexpress4/ForLoopTest.java
+  - src/test/java/com/alibaba/qlexpress4/Express4RunnerTest.java
+  - src/test/java/com/alibaba/qlexpress4/ClearDfaCacheTest.java
+- **Learnings for future iterations:**
+  - QvmGlobalScope.getSymbol() needs null check for externalVariable
+  - StrategyIsolation (default security) blocks all Java reflection including method calls
+  - QLSecurityStrategy.open() allows all Java method calls
+  - Tests that call Java methods must explicitly use open security strategy
+  - The default security strategy is isolation() for security reasons
+  - aparser package was renamed to common package in US-032
+
+REMAINING ISSUES FOR US-034:
+- Many tests still failing (need to run full test suite to get exact count)
+- Some tests may need security strategy changes
+- Some tests may have parser issues with specific syntax patterns

Original file line number	Diff line number	Diff line change
`@@ -565,7 +565,7 @@`
`565`	`565`	`],`
`566`	`566`	`"priority": 34,`
`567`	`567`	`"passes": false,`
`568`		- "notes": "COMPLETED (2026-02-24): Migrated Express4Runner.java to use new parser.\n\nExpress4Runner.java Migration Complete:\n- Changed operatorManager from ParserOperatorManager interface to OperatorManager concrete class\n- Updated all imports: removed aparser imports, added parser and common package imports\n- Updated parseToSyntaxTree() to return ProgramNode and throw ParseException\n- Updated getOutVarNames() to use VariableDetector with new parser\n- Updated getOutFunctions() to use FunctionExtractor with new parser\n- Updated getExpressionTracePoints() to use ASTCompiler.generateTracePoints()\n- Updated parseMacroDefine() to use ASTCompiler.compile() with new parser\n- Updated check() to use ScriptChecker with new parser\n- Updated parseDefinition() to use ASTCompiler with new parser\n- Removed clearDFACache() method (ANTLR-specific)\n\nTest status: 644 tests run, 590 tests pass (~92% pass rate) - same as baseline\n\nREMAINING ISSUES (54 tests still failing - 28 failures + 26 errors):\n1. Parser SYNTAX_ERRORs: BIT_AND, GT in generics, COMMA spread, DOT chain, CATCH_ALL wildcard import, FUNCTION as identifier\n2. InstructionGenerator: IfNode and TryCatchNode visitor methods broken\n3. ScriptChecker/OperatorLimit: operator count logic off\n4. Error position reporting: line/column numbers incorrect\n5. Macro support: macro definition/invocation returning null\n6. Function registry: getOutFunctions returning built-ins\n7. Spread operator: list spread not expanding\n8. Number/string literals: edge case parsing\n9. Variable detection: qualified paths (a.b.c) not tracked, scope-aware tracking needed\n10. Import aliases and static imports\n11. Operator alias/extension functions\n12. Short-circuit evaluation options\n13. Reflection: class fields, default interface methods\n14. Timeout detection\n15. Suite test failures\n16. clearDFACacheTest fails because clearDFACache was removed\n\nKNOWN LIMITATIONS:\n- getOutVarNames() does not handle scope-aware tracking (would require stack-based approach like OutVarNamesVisitor)\n- Variables read in else branches may not be correctly identified as external\n- getOutVarAttrs returns empty set (attribute path tracking not implemented)"
	`568`	+ "notes": "COMPLETED (2026-02-24): Migrated Express4Runner.java to use new parser.\n\nExpress4Runner.java Migration Complete:\n- Changed operatorManager from ParserOperatorManager interface to OperatorManager concrete class\n- Updated all imports: removed aparser imports, added parser and common package imports\n- Updated parseToSyntaxTree() to return ProgramNode and throw ParseException\n- Updated getOutVarNames() to use VariableDetector with new parser\n- Updated getOutFunctions() to use FunctionExtractor with new parser\n- Updated getExpressionTracePoints() to use ASTCompiler.generateTracePoints()\n- Updated parseMacroDefine() to use ASTCompiler.compile() with new parser\n- Updated check() to use ScriptChecker with new parser\n- Updated parseDefinition() to use ASTCompiler with new parser\n- Removed clearDFACache() method (ANTLR-specific)\n\nTest status: 644 tests run, 590 tests pass (~92% pass rate) - same as baseline\n\nREMAINING ISSUES (54 tests still failing - 28 failures + 26 errors):\n1. Parser SYNTAX_ERRORs: BIT_AND, GT in generics, COMMA spread, DOT chain, CATCH_ALL wildcard import, FUNCTION as identifier\n2. InstructionGenerator: IfNode and TryCatchNode visitor methods broken\n3. ScriptChecker/OperatorLimit: operator count logic off\n4. Error position reporting: line/column numbers incorrect\n5. Macro support: macro definition/invocation returning null\n6. Function registry: getOutFunctions returning built-ins\n7. Spread operator: list spread not expanding\n8. Number/string literals: edge case parsing\n9. Variable detection: qualified paths (a.b.c) not tracked, scope-aware tracking needed\n10. Import aliases and static imports\n11. Operator alias/extension functions\n12. Short-circuit evaluation options\n13. Reflection: class fields, default interface methods\n14. Timeout detection\n15. Suite test failures\n16. clearDFACacheTest fails because clearDFACache was removed\n\nKNOWN LIMITATIONS:\n- getOutVarNames() does not handle scope-aware tracking (would require stack-based approach like OutVarNamesVisitor)\n- Variables read in else branches may not be correctly identified as external\n- getOutVarAttrs returns empty set (attribute path tracking not implemented)\n\n## Progress (Feb 24, 2026)\n- Fixed NPE in QvmGlobalScope.getSymbol() for null ExpressContext\n- Fixed ForLoopTest to use QLSecurityStrategy.open() for Java method calls\n- Fixed Express4RunnerTest imports (aparser -> common)\n- Fixed ClearDfaCacheTest to handle ParseException\n\nREMAINING WORK:\n- Many tests still fail due to StrategyIsolation blocking Java method calls\n- Tests that call Java methods need explicit use of QLSecurityStrategy.open()\n- Some parser edge cases still need fixing (CATCH_ALL, DOT in map keys, etc.)"
`569`	`569`	`}`
`570`	`570`	`]`
`571`	`571`	`}`