Skip to content

Create Open Source "Eval" Suite #49

@handstandsam

Description

@handstandsam

We currently don't have an open source "Eval" suite to verify basic interactions. There is a test suite that is not public, but it would be good to create a public test suite to ensure evaluations are run on open source changes.

This change relies on #48 being done first.

Examples:

Level 1 - Basic Screen Interaction

  • Tap on UI elements
  • Verify that a string exists or does not exist on screen
  • Enter text into a text field
  • Verify up to 2-3 commands in a prompt

Level 2 - Screen Navigation and Basic Logic

  • Manipulate the correct screen or transition to the next screen:
    • Scroll to find something in a list
    • Tap on a button (e.g., Review Sale, Charge)
  • Perform basic math verifications:
    • Example: Stock quantity increases correctly after an action
  • Handle use of the back button
  • Navigate a ten-key (numeric keypad)
  • Fail gracefully if:
    • Element is impossible to tap
    • Text is impossible to enter
  • Edit existing text
  • Verify up to 5 commands in a prompt

Level 3 - Complex Interaction and Timing-Based Verification

  • Locate and select the correct text input field when multiple fields exist
  • Navigate screens with large number of interactive elements
  • Verify math or data consistency on a busy screen with many updates
  • Verify up to 10 commands in a prompt

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions