monkey test plans

Test plans for Monkey based NetSurf testing

Schema

Top level test mapping must have:

• Test title string.
• Test group string.
• Test steps sequence (of sequence step mappings).

Top level test mapping might optionally have:

• Test expected-result string. When not supplied, default is "pass".

The sequence step mappings must have:

• Step action string.

Depending on the action, there may be other keys in the mapping for that specific action type.

Step action types

launch

Starts NetSurf.

Optional parameters:

args: sequence of strings (command line arguments).

quit

Exits NetSurf.

Behaviour

Standard modes of failure (unless expected-result: fail):

• Test exits with non-zero return code.
• NetSurf logs something at ERROR or above.
• Leaked LWC strings on exit.
• TODO: Maybe fail if there are any Valgrind or sanitizer errors at any point?

Tests are composed of a sequence of actions, which can optionally specify failure conditions.

Examples

A test will fail if NetSurf exits with a non-zero return code.

title: start and stop browser
group: basic
steps:
- action: launch
- action: quit

There are launch and quit actions.

title: start and stop browser without JS
group: basic
steps:
- action: launch
  args:
  - enable_javascript=0
- action: quit

Use actions to get the browser to do things.

title: resource scheme
group: basic
steps:
- action: launch
  language: en
- action: window-new
  tag: win1
- action: navigate
  window: win1
  url: resource:does-not-exist
- action: block
  conditions:
  - window: win1
    status: complete
- action: plot-check
  - text-contains: Not found
  - text-contains: Error 404
- action: navigate
  window: win1
  url: resource:netsurf.png
- action: block
  conditions:
  - window: win1
    status: complete
- action: plot-check
  - bitmap-count: 1
- action: window-close
  - window: win1
- action: quit

Can make multiple windows fetch stuff at the same time, without block actions.

title: simultanious page fetches
group: real-world
steps:
- action: launch
  language: en
- action: window-new
  tag: win1
- action: window-new
  tag: win2
- action: window-new
  tag: win3
- action: window-new
  tag: win4
- action: navigate
  window: win1
  url: http://www.bbc.co.uk/news
- action: navigate
  window: win2
  url: http://www.amazon.co.uk/
- action: navigate
  window: win3
  url: http://www.theregister.co.uk/
- action: navigate
  window: win4
  url: http://www.arstechnica.co.uk/
- action: block
  conditions:
  - window: *all*
    status: complete
- action: window-close
  window: win1
- action: window-close
  window: win2
- action: quit

Can quit during navigate, to check cleanup in intermediate states. Note: this requires the test to run multiple times.

title: quitting mid-fetch
group: cleanup
steps:
- action: repeat
  min: 0
  step: 50
  name: sleepytimer
  actions:
  - action: launch
  - action: window-new
    tag: win1
  - action: navigate
    window: win1
    url: http://www.bbc.co.uk/news
  - action: sleep-ms
    time: sleepytimer
    conditions:
    - window: win1
      status: complete
    breaks: sleepytimer
  - action: quit

Can start and stop timers.

title: cache test
group: performance
steps:
- action: launch
  language: en
- action: timer-start
  tag: timer1
- action: window-new
  tag: win1
- action: navigate
  window: win1
  url: http://www.bbc.co.uk/news
- action: block
  conditions:
  - window: win1
    status: complete
- action: timer-stop
  timer: timer1
- action: timer-start
  tag: timer2
- action: window-new
  tag: win2
- action: navigate
  window: win2
  url: http://www.bbc.co.uk/news
- action: block
  conditions:
  - window: win2
    status: complete
- action: timer-stop
  timer: timer2
- action: timer-check
  condition: timer2 < timer1
- action: quit