|
| 1 | +# Running AutoPTSClientBot |
| 2 | + |
| 3 | +AutoPTSClientBot has been added to automate running test cases on various |
| 4 | +configurations and reporting the results. |
| 5 | + |
| 6 | +**Key features** |
| 7 | + |
| 8 | +- Fetching recent project sources |
| 9 | +- Applying changes to project configuration files via "iut_config" |
| 10 | +with "overlay" that need to be applied for "test_cases" |
| 11 | +- setting test-specific retry count via "retry_config" |
| 12 | +- Building ZephyrOS/MynewtOS image |
| 13 | +- Flashing board |
| 14 | +- Running all the test cases |
| 15 | +- Archiving test execution logs |
| 16 | +- Storing the results in Excel spreadsheet file |
| 17 | + - test case statuses |
| 18 | + - pie chart |
| 19 | +- Sending the files to the Google Drive |
| 20 | +- Sending e-mail |
| 21 | + |
| 22 | +**Configuration** |
| 23 | + |
| 24 | +The Bot configuration file is located in `bot` . Sample files `bot/config.py.zephyr.sample` |
| 25 | +and `bot/config.py.mynewt.sample` are provided. The file contains setup and project-specific configuration. |
| 26 | + |
| 27 | +This may contain few sections: |
| 28 | +- `name` - AutoPTS project name |
| 29 | +* `auto_pts` - AutoPTS configuration |
| 30 | + - `iut_mode` - Specifies the mode of the IUT (e.g. native, qemu, tty) |
| 31 | + - `workspace` - Path to PTS workspace .pqw6 file to use for testing. THe file should be located where automation |
| 32 | + server is running. |
| 33 | + - `project_path` - path to project source directory |
| 34 | + - `cli_port` - AutoPTSClient port(s). If running with multiple servers(PTS dongles), specify the ports in |
| 35 | + config_project.py as follows: 'srv_port': [65001, 65003, 65005] |
| 36 | + - `srv_port` - AutoPTSServer port(s). If running with multiple servers(PTS dongles), specify the ports in |
| 37 | + config_project.py as follows: 'srv_port': [65000, 65002, 65004] |
| 38 | + - `ip_addr` - IP address of the PTS automation servers. |
| 39 | + - `client_ip` - AutoPTSClient IP address/es |
| 40 | + - `server_ip` - AutoPTSServer IP address/es |
| 41 | + - `store` - set True to save run results in a database .db file (default False) |
| 42 | + - `rtt_log` - collect IUT logs via RTT J-Link buffer named "Logger" |
| 43 | + - `btmon` - collect IUT btsnoops with btmon |
| 44 | + - `device_core` - Specify the device core for JLink related features, e.g. BTMON or RTT logging |
| 45 | + - `qemu_bin` - The path to the QEMU executable, e.g. /usr/bin/qemu-system-arm |
| 46 | + - `qemu_options` - Additional options for the qemu, e.g. -cpu cortex-m3 -machine lm3s6965evb |
| 47 | + - `btattach_bin` - The path to the btattach executable, e.g. /usr/bin/btattach |
| 48 | + - `btattach_at_every_test_case` - Use btattach at every test case |
| 49 | + - `btproxy_bin` - The path to the btproxy executable, e.g. /usr/bin/btproxy |
| 50 | + - `btmgmt_bin` - The path to the btmgmt executable, e.g. /usr/bin/btmgmt |
| 51 | + - `hid_vid` - Specify the VID of the USB device used as a HCI controller (hexadecimal string, e.g. '2fe3') |
| 52 | + - `hid_pid` - Specify the PID of the USB device used as a HCI controller (hexadecimal string, e.g. '000b') |
| 53 | + - `hid_serial` - Specify the serial number of the USB device used as a HCI controller |
| 54 | + - `kernel_cpu` - The type of CPU that will be used for building an image, e.g. qemu_cortex_m3 |
| 55 | + - `setcap_cmd` - Command to set HCI access permissions for zephyr.exe in native mode, |
| 56 | + e.g. sudo /usr/sbin/setcap cap_net_raw,cap_net_admin,cap_sys_admin+ep /path/to/zephyr.exe. To allow sudo setcap |
| 57 | + without password, add to visudo a line like this: youruser ALL=(ALL) NOPASSWD: /usr/sbin/setcap |
| 58 | + - `hci` - Specify the number of the HCI controller |
| 59 | + - `database_file` - custom path to database .db file (default path: <project-dir>/TestCase.db) |
| 60 | + - `board` - IUT used. Currently nrf52 is supported only |
| 61 | + - `enable_max_logs` - enable debug logs |
| 62 | + - `retry` - maximum repeat count per test |
| 63 | + - `no_retry_on_regression` - When skip_retry is used, failed test cases are handled as follows: if test failure is |
| 64 | + not a regression, test case will not be retried (i.e. retry is ignored). If the failure is regression, test case will |
| 65 | + be retried for retry number of times. If you set retry to zero, no failed test cases will be retried. |
| 66 | + - `stress test` - repeat every test `retry` number of times, even if result was PASS |
| 67 | + - `bd_addr` - IUT Bluetooth Address (optional) |
| 68 | + - `recovery` - enable recovery after non-valid result (optional) |
| 69 | + - `superguard` - force recovery when server has been idle for the given time (optional) |
| 70 | + - `ykush` - reconnect board/PTS dongle during recovery, if YKUSH Switchable Hub is used (optional) |
| 71 | + - `ykush_replug_delay` - delay ykush replug |
| 72 | + - `repeat_until_fail` - keep repeating test case until fail verdict |
| 73 | + - `test_case_limit` - limits number of test cases to be run. Useful when passing test group as an argument |
| 74 | + - `pylink_reset' - Use pylink reset |
| 75 | + - `no_build` - Skip build and flash in bot mode |
| 76 | + - `dongle_init_retry` - number of times autoptsserver will try to launch |
| 77 | + - `build_env_cmd` - path to virtual environment needed for build and flash step |
| 78 | + - `copy` - Create a copy of workspace file |
| 79 | + - `wid_usage` - Create a report of wid usage |
| 80 | +- `git` - Git repositories configuration (optional) |
| 81 | + - `path` - path to project repo |
| 82 | + - `remote` - git remote repo name |
| 83 | + - `branch` - branch selected at git checkout |
| 84 | + - `stash_changes` - stash changes if local repo is dirty |
| 85 | + - `update_repo` - if False, prevents bot from updating the repo |
| 86 | +- `mail` - Mail configuration (optional) |
| 87 | + - `sender` - sender e-mail address |
| 88 | + - `smtp_host`, `smtp_port` - sender SMTP configuration |
| 89 | + - `subject` - overrides default email title |
| 90 | + - `name` - to be used in message footer |
| 91 | + - `passwd` - sender mailbox password. When Google account is used [allow |
| 92 | + less secure apps to access account](https://myaccount.google.com/lesssecureapps) |
| 93 | + - `recipients` - list of e-mail addresses |
| 94 | + - `start_tls` - put SMTP server into TLS mode |
| 95 | +* `gdrive` - GDrive configuration (optional) |
| 96 | + - `root_directory_id` - Root Directory ID, can be obtained from URL, |
| 97 | + `https://drive.google.com/drive/u/0/folders/<GoogleDriveDirID>` |
| 98 | + - `credentials_file` - path to credentials file to access Google Account. |
| 99 | + Read more [here](https://developers.google.com/drive/v3/web/quickstart/python). |
| 100 | +- `iut_config` - IUT configuration overlay. This is used to apply test case |
| 101 | +specific changes in IUT configuration. It consists of dict of configuration |
| 102 | +names and related key: value pairs: |
| 103 | + - `overlay` - changes in config to be applied |
| 104 | + - `test_cases` - test cases to be ran with this config |
| 105 | + - `excluded` - Names of test cases to exclude. Groups of test cases can be specified by profile names. |
| 106 | +- `retry_config` - allows to set test-specific retry count. This value overrides |
| 107 | +default setting `retry`, allowing to both increase and decrease it. This setting |
| 108 | +is in form of dictionary of testcase names and retry counts (as `int`) |
| 109 | +* `scheduler` - Scheduler configuration (optional) |
| 110 | + - `weekday`: "time" dictionary. |
| 111 | + |
| 112 | +**Installation** |
| 113 | + |
| 114 | +Install required Python modules with: |
| 115 | + |
| 116 | + pip3 install --user -r bot/requirements.txt |
| 117 | + |
| 118 | +**Usage** |
| 119 | + |
| 120 | + ./autoptsclient_bot.py |
0 commit comments