Add initial test procedures for WebUSB & BLE flashing.

docs/release-tests/ble-flashing.md

@@ -0,0 +1,120 @@
+# Bluetooth Tests
+
+## Test preparations
+
+Four hex files needed:
+- "Meet the micro:bit": This is the programme that runs on the micro:bit when
+  it is first powered on from factory.
+    - Downloadable from the micro:bit website:
+      https://microbit.org/get-started/user-guide/meet-the-microbit-program/
+- "Out of Box": The previous factory programme before the
+  "Meet the micro:bit" was released.
+    - Downloadable from the micro:bit website:
+      https://microbit.org/get-started/user-guide/meet-the-microbit-program/
+- MakeCode Live Editor hex file: To ensure micro:bits with a recent MakeCode
+  project can be flashed via Bluetooth with the MakeCode editor under test.
+    - Create a hex file from the latest MakeCode live editor:
+      https://makecode.microbit.org/
+- Python Editor hex file: To ensure the MakeCode editor under test can
+  BLE flash micro:bits after using the Python Editor.
+    - Go to https://python.microbit.org and download the hex file
+      with the default project.
+    - This hex file can only be tested with micro:bit V2
+- MakeCode Under Test hex file: To ensure it can be flashed over.
+
+Two mobile/tablet devices needed:
+- Android device with the micro:bit app installed.
+    - To load a custom MakeCode URL long touch the micro:bit logo in the app
+      main screen. A modal will appear with the option change the MakeCode URL.
+    - This URL setting will be reset every time the app is quit
+- iOS device with the micro:bit app installed.
+    - To load a different MakeCode version, long press the MakeCode back
+      chevron to choose beta.
+    - The iOS app can only choose between beta and live.
+
+Two micro:bits needed:
+- micro:bit V1
+- micro:bit V2
+
+The micro:bits can be powered via USB for these tests, as that avoids any
+possible issues or errors due to low batteries.
+
+## Test 1: Full & partial flashing with Bluetooth
+
+This test should be carried out with:
+- Android app
+- iOS app
+
+And the following micro:bit versions:
+- micro:bit V1
+- micro:bit V2
+
+A test run for each of these hex files is required:
+- "Out of Box" hex file
+- "Meet the micro:bit" hex file
+- MakeCode Live Editor hex file
+- Python Editor hex file (micro:bit V2 only)
+- MakeCode Under Test hex file
+
+Steps:
+1. Connect the micro:bit to the computer via USB.
+2. Copy the hex file under test to the `MICROBIT` USB drive.
+3. Remove the previous micro:bit pairing from the iOS or Android device.
+4. Open the micro:bit app and load the MakeCode editor under test.
+    - From the front page, select "Create Code"
+    - Click on the cog, and the "About" menu entry, check the MakeCode version
+      in this modal
+5. Create a new MakeCode project with a simple programme (e.g. display
+   something on the LED matrix).
+    - If a programme has already been created from a previous test run,
+      it can be reused.
+6. Tap the "Download" button and use "My pattern is different" to force pairing,
+   then follow the instruction to flash the programme.
+7. Verify the programme is running on the micro:bit.
+8. Change the blocks in the MakeCode editor to do something different
+   (e.g. display a different icon/message on the LED matrix).
+9. Tap the "Download" button and follow the instructions to flash the programme.
+10. Confirm the BLE flashing process was shorter than the initial download from
+   step 5.
+11. Verify that the new programme is running on the micro:bit.
+
+## Test 2: Utility Service
+
+This test checks that the Utility Service is available and working on
+the builds of the MakeCode editor under test.
+
+This test should be carried out with:
+- Android app
+- iOS app
+
+And the following micro:bit versions:
+- micro:bit V2 only
+
+Steps:
+1. Flash the micro:bit with this programme usin the MakeCode Under Test.
+   This project needs the datalogger extension to be added.
+   Ensure this project does not have the Bluetooth extension.
+   The board can be flashed using any method.
+    ```javascript
+    basic.forever(function () {
+        datalogger.log(
+        datalogger.createCV("y", input.acceleration(Dimension.Y)),
+        datalogger.createCV("x", input.acceleration(Dimension.X))
+        )
+        basic.pause(100)
+    })
+    ```
+2. Open the micro:bit app, go to "My Programs".
+3. Tap on the "Fetch MY_DATA (V2 only)" button and follow the instructions
+   to connect the micro:bit via Bluetooth.
+4. Verify that the data as a table of Time, X and Y acceleration values
+   is shown in the app.
+5. Add the Bluetooth extension to the project and flash the micro:bit again.
+6. Back to the app main menu, tap on the "Pair" button to pair the micro:bit.
+    - A WebUSB flash would have deleted the previous pair.
+7. Press the reset button on the micro:bit to ensure the programme is running.
+8. Back to the app main menu, go to "My Programs" and tap on the
+   "Fetch MY_DATA (V2 only)" button.
+9. In the "Fecth MY_DATA" screen, select "During Logging" at the top and
+   tap the "Fetch" button.
+10. Verify data is displayed and the fetch has not failed.

docs/release-tests/webusb.md

@@ -0,0 +1,149 @@
+# WebUSB Tests
+
+## Test matrix
+
+WebUSB tests should be carried out with the following combination of
+micro:bit boards, DAPLink versions, and browsers/apps to ensure full coverage
+of the supported configurations.
+
+### Test boards and DAPLink versions
+
+Each WebUSB test should be carried out with these micro:bit boards and
+DAPLink versions.
+The goal is to test each DAPLink port with the first factory release,
+the latest official release, and the latest beta release.
+
+| Board | DAPLink   | DAPLink MCU  | Description                       |
+| ----- | --------- | ------------ | --------------------------------- |
+| V1.x  | 0249      | KL26         | V1.5 factory release              |
+| V1.x  | 0253      | KL26         | Latest V1 official release        |
+| V2.00 | 0255      | KL27         | Factory & latest official release |
+| V2.00 | 0258-beta | KL27         | Latest beta release               |
+| V2.2x | 0257      | nRF52833/820 | Factory & latest official release |
+| V2.2x | 0258-beta | nRF52833/820 | Latest beta release               |
+
+### Test browsers, apps & operating systems
+
+Each WebUSB test should be carried out with the latest version of these
+Chrome-based browsers and operating systems:
+
+- Chrome on Windows
+- Chrome on macOS
+- Chrome on ChromeOS
+- Chrome on Linux
+- Chrome on Android
+- Edge on Windows
+- MakeCode Offline App for Windows
+    - https://makecode.microbit.org/offline-app
+    - This app needs to be packaged with the version of MakeCode under test.
+- MakeCode Offline App for macOS
+    - https://makecode.microbit.org/offline-app
+    - This app needs to be packaged with the version of MakeCode under test.
+- MakeCode Windows Store App
+    - https://apps.microsoft.com/detail/9nmqdq2xzkwk
+    - To load MakeCode beta, type `/@beta` in the extension search box.
+
+## Tests to run on all matrix variations
+
+The following tests should be carried out with all combinations of the
+micro:bit boards, DAPLink versions, and browsers/apps listed in the
+"Test matrix" section.
+
+As these tests can be repeated multiple times in the same browser/app with
+different micro:bit boards, when a test step indicates to create a new project,
+it is acceptable to use an existing project created in a previous test run.
+
+### Test 1: WebUSB full flash, partial flash, and serial
+
+1. Connect the micro:bit to the computer via USB.
+2. Flash a hex file that has not been generated by the MakeCode editor under
+   test.
+    - For example, the "Meet the micro:bit" hex file from:
+      https://microbit.org/get-started/user-guide/meet-the-microbit-program/
+3. Open the micro:bit MakeCode editor under test in a Chrome-based browser.
+4. Create a new project, and add a block to display an icon on the LED matrix.
+5. If the micro:bit was automatically connected via WebUSB, disconnect it from
+   the editor by clicking on the "Disconnect" button in "Download" menu
+   (three dots icon at the right of the "Download" button).
+6. Click the "Download" button.
+7. Follow the connection instructions to connect the micro:bit via WebUSB and
+   download the programme.
+8. Verify that the programme runs on the micro:bit.
+9. Change the code to this one:
+    ```javascript
+    basic.forever(function () {
+        serial.writeLine("hello")
+        basic.pause(100)
+    })
+    ```
+10. Click the "Download" button again.
+11. No connection instructions should be shown this time and the programme
+    should be downloaded directly to the micro:bit in a shorter amount of time.
+12. A "Show Data Device" button should appear, click it to open the serial
+    console.
+13. Verify that the serial console shows "hello" every second.
+
+## Tests on specific variations
+
+These tests should only be carried out with the configuration listed at the
+beginning of each test.
+
+### Test 2: WebUSB with incompatible DAPLink
+
+The following DAPLink versions are not expected to work with WebUSB, and
+this test is to ensure an error message suggesting a firmware update is shown.
+
+This test should be carried out with the following DAPLink versions:
+- micro:bit V1.x with DAPLink 0234
+    - This is the V1.03 factory version from the original BBC school drop
+- micro:bit V1.x with DAPLink 0241
+    - This is the V1.3B factory version, from the first commercial release
+
+1. Connect the micro:bit to the computer via USB.
+2. Open the micro:bit MakeCode editor under test in a Chrome-based browser.
+3. Create a new project.
+4. Add a block to the project (e.g. something simple to display on the LED
+   matrix).
+5. Click the "Download" button.
+6. Follow the connection instructions to connect the micro:bit via WebUSB.
+7. Verify that an error message is shown suggesting a firmware update is
+   required.
+
+### Test 3: WebUSB within micro:bit Classroom
+
+This test can be executed only once, using any micro:bit board with any
+DAPLink WebUSB compatible version, using Chrome on Windows.
+The goal of this test is to ensure that WebUSB feature is not broken due to
+the MakeCode editor being embedded in classroom.
+
+1. Connect the micro:bit to the computer via USB.
+2. Open the micro:bit Classroom with this URL:
+    https://classroom.microbit.org/?editorVersion=beta
+3. Create and start a blank session.
+4. Click on "Edit Code", which should open the embedded MakeCode editor.
+5. Drag a block into the workspace (e.g. something simple to display on the LED
+   matrix).
+6. Click the "Download" button.
+7. Follow the connection instructions to connect the micro:bit via WebUSB and
+   download the programme.
+8. Verify that the programme runs on the micro:bit.
+
+### Test 4: WebUSB within micro:bit CreateAI
+
+This test can be executed only once, using any micro:bit board with any
+DAPLink WebUSB compatible version, using Chrome on Windows.
+The goal of this test is to ensure that WebUSB feature is not broken due to
+the MakeCode editor being embedded in CreateAI.
+
+1. Connect the micro:bit to the computer via USB.
+2. Open the micro:bit CreateAI with this URL, which includes a project:
+     https://review-createai.microbit.org/makecode-lang-picker/import?id=simple-ai-exercise-timer&project=Project%3A%20Simple%20AI%20exercise%20timer&name=Simple%20AI%20exercise%20timer&editors=makecode&editorVersion=beta
+3. A "New session setup" screen should appear, click on "Start session".
+4. On the next screen, click on "Train Model".
+5. Wait for the model to be trained, which should take a few seconds.
+6. On the next screen, click on "Edit in MakeCode".
+7. The MakeCode editor should open with the project loaded. Click on the
+   "Download" button.
+8. Follow the connection instructions to connect the micro:bit via WebUSB and
+   download the programme.
+9. Verify that the programme runs on the micro:bit.