diff --git a/docs/release-tests/ble-flashing.md b/docs/release-tests/ble-flashing.md new file mode 100644 index 00000000000..2eb1ea91a5b --- /dev/null +++ b/docs/release-tests/ble-flashing.md @@ -0,0 +1,120 @@ +# Bluetooth Tests + +## Test preparations + +Four hex files needed: +- "Meet the micro:bit": This is the programme that runs on the micro:bit when + it is first powered on from factory. + - Downloadable from the micro:bit website: + https://microbit.org/get-started/user-guide/meet-the-microbit-program/ +- "Out of Box": The previous factory programme before the + "Meet the micro:bit" was released. + - Downloadable from the micro:bit website: + https://microbit.org/get-started/user-guide/meet-the-microbit-program/ +- MakeCode Live Editor hex file: To ensure micro:bits with a recent MakeCode + project can be flashed via Bluetooth with the MakeCode editor under test. + - Create a hex file from the latest MakeCode live editor: + https://makecode.microbit.org/ +- Python Editor hex file: To ensure the MakeCode editor under test can + BLE flash micro:bits after using the Python Editor. + - Go to https://python.microbit.org and download the hex file + with the default project. + - This hex file can only be tested with micro:bit V2 +- MakeCode Under Test hex file: To ensure it can be flashed over. + +Two mobile/tablet devices needed: +- Android device with the micro:bit app installed. + - To load a custom MakeCode URL long touch the micro:bit logo in the app + main screen. A modal will appear with the option change the MakeCode URL. + - This URL setting will be reset every time the app is quit +- iOS device with the micro:bit app installed. + - To load a different MakeCode version, long press the MakeCode back + chevron to choose beta. + - The iOS app can only choose between beta and live. + +Two micro:bits needed: +- micro:bit V1 +- micro:bit V2 + +The micro:bits can be powered via USB for these tests, as that avoids any +possible issues or errors due to low batteries. + +## Test 1: Full & partial flashing with Bluetooth + +This test should be carried out with: +- Android app +- iOS app + +And the following micro:bit versions: +- micro:bit V1 +- micro:bit V2 + +A test run for each of these hex files is required: +- "Out of Box" hex file +- "Meet the micro:bit" hex file +- MakeCode Live Editor hex file +- Python Editor hex file (micro:bit V2 only) +- MakeCode Under Test hex file + +Steps: +1. Connect the micro:bit to the computer via USB. +2. Copy the hex file under test to the `MICROBIT` USB drive. +3. Remove the previous micro:bit pairing from the iOS or Android device. +4. Open the micro:bit app and load the MakeCode editor under test. + - From the front page, select "Create Code" + - Click on the cog, and the "About" menu entry, check the MakeCode version + in this modal +5. Create a new MakeCode project with a simple programme (e.g. display + something on the LED matrix). + - If a programme has already been created from a previous test run, + it can be reused. +6. Tap the "Download" button and use "My pattern is different" to force pairing, + then follow the instruction to flash the programme. +7. Verify the programme is running on the micro:bit. +8. Change the blocks in the MakeCode editor to do something different + (e.g. display a different icon/message on the LED matrix). +9. Tap the "Download" button and follow the instructions to flash the programme. +10. Confirm the BLE flashing process was shorter than the initial download from + step 5. +11. Verify that the new programme is running on the micro:bit. + +## Test 2: Utility Service + +This test checks that the Utility Service is available and working on +the builds of the MakeCode editor under test. + +This test should be carried out with: +- Android app +- iOS app + +And the following micro:bit versions: +- micro:bit V2 only + +Steps: +1. Flash the micro:bit with this programme usin the MakeCode Under Test. + This project needs the datalogger extension to be added. + Ensure this project does not have the Bluetooth extension. + The board can be flashed using any method. + ```javascript + basic.forever(function () { + datalogger.log( + datalogger.createCV("y", input.acceleration(Dimension.Y)), + datalogger.createCV("x", input.acceleration(Dimension.X)) + ) + basic.pause(100) + }) + ``` +2. Open the micro:bit app, go to "My Programs". +3. Tap on the "Fetch MY_DATA (V2 only)" button and follow the instructions + to connect the micro:bit via Bluetooth. +4. Verify that the data as a table of Time, X and Y acceleration values + is shown in the app. +5. Add the Bluetooth extension to the project and flash the micro:bit again. +6. Back to the app main menu, tap on the "Pair" button to pair the micro:bit. + - A WebUSB flash would have deleted the previous pair. +7. Press the reset button on the micro:bit to ensure the programme is running. +8. Back to the app main menu, go to "My Programs" and tap on the + "Fetch MY_DATA (V2 only)" button. +9. In the "Fecth MY_DATA" screen, select "During Logging" at the top and + tap the "Fetch" button. +10. Verify data is displayed and the fetch has not failed. diff --git a/docs/release-tests/webusb.md b/docs/release-tests/webusb.md new file mode 100644 index 00000000000..35f43d24e81 --- /dev/null +++ b/docs/release-tests/webusb.md @@ -0,0 +1,149 @@ +# WebUSB Tests + +## Test matrix + +WebUSB tests should be carried out with the following combination of +micro:bit boards, DAPLink versions, and browsers/apps to ensure full coverage +of the supported configurations. + +### Test boards and DAPLink versions + +Each WebUSB test should be carried out with these micro:bit boards and +DAPLink versions. +The goal is to test each DAPLink port with the first factory release, +the latest official release, and the latest beta release. + +| Board | DAPLink | DAPLink MCU | Description | +| ----- | --------- | ------------ | --------------------------------- | +| V1.x | 0249 | KL26 | V1.5 factory release | +| V1.x | 0253 | KL26 | Latest V1 official release | +| V2.00 | 0255 | KL27 | Factory & latest official release | +| V2.00 | 0258-beta | KL27 | Latest beta release | +| V2.2x | 0257 | nRF52833/820 | Factory & latest official release | +| V2.2x | 0258-beta | nRF52833/820 | Latest beta release | + +### Test browsers, apps & operating systems + +Each WebUSB test should be carried out with the latest version of these +Chrome-based browsers and operating systems: + +- Chrome on Windows +- Chrome on macOS +- Chrome on ChromeOS +- Chrome on Linux +- Chrome on Android +- Edge on Windows +- MakeCode Offline App for Windows + - https://makecode.microbit.org/offline-app + - This app needs to be packaged with the version of MakeCode under test. +- MakeCode Offline App for macOS + - https://makecode.microbit.org/offline-app + - This app needs to be packaged with the version of MakeCode under test. +- MakeCode Windows Store App + - https://apps.microsoft.com/detail/9nmqdq2xzkwk + - To load MakeCode beta, type `/@beta` in the extension search box. + +## Tests to run on all matrix variations + +The following tests should be carried out with all combinations of the +micro:bit boards, DAPLink versions, and browsers/apps listed in the +"Test matrix" section. + +As these tests can be repeated multiple times in the same browser/app with +different micro:bit boards, when a test step indicates to create a new project, +it is acceptable to use an existing project created in a previous test run. + +### Test 1: WebUSB full flash, partial flash, and serial + +1. Connect the micro:bit to the computer via USB. +2. Flash a hex file that has not been generated by the MakeCode editor under + test. + - For example, the "Meet the micro:bit" hex file from: + https://microbit.org/get-started/user-guide/meet-the-microbit-program/ +3. Open the micro:bit MakeCode editor under test in a Chrome-based browser. +4. Create a new project, and add a block to display an icon on the LED matrix. +5. If the micro:bit was automatically connected via WebUSB, disconnect it from + the editor by clicking on the "Disconnect" button in "Download" menu + (three dots icon at the right of the "Download" button). +6. Click the "Download" button. +7. Follow the connection instructions to connect the micro:bit via WebUSB and + download the programme. +8. Verify that the programme runs on the micro:bit. +9. Change the code to this one: + ```javascript + basic.forever(function () { + serial.writeLine("hello") + basic.pause(100) + }) + ``` +10. Click the "Download" button again. +11. No connection instructions should be shown this time and the programme + should be downloaded directly to the micro:bit in a shorter amount of time. +12. A "Show Data Device" button should appear, click it to open the serial + console. +13. Verify that the serial console shows "hello" every second. + +## Tests on specific variations + +These tests should only be carried out with the configuration listed at the +beginning of each test. + +### Test 2: WebUSB with incompatible DAPLink + +The following DAPLink versions are not expected to work with WebUSB, and +this test is to ensure an error message suggesting a firmware update is shown. + +This test should be carried out with the following DAPLink versions: +- micro:bit V1.x with DAPLink 0234 + - This is the V1.03 factory version from the original BBC school drop +- micro:bit V1.x with DAPLink 0241 + - This is the V1.3B factory version, from the first commercial release + +1. Connect the micro:bit to the computer via USB. +2. Open the micro:bit MakeCode editor under test in a Chrome-based browser. +3. Create a new project. +4. Add a block to the project (e.g. something simple to display on the LED + matrix). +5. Click the "Download" button. +6. Follow the connection instructions to connect the micro:bit via WebUSB. +7. Verify that an error message is shown suggesting a firmware update is + required. + +### Test 3: WebUSB within micro:bit Classroom + +This test can be executed only once, using any micro:bit board with any +DAPLink WebUSB compatible version, using Chrome on Windows. +The goal of this test is to ensure that WebUSB feature is not broken due to +the MakeCode editor being embedded in classroom. + +1. Connect the micro:bit to the computer via USB. +2. Open the micro:bit Classroom with this URL: + https://classroom.microbit.org/?editorVersion=beta +3. Create and start a blank session. +4. Click on "Edit Code", which should open the embedded MakeCode editor. +5. Drag a block into the workspace (e.g. something simple to display on the LED + matrix). +6. Click the "Download" button. +7. Follow the connection instructions to connect the micro:bit via WebUSB and + download the programme. +8. Verify that the programme runs on the micro:bit. + +### Test 4: WebUSB within micro:bit CreateAI + +This test can be executed only once, using any micro:bit board with any +DAPLink WebUSB compatible version, using Chrome on Windows. +The goal of this test is to ensure that WebUSB feature is not broken due to +the MakeCode editor being embedded in CreateAI. + +1. Connect the micro:bit to the computer via USB. +2. Open the micro:bit CreateAI with this URL, which includes a project: + https://review-createai.microbit.org/makecode-lang-picker/import?id=simple-ai-exercise-timer&project=Project%3A%20Simple%20AI%20exercise%20timer&name=Simple%20AI%20exercise%20timer&editors=makecode&editorVersion=beta +3. A "New session setup" screen should appear, click on "Start session". +4. On the next screen, click on "Train Model". +5. Wait for the model to be trained, which should take a few seconds. +6. On the next screen, click on "Edit in MakeCode". +7. The MakeCode editor should open with the project loaded. Click on the + "Download" button. +8. Follow the connection instructions to connect the micro:bit via WebUSB and + download the programme. +9. Verify that the programme runs on the micro:bit.