Use GHA with a custom runner to run automated tests

[Josverl]

to improve quality ive started to set-up a simple hardware CI test

• Pi4
  • configured with DietPi
  • installed GHA self hosted runner
  • two micropython boards connected via USB
  • created a CI workflow

Initial results:

Similar could be run on linux x64 , windows , and mac

additional work is needed to log the test results in a standard format (JUnitXML seems the common supported format) so that other GHA tools can work with it to provide better feedback and to join additional platforms.

Also currently the action-runner is only installed for my fork of the repo, but it can also be installed for an organization (PyCom)

[Josverl]

@jakobrosenberg ,
note that this also may help simplify validation of version dependencies.
It appears that some unit tests are failing on node 16,x which might be relevant as VSCode 1.66.2 is also based on node 16.

Version: 1.66.2 (user setup)
Electron: 17.2.0
Node.js: 16.13.0

[jakobrosenberg]

This is great!! I'll fix the Node 16 issue.

[jakobrosenberg]

Do you know if it's possible to connect a self hosted runner to Github? If so we could replicate your setup and have full integration tests for PRs.

[Josverl]

Do you know if it's possible to connect a self hosted runner to Github? If so we could replicate your setup and have full integration tests for PRs.

Yes , that is what I have done,
I found it really much less complex than I expected, and it does not take much resources either.

I think it would make sense to run a few different GHA-SHRs with different setups and devices connected to them.
If you set this up we could work togetherby hooking up my runner to the pycom repo ( willl need to pass the setupkey securely )

The GH documentation is here :

• https://docs.github.com/en/actions/hosting-your-own-runners

I have a small writeup of my setup that I will check-in to thenext-gha branch as well.

• https://github.com/Josverl/pymakr-vsc/blob/next-gha/GHA-runner.md

[jakobrosenberg]

I have solved the Node 16 issues locally. You can try the branch here https://github.com/pycom/pymakr-vsc/tree/node-16

If it works for you, I'll merge it into the next-staging branch.

[jakobrosenberg]

Great news! What kind of errors do you get with the integration test?

I'm currently rewriting the implementation. I'm hoping to get watch mode and interactive test filtering done by tomorrow. Can't make any promises though. Still a lot of gotchas when running the test runner inside the VSCode thread.

[Josverl]

note: I have a hunch that the failed test resolve mostly around :

• the setup of the filesystem ( root = \ vs root = \flash )
• MCU communication specifics ( ie send the reponse to p[rint("foo") in a single write versus multiple writes
  possibly
• MCU rest
• the use of MCU specific micopython calls ?

it would be very good to have the test results in a structure (JUnit) test results document to find what tests are failing when

[Josverl]

What kind of errors do you get with the integration test?
Integration Log: https://gist.github.com/Josverl/ee539553fc31ad097d6d687aa1a3980d

[Josverl]

Is there a way to flash a pycom firmware to a non-pycom esp32 device ?
as I do not have any pycom devices that may be one of the causes of inconcistency

[Josverl]

I found a generic pycom firmware to test with a pycom stock filesystem
https://github.com/LIHUAWRITER/pycom-esp32-universal
(I could not find firmwares to use in testing on the pycom site , is there something similar ?)

Much better, only the print("foo") test fails

Tests passed: 24 / 25
Tests failed: 1 / 25

https://gist.github.com/Josverl/82ed4e7fae41b39d1ea1c4ee38c5351c

[jakobrosenberg]

Much better, only the print("foo") test fails

So close now. Any idea what could be the cause of this? What happens if you print("foo") manually in the terminal?

[Josverl]

I think the problem may be that:

• the MCU send the response in multiple chunks , and not in a single string.
  • chunck 1 : p
  • chunck 2 : rint("foo")\r\n
  • chunck 3 : 'foo
  • chunck 4 : '\r\n>>>
• that the function that is called in the test nextTerminalData(device) only recieves the first chunck or even stops after a single char

Integration test on Winx64:

 ❌ can use print command   13 ms
node_modules/probs/lib/reporters/consoleReporter/consoleReporter.js:12
AssertionError: 'p' == 'print("foo")\r\nfoo\r\n>>> ' AssertionError [ERR_ASSERTION]: 'p' == 'print("foo")\r\nfoo\r\n>>> '
	at TestInstance.callback (c:\develop\vscode\pymakr-vsc\test\suite\integration\devices.test.js:44:16)
	at async TestInstance.runTestCallback (file:///c:/develop/vscode/pymakr-vsc/node_modules/probs/lib/Framework/TestRunner/TestInstance.js:76:13)
	at async TestInstance.run (file:///c:/develop/vscode/pymakr-vsc/node_modules/probs/lib/Framework/TestRunner/TestInstance.js:120:9)
	at async TestInstance.runChildren (file:///c:/develop/vscode/pymakr-vsc/node_modules/probs/lib/Framework/TestRunner/TestInstance.js:108:17)
	at async TestInstance.run (file:///c:/develop/vscode/pymakr-vsc/node_modules/probs/lib/Framework/TestRunner/TestInstance.js:121:9)
	at async TestInstance.runChildren (file:///c:/develop/vscode/pymakr-vsc/node_modules/probs/lib/Framework/TestRunner/TestInstance.js:108:17)
	at async TestInstance.run (file:///c:/develop/vscode/pymakr-vsc/node_modules/probs/lib/Framework/TestRunner/TestInstance.js:121:9)
	at async TestInstance.runChildren (file:///c:/develop/vscode/pymakr-vsc/node_modules/probs/lib/Framework/TestRunner/TestInstance.js:108:17)
	at async TestInstance.run (file:///c:/develop/vscode/pymakr-vsc/node_modules/probs/lib/Framework/TestRunner/TestInstance.js:121:9)
	at async TestInstance.runChildren (file:///c:/develop/vscode/pymakr-vsc/node_modules/probs/lib/Framework/TestRunner/TestInstance.js:108:17)
	at async TestInstance.run (file:///c:/develop/vscode/pymakr-vsc/node_modules/probs/lib/Framework/TestRunner/TestInstance.js:121:9)
	at async TestFile.run (file:///c:/develop/vscode/pymakr-vsc/node_modules/probs/lib/Framework/TestRunner/TestFile.js:28:9)
	at async QueueItem.run (file:///c:/develop/vscode/pymakr-vsc/node_modules/probs/lib/utils/QueueManager.js:8:36)
	at async QueueManager.checkQueue (file:///c:/develop/vscode/pymakr-vsc/node_modules/probs/lib/utils/QueueManager.js:41:13)
node_modules/probs/lib/reporters/consoleReporter/consoleReporter.js:14
          ✅ can disconnect    2 ms
no

[Josverl]

a quick update:
I gave Jest a try , first for the Unit tests.

Unit Tests

The Unit tests were very simple to update to run under Jest, with the follow pro's ::

• Test explorer
• run tests one-by-one / watch / only failed etc
• reports in multiple formats, Junit, github action
• code coverage report
• raises exitcode=1 so GHA can detect failures

Integration tests
The integration tests are more a bit more work to integrate, mostly as there is no formal API for Jest

But there are a few templates in typescript that implement this, that need to be reworked into .js
this is work in progress

test explorer

reports in Github Actions

this simplifies spotting failing tests

• there two tests are failing only on Mac and Node 14 or 16
  -see the results of this run: https://github.com/Josverl/pymakr-vsc/actions/runs/2236176529

[jakobrosenberg]

the MCU send the response in multiple chunks , and not in a single string.
chunck 1 : p
chunck 2 : rint("foo")\r\n
chunck 3 : 'foo
chunck 4 : '\r\n>>>

I came across this too. It complicated detection of whether a device is running since the chunks can range from single characters, single lines and multiple lines. I think the affected test has been updated, so it might not affect you any more. If it does let me know and I'll have a look.

As for Jest, I started migrating my projects away from Jest last year because of scalability, performance and ESM issues. It's actually why I have started the probs project. Except for code coverage, probs has the same pros, though some are still in development. The lack of exitcode=1 was accidentally removed, but has been readded. I'll try get a release out asap. Apologies for the lack of documentation. This is on my todo list. Enabling watch mode (--watch) will provide a UI that lets you filter and watch tests.

[Josverl]

I believe you that jest ain't perfect , but after switching from probs to jest:

• I could run individual tests
• found and fixed dependencies between these tests (no docs that probs also could do that )
• I was able to find a few new issues.
• create and use mocks

As there are other extension authors that also use jest , which allows for some re-use.
I don't really care which testing framework is used , as long as
P1:

• it can test,

• mock, spy

• there is some level documentation ( even if its : most is like jest / jasmine / but not foo and bar )
  P2:

• report in a standard format ( to get structured reports via a tool such as testspace

• preferably with IDE integration in VSCode

P3:

• code coverage
• there is some IP to reuse ( I can borrow from others working projects)

I know that jest won't solve all issues either, for example: Currently working on mocking sufficient of vscode to allow for unit testing of the devices and devices store in order to find and resolve the issues on windows.

Not sure what the way forward would be on this. Its not for me to decide, but currently the lack of mocking / spying / stubbing holds me back from contributing.

One way could be that probs and jest can both be used for unit testing with a naming convention & configuration to seperate)
Once probs has the needed/used capabilities functionality, I suppose it should be as easy to convert from jest back to probs as it was for the reverse.
To me that seems to be the most effective way to collaborate on PyMakr v-next