Releases: google/testrun
Releases · google/testrun
v2.3.0
Top Highlight: Intermediate Certificate Support
We now fully support chained certificate systems. The validation engine has been upgraded to correctly distinguish between Root and Intermediate certificates, ensuring security compliance for modern devices.
✨ Key Improvements
- Reports: Module test reports now render HTML correctly (fixed the raw Jinja2 template issue).
- Accuracy: The "Outbound Connections" section is now permanently visible in reports to prevent ambiguity.
- UX: The duration timer works only while testing (does not include the Monitoring and Device discovery time).
🛠️ Critical Bug Fixes
- Connectivity Logic: Improved
ping: for connection module tests added 5 attempts to ping low-performance devices. - UI: Selecting "None of the above" now properly clears other options in multi-select forms.
- Visibility: All non-compliant tests (including "Recommended" and "Roadmap") are now highlighted Red in results.
- PDFs: Fixed text alignment and formatting issues in PDF export.
v2.2.2
Version 2.2.2 is out with some bug fixes and minor UX improvements. This release is a result of backend refactoring and addresses issues reported by users.
Improvements
- Added more context to TLS test results to explain why a certificate is reported as invalid.
Bug fixes
- Fixed a deprecation warning related to a third-party cryptography object, ensuring future compatibility.
- Resolved a TLS module timeout issue that occurred when a firewall was used to close all ports.
- Updated layout in PDF report including text concatenation, truncated url fixes.
- Addressed a UI bug where specific port setting values were lost.
- Made minor cosmetic fixes to improve the user interface.
v2.2.1
Version 2.2.1 provides a number of fixes to test logic and improvements for the test review process.
Improvements
- The PDF report is regenerated with any changes to the device profile
- Filtered the NTP test module packet capture to just NTP packets
- Updated test logic in the TLS server tests
- Improved validation of the device manufacturer and model strings
- Added automated testing for the NTP test module
- Added automated testing for the Protocol test module
Bug fixes
- Resolved an issue with stopping Testrun in Firefox
- Corrected the test description for dns.mdns
- Resolved formatting issue in outbound connections table
- When loading old Testrun reports, display the correct test result
v2.2
This release sees Testrun receive a full user interface refresh, upgrading from Material 2 to Material 3.
NOTE: Ubuntu 20 is no longer supported. Users should update to Ubuntu 22 or 24 to continue using Testrun.
New features
- Design system update (Material 2 to 3):
- Implemented Google Material Design 3 across all pages, enhancing usability and maintainability.
- Angular framework version update (18 to 19): performance improvements and leveraging the power of Signals for reactivity
- Introduced a new streamlined first-time user flow.
- Added a new "Actions" button for easy access to start testing, create devices, or manage risk profiles from any screen.
- The General settings menu now integrates both Settings and Certificates functionalities.
Improvements
- Switched from ethtool to ifconfig for connection tests
- Added new testing for test accuracy
- Moved additional configuration options to its own documentation page
- Added log levels for test modules
- Added configuration option to prevent canceling test attempt on port disconnect
- Increased the maximum size permitted for ZIP downloads
Bug fixes
- Fixed test duration formatting in the report
- Caught OSError in the TLS test correctly
- Fixed test description in DNS from DHCP test
- Added exception handling if packet captures are missing
- Fixed issue with DNS HTML report not generating
- Resolved issue where incorrect IP was scanned during services tests
v2.1.1
This release improves the user experience when running Testrun for pilot assessments whilst also introducing a number of smaller improvements and bug fixes.
New features
- Starting status is now shown whilst Testrun is starting
- When an unexpected device is connected this is logged in the command line
Improvements
- Language used for pilot assessments is now more appropriate
- TLS server tests scans for TLS services on every TCP port
- NTPv3 is now allowed provided that NTPv4 is also used
- The device profile is updated on every ZIP export (in case device information has changed)
- Documentation now states which tests are affected by single interface mode
Bug fixes
- Risk profile name is now validated correctly
- Updated Cryptogaphy and Jinja2 libraries
v2.1
This release provides a number of bug fixes and feature improvements aiming to increase the stability of Testrun and improving the user experience.
New features
- Provided documentation in additional configuration options
- Allow firmware and target MAC address to be passed in the command line
- New communication.network.type test implemented
- Extended TCP port scan to cover all ports
- New module report for the TLS module
- Outbound connections are listed in the TLS module report
Improvements
- Upgraded the docker base image to Ubuntu 22
- Most modules use Python 3.10 slim docker image, decreasing installation times
- Greater coverage in unit testing
- Updated multiple Python dependencies
- Added automated actions to verify that Testrun starts on different operating systems (Ubuntu 20, 22, 24)
- Remove excess logging from MQTT client
- Don't start the UI when running in net only mode
- Only run tests if they are included in the chosen test pack
- Added the BACnet port to the services open ports table
Bug fixes
- Locked some sub dependencies
- Corrected permissions of the resources folder
- Check for certificate properties when uploading a CA certificate
- Added missing tests to the documentation
- Better exception handling if the USB adapter does not support port stats
- Added exception handling around missing leases during tests
- Better exception handling in the connection test module
- More accurate result if device does not respond to BACnet discovery
- Fixed results for TLS server if public certificate could not be resolved
- Allow reports to be downloaded from external hosts
- Adapter changes between test attempts are now handled correctly
v2.0.1
v2.0
Version 2.0 of Testrun provides new functionality to support new users hoping to submit a device for a pilot assessment. In addition, new features support a number of use cases for both Google and non-Google users.
New features
- Device create and edit modal has been updated and requires further device information
- Device testing type (Device qualification or pilot assessment) has been added to device object
- DHCP disconnect and DHCP disconnect IP change tests have been implemented
- Test module reports now provide further information
- Device testing type has been added to the reports list page
- Optional recommendations have been added to the pilot assessment device report
- TLS v1.0 test has been implemented
Improvements
- Documentation has been reviewed and updated
- UI is now built during package, not installation
- Informational test result color has changed
- PDF report design has been improved
- Unit testing has been enabled in GitHub actions
- Installation time is reduced by using slim Python images for test modules
Bug fixes
- DHCP NAK messages no longer trigger a Non Compliant result in the DHCP snooping test
- Steps to resolve are no longer present on tests with Error or Informational test result
- Single IP test will no longer detect the testing adapters
Known issues
- We currently have an accessibility issue in the create/edit device modal where some text does not display correctly when using accessibility tools.
- If you are using a python version lower than 3.9, you will experience the following error:
Traceback (most recent call last):
File "framework/python/src/core/test_runner.py", line 25, in <module>
from testrun import Testrun
File "/usr/local/testrun/framework/python/src/core/testrun.py", line 30, in <module>
from session import TestrunSession
File "/usr/local/testrun/framework/python/src/core/session.py", line 80, in <module>
class TestrunSession():
File "/usr/local/testrun/framework/python/src/core/session.py", line 638, in TestrunSession
def _remove_invalid_questions(self, questions: list[dict]) -> list[dict]:
TypeError: 'type' object is not subscriptable
To resolve, you will need to upgrade to 3.9 by doing the following:
-
Upgrade python:
apt install python3.9 -
Set python3.9 as default python installation by running the following:
sudo update-alternatives --install /usr/bin/python3 python3 /usr/bin/python3.9 1
sudo update-alternatives --install /usr/bin/python3 python3 /usr/bin/python3.9 2
sudo update-alternatives --config python3
- Confirm python3.9 is now the default:
python3 --version
Python 3.9.5
- Install 3.9 venv:
sudo apt install python3.9-venv
v1.4
During Q3 of 2024, Testrun will see a number of minor releases containing just a handful of new features, improvements and bug fixes. Whilst it is not required that you update your installation of Testrun, it is recommended so that you have a better experience with the application.
New features
- Addition of a web-sockets server for real-time communication between front and backend
- Receive a notification when an adapter is connected or disconnected
- Implemented automated unit testing to improve stability
- Addition of the informational dns.mdns test
- Added GRPC server to allow host network control
- TLS 1.3 server and client informational tests have been added
Improvements
- Duplicate DNS queries are now combined in the PDF report
- Improved security of the package workflow
- Disabled the UI during network only mode
- Disabled the monitoring period when in network only mode
- Tests always report Informational if they are informational tests
- All pylint issues have been resolved
- Google CLA, pylint and ESLint checks are now enforced
- Added a no profile option in the Testrun zip download
- Backend not responding error message has been rephrased
Bug fixes
- Individual test module configuration changes are no longer ignored
- DNS queries from the DNS server to 8.8.8.8 are now hidden
- Draft risk profiles can now expire after 1 year
v1.4-alpha
NOTE: This is a pre-release. It is not ready for official use. Feel free to give it a go and let us know if you find any bugs ahead of the release. For official use, please always download the latest version.
New features
- Addition of a web-sockets server for real-time communication between front and backend
- Implemented automated unit testing to improve stability
- Addition of the informational dns.mdns test
- Addition of the TLS client and server v1.3 tests
- TLS 1.3 server and client informational tests have been added
- Added risk level to each individual question in the PDF report
- Check for internet connectivity during a Testrun (wifi icon in top right)
Improvements
- Duplicate DNS queries are now combined in the PDF report
- Improved security of the package workflow
- Disabled the UI during network only mode
- Disabled the monitoring period when in network only mode
- Tests always report Informational if they are informational tests
- UI is now pre-built to speed up installation times
- All pylint issues have been resolved
- Google CLA, pylint and ESLint checks are now enforced
- Added a no profile option in the Testrun zip download
- Backend not responding error message has been rephrased
- Added automated unit testing
- More complete documentation
Bug fixes
- Individual test module configuration changes are no longer ignored
- DNS queries from the DNS server to 8.8.8.8 are now hidden
- Draft risk profiles can now expire after 1 year