Current version is used for the manual assessment.
#TBA: manual scoring details
#TBA: values' invariants
#TBD: automation
_Does the system deliver the right capabilities, correctly and completely, to satisfy user needs under expected usage conditions?
- 1.1 Functional Completeness: Are all required use cases, edge cases, and user objectives fully supported by the implemented feature set? Signals: traceability matrix alignment.
- 1.2 Functional Correctness: Do the implemented functions consistently produce accurate, reliable, and expected results for intended inputs and scenarios? Signals: business logic defects density, test pass rate correctness, traceability matrix alignment.
- 1.3 Functional Appropriateness: Do the available functions efficiently enable users to accomplish their goals without unnecessary steps or workarounds? Signals: workflow efficiency.
_Does the system execute its functions within defined time, throughput, and resource usage limits under expected workloads?
- 2.1 Time Behaviour: Are response times, latency, and throughput consistently within agreed performance targets under normal and peak conditions? Signals: P95/P99 latency metrics, TPS, SLA compliance rate.
- 2.2 Resource Utilization: Does the system use CPU, memory, storage, network, and other resources efficiently while maintaining required performance levels? Signals: Resource consumption thresholds violations, cost per transaction.
- 2.3 Capacity: Can the system handle the maximum expected load, data volume, and concurrent users without degradation beyond acceptable thresholds? Signals:: load test breakpoints.
_Can the system operate effectively within a shared ecosystem and seamlessly interact with other systems without negative impact?
- 3.1 Co-existence: Does the system function correctly and efficiently in a shared environment without degrading or interfering with other applications or services? Signals: performance in multi-tenant environments, cross-system conflicts.
- 3.2 Interoperability: Can the system reliably exchange, interpret, and use data with external systems according to defined interfaces and protocols? Signals: API contract compliance, data exchange validation, integration test pass rate.
_Can intended users effectively, efficiently, and confidently interact with the system’s interface to accomplish tasks across different usage contexts?
- 4.1 Appropriateness Recognizability: Can users quickly determine whether the system meets their needs and supports their intended tasks? Signals: value proposition clarity, entry points clarity, lifecycles clarity, drop-off rate.
- 4.2 Learnability: How quickly can new users understand and start using core features successfully? Signals: Time-to-first-success, onboarding completion rate.
- 4.3 Operability: Is the system easy to navigate, control, and operate during routine and advanced tasks? Signals: Task completion rate, navigation friction, usability defect reports.
- 4.4 User Error Protection: Does the system prevent, detect, and help recover from user mistakes? Signals: Input validation coverage, error messages, recovery success rate.
- 4.5 User Engagement: Does the interface encourage continued and meaningful interaction? Signals: Session duration, feature adoption rate, user retention metrics.
- 4.6 Inclusivity: Can users from diverse backgrounds and abilities effectively use the system? Signals: Accessibility compliance (e.g., WCAG), localization support, usability across demographics.
- 4.7 User Assistance: Are contextual help, guidance, and support mechanisms available and effective when needed? Signals: Help feature usage success rate, support tickets.
- 4.8 Self-Descriptiveness: Does the interface clearly communicate its functionality without requiring external documentation? Signals: labeling, discoverability score, reliance on manuals or external support.
_Does the system consistently perform its intended functions correctly and remain dependable over time under defined operating conditions?
- 5.1 Faultlessness: Does the system execute required functionality without defects or unexpected failures during normal operations? Signals: production defect rate, incidents levels, test pass stability.
- 5.2 Availability: Is the system operational and accessible whenever users or dependent systems need it? Signals: Uptime percentage, SLA adherence.
- 5.3 Fault Tolerance: Can the system continue operating correctly even when components fail or unexpected issues occur? Signals: degradation behavior, redundancy validation, failover test results.
- 5.4 Recoverability: After a failure or interruption, can the system restore affected data and return to a consistent, operational state within acceptable time limits? Signals: RTO/RPO compliance, backup restoration success rate, incident recovery time metrics.
_Does the system adequately protect data, enforce proper access control, and defend against malicious activity while maintaining trusted operations?
- 6.1 Confidentiality: Is sensitive data accessible only to authorized users, systems, or roles? Signals: access control enforcement, encryption in transit, unauthorized access findings.
- 6.2 Integrity: Is system and data state protected against unauthorized or accidental modification or deletion? Signals: detection mechanisms, hash validation, integrity violations in audits.
- 6.3 Non-Repudiation: Can actions and transactions be reliably proven to have occurred, preventing denial by involved parties? Signals: signed transactions log, audit logs, event trails.
- 6.4 Accountability: Can every significant action be uniquely traced back to a specific user, system, or entity? Signals: Comprehensive coverage, identity enforcement, audit records.
- 6.5 Authenticity: Can the identity of users, services, or resources be verified with high confidence? Signals: authentication mechanisms, certificate validation, identity verification success rate.
- 6.7 Resistance: Can the system maintain essential operations and minimize impact while under attack? Signals: penetration test outcomes, DDoS resilience validation.
_Can the system be efficiently analyzed, modified, extended, and validated over time without degrading quality or introducing excessive risk?
- 7.1 Modularity: Is the system structured into well-defined, loosely coupled components so that changes in one area have minimal impact on others? Signals: separation of concerns, coupling and cohesion metrics, regression impact radius.
- 7.2 Reusability: Can components, services, or modules be leveraged across multiple systems or features without significant rework? Signals: library adoption rate, APIs standards, duplicate code.
- 7.3 Analysability: How easily can engineers understand the system, assess change impact, and diagnose defects or failures? Signals: Code readability, documentation coverage, mean time to identify root cause.
- 7.4 Modifiability: Can enhancements, fixes, or configuration changes be implemented quickly and safely without introducing regressions? Signals: Change failure rate, deployment frequency, post-release defect leakage.
- 7.5 Testability: How effectively can test conditions be defined and executed to validate system behavior? Signals: Unit/integration test coverage, automated test reliability, CI/CD validation pass rate.
_Can the system adapt efficiently to evolving requirements, varying workloads, and different operational environments without significant rework or degradation?
- 8.1 Adaptability: How easily can the system be configured or adjusted to operate across different hardware, software platforms, or usage contexts? : Environment configuration effort, cross-platform compatibility success rate, minimal environment-specific code changes.
- 8.2 Scalability: Can the system effectively handle increases or decreases in workload while maintaining performance and stability? : Linear scaling validation, auto-scaling effectiveness, performance stability under variable load.
- 8.3 Installability: How efficiently and reliably can the system be installed, configured, upgraded, or removed in a target environment? : Installation success rate, setup time, rollback/upgrade reliability.
- 8.4 Replaceability: Can the system substitute another product in the same environment with minimal disruption and integration effort? : Standards compliance, migration effort estimation, compatibility with existing interfaces and dependencies.
_Does the system operate in a manner that prevents harm to people, property, or the environment under defined and abnormal conditions?
- 9.1 Operational Constraint: Does the system restrict its behavior to safe limits when encountering hazardous or abnormal conditions? : Enforcement of safety thresholds, boundary condition validation, automatic shutdown outside safe parameters.
- 9.2 Risk Identification: Can the system detect scenarios, states, or sequences of actions that may introduce unacceptable safety risks? : Hazard detection coverage, real-time monitoring alerts, documented risk assessment mapping.
- 9.3 Fail Safe: In the event of malfunction or failure, does the system transition to a predefined safe state automatically? : Verified fail-safe mechanisms, safe-mode activation success rate, no uncontrolled failure states.
- 9.4 Hazard Warning: Does the system provide timely and clear warnings about unsafe conditions to allow preventive action? : Alert accuracy, warning response time, signal clarity and escalation effectiveness.
- 9.5 Safe Integration: Can the system maintain required safety levels when integrated with other components or systems? : Integration safety validation results, no new hazard introduction during system coupling, compliance with safety interface contracts.