Operational Stability

Go up to Terms and Definitions

Operational Stability is a Quality Characteristic concerning the degree to which the operational characteristics of a subject remain within permitted variation under specified conditions for a specified period.

Operational Stability can apply to:

• A Candidate Solution
• A Distributed System
• A Node
• A Node Set
• A service
• A software component
• A hardware component
• An Execution Facility
• An Operational Resource
• Another subject having observable operational characteristics

Operational characteristics can include:

• Response time
• Throughput
• Resource utilization
• Availability
• Error frequency
• Failure frequency
• Queue depth
• Data-processing rate
• State-transition rate
• Communication latency
• Storage utilization
• Energy consumption
• Another characteristic established by the Acceptance Criteria

Operational Stability can identify:

• The evaluated subject
• The selected operational characteristics
• The specified conditions
• The evaluation period
• The applicable workload
• The applicable Test Environment
• The applicable Configuration
• The applicable reference values
• The permitted variation
• The measurement intervals
• The measurement methods
• The measurement units
• The required measurement precision
• The treatment of transient behavior
• The treatment of warm-up periods
• The treatment of planned changes
• The treatment of nondeterministic behavior
• The applicable DIDO Baseline
• The applicable Validation Criteria
• The applicable Acceptance Criteria
• Its Evidence
• Its Provenance
• Its Traceability

Operational Stability can be evaluated using:

• Variation around a reference value
• Variation over time
• Frequency of excursions outside a permitted range
• Duration of excursions outside a permitted range
• Trend analysis
• Statistical dispersion
• Change-point detection
• Comparison with a controlling DIDO Baseline
• Another method established by the Acceptance Criteria

Quality Characteristic concerning the degree to which the operational characteristics of a subject remain within permitted variation under specified conditions for a specified period

Adapted from:

• ISO/IEC 25010:2023, Systems and software Quality Requirements and Evaluation (SQuaRE) — Product quality model
• Quality Characteristic
• Reliability
• Performance Efficiency
• DIDO Reference Implementation Conceptual Model
• DIDO-TE draft Requirements Register
• DIDO-TE Broad Agency Announcement source material

The DIDO-TE source material identifies Stability as a central evaluation value but does not establish an independently measurable definition.

ISO/IEC 25010 does not identify Operational Stability as a separate top-level product-quality characteristic. This definition introduces Operational Stability to provide a measurable interpretation of stability for DIDO-TE comparative evaluation.

Operational Stability differs from Reliability:

• Operational Stability concerns variation in selected operational characteristics
• Reliability concerns performance of specified functions under specified conditions for a specified period

A subject can remain operational and reliable while exhibiting unacceptable variation in response time or resource utilization.

A subject can exhibit stable performance while consistently producing incorrect results.

Operational Stability differs from Performance Efficiency:

• Operational Stability concerns variation over time or operating conditions
• Performance Efficiency concerns performance relative to resources used

Operational Stability does not mean the operational characteristics remain constant.

Normal operation can include permitted variation caused by:

• Workload changes
• Scheduling
• Communication delays
• Resource allocation
• Nondeterministic processing
• Maintenance activity
• Planned state transitions

The applicable Acceptance Criteria establish which variation is permitted.

The term Stability should not appear alone in a requirement when its intended meaning is Operational Stability.

Other meanings of stability can include:

• Numerical stability
• Configuration stability
• Interface stability
• Organizational stability
• Financial stability

Those meanings require separate terms or explicit qualification.

DIDO-TE evaluates three Candidate Solutions for 24 hours under the same variable workload.

The Acceptance Criteria establish permitted ranges for:

• Response time
• Throughput
• Processor utilization
• Memory utilization
• Queue depth
• Communication latency

DIDO-TE records each characteristic at defined measurement intervals.

One Candidate Solution maintains every characteristic within the permitted variation. Another Candidate Solution completes the same work but exhibits repeated response-time and queue-depth excursions.

The comparative evaluation identifies the first Candidate Solution as having greater Operational Stability under the evaluated conditions.