Performance Efficiency

Go up to Terms and Definitions

Performance Efficiency is a Quality Characteristic concerning performance relative to the resources used under specified conditions.

Performance Efficiency can apply to:

• A Candidate Solution
• A Distributed System
• A Node
• A Node Set
• A service
• A software component
• A hardware component
• An Execution Facility
• An Operational Resource
• Another subject performing work or consuming resources

Performance Efficiency can concern:

• Response time
• Processing time
• Latency
• Throughput
• Transaction rate
• Event-processing rate
• Data-transfer rate
• Capacity
• Concurrency
• Resource utilization
• Storage utilization
• Network utilization
• Processor utilization
• Memory utilization
• Energy consumption
• Another performance or resource-use characteristic

Performance Efficiency can identify:

• The evaluated operation
• The evaluated workload
• The applicable Test Environment
• The applicable Configuration
• The applicable Operational Resources
• The applicable resource limits
• The applicable measurement interval
• The applicable performance measures
• The applicable resource-consumption measures
• The measurement units
• The measurement precision
• The permitted tolerance
• The applicable DIDO Baseline
• The applicable Validation Criteria
• The applicable Acceptance Criteria
• Its Evidence
• Its Provenance
• Its Traceability

Performance Efficiency can be evaluated using relationships such as:

• Work completed per unit of time
• Work completed per unit of processor utilization
• Work completed per unit of memory
• Work completed per unit of storage
• Work completed per unit of network capacity
• Work completed per unit of energy
• Response time under a specified workload
• Throughput under a specified resource allocation
• Maximum supported workload under specified performance constraints

Quality Characteristic concerning performance relative to the resources used under specified conditions

Adapted from:

• ISO/IEC 25010:2023, Systems and software Quality Requirements and Evaluation (SQuaRE) — Product quality model
• ISO/IEC 25023:2016, Systems and software Quality Requirements and Evaluation (SQuaRE) — Measurement of system and software product quality
• Quality Characteristic
• Operational Resource
• DIDO Reference Implementation Conceptual Model
• DIDO-TE draft Requirements Register

ISO/IEC 25010 identifies Performance Efficiency as a product-quality characteristic concerning performance relative to the amount of resources used under stated conditions.

ISO/IEC 25023 provides quality measures for quantitatively evaluating system and software-product quality characteristics and subcharacteristics.

This definition adapts Performance Efficiency for DIDO-TE evaluation of Candidate Solutions and distributed-system subjects under controlled Test Environment, workload, Configuration, and resource conditions.

Performance Efficiency differs from performance:

• Performance can identify an observed accomplishment, timing value, rate, or capacity
• Performance Efficiency relates performance to the resources used under specified conditions

Performance Efficiency differs from Resource consumption:

• Performance Efficiency evaluates the relationship between performance and resource use
• Resource consumption identifies the amount of a Resource consumed

Two Candidate Solutions can have equal throughput but different Performance Efficiency when one consumes more resources.

Two Candidate Solutions can consume equal resources but have different Performance Efficiency when one completes more work.

Performance Efficiency cannot be compared objectively unless the evaluation identifies:

• The operation or workload
• The Test Environment
• The Configuration
• The allocated Operational Resources
• The measurement interval
• The measurement method
• The measurement units
• The required precision
• The permitted tolerance

A single performance measurement does not necessarily establish Performance Efficiency.

For example, response time without corresponding workload and resource conditions does not provide a complete Performance Efficiency evaluation.

A higher value does not always indicate greater Performance Efficiency:

• Higher throughput generally indicates greater efficiency under equivalent resource conditions
• Higher response time generally indicates lower efficiency under equivalent workload conditions
• Higher resource consumption generally indicates lower efficiency when performance remains equivalent

The applicable Acceptance Criteria establish the interpretation and preferred direction of each measure.

Performance Efficiency can include more specific characteristics concerning:

• Time behavior
• Resource utilization
• Capacity

These characteristics can require separate glossary definitions when DIDO-TE requirements refer to them directly.

DIDO-TE evaluates three Candidate Solutions using the same Test Definition, Test Environment, workload, Configuration, and allocated Operational Resources.

For each Candidate Solution, DIDO-TE measures:

• Transactions completed
• Elapsed time
• Processor utilization
• Memory utilization
• Storage utilization
• Network utilization
• Energy consumption

The Acceptance Criteria establish the measurement interval, units, precision, tolerances, and normalization rules.

DIDO-TE uses the resulting Test Results to compare:

• Transactions per second
• Transactions per processor unit
• Transactions per unit of memory
• Transactions per unit of energy

These comparisons provide Evidence concerning the Performance Efficiency of each Candidate Solution.