ENV-006 — Reproduce Real-World Conditions

Go to C.2 Test Environment Requirements

The DIDO-TE SHALL reproduce the real-world conditions specified by the applicable Test Definition.

• [DTE5] DIDO-TE Requirements Register, source requirement identifier and obligation to be assigned
• [DTE6] Structured Information Processing Reference Architecture (SIP-RA)
• [DTE7] Federated Data Interpretation Systems Reference Architecture (FDIS-RA)

A test performed under idealised or unrepresentative conditions may produce results that do not reflect the behaviour of the Test Object in its intended operational context.

Real-world conditions include the operational, physical, computational, network, temporal, data, workload, dependency, security, environmental, organisational, and jurisdictional conditions material to the purpose of a test. Relevant conditions may include realistic transaction volumes, user activity, data distributions, resource contention, network characteristics, service dependencies, geographic distribution, clock behaviour, infrastructure limitations, access controls, component failures, and recovery events.

Reproduction does not require unrestricted duplication of a production environment. The applicable Test Definition identifies the real-world conditions material to the test, the method used to reproduce each condition, and the acceptance criteria and tolerances governing adequate reproduction.

The DIDO-TE records the conditions actually present during Test Execution and validates them against the specified conditions. This comparison establishes whether the Test Results support conclusions about behaviour under the intended real-world conditions.

• Test Environment
• Test Definition
• Test Execution
• Test Object
• Node
• Test Resource
• Configuration
• Constraint
• Evidence
• Test Result

Verification confirms that:

• The applicable Test Definition identifies each real-world condition subject to reproduction.
• Each specified real-world condition has a unique identity.
• The Test Definition states why each condition is material to the purpose of the test.
• The Test Definition identifies the operational context represented by the specified conditions.
• The Test Definition identifies the Nodes, Test Resources, interfaces, dependencies, services, users, data, workloads, locations, and environmental elements associated with each condition.
• The Test Definition specifies the expected value, state, range, distribution, pattern, sequence, duration, frequency, or behaviour associated with each condition.
• The Test Definition specifies the applicable acceptance criteria, tolerances, resolutions, timing limits, and permitted deviations.
• The Test Definition specifies the method used to establish, control, observe, measure, and validate each condition.
• The Test Definition identifies each condition reproduced through physical resources, virtual resources, simulation, emulation, recorded data, generated data, operator action, or an external service.
• The Test Definition identifies assumptions and abstractions associated with the reproduction of each condition.
• The Test Definition identifies sensitive, proprietary, regulated, or personally identifiable information excluded, transformed, substituted, masked, or synthesised for Test Execution.
• The Test Definition identifies production dependencies replaced by test doubles, simulators, emulators, replicas, or controlled services.
• The Test Definition specifies the required relationships and interactions among concurrently reproduced conditions.
• The Test Definition identifies the conditions that must exist before Test Execution begins.
• The Test Definition identifies the conditions that must remain within specified limits during Test Execution.
• The Test Definition identifies the conditions assessed after Test Execution.
• The DIDO-TE establishes each required pre-execution condition before beginning the applicable Test Execution.
• The DIDO-TE controls each reproduced condition throughout its specified period.
• The DIDO-TE observes or measures each reproduced condition at the frequency and resolution specified by the Test Definition.
• The DIDO-TE records the planned and actual values, states, ranges, distributions, patterns, sequences, durations, frequencies, and behaviours associated with each reproduced condition.
• The DIDO-TE preserves the relationships and interactions specified among concurrently reproduced conditions.
• The DIDO-TE distinguishes reproduced conditions from conditions introduced by the Test Object, an operator, the DIDO-TE, or an unintended environmental event.
• The DIDO-TE detects a reproduced condition that does not occur, occurs at the wrong time, exceeds its specified scope, falls outside its specified tolerance, or remains active beyond its specified duration.
• The DIDO-TE identifies any required condition that cannot be reproduced, observed, measured, controlled, or validated.
• The DIDO-TE does not treat an unevaluated mandatory condition as satisfied.
• The DIDO-TE determines whether each reproduced condition satisfies its specified acceptance criteria.
• The DIDO-TE determines whether a deviation invalidates the Test Execution, limits interpretation of a Test Result, or remains within a permitted exception.
• The DIDO-TE distinguishes a failure to reproduce a specified condition from a failure or nonconformance of the Test Object.
• The DIDO-TE records condition identities, sources, methods, expected values, actual values, tolerances, measurements, observations, assumptions, abstractions, substitutions, deviations, exceptions, and determinations as Evidence.
• The DIDO-TE maintains Traceability among each real-world condition, its source or justification, reproduction method, Test Definition, Test Environment, Test Execution, recorded Evidence, deviation, and Test Result.
• A missing, uncontrolled, unobserved, inaccurately reproduced, unsupported, or untraceable mandatory real-world condition constitutes nonconformance with this requirement.

Verification includes:

• Inspection of the real-world conditions specified by the applicable Test Definition
• Inspection of the operational context, rationale, acceptance criteria, tolerances, and permitted deviations associated with each condition
• Inspection of the methods used to establish, control, observe, measure, and validate each condition
• Comparison of specified real-world conditions with the conditions present during Test Execution
• Observation of representative workloads, data distributions, user activity, network behaviour, resource limitations, service dependencies,