ENV-005m — Validate Emulation Fidelity

Go to ENV-005 — Emulate Distributed Systems

The DIDO-TE SHALL validate the fidelity of distributed-system emulation against the criteria specified by the applicable Test Definition.

• ENV-005 — Emulate Distributed Systems
• ENV-005a — Emulate Distributed Topology
• ENV-005b — Emulate Participating Nodes
• ENV-005c — Emulate Communication Behaviour
• ENV-005d — Emulate Timing and Synchronisation
• ENV-005e — Emulate Resource Conditions
• ENV-005f — Emulate Node Failures
• ENV-005g — Emulate Communication Failures
• ENV-005h — Emulate Network Partitions
• ENV-005i — Emulate Membership and Topology Changes
• ENV-005j — Emulate Recovery Behaviour
• ENV-005k — Observe Distributed Interactions
• ENV-005l — Record Distributed-System Emulation
• [DTE5] DIDO-TE Requirements Register, source requirement identifier and obligation to be assigned
• [DTE6] Structured Information Processing Reference Architecture (SIP-RA)
• [DTE7] Federated Data Interpretation Systems Reference Architecture (FDIS-RA)

Emulation fidelity expresses the degree to which an emulated distributed system reproduces the conditions, characteristics, events, and behaviours required by the applicable Test Definition.

Fidelity does not require unrestricted similarity to a production system. The Test Definition identifies the characteristics material to the purpose of the test and specifies the applicable acceptance criteria, tolerances, resolutions, timing limits, and permitted deviations.

Relevant characteristics include topology, Node participation, communication behaviour, timing and synchronisation, resource conditions, failure behaviour, network partitions, membership and topology changes, recovery behaviour, and observability.

Validation compares the specified conditions with the conditions recorded during Test Execution. A test result lacks adequate support when the emulation record does not demonstrate that all material conditions remained within their specified fidelity criteria.

• Test Environment
• Test Definition
• Test Execution
• Test Object
• Node
• Test Resource
• Configuration
• Constraint
• Validation
• Evidence
• Test Result

Verification confirms that:

• The applicable Test Definition identifies each emulation characteristic subject to fidelity validation.
• Each specified fidelity criterion has a unique identity.
• The Test Definition identifies the purpose and significance of each fidelity criterion.
• The Test Definition specifies the expected value, condition, state, sequence, distribution, or behaviour associated with each criterion.
• The Test Definition specifies the applicable tolerance, range, resolution, precision, accuracy, duration, frequency, or timing limit for each criterion.
• The Test Definition identifies the measurement or observation method used to assess each criterion.
• The Test Definition identifies the observation points, data sources, sampling conditions, units, and comparison methods required for fidelity validation.
• The Test Definition identifies the fidelity criteria that must be satisfied before Test Execution begins.
• The Test Definition identifies the fidelity criteria that must remain satisfied during Test Execution.
• The Test Definition identifies the fidelity criteria assessed after Test Execution.
• The DIDO-TE validates the initial emulated topology, participating Nodes, communication paths, configurations, and resource conditions before beginning the applicable Test Execution.
• The DIDO-TE validates communication behaviour, timing, synchronisation, resource conditions, failures, partitions, membership changes, topology changes, and recovery behaviour, as applicable.
• The DIDO-TE compares each specified fidelity criterion with measured or observed values recorded during Test Execution.
• The DIDO-TE accounts for measurement accuracy, resolution, uncertainty, sampling limitations, clock error, and observation interference when evaluating fidelity.
• The DIDO-TE determines whether each measured or observed value falls within its specified tolerance or acceptance range.
• The DIDO-TE validates the sequence, activation time, duration, scope, and termination of each specified emulation event.
• The DIDO-TE validates the relationships among emulated conditions when the Test Definition specifies combined or dependent fidelity criteria.
• The DIDO-TE identifies each fidelity criterion that cannot be evaluated because of missing, incomplete, inaccurate, conflicting, or unavailable data.
• The DIDO-TE does not treat an unevaluated mandatory fidelity criterion as satisfied.
• The DIDO-TE identifies each condition that remained outside its specified fidelity tolerance.
• The DIDO-TE identifies the start time, end time, duration, magnitude, scope, and affected elements associated with each fidelity deviation.
• The DIDO-TE determines whether each fidelity deviation invalidates the Test Execution, limits the interpretation of a Test Result, or remains within a permitted exception.
• The DIDO-TE distinguishes an emulation-fidelity deviation from a failure or nonconformance of the Test Object.
• The DIDO-TE identifies any operator intervention or environmental event that affects emulation fidelity.
• The DIDO-TE produces an overall emulation-fidelity determination for the Test Execution.
• The DIDO-TE records the applicable criteria, expected values, measured or observed values, tolerances, comparison methods, uncertainties, deviations, exceptions, and determinations as Evidence.
• The DIDO-TE maintains Traceability among each fidelity criterion, emulated condition, recorded value, comparison, deviation, Test Definition, Test Execution, Evidence, and Test Result.
• An unvalidated, unsatisfied, unsupported, incorrectly assessed, or untraceable mandatory fidelity criterion constitutes nonconformance with this requirement.

Verification includes:

• Inspection of the fidelity criteria specified by the applicable Test Definition
• Inspection of expected values, tolerances, ranges, resolutions, accuracies, timing limits, and acceptance criteria
• Inspection of the measurement methods, observation points, data sources, units, sampling conditions, and comparison methods
• Comparison of specified and recorded topology, Node membership, communication behaviour, timing, resource conditions, failures, partitions, changes, and recovery behaviour
• Confirmation that required pre-execution fidelity criteria are satisfied before Test Execution begins
• Confirmation that required runtime fidelity criteria remain within their specified tolerances
• Confirmation that required post-execution fidelity criteria are assessed
• Testing of a value at, within, and outside a specified fidelity tolerance
• Testing of an emulation event that starts late, ends early, exceeds its scope, or produces an incorrect condition
• A negative assessment involving missing, inaccurate, conflicting, or unavailable fidelity data
• Confirmation that an unevaluated mandatory fidelity criterion does not receive a satisfied determination
• Confirmation that measurement uncertainty and observation interference are included in the fidelity determination
• Confirmation that the DIDO-TE distinguishes an emulation-fidelity deviation from Test Object behaviour
• Confirmation that each fidelity deviation identifies its duration, magnitude, scope, and effect on the Test Result
• Inspection of the overall emulation-fidelity determination, supporting Evidence, and Traceability records
• Confirmation that each mandatory fidelity deviation produces a nonconformance result

• Add links to the architecture sections governing emulation fidelity, measurement, calibration, uncertainty, acceptance criteria, Validation, Evidence, Traceability, and Test Result qualification.

Assign the applicable delivery phase.

Draft

Use the following syntax to reference this requirement’s Statement section from another DokuWiki page:

{{section>dido:03-dido-te:99-annexes:annex-c-requirements:02-test-environment-requirements:env-005-emulate-distributed-systems:env-005m-validate-emulation-fidelity#Statement&noheader&nofooter&noeditbtn}}