ENV-005f — Emulate Node Failures

Go to ENV-005 — Emulate Distributed Systems

The DIDO-TE SHALL emulate Node failures in accordance with the applicable Test Definition.

• ENV-005 — Emulate Distributed Systems
• ENV-005b — Emulate Participating Nodes
• ENV-005e — Emulate Resource Conditions
• [DTE5] DIDO-TE Requirements Register, source requirement identifier and obligation to be assigned
• [DTE6] Structured Information Processing Reference Architecture (SIP-RA)
• [DTE7] Federated Data Interpretation Systems Reference Architecture (FDIS-RA)

A participating Node may cease operation, become unresponsive, produce incorrect output, lose state, operate intermittently, or exhibit degraded behaviour.

Distributed systems respond differently to crash failures, omission failures, timing failures, state corruption, resource-induced failures, and uncontrolled restart. Controlled emulation of these conditions supports assessment of failure detection, containment, failover, redundancy, consistency, recovery, and continuity behaviour.

A Node failure differs from a communication failure. This requirement governs failure of a participating Node or one of its required functions. ENV-005g governs failure of communication between otherwise operating Nodes.

The applicable Test Definition identifies the affected Node, failure type, triggering condition, initiation point, duration, observable effects, permitted propagation, and required post-failure state.

• Test Environment
• Test Definition
• Test Execution
• Test Object
• Node
• Test Resource
• Configuration
• Constraint
• Evidence

Verification confirms that:

• The applicable Test Definition identifies each Node subject to failure emulation.
• Each specified Node failure has a unique identity.
• The Test Definition specifies the failure type, affected function, triggering condition, initiation point, duration, observable effects, and required post-failure state.
• The Test Definition specifies the expected scope and permitted propagation of each Node failure.
• The Test Definition specifies whether the failed Node remains failed, restarts, or proceeds to recovery.
• The DIDO-TE establishes the required pre-failure Node state.
• The DIDO-TE initiates each specified Node failure at the specified event, state, or time.
• The emulated failure affects only the specified Node, functions, and resources unless the Test Definition permits propagation.
• The DIDO-TE maintains each Node failure for its specified duration or until its specified termination condition occurs.
• The DIDO-TE observes the affected Node and the responses of other participating Nodes.
• The DIDO-TE distinguishes the injected Node failure from an unintended Test Environment failure.
• The DIDO-TE detects a failure that does not begin, persist, terminate, or propagate as specified.
• The DIDO-TE records the failure identity, affected Node, failure type, trigger, initiation time, duration, observed effects, propagation, termination, and resulting state as Evidence.
• The DIDO-TE maintains Traceability among each Node failure, affected Node, Test Definition, Test Execution, Evidence, and Test Results.
• A missing, incorrect, uncontrolled, unobserved, unsupported, or untraceable mandatory Node failure constitutes nonconformance with this requirement.

Verification includes:

• Inspection of the Node failures specified by the Test Definition
• Inspection of failure identities, affected Nodes, failure types, triggers, durations, effects, propagation rules, and resulting states
• Observation of the affected Node before, during, and after failure emulation
• Emulation of a Node crash or loss of responsiveness
• Emulation of an omission, timing, intermittent, degraded, or state-related failure when specified
• Testing that the failure remains within its specified scope
• Testing that an intended propagated effect reaches each specified Node or resource
• Confirmation that the DIDO-TE detects an incorrectly initiated, scoped, sustained, or terminated failure
• Inspection of Node-failure Evidence and Traceability records
• Confirmation that each mandatory Node-failure deviation produces a nonconformance result

• Add links to the architecture sections governing Node lifecycle, fault injection, failure containment, redundancy, recovery, Evidence, and Traceability.

Assign the applicable delivery phase.

Draft

Use the following syntax to reference this requirement’s Statement section from another DokuWiki page:

{{section>dido:03-dido-te:99-annexes:annex-c-requirements:02-test-environment-requirements:env-005-emulate-distributed-systems:env-005f-emulate-node-failures#Statement&noheader&nofooter&noeditbtn}}