ENV-005j — Emulate Recovery Behaviour

Go to ENV-005 — Emulate Distributed Systems

The DIDO-TE SHALL emulate the recovery behaviour specified by the applicable Test Definition.

• ENV-005 — Emulate Distributed Systems
• ENV-005b — Emulate Participating Nodes
• ENV-005f — Emulate Node Failures
• ENV-005g — Emulate Communication Failures
• ENV-005h — Emulate Network Partitions
• ENV-005i — Emulate Membership and Topology Changes
• [DTE5] DIDO-TE Requirements Register, source requirement identifier and obligation to be assigned
• [DTE6] Structured Information Processing Reference Architecture (SIP-RA)
• [DTE7] Federated Data Interpretation Systems Reference Architecture (FDIS-RA)

Recovery behaviour determines how a distributed system returns to an acceptable operational state following a failure, isolation event, resource constraint, membership change, topology change, or other disruption.

Relevant behaviour includes failure detection, restart, reconnection, failover, reconfiguration, state restoration, replay, resynchronisation, reconciliation, reintegration, leadership selection, quorum restoration, backlog processing, and return to service.

Recovery does not necessarily restore the exact pre-disruption state. The applicable Test Definition identifies the required recovery objective, acceptable resulting state, permitted data loss or duplication, timing criteria, consistency criteria, and completion conditions.

Controlled recovery emulation permits repeatable assessment of recovery sequences and their effects on participating Nodes, communication paths, state, data, and services.

• Test Environment
• Test Definition
• Test Execution
• Test Object
• Node
• Test Resource
• Configuration
• Constraint
• Evidence

Verification confirms that:

• The applicable Test Definition identifies each recovery behaviour subject to emulation.
• Each specified recovery behaviour has a unique identity.
• The Test Definition identifies the disruption, failure, or changed condition from which recovery begins.
• The Test Definition identifies each Node, Test Resource, communication path, service, state, or data set participating in the recovery.
• The Test Definition specifies the required pre-disruption state, disrupted state, and acceptable recovered state.
• The Test Definition specifies the recovery trigger, activation condition, sequence, scope, timing criteria, and completion criteria.
• The Test Definition specifies the applicable restart, reconnection, failover, reconfiguration, restoration, replay, resynchronisation, reconciliation, reintegration, and return-to-service behaviour.
• The Test Definition specifies the permitted loss, duplication, reordering, inconsistency, or unavailability during recovery.
• The Test Definition specifies the required post-recovery membership, topology, configuration, service availability, and data state.
• The DIDO-TE confirms the required disrupted state before activating the recovery behaviour.
• The DIDO-TE activates each recovery action under its specified conditions.
• The DIDO-TE performs recovery actions in the specified order.
• The DIDO-TE records the planned and actual start and completion times for each recovery action.
• The DIDO-TE observes participating Nodes, communication paths, services, state transitions, and data reconciliation throughout recovery.
• The DIDO-TE determines whether the recovery reaches the specified completion criteria.
• The DIDO-TE detects a recovery action that does not occur, occurs in the wrong order, exceeds its timing criteria, produces an unacceptable state, or remains incomplete.
• The DIDO-TE distinguishes specified recovery behaviour from an unintended environmental intervention.
• The DIDO-TE records recovery identities, triggers, actions, sequences, state transitions, timing measurements, reconciliation outcomes, resulting states, and detected deviations as Evidence.
• The DIDO-TE maintains Traceability among each recovery behaviour, originating disruption, affected Nodes and resources, Test Definition, Test Execution, Evidence, and Test Results.
• Missing, incomplete, incorrectly sequenced, untimely, uncontrolled, unobserved, unsupported, or untraceable mandatory recovery behaviour constitutes nonconformance with this requirement.

Verification includes:

• Inspection of the recovery behaviours specified by the Test Definition
• Inspection of recovery identities, triggers, affected elements, actions, sequences, timing criteria, and completion criteria
• Comparison of the specified pre-disruption, disrupted, and recovered states
• Observation of recovery following a Node failure
• Observation of recovery following a communication failure or network partition
• Observation of recovery following a resource constraint or membership change, as applicable
• Testing of restart, reconnection, failover, state restoration, replay, resynchronisation, reconciliation, or reintegration, as applicable
• Confirmation that recovery actions occur in the specified order
• Measurement of recovery timing against the specified criteria
• Confirmation that the recovered state satisfies the specified availability, consistency, membership, topology, configuration, and data criteria
• Confirmation that the DIDO-TE detects an incomplete, incorrectly sequenced, untimely, or otherwise unacceptable recovery
• Inspection of recovery Evidence and Traceability records
• Confirmation that each mandatory recovery deviation produces a nonconformance result

• Add links to the architecture sections governing failure detection, recovery control, restart, failover, state restoration, reconciliation, reintegration, service continuity, Evidence, and Traceability.

Assign the applicable delivery phase.

Draft

Use the following syntax to reference this requirement’s Statement section from another DokuWiki page:

{{section>dido:03-dido-te:99-annexes:annex-c-requirements:02-test-environment-requirements:env-005-emulate-distributed-systems:env-005j-emulate-recovery-behaviour#Statement&noheader&nofooter&noeditbtn}}