MO-002b — Reproducible Test Execution

Go to MO-002 — Reproducible Test Environments and Test Execution

DIDO-TE SHALL repeat a selected Test Execution within the permitted variation established by the Acceptance Criteria.

This requirement derives from:

• MO-002 — Reproducible Test Environments and Test Execution

The Original Requirement states:

Create repeatable testing scenarios, including regression testing, that allow rigorous evaluation of distributed systems under different conditions and configurations.

MO-002b preserves the portion of the Original Requirement addressing repeatable performance of testing scenarios.

The separate requirement derived from MO-002 addresses:

• MO-002a — Reproducible Test Environments

Organizations require reproducible Test Execution to evaluate Distributed Systems, compare candidate solutions, investigate failures, perform regression testing, validate changes, and confirm previously observed behavior.

A reproducible Test Environment does not by itself establish reproducible Test Execution. Test Execution can vary because of differences in:

• The selected Test Definition
• The Test Definition Version
• The selected Test Sequence
• The applicable Sequence Steps
• The selected Test Items
• The applicable Test Argument Values
• The applicable Starting Conditions
• The applicable Timing Conditions
• The permitted Execution Paths
• The applicable Timeouts
• The selected Test Executables
• The Test Executable Versions
• The selected Executable Artifacts
• The Executable Artifact Versions
• The applicable Dependencies
• The applicable Execution Facilities
• The selected Nodes
• The selected Node Sets
• The order and timing of inputs delivered to the Nodes
• The permitted nondeterministic behavior

The phrase within the permitted variation does not require every repeated Test Execution to produce an identical sequence of events, measurements, outputs, or Test Results.

The Acceptance Criteria establish which Test Execution characteristics must remain equivalent and which characteristics can vary within defined limits.

The Acceptance Criteria establish:

• The selected Test Execution
• The controlling Test Definition and Version
• The controlling Test Sequence and Sequence Steps
• The selected Test Items
• The controlling Test Argument Values
• The applicable Starting Conditions
• The applicable Timing Conditions
• The permitted Execution Paths
• The applicable Timeouts
• The Test Executables and Executable Artifacts
• The applicable Dependencies
• The applicable Execution Facilities
• The selected Test Environment
• The Test Execution characteristics subject to comparison
• The permitted variation for each compared characteristic
• The comparison method
• The measurement conditions
• The required measurement precision
• The treatment of nondeterministic behavior
• The treatment of failed, incomplete, interrupted, timed-out, canceled, and repeated Test Runs

MO-002b addresses repetition of a selected Test Execution. It does not establish whether the resulting Test Results satisfy the applicable Expected Results or receive a particular Verdict.

Requirements addressing Test Result evaluation, Baseline comparison, and Validation establish those obligations separately.

This requirement applies to:

• DIDO-TE
• Selected Test Execution
• Repeated Test Execution
• Reproducibility
• Acceptance Criteria
• Test Definitions
• Test Sequences
• Sequence Steps
• Test Items
• Test Arguments
• Test Argument Values
• Starting Conditions
• Timing Conditions
• Execution Paths
• Timeouts
• Test Executables
• Executable Instructions
• Executable Artifacts
• Dependencies
• Execution Facilities
• Test Environments
• Nodes
• Node Sets
• Test Runs
• Test Results
• Failed Test Runs
• Incomplete Test Runs
• Interrupted Test Runs
• Timed-out Test Runs
• Canceled Test Runs
• Repeated Test Runs
• Nondeterministic behavior
• Evidence
• Provenance
• Traceability

Verification confirms that:

1. The selected Test Execution is identified
2. The Acceptance Criteria identify the controlling Test Definition and Version
3. The Acceptance Criteria identify the controlling Test Sequence and applicable Sequence Steps
4. The Acceptance Criteria identify the applicable Test Items
5. The Acceptance Criteria identify the applicable Test Argument Values
6. The Acceptance Criteria identify the applicable Starting Conditions
7. The Acceptance Criteria identify the applicable Timing Conditions
8. The Acceptance Criteria identify the permitted Execution Paths
9. The Acceptance Criteria identify the applicable Timeouts
10. The Acceptance Criteria identify the applicable Test Executables, Executable Artifacts, and their Versions
11. The Acceptance Criteria identify the applicable Dependencies
12. The Acceptance Criteria identify the applicable Execution Facilities
13. The Acceptance Criteria identify the selected Test Environment
14. The Acceptance Criteria identify the Test Execution characteristics subject to comparison
15. The Acceptance Criteria establish the permitted variation for each compared characteristic
16. The Acceptance Criteria establish the comparison method and required measurement precision
17. The Acceptance Criteria establish the treatment of nondeterministic behavior
18. The Acceptance Criteria establish the treatment of failed, incomplete, interrupted, timed-out, canceled, and repeated Test Runs
19. DIDO-TE performs the selected Test Execution more than once under the conditions established by the Acceptance Criteria
20. Each repeated Test Execution uses the controlling Test Definition and Version
21. Each repeated Test Execution uses the applicable Test Argument Values
22. Each repeated Test Execution satisfies the applicable Starting Conditions
23. Each repeated Test Execution satisfies the applicable Timing Conditions
24. Each repeated Test Execution follows a permitted Execution Path
25. Each compared Test Execution characteristic remains within the permitted variation established by the Acceptance Criteria
26. The verification record preserves Traceability among the Acceptance Criteria, selected Test Execution, repeated Test Executions, Test Runs, compared characteristics, measured variations, comparison results, and supporting Evidence

The following pages reference this requirement:

TBD

Assess the current DIDO-TE implementation against the Test Execution Reproducibility obligation established by MO-002b.

Review and accept MO-002b as a proposed derived requirement created from the evaluation and decomposition of MO-002.

The following unresolved issues affect this requirement:

Confirm whether repeatability or reproducibility identifies the approved quality required for repeated Test Execution.

Define the Test Execution characteristics subject to comparison.

Define the permitted variation for each compared Test Execution characteristic.

Define the comparison methods, measurement conditions, precision, exclusions, and tolerances used to determine Test Execution Reproducibility.

Define the Test Definition, Test Sequence, Sequence Steps, Test Argument Values, Starting Conditions, Timing Conditions, Execution Paths, and Timeouts that must remain controlled during repeated Test Execution.

Define which Test Executable, Executable Artifact, Dependency, and Execution Facility Versions must remain unchanged during repeated Test Execution.

Define the permitted differences among Operational Resources used for repeated Test Execution.

Define how DIDO-TE identifies and evaluates nondeterministic behavior.

Define how failed, incomplete, interrupted, timed-out, canceled, and repeated Test Runs affect the determination of Test Execution Reproducibility.

Define the Evidence required to demonstrate Test Execution Reproducibility.

This requirement page should retain the stable requirement identifier MO-002b.

This page is a leaf requirement page and omits a trailing :start from its namespace.

The parent MO-002 page is a non-leaf page and retains a trailing :start in its namespace.

Changes to the Statement should preserve:

• DIDO-TE as the responsible Actor
• The selected Test Execution as the repeated activity
• Repetition of the selected Test Execution as the required behavior
• The permitted variation as the reproducibility constraint
• The Acceptance Criteria as the source of the permitted variation

The Acceptance Criteria should identify:

• The selected Test Execution
• The controlling Test Definition and Version
• The controlling Test Sequence and Sequence Steps
• The applicable Test Items
• The applicable Test Argument Values
• The applicable Starting Conditions
• The applicable Timing Conditions
• The permitted Execution Paths
• The applicable Timeouts
• The applicable Test Executables and Versions
• The applicable Executable Artifacts and Versions
• The applicable Dependencies and Versions
• The applicable Execution Facilities
• The selected Test Environment
• The Test Execution characteristics subject to comparison
• The permitted variation for each compared characteristic
• The comparison method
• The measurement conditions
• The required measurement precision
• Any permitted exclusions or tolerances
• The treatment of nondeterministic behavior
• The treatment of failed, incomplete, interrupted, timed-out, canceled, and repeated Test Runs
• The required Evidence
• The required Provenance
• The required Traceability

MO-002b does not require every repeated Test Execution to produce identical Test Results unless the Acceptance Criteria establish exact equality as the permitted variation.

MO-002b does not determine whether a Test Result satisfies an Expected Result or receives a particular Verdict.

Material changes should receive review and should update the verification criteria, source records, parent decomposition record, and Issues section.

To reference this requirement Statement from another Wiki page, insert:

{{section>dido:03-dido-te:99-annexes:annex-c-requirements:01-mission-objectives:mo-002:mo-002b#Statement&noheader&nofooter&noeditbtn}}

Do not rename this page after an external citation unless a redirect or move plan is in place.