MO-002 — Reproducible Test Environments and Test Execution

Go to C.1 Mission Objectives

The draft DIDO-TE source material states:

Create repeatable testing scenarios, including regression testing, that allow rigorous evaluation of distributed systems under different conditions and configurations.

Identify the authoritative source, original requirement identifier, and controlling citation for this Original Requirement.

The Original Requirement preserves the approved mission-level intent that DIDO-TE provide repeatable or reproducible testing under different conditions and Configurations.

The Original Requirement does not express an independently interpretable or independently verifiable normative obligation.

The following Specification Discipline and Authoring findings apply:

• Create does not identify an explicit responsible Actor
• Create does not use an explicit normative modal
• Repeatable does not identify the required degree or method of Reproducibility
• Testing scenarios does not use defined terms such as Test Definition, Test Sequence, Test Environment, or Test Execution
• Including regression testing provides an example but does not establish a bounded scope
• Allow is a weak verb that does not identify an observable DIDO-TE behavior
• Rigorous evaluation is subjective and does not identify measurable evaluation criteria
• Distributed systems does not identify the selected Distributed System, Test Items, Nodes, or Node Sets subject to testing
• Different conditions does not identify the controlled conditions, variable conditions, or permitted differences
• Different configurations does not identify the Configurations that DIDO-TE must preserve, recreate, or vary
• The Original Requirement does not identify the applicable Acceptance Criteria
• The Original Requirement does not identify the Evidence required to demonstrate Reproducibility

The source uses the word repeatable, while the proposed canonical requirements use reproducible and Reproducibility. The requirement owner must confirm whether:

• Repeatability and Reproducibility express the same approved intent in this context
• Repeatability identifies a narrower controlled-condition obligation
• Reproducibility identifies a broader obligation that permits defined changes to operators, resources, locations, or other conditions
• Separate requirements for Repeatability and Reproducibility are necessary

The Original Requirement combines at least two independently verifiable outcomes:

• Recreating a selected Test Environment within an established permitted variation
• Repeating selected Test Execution within an established permitted variation

These outcomes require separate verification.

A Test Environment can be recreated successfully even when repeated Test Execution produces unacceptable differences in the resulting Test Results.

Test Execution can also produce acceptable Test Results even when DIDO-TE has not demonstrated that the Test Environment was recreated according to its controlling Configuration.

The Original Requirement therefore requires:

• Identification of DIDO-TE as the responsible Actor
• Replacement of nonnormative language with explicit normative obligations
• Selection of Repeatability, Reproducibility, or both as the controlling quality
• Definition of the Test Environment characteristics subject to reproduction
• Definition of the Test Execution information subject to reproduction
• Separation of Test Environment Reproducibility from Test Execution Reproducibility
• Identification of the permitted variation for each outcome
• Identification of the applicable Acceptance Criteria
• Identification of the Evidence required to verify each outcome
• Confirmation that the derived requirements collectively preserve the complete intent of MO-002

The decomposition preserves MO-002 as the stable parent requirement identifier. Each proposed replacement requirement receives a lettered identifier and a separate leaf requirement page.

The Original Requirement is decomposed into the following proposed replacement requirements:

Review and approve the decomposition of MO-002 into MO-002a and MO-002b.

Confirm whether Repeatability, Reproducibility, or separate requirements for both qualities preserve the approved intent of the Original Requirement.

Determine whether MO-002a and MO-002b collectively supersede MO-002.

If MO-002a and MO-002b are accepted as the complete replacement for MO-002, mark MO-002 as superseded and preserve this page as the parent [[dido:99_annexes:annex-b-terms-and-definitions:t:traceability|Traceability]] record.

The following unresolved issues affect the decomposition of MO-002:

Identify the authoritative source, original requirement identifier, and controlling citation for the Original Requirement.

Confirm the relationship between Repeatability and [[dido:99_annexes:annex-b-terms-and-definitions:r:reproducibility|Reproducibility]] within DIDO-TE.

Determine whether Repeatability requires a separate shared Terms and Definitions entry.

Define the controlled and variable characteristics of a reproducible [[dido:99_annexes:annex-b-terms-and-definitions:t:test_environment|Test Environment]].

Define the [[dido:99_annexes:annex-b-terms-and-definitions:c:configuration|Configuration]], [[dido:99_annexes:annex-b-terms-and-definitions:v:version|Versions]], [[dido:99_annexes:annex-b-terms-and-definitions:d:dependency|Dependencies]], [[dido:99_annexes:annex-b-terms-and-definitions:o:operational_resource|Operational Resources]], and [[dido:99_annexes:annex-b-terms-and-definitions:d:deployment_target|Deployment Targets]] that DIDO-TE must preserve or recreate.

Define the [[dido:99_annexes:annex-b-terms-and-definitions:t:test_definition|Test Definitions]], [[dido:99_annexes:annex-b-terms-and-definitions:t:test_sequence|Test Sequences]], [[dido:99_annexes:annex-b-terms-and-definitions:s:sequence_step|Sequence Steps]], [[dido:99_annexes:annex-b-terms-and-definitions:t:test_argument_value|Test Argument Values]], [[dido:99_annexes:annex-b-terms-and-definitions:s:starting_condition|Starting Conditions]], [[dido:99_annexes:annex-b-terms-and-definitions:t:timing_condition|Timing Conditions]], and [[dido:99_annexes:annex-b-terms-and-definitions:e:execution_path|Execution Paths]] that DIDO-TE must preserve for reproducible Test Execution.

Define whether Reproducibility requires exact equality or permits bounded variation.

Define the permitted variation for Test Environment recreation.

Define the permitted variation for repeated Test Execution and resulting [[dido:99_annexes:annex-b-terms-and-definitions:t:test_result|Test Results]].

Define the comparison methods, tolerances, precision, and treatment of nondeterministic behavior.

Define how failed, incomplete, interrupted, timed-out, canceled, and repeated [[dido:99_annexes:annex-b-terms-and-definitions:t:test_run|Test Runs]] affect the determination of Reproducibility.

Define the [[dido:99_annexes:annex-b-terms-and-definitions:e:evidence|Evidence]], [[dido:99_annexes:annex-b-terms-and-definitions:p:provenance|Provenance]], and Traceability required to demonstrate Test Environment and Test Execution Reproducibility.

Determine whether MO-002a and MO-002b provide complete coverage of the approved intent of MO-002.

This page should retain the stable parent requirement identifier MO-002.

This page is a non-leaf requirement page and retains a trailing :start in its namespace.

The child requirements are leaf requirement pages and omit a trailing :start from their namespaces.

This page preserves:

• The Original Requirement
• The assessment of the Original Requirement
• The reason for decomposition
• Traceability to the proposed replacement requirements
• The unresolved cross-cutting issues affecting the decomposition
• The supersession decision

MO-002a addresses the Reproducibility of a selected Test Environment.

MO-002b addresses the Reproducibility of selected Test Execution.

The child requirements must not combine Test Environment recreation and Test Execution repetition into a single normative Statement.

The child requirements should identify independently verifiable Acceptance Criteria for:

• The selected Test Environment
• The selected Test Execution
• The controlling Configurations
• The controlling Versions
• The applicable Test Argument Values
• The Starting Conditions
• The Timing Conditions
• The permitted Execution Paths
• The permitted variation
• The comparison method
• The treatment of nondeterministic behavior
• The required Evidence
• The required Provenance
• The required Traceability

MO-002 should not be marked as superseded until the requirement owner:

• Confirms the authoritative Original Requirement and its controlling citation
• Approves the interpretation of Repeatability and Reproducibility
• Approves MO-002a
• Approves MO-002b
• Confirms that the child requirements collectively preserve the complete approved intent of MO-002
• Confirms that no additional child requirements are required

After supersession, this page should remain as the parent Traceability record and should continue to preserve the Original Requirement and its assessment.

Material changes to the decomposition should update the child requirements, Requirement Status, Issues, source records, and Traceability records.

Do not rename this page after an external citation unless a redirect or move plan is in place.