MO-002a — Reproducible Test Environments

Go to MO-002 — Reproducible Test Environments and Test Execution

DIDO-TE SHALL recreate a selected Test Environment within the permitted variation established by the Acceptance Criteria.

This requirement derives from:

• MO-002 — Reproducible Test Environments and Test Execution

MO-002 combines two independently verifiable outcomes:

• Reproduction of a selected Test Environment
• Repetition of selected Test Execution

MO-002a preserves the portion of the source mission-level intent concerning Reproducible Test Environments.

The separate requirement derived from MO-002 addresses:

• MO-002b — Reproducible Test Execution

The Broad Agency Announcement material identifies mission-level intent concerning:

• Realistic and controllable Test Environments
• Deterministic Test Environment states
• Reusable Node configurations
• Reusable Node Sets
• Repeatable testing scenarios
• Consistent testing across different environments

The Broad Agency Announcement material states:

Create repeatable testing scenarios, including regression testing, that allow rigorous evaluation of distributed systems under different conditions and configurations.

The Original Requirement identifiers in the draft DIDO-TE Requirements Register require confirmation before inclusion on this derived requirement page.

Reproducibility requires DIDO-TE to preserve sufficient information to recreate a selected Test Environment and determine whether the recreated Test Environment remains within permitted variation.

The selected Test Environment provides the source against which DIDO-TE evaluates the recreated Test Environment.

The Acceptance Criteria identify:

• The selected Test Environment
• The Test Environment Version
• The applicable Baseline
• The characteristics subject to reproduction
• The characteristics requiring exact equivalence
• The characteristics permitting variation
• The permitted variation
• The comparison method
• The required Evidence
• The treatment of unavailable or substituted Resources
• The treatment of failed, incomplete, or interrupted recreation attempts

Characteristics subject to reproduction can include:

• Configuration
• Resources
• Operational Resources
• Nodes
• Node Sets
• Deployment Targets
• Execution Facilities
• Executable Artifacts
• Dependencies
• Network configuration
• Storage configuration
• Security configuration
• Applicable Starting Conditions
• Applicable Governance Policies

Reproducibility does not necessarily require physical identity or exact equality for every characteristic.

For example:

• A recreated Test Environment can use a different physical machine when the Acceptance Criteria require equivalent processing capability
• A recreated Test Environment can use a different network location when the Acceptance Criteria require the same defined network behavior
• A recreated Test Environment can use a replacement Resource when the Acceptance Criteria permit substitution and define the required equivalence
• A recreated Test Environment can receive a new identity while preserving Traceability to the selected Test Environment

The Acceptance Criteria determine whether a difference is:

• Permitted variation
• An acceptable substitution
• An unresolved discrepancy
• A failure to reproduce the selected Test Environment

MO-002a addresses reproduction of the Test Environment. MO-002b separately addresses whether repeated Test Execution produces Test Results within permitted variation.

This requirement applies to:

• DIDO-TE
• Selected Test Environments
• Recreated Test Environments
• Test Environment identities
• Test Environment Versions
• Baselines
• Configurations
• Resources
• Operational Resources
• Nodes
• Node Sets
• Deployment Targets
• Execution Facilities
• Executable Artifacts
• Dependencies
• Network configurations
• Storage configurations
• Security configurations
• Starting Conditions
• Governance Policies
• Permitted variations
• Substituted Resources
• Acceptance Criteria
• Evidence
• Provenance
• Traceability
• Connected Environments
• Disconnected Environments
• Air-Gapped Environments

Verification confirms that:

1. The selected Test Environment is identified
2. The selected Test Environment Version is identified
3. The applicable Baseline is identified
4. The Acceptance Criteria identify the Test Environment characteristics subject to reproduction
5. The Acceptance Criteria identify the characteristics requiring exact equivalence
6. The Acceptance Criteria identify the characteristics permitting variation
7. The Acceptance Criteria define the permitted variation
8. The Acceptance Criteria define the comparison method
9. DIDO-TE preserves the information required to recreate the selected Test Environment
10. DIDO-TE recreates the selected Test Environment
11. The recreated Test Environment receives an identifiable relationship to the selected Test Environment
12. The recreated Test Environment identifies its Configuration
13. The recreated Test Environment identifies its Resources and Operational Resources
14. The recreated Test Environment identifies its Nodes and Node Sets
15. The recreated Test Environment identifies its Deployment Targets
16. The recreated Test Environment identifies its Execution Facilities
17. The recreated Test Environment identifies its Executable Artifacts and their Versions
18. The recreated Test Environment identifies its Dependencies and their Versions
19. The recreated Test Environment identifies its applicable Starting Conditions
20. The recreated Test Environment identifies its applicable Governance Policies
21. DIDO-TE compares the recreated Test Environment with the selected Test Environment
22. Characteristics requiring exact equivalence satisfy that requirement
23. Characteristics permitting variation remain within the permitted variation
24. Substituted Resources satisfy the equivalence conditions established by the Acceptance Criteria
25. DIDO-TE records discrepancies between the selected and recreated Test Environments
26. DIDO-TE treats failed, incomplete, and interrupted recreation attempts according to the Acceptance Criteria
27. DIDO-TE preserves the Evidence required to evaluate Reproducibility
28. The verification record preserves Traceability among the selected Test Environment, recreated Test Environment, applicable Versions, Baseline, Configurations, Resources, Nodes, Node Sets, Deployment Targets, Execution Facilities, Executable Artifacts, Dependencies, Starting Conditions, Acceptance Criteria, Evidence, and verification result

The following pages reference this requirement:

TBD

Assess the current DIDO-TE implementation against the Test Environment Reproducibility conditions established by the Acceptance Criteria.

Review and approve MO-002a as a proposed derived requirement created from the decomposition of MO-002.

The following unresolved issues affect this requirement:

Confirm the Original Requirement identifiers in the draft DIDO-TE Requirements Register that provide the source for MO-002a.

Define the Test Environment characteristics subject to reproduction.

Define which Test Environment characteristics require exact equivalence.

Define which Test Environment characteristics permit variation.

Define the permitted variation for each applicable Test Environment characteristic.

Define the method used to compare the selected and recreated Test Environments.

Define the equivalence criteria for substituted Resources.

Define the treatment of unavailable Resources and Dependencies.

Define the treatment of failed, incomplete, and interrupted Test Environment recreation attempts.

Define the Evidence required to demonstrate Test Environment Reproducibility.

Define the Test Environment Reproducibility requirements for Connected, Disconnected, and Air-Gapped Environments.

This requirement page retains the stable requirement identifier MO-002a.

This page is a leaf requirement page and omits a trailing :start from its namespace.

The parent MO-002 page is a non-leaf page and retains a trailing :start in its namespace.

Changes to the Statement preserve:

• DIDO-TE as the responsible Actor
• The selected Test Environment as the source subject
• Recreation of the selected Test Environment as the required outcome
• Permitted variation as the applicable Reproducibility boundary
• The Acceptance Criteria as the source of the permitted variation

The Acceptance Criteria identify:

• The selected Test Environment
• The selected Test Environment Version
• The applicable Baseline
• The characteristics subject to reproduction
• The characteristics requiring exact equivalence
• The characteristics permitting variation
• The permitted variation
• The comparison method
• The permitted Resource substitutions
• The treatment of unavailable Resources and Dependencies
• The treatment of failed, incomplete, and interrupted recreation attempts
• The required Evidence

Material changes receive review and update the verification criteria, source records, parent requirement, sibling requirement, and Issues section.

To reference this requirement Statement from another Wiki page, insert:

{{section>dido:03-dido-te:99-annexes:annex-c-requirements:01-mission-objectives:mo-002:mo-002a#Statement&noheader&nofooter&noeditbtn}}

Do not rename this page after an external citation unless a redirect or move plan is in place.