MO-002 — Reproducible Test Environments and Test Execution

Go to C.1 Mission Objectives

MO-002 identifies the mission-level objective for Reproducible Test Environments and Test Execution.

The objective contains two independently verifiable outcomes and therefore serves as a non-leaf parent requirement.

• MO-002a — Reproducible Test Environments
• MO-002b — Reproducible Test Execution

This Mission Objective consolidates mission-level intent from the draft DIDO-TE Requirements Register and the Broad Agency Announcement material concerning:

• Realistic and controllable Test Environments
• Deterministic Test Environment states
• Repeatable testing scenarios
• Consistent Test Execution
• Automated scenario execution
• Recording inputs supplied to Nodes
• Playing recorded inputs back to Nodes
• Coordinating playback across a Node Set
• Preserving Test Environment Configurations
• Preserving Test Definitions
• Preserving Test Argument Values
• Preserving Test Run context
• Comparing repeated Test Results

The Broad Agency Announcement material states the following objective:

Create repeatable testing scenarios, including regression testing, that allow rigorous evaluation of distributed systems under different conditions and configurations.

The source material also identifies:

• Deterministic states
• State consistency
• Predictable outcomes
• Scenario automation
• Repeatable and consistent execution
• Recorded input playback
• Coordinated playback across Nodes

The draft Requirements Register provides the controlling source requirement identifiers. The applicable identifiers require confirmation from a readable copy of the register before inclusion on the derived leaf pages.

The source mission-level intent combines two independently verifiable outcomes:

• Reproduction of a selected Test Environment
• Repetition of selected Test Execution

MO-002 decomposes into:

• MO-002a, which addresses reproduction of a selected Test Environment
• MO-002b, which addresses repetition of selected Test Execution and evaluation of the resulting Test Results

The decomposition permits DIDO-TE to satisfy one derived requirement while independently passing or failing the other.

MO-002a addresses whether DIDO-TE can recreate the applicable conditions, Configurations, Resources, Nodes, Node Sets, Deployment Targets, Execution Facilities, and Dependencies that constitute a selected Test Environment.

MO-002b addresses whether DIDO-TE can repeat selected Test Execution under defined conditions and produce Test Results within the permitted variation established by the Acceptance Criteria.

Reproducibility requires more than rerunning the same command or reusing a Test Definition.

A reproducible Test Environment requires identification and preservation of the applicable:

• Test Environment identity
• Test Environment Version
• Baseline
• Configuration
• Resources
• Operational Resources
• Nodes
• Node Sets
• Deployment Targets
• Execution Facilities
• Executable Artifacts
• Dependencies
• Network conditions
• Starting Conditions
• Applicable Governance Policies

Reproducible Test Execution additionally requires identification and preservation of the applicable:

• Test Definition Version
• Test Sequence
• Sequence Steps
• Test Argument Values
• Expected Results
• Starting Conditions
• Timing Conditions
• External interactions
• Execution Path
• Test Results
• Verdicts
• Supporting Evidence

Distributed systems can contain nondeterministic behaviors caused by concurrency, timing, scheduling, communication latency, external services, random values, and failure conditions. Reproducibility therefore does not necessarily require identical Test Results.

The Acceptance Criteria establish:

• Which conditions remain controlled
• Which conditions can vary
• Which Test Results require exact equality
• Which Test Results permit bounded variation
• The permitted tolerances
• The required comparison method
• The treatment of incomplete, interrupted, failed, timed-out, canceled, or repeated Test Runs

This Mission Objective applies to:

• DIDO-TE
• Test Environments
• Test Execution
• Test Definitions
• Versions
• Configurations
• Test Sequences
• Sequence Steps
• Test Argument Values
• Starting Conditions
• Timing Conditions
• Execution Paths
• Timeouts
• Test Runs
• Test Results
• Expected Results
• Verdicts
• Baselines
• Resources
• Operational Resources
• Nodes
• Node Sets
• Deployment Targets
• Execution Facilities
• Executable Artifacts
• Dependencies
• Network conditions
• External interactions
• Acceptance Criteria
• Evidence
• Provenance
• Traceability
• Connected Environments
• Disconnected Environments
• Air-Gapped Environments

Verification of MO-002 is allocated to:

• MO-002a — Reproducible Test Environments
• MO-002b — Reproducible Test Execution

MO-002 is satisfied only when DIDO-TE satisfies both derived requirements.

The following pages reference this requirement:

TBD

Assess the current DIDO-TE implementation against MO-002a, MO-002b, and their supporting Operational, Functional, Logging and Auditing, Reliability, Performance, Data Management, and Interoperability requirements.

Review and approve the decomposition of MO-002 into MO-002a and MO-002b.

The following unresolved issues affect this requirement:

Confirm the Original Requirement identifiers in the draft DIDO-TE Requirements Register that provide the source for MO-002.

Define the conditions under which DIDO-TE considers a [[dido:99_annexes:annex-b-terms-and-definitions:t:test_environment|Test Environment]] reproduced.

Define the conditions under which DIDO-TE considers [[dido:99_annexes:annex-b-terms-and-definitions:t:test_execution|Test Execution]] reproduced.

Define which Test Environment characteristics remain controlled during reproduction.

Define which Test Environment characteristics can vary during reproduction.

Define which [[dido:99_annexes:annex-b-terms-and-definitions:t:test_result|Test Results]] require exact equality and which permit bounded variation.

Define the permitted Test Result tolerances and comparison methods.

Define the treatment of nondeterministic distributed-system behavior.

Define the treatment of failed, incomplete, interrupted, timed-out, canceled, or repeated [[dido:99_annexes:annex-b-terms-and-definitions:t:test_run|Test Runs]].

Define the [[dido:99_annexes:annex-b-terms-and-definitions:e:evidence|Evidence]] required to demonstrate Test Environment and Test Execution reproducibility.

This page retains the stable parent requirement identifier MO-002.

This page is a non-leaf requirement page and retains a trailing :start in its namespace:

dido:03-dido-te:99-annexes:annex-c-requirements:01-mission-objectives:mo-002:start

MO-002 does not contain a normative Statement. Its derived leaf requirements contain the independently verifiable Statements.

The derived leaf requirements use a Derived From section rather than a Source section.

The derived requirement identifiers remain:

• MO-002a
• MO-002b

Do not assign these identifiers to unrelated requirements.

Changes to the decomposition preserve:

• Test Environment reproducibility as the subject of MO-002a
• Test Execution reproducibility as the subject of MO-002b
• Independent verification of the two outcomes
• Traceability to the source material
• Traceability between MO-002 and both derived requirements

Incoming Traceability uses the Wiki Backlinks function rather than a manually maintained list.

Do not rename this page or either derived requirement page after an external citation unless a redirect or move plan is in place.