MO-005 — Comparative Evaluation of Candidate Solutions

Go to C.1 Mission Objectives

DIDO-TE SHALL produce a comparative evaluation of the selected Candidate Solutions according to the Acceptance Criteria.

This requirement consolidates mission-level intent from the DIDO-TE source material addressing:

Creating baseline results using validated nodes and comparing them against tests with modified nodes to identify discrepancies or improvements.

The source material further identifies:

The ability to emulate real-world conditions, simulate different node configurations, and conduct baseline and comparison testing will be assessed for thoroughness and coverage.

The source material describes the use of DIDO-TE to evaluate different Node configurations, implementations, technologies, and candidate approaches under controlled testing conditions.

Identify the authoritative source requirements, original requirement identifiers, and controlling citations for the comparative evaluation of Candidate Solutions.

MO-005 expresses the common mission-level outcome. Detailed obligations addressing Candidate Solution selection, common evaluation conditions, Test Execution, Test Result collection, comparison methods, evaluation criteria, ranking, reporting, and Evidence preservation remain separate Operational, Functional, Performance, Data Management, and Interoperability Requirements.

Organizations need an objective means of comparing alternative implementations, technologies, products, configurations, architectures, services, or other proposed solutions.

A comparative evaluation permits an organization to determine how selected Candidate Solutions differ according to defined characteristics such as:

• Functional behavior
• Correctness
• Performance
• Reliability
• Stability
• Security
• Storage
• Energy consumption
• Interoperability
• Resource consumption
• Cost
• Scalability
• Maintainability
• Recoverability
• Compliance
• Another characteristic established by the Acceptance Criteria

A comparative evaluation requires a common evaluation basis. Differences unrelated to the Candidate Solutions can confound the evaluation and prevent an objective comparison.

The Acceptance Criteria identify:

• The selected Candidate Solutions
• The characteristics subject to evaluation
• The applicable Test Definitions
• The applicable Test Environments
• The applicable Test Argument Values
• The applicable Starting Conditions
• The applicable Timing Conditions
• The permitted Execution Paths
• The applicable Configurations
• The characteristics held constant
• The characteristics permitted to vary
• The comparison methods
• The measurement conditions
• The measurement units
• The required measurement precision
• The permitted tolerances
• The permitted exclusions
• The rules for normalizing measurements
• The rules for combining evaluation characteristics
• The treatment of missing, invalid, incompatible, or unavailable information
• The treatment of nondeterministic behavior
• The permitted comparative-evaluation outcomes

The comparative evaluation can identify:

• Equality
• Difference
• Direction of difference
• Magnitude of difference
• Relative advantage
• Relative disadvantage
• Trade-off
• Rank
• Equivalence within an applicable tolerance
• Insufficient information to complete the evaluation

MO-005 does not require DIDO-TE to select a preferred Candidate Solution unless the Acceptance Criteria explicitly establish:

• A selection method
• The characteristics contributing to selection
• The relative weight of each characteristic
• The rules for resolving equal or conflicting results
• The applicable decision Authority

A comparative evaluation can identify relative differences without producing a selection or acquisition decision.

MO-005 differs from MO-004a:

• MO-004a compares a selected Test Result with its corresponding Test Result in a controlling DIDO Baseline
• MO-005 compares the evaluation outcomes of multiple selected Candidate Solutions using a common evaluation basis

A comparative evaluation can use DIDO Baseline comparisons as inputs, but MO-005 does not require every Candidate Solution evaluation to use a DIDO Baseline unless the Acceptance Criteria establish that requirement.

This requirement applies to:

• DIDO-TE
• Candidate Solutions
• Selected Candidate Solutions
• Comparative evaluations
• Common evaluation bases
• Evaluation characteristics
• Evaluation outcomes
• Comparison methods
• Measurement conditions
• Measurement units
• Measurement precision
• Permitted tolerances
• Permitted exclusions
• Measurement normalization
• Relative advantages
• Relative disadvantages
• Trade-offs
• Rankings
• Equivalence determinations
• Inconclusive evaluations
• Acceptance Criteria
• Authorities
• Governance Policies
• Test Definitions
• Test Environments
• Test Execution
• Test Argument Values
• Starting Conditions
• Timing Conditions
• Execution Paths
• Configurations
• Versions
• Test Runs
• Test Run States
• Test Run Termination Reasons
• Incomplete Test Runs
• Test Results
• DIDO Baselines
• DIDO Baseline comparison results
• Nodes
• Node Sets
• Deployment Targets
• Operational Resources
• Evidence
• Provenance
• Traceability

Verification confirms that:

1. The selected Candidate Solutions are identified
2. At least two Candidate Solutions participate in the comparative evaluation
3. The Acceptance Criteria identify the characteristics subject to evaluation
4. The Acceptance Criteria identify the common evaluation basis
5. The Acceptance Criteria identify the applicable Test Definitions
6. The Acceptance Criteria identify the applicable Test Environments
7. The Acceptance Criteria identify the applicable Test Argument Values
8. The Acceptance Criteria identify the applicable Starting Conditions
9. The Acceptance Criteria identify the applicable Timing Conditions
10. The Acceptance Criteria identify the permitted Execution Paths
11. The Acceptance Criteria identify the applicable Configurations
12. The Acceptance Criteria identify the characteristics held constant among Candidate Solution evaluations
13. The Acceptance Criteria identify the characteristics permitted to vary among Candidate Solution evaluations
14. The Acceptance Criteria establish the comparison method for each evaluated characteristic
15. The Acceptance Criteria establish the applicable measurement conditions
16. The Acceptance Criteria establish the applicable measurement units
17. The Acceptance Criteria establish the required measurement precision
18. The Acceptance Criteria establish the permitted tolerances
19. The Acceptance Criteria establish the permitted exclusions
20. The Acceptance Criteria establish the rules for normalizing measurements
21. The Acceptance Criteria establish the rules for combining evaluation characteristics
22. The Acceptance Criteria establish the treatment of missing, invalid, incompatible, and unavailable information
23. The Acceptance Criteria establish the treatment of nondeterministic behavior
24. The Acceptance Criteria establish the treatment of Incomplete Test Runs for each applicable Test Run Termination Reason
25. The Acceptance Criteria establish the permitted comparative-evaluation outcomes
26. DIDO-TE associates each selected Candidate Solution with its applicable Test Runs and Test Results
27. DIDO-TE evaluates each selected Candidate Solution using the common evaluation basis
28. DIDO-TE applies the comparison method established for each evaluated characteristic
29. DIDO-TE normalizes measurements according to the Acceptance Criteria
30. DIDO-TE compares the evaluation outcomes of the selected Candidate Solutions
31. DIDO-TE identifies the relative difference for each evaluated characteristic
32. DIDO-TE identifies each comparative outcome supported by the available Test Results
33. DIDO-TE identifies each characteristic for which the available information is insufficient
34. DIDO-TE records the comparative evaluation
35. The verification record preserves Traceability among the Acceptance Criteria, selected Candidate Solutions, common evaluation basis, Test Definitions, Test Environments, Test Runs, Test Results, evaluated characteristics, measurement normalization, comparison methods, comparative outcomes, and supporting Evidence

The following pages reference this requirement:

TBD

Assess the current DIDO-TE implementation against the comparative evaluation obligation established by MO-005.

Review and accept MO-005 as a proposed Mission Objective consolidated from the comparative-evaluation intent in the DIDO-TE source material.

The following unresolved issues affect this requirement:

Identify the authoritative source requirements, original requirement identifiers, and controlling citations for the comparative evaluation of Candidate Solutions.

Define Candidate Solution in the shared Terms and Definitions corpus.

Determine whether Comparative Evaluation requires a separate shared Terms and Definitions entry.

Define the Candidate Solution characteristics subject to comparative evaluation.

Define the common evaluation basis used for selected Candidate Solutions.

Define the characteristics held constant and the characteristics permitted to vary among Candidate Solution evaluations.

Define the supported comparison methods.

Define the measurement conditions, units, precision, exclusions, and tolerances used for each evaluated characteristic.

Define the rules for normalizing measurements produced by different Candidate Solutions.

Define the rules for combining evaluation characteristics.

Define the treatment of missing, invalid, incompatible, or unavailable information.

Define the treatment of nondeterministic behavior.

Define the treatment of Incomplete Test Runs for each applicable Test Run Termination Reason.

Define the permitted comparative-evaluation outcomes.

Determine whether MO-005 permits DIDO-TE to rank Candidate Solutions.

Determine whether MO-005 permits DIDO-TE to recommend or select a preferred Candidate Solution.

Define the Evidence DIDO-TE must preserve for each comparative evaluation.

This requirement page should retain the stable requirement identifier MO-005.

This page is a leaf requirement page and omits a trailing :start from its namespace.

MO-005 consolidates mission-level intent from multiple source statements. MO-005 does not result from splitting a single parent requirement and therefore uses a Source section rather than a Derived From section.

Changes to the Statement should preserve:

• DIDO-TE as the responsible Actor
• At least two selected Candidate Solutions as the subjects of the comparative evaluation
• The comparative evaluation as the required outcome
• The Acceptance Criteria as the source of the evaluation basis and comparison rules

The Acceptance Criteria should identify:

• The selected Candidate Solutions
• The characteristics subject to evaluation
• The common evaluation basis
• The applicable Test Definitions
• The applicable Test Environments
• The applicable Test Argument Values
• The applicable Starting Conditions
• The applicable Timing Conditions
• The permitted Execution Paths
• The applicable Configurations
• The characteristics held constant
• The characteristics permitted to vary
• The comparison methods
• The measurement conditions
• The measurement units
• The required measurement precision
• The permitted tolerances
• The permitted exclusions
• The measurement-normalization rules
• The rules for combining evaluation characteristics
• The treatment of missing, invalid, incompatible, or unavailable information
• The treatment of nondeterministic behavior
• The treatment of Incomplete Test Runs
• The permitted comparative-evaluation outcomes
• The required Evidence
• The required Provenance
• The required Traceability

MO-005 does not require a DIDO Baseline comparison unless the Acceptance Criteria establish that requirement.

MO-005 does not authorize DIDO-TE to recommend, select, approve, acquire, or deploy a Candidate Solution unless a separate Requirement and the applicable Authority establish that obligation.

Material changes should receive review and should update the verification criteria, source records, supporting requirements, and Issues section.

To reference this requirement Statement from another Wiki page, insert:

{{section>dido:03-dido-te:99-annexes:annex-c-requirements:01-mission-objectives:mo-005#Statement&noheader&nofooter&noeditbtn}}

Do not rename this page after an external citation unless a redirect or move plan is in place.