Test Result

Go up to Terms and Definitions

A Test Result is a Test Object that records an outcome associated with a Sequenced Test Object or Sequence Step.

A Test Result can record an outcome produced during Test Execution or derived later through evaluation or arbitration.

A Test Result can identify:

• Its identity
• Its name
• Its description
• Its result value
• Its applicable Verdict
• Its start time
• Its completion time
• Its completion status
• Its related Sequenced Test Object
• Its related Sequence Step
• Its applicable Test Run
• Its applicable Test Items
• Its observed values
• Its applicable Expected Results
• Its evaluation criteria
• Its error information
• Its supporting Evidence
• Its applicable Governance Policies
• Its Provenance
• Its Traceability

A Test Result can represent an outcome for a:

• Test Set
• Test Case
• Test Step
• Test Executable
• Sequence Step

Test Object that records an outcome associated with a Sequenced Test Object or Sequence Step

Adapted from:

• OMG Test Information Interchange Format (TestIF), Version 1.0 Beta 3, Section 7.3.11, TestResult
• Test Object
• Sequenced Test Object
• Sequence Step
• Test Run
• DIDO Reference Implementation Conceptual Model
• DIDO-TE draft Requirements Register

OMG TestIF defines TestResult as a TestObject that provides an outcome from executing a test and maintains a reference to the applicable SequencedTestObject and, when applicable, SequenceStep.

OMG TestIF permits a tool or external arbiter to assign a result value or Verdict during or after execution. This definition preserves that capability.

Every Test Result is a Test Object.

A Test Result differs from a Test Run:

• A Test Result records a particular outcome
• A Test Run records the execution context and collects the applicable Test Results

A Test Result differs from an Expected Result:

• An Expected Result specifies an anticipated value, state, event, or condition
• A Test Result records an observed or derived outcome

A Test Result differs from a Verdict:

• A Test Result records the outcome and its supporting information
• A Verdict classifies the Test Result according to defined criteria

A Test Result can exist without a Verdict when:

• Test Execution remains incomplete
• The required Evidence remains unavailable
• Evaluation has not occurred
• Arbitration remains pending
• The applicable criteria do not require a Verdict

A Test Result can receive a Verdict during Test Execution or through later evaluation or arbitration.

A Test Result for a higher-level Sequenced Test Object can derive from multiple lower-level Test Results. The applicable arbitration rules determine how lower-level Test Results contribute to the higher-level Test Result.

A failed lower-level Test Result does not necessarily cause every related higher-level Test Result to fail. The applicable evaluation or arbitration criteria determine that outcome.

A Test Result does not by itself constitute a Validation Decision. A Validation Decision uses applicable Test Results, Verdicts, Evidence, and decision criteria to determine whether the evaluated subject satisfies its defined purpose.

A Test Step requires a Node to accept a valid transaction and return its transaction identifier.

The Test Result records:

• The related Test Step
• The related Sequence Step
• The submitted transaction
• The observed response
• The returned transaction identifier
• The start time and completion time
• The applicable Expected Result
• The comparison between the observed and expected outcomes
• The assigned Verdict
• The supporting Evidence

The applicable Test Run contains the Test Result and preserves its relationship to the performed Test Step and Sequence Step.