Incomplete Test Run

Go up to Terms and Definitions

An Incomplete Test Run is a Test Run that terminated without satisfying its defined completion condition.

An Incomplete Test Run has:

• A Terminated Test Run State
• A Test Run Termination Reason
• An unsatisfied completion condition

An Incomplete Test Run can have the following Test Run Termination Reason:

• Failed
• Interrupted
• Timed Out
• Canceled

An Incomplete Test Run can preserve valid information produced before termination, including:

• Completed Sequence Steps
• Completed Test Steps
• Performed Test Executables
• Applied Test Argument Values
• Observations
• Measurements
• Events
• Outputs
• Test Results
• Verdicts
• Evidence
• Provenance
• Traceability

The applicable Acceptance Criteria determine whether information produced by an Incomplete Test Run can be used for:

• Analysis
• Diagnosis
• Comparison
• Regression analysis
• Baseline establishment
• Validation
• A Validation Decision
• Another defined purpose

Test Run that entered the Terminated Test Run State without satisfying its defined completion condition

Adapted from:

• OMG Test Information Interchange Format (TestIF), Version 1.0
• Test Run
• Test Execution
• Test Run State
• Test Run Termination Reason
• DIDO Reference Implementation Conceptual Model
• DIDO-TE draft Requirements Register

OMG TestIF provides Test Run, Test Result, and Verdict concepts but does not define Incomplete Test Run as a separate TestIF model element.

This definition introduces Incomplete Test Run to distinguish a Test Run that failed to satisfy its completion condition from a completed Test Run producing a Test Result assigned a failing Verdict.

An Incomplete Test Run is a Test Run in the Terminated Test Run State.

A Test Run that is Not Started, Ready, Running, or Paused has not yet completed, but it is not classified as an Incomplete Test Run because its Test Execution has not terminated.

A Test Run becomes an Incomplete Test Run when:

• Test Execution stops
• The Test Run enters the Terminated Test Run State
• The defined completion condition remains unsatisfied

An Incomplete Test Run differs from a completed Test Run assigned a failing Verdict:

• An Incomplete Test Run did not satisfy its completion condition
• A completed Test Run assigned a failing Verdict satisfied its completion condition and produced a Test Result that did not satisfy the applicable Expected Result or Validation Criteria

An Incomplete Test Run does not imply that every Test Result produced before termination is invalid.

The applicable Acceptance Criteria determine:

• Which partial Test Results remain valid
• Which Evidence remains usable
• Whether the Test Run can resume
• Whether DIDO-TE must create a new Test Run
• Whether the Test Run can participate in a comparison
• Whether the Test Run can contribute to a Baseline
• Whether the Test Run can support a Validation Decision
• Whether the Test Run must be repeated

A paused Test Run is not an Incomplete Test Run because it remains eligible to resume.

A canceled Test Run is an Incomplete Test Run when cancellation causes it to enter the Terminated state before satisfying its completion condition.

A timed-out Test Run is an Incomplete Test Run when expiration of an applicable Timeout causes it to enter the Terminated state before satisfying its completion condition.

A Test Run contains ten Sequence Steps.

The Test Run successfully completes the first six Sequence Steps. During the seventh Sequence Step, an Execution Facility becomes unavailable and Test Execution cannot continue.

DIDO-TE:

• Stops Test Execution
• Changes the Test Run State from Running to Terminated
• Assigns Interrupted as the Test Run Termination Reason
• Classifies the Test Run as an Incomplete Test Run
• Preserves the Test Results produced by the first six Sequence Steps
• Records the affected Sequence Step
• Records the unavailable Execution Facility
• Preserves the supporting Evidence
• Maintains Provenance and Traceability

The applicable Acceptance Criteria determine whether the preserved Test Results can be used and whether DIDO-TE must repeat the entire Test Run or resume from an approved Sequence Step.