Test Execution Exception

Go up to Terms and Definitions

A Test Execution Exception identifies an event or condition that disrupts, prevents, or requires deviation from the defined progression of Test Execution.

A Test Execution Exception can arise from:

• An invalid Test Argument Value
• An unsatisfied Starting Condition
• An unsatisfied Timing Condition
• Expiration of a Timeout
• An unavailable Test Item
• An unavailable Node
• An unavailable Operational Resource
• An unavailable Execution Facility
• An unavailable Executable Artifact
• An unsatisfied Dependency
• An invalid Configuration
• A communication failure
• A resource failure
• A security event
• A policy violation
• An invalid state transition
• An unexpected Test Result
• An unauthorized operation
• An external event
• Another defined exceptional condition

A Test Execution Exception can identify:

• Its identity
• Its name
• Its description
• Its exception type
• Its severity
• Its detection time
• Its detection source
• Its originating event or condition
• The affected Test Run
• The affected Test Sequence
• The affected Sequence Step
• The affected Test Step
• The affected Test Executable
• The affected Operational Resources
• The applicable Test Run State
• The applicable Exception Handling
• The applicable Governance Policies
• Its disposition
• Its resulting Test Run State
• Its applicable Test Run Termination Reason
• Its Evidence
• Its Provenance
• Its Traceability

A Test Execution Exception can cause DIDO-TE to:

• Continue Test Execution
• Retry an operation
• Repeat a Sequence Step
• Skip a permitted Sequence Step
• Select an alternative Execution Path
• Restore a defined state
• Perform a compensating operation
• Pause the Test Run
• Resume the Test Run
• Cancel the Test Run
• Terminate the Test Run
• Request an authorized Actor decision
• Record the exception without changing the Test Run State

The applicable Exception Handling determines the response to a detected Test Execution Exception.

event or condition that disrupts, prevents, or requires deviation from the defined progression of Test Execution

Adapted from:

• OMG Test Information Interchange Format (TestIF), Version 1.0
• Test Execution
• Test Run
• Test Run State
• Test Run Termination Reason
• Exception Handling
• DIDO Reference Implementation Conceptual Model
• DIDO-TE draft Requirements Register

OMG TestIF defines information used to describe and exchange testing activities and execution results.

OMG TestIF does not define Test Execution Exception as a separate TestIF model element. This definition introduces the term to identify exceptional events and conditions affecting DIDO-TE Test Execution and to associate those events and conditions with their handling, resulting Test Run States, Test Results, Evidence, Provenance, and Traceability.

A Test Execution Exception is specific to Test Execution. It does not mean:

• An approved exception to a Governance Policy
• A waiver from a Requirement
• A legal or regulatory exception
• A programming-language exception unless the programming-language exception affects Test Execution
• Any unusual event unrelated to Test Execution

A Test Execution Exception differs from an error:

• An error identifies an incorrect state, value, action, or result
• A Test Execution Exception identifies an event or condition requiring a Test Execution response

An error can cause a Test Execution Exception. An error that does not affect Test Execution does not necessarily constitute a Test Execution Exception.

A Test Execution Exception differs from a failure:

• A failure identifies an inability to perform a required function or satisfy a required condition
• A Test Execution Exception identifies the effect of an event or condition on Test Execution progression

A failure can cause a Test Execution Exception. A Test Execution Exception does not necessarily indicate a failure.

A Test Execution Exception differs from a Test Run Termination Reason:

• A Test Execution Exception identifies the event or condition affecting Test Execution
• A Test Run Termination Reason classifies why the Test Run entered the Terminated Test Run State

A handled Test Execution Exception does not necessarily terminate the Test Run.

An unhandled or unrecoverable Test Execution Exception can cause a Test Run to enter the Terminated Test Run State.

A single Test Execution Exception can result in:

• No Test Run State transition
• One Test Run State transition
• Multiple Test Run State transitions
• One or more additional Test Results
• One or more additional Evidence records

Multiple Test Execution Exceptions can contribute to one Test Run termination.

The applicable Acceptance Criteria or Governance Policies should determine:

• Which events and conditions constitute Test Execution Exceptions
• How DIDO-TE classifies each Test Execution Exception
• Which Exception Handling applies
• Whether Test Execution can continue
• Whether human intervention is permitted or required
• Which Test Run State transition occurs
• Which Test Run Termination Reason applies
• Which Evidence DIDO-TE preserves

A Test Step requires a Node to return a response within 30 seconds.

The Node does not return the response before the applicable Timeout expires.

DIDO-TE detects a Test Execution Exception identifying:

• The expired Timeout
• The affected Test Run
• The affected Sequence Step
• The affected Test Step
• The affected Node
• The time of detection
• The applicable Exception Handling

The applicable Exception Handling permits one retry. If the retry succeeds, DIDO-TE continues Test Execution and records the Test Execution Exception and its disposition.

If the retry does not succeed, DIDO-TE:

• Changes the Test Run State from Running to Terminated
• Assigns Timed Out as the Test Run Termination Reason
• Identifies the Test Run as an Incomplete Test Run
• Preserves the applicable Evidence
• Maintains Provenance and Traceability