Test Execution

Go up to Terms and Definitions

Test Execution is the performance of one or more Sequenced Test Objects within a Test Environment to produce one or more Test Results.

Test Execution can perform:

• A Test Set
• A Test Case
• A Test Step
• A Test Executable

Test Execution can include:

• Selecting the applicable Sequenced Test Objects
• Establishing the applicable Test Environment
• Confirming applicable preconditions
• Selecting the applicable Test Items
• Selecting the applicable Node Sets
• Selecting the applicable Deployment Targets
• Resolving the applicable Test Arguments
• Supplying the applicable Test Argument Values
• Selecting the applicable Execution Facilities
• Retrieving the applicable Executable Artifacts
• Interpreting or performing the applicable Executable Instructions
• Observing the behavior of the Test Items
• Measuring applicable conditions and outcomes
• Comparing observed outcomes with Expected Results
• Assigning applicable Verdicts
• Producing Test Results
• Collecting Evidence
• Preserving Provenance
• Maintaining Traceability

A Test Sequence can establish the order, conditional progression, repetition, and termination of the Sequenced Test Objects performed during Test Execution.

A Test Run identifies and records a particular occurrence of Test Execution.

performance of one or more Sequenced Test Objects within a Test Environment to produce one or more Test Results

Adapted from:

• OMG Test Information Interchange Format (TestIF), Version 1.0 Beta 3, Sections 1.3, 7.3.2, 7.3.9, 7.3.10, and 7.3.11
• Sequenced Test Object
• Test Sequence
• Test Run
• Test Result
• Test Environment
• DIDO Reference Implementation Conceptual Model
• DIDO-TE draft Requirements Register

OMG TestIF models the information used to define tests and the TestRuns and TestResults produced when those tests execute.

OMG TestIF does not define Test Execution as a separate TestIF model element. This definition introduces Test Execution as the activity that occurs between the applicable test definitions and the resulting Test Run and Test Results.

Test Execution differs from a Test Run:

• Test Execution is the activity of performing the applicable Sequenced Test Objects
• A Test Run identifies and records a particular occurrence of that activity

Test Execution differs from a Test Result:

• Test Execution performs the applicable test activities
• A Test Result records an outcome produced by those activities

Test Execution differs from a Test Sequence:

• A Test Sequence specifies the order and progression of Sequenced Test Objects
• Test Execution performs those objects according to the applicable Test Sequence

Test Execution differs from a Test Environment:

• A Test Environment identifies the conditions, resources, configurations, and dependencies required to conduct testing
• Test Execution is the activity conducted within those conditions and using those resources

Test Execution differs from a Test Executable:

• A Test Executable contains or references Executable Instructions
• Test Execution causes the applicable Execution Facilities to interpret or perform those instructions

Test Execution can use automated, manual, or combined procedures.

Manual Test Execution can produce valid Test Results when the applicable test definition identifies:

• The required actions
• The responsible participant
• The required observations
• The applicable Expected Results
• The applicable Verdict criteria
• The required Evidence

Test Execution can perform Sequenced Test Objects:

• Sequentially
• Concurrently
• Conditionally
• Iteratively
• In response to an event
• According to another progression established by the applicable Test Sequence

Starting Test Execution does not establish successful completion. Test Execution can:

• Complete successfully
• Complete with one or more failed Test Results
• Terminate because of an error
• Terminate because of a timeout
• Stop because a precondition is not satisfied
• Stop because a required resource is unavailable
• Be interrupted
• Be canceled
• Remain incomplete

The applicable Acceptance Criteria or Governance Policy determines how incomplete, interrupted, canceled, repeated, or failed Test Execution affects the resulting Verdict or Validation Decision.

Repeated Test Execution does not necessarily reproduce the same Test Results. Reproducibility depends on preserving or controlling the applicable:

• Sequenced Test Object Versions
• Test Item Versions and states
• Test Environment
• Test Argument Values
• Executable Artifact Versions and content digests
• Execution Facility Versions and configurations
• Dependencies
• Starting conditions
• Timing conditions
• External interactions
• Governance Policies

A reproducible Test Run should preserve sufficient Evidence, Provenance, and Traceability to identify these conditions.

A Test Case evaluates whether a selected Node Set reaches the expected distributed state after submission of a proposed transaction.

Test Execution:

• Establishes the applicable Test Environment
• Confirms the required Nodes and Execution Facilities are available
• Loads the applicable Executable Artifacts
• Supplies the applicable Test Argument Values
• Performs the applicable Test Steps
• Submits the proposed transaction through the applicable Endpoint
• Observes the resulting state of each applicable Node
• Compares the observed states with the applicable Expected Results
• Produces the applicable Test Results
• Preserves the supporting Evidence

The Test Run records this occurrence of Test Execution and maintains Traceability among:

• The Test Case
• The performed Test Steps
• The applicable Test Items
• The applicable Node Set
• The Test Environment
• The Test Argument Values
• The Executable Artifacts
• The Execution Facilities
• The Test Results
• The Evidence
• The resulting Validation Decision