Verdict

Go up to Terms and Definitions

A Verdict is a classification assigned to a Test Result according to defined evaluation criteria.

A Verdict communicates the evaluated status of a Test Result independently of the detailed values, observations, measurements, and Evidence recorded by that Test Result.

A Verdict set can include values such as:

• PASS
• FAIL
• INCONCLUSIVE
• ERROR
• NOT EXECUTED
• NOT APPLICABLE

The applicable Acceptance Criteria, evaluation criteria, or Governance Policy defines:

• The permitted Verdict values
• The meaning of each Verdict value
• The conditions for assigning each Verdict value
• The authority permitted to assign or revise a Verdict
• The treatment of missing or incomplete results
• The arbitration of multiple Test Results
• The precedence of conflicting Verdicts
• The use of Verdicts in a Validation Decision

A Verdict can apply to a Test Result associated with a:

• Test Set
• Test Case
• Test Step
• Test Executable
• Sequence Step

classification assigned to a Test Result according to defined evaluation criteria

Adapted from:

• OMG Test Information Interchange Format (TestIF), Version 1.0 Beta 3, Sections 7.3.11 and 7.3.18, Verdict
• Test Result
• Expected Result
• Acceptance Criteria
• DIDO Reference Implementation Conceptual Model
• DIDO-TE draft Requirements Register

OMG TestIF defines Verdict as an enumeration of result Verdicts that can be assigned to a TestResult.

OMG TestIF supports the storage and interchange of Verdicts but leaves Test Result arbitration rules outside the scope of the specification. This definition therefore requires the applicable Acceptance Criteria, evaluation criteria, or Governance Policy to establish the meaning and assignment rules for each permitted Verdict.

Use the singular glossary term Verdict. Link plural usage to the same controlling page:

[[dido:99_annexes:annex-b-terms-and-definitions:v:verdict|Verdicts]]

A Verdict differs from a Test Result:

• A Test Result records an outcome and its supporting information
• A Verdict classifies that outcome according to defined criteria

A Verdict differs from an Expected Result:

• An Expected Result specifies an anticipated value, state, event, or condition
• A Verdict classifies the relationship between the applicable outcome and the evaluation criteria

A Verdict differs from a Validation Decision:

• A Verdict classifies an individual or aggregated Test Result
• A Validation Decision determines whether the evaluated subject satisfies the applicable validation purpose or decision criteria

A PASS Verdict for one Test Result does not establish that:

• The entire Test Run passed
• Every related Test Case passed
• Every applicable requirement was satisfied
• The evaluated subject is valid
• The evaluated subject is accepted
• The evaluated subject is authorized for use

Those conclusions require the applicable aggregation, arbitration, acceptance, validation, or authorization criteria.

A FAIL Verdict indicates failure to satisfy the criteria applicable to the classified Test Result. It does not necessarily identify the cause of the failure.

An ERROR Verdict ordinarily indicates that an execution, infrastructure, configuration, or evaluation problem prevented the expected evaluation. An ERROR Verdict does not necessarily indicate that the Test Item failed the tested condition.

An INCONCLUSIVE Verdict ordinarily indicates that the available Test Result and Evidence do not support a PASS or FAIL determination.

A Verdict can be assigned:

• During Test Execution
• After Test Execution
• By an automated mechanism
• By an authorized evaluator
• By an arbitration process
• By another mechanism established by the applicable Governance Policy

A change to a Verdict should preserve the previous Verdict, the reason for the change, the responsible authority, the supporting Evidence, the applicable criteria, Provenance, and Traceability.

A Test Step requires a selected Node to return a defined state within five seconds.

The resulting Test Result records:

• The returned state
• The measured response time
• The applicable Expected Result
• The applicable Acceptance Criteria
• The supporting Evidence

The evaluator assigns:

• PASS when the Node returns the required state within five seconds
• FAIL when the Node returns an incorrect state or exceeds five seconds
• ERROR when the Test Environment cannot submit the request
• INCONCLUSIVE when the available Evidence cannot establish the returned state or response time

The Verdict classifies the Test Result. A separate Validation Decision determines whether the evaluated Node Set satisfies the broader validation purpose.