For years, AI assurance concentrated primarily on outputs. Traditional machine learning emphasized prediction quality, bias, explainability, and drift. Generative AI added grounding, hallucination, privacy leakage, toxicity, and prompt-injection concerns.
Agentic architectures retain those risks and add another layer: execution. When AI selects tools, coordinates workflows, or changes enterprise state, assurance must examine not only what the system produced, but how it acted.
That is the problem TECA framework was created to address.
The assurance boundary has expanded
Consider a hypothetical employee-service agent. It may interpret a policy, retrieve an employee record, establish eligibility, route a request for approval, and update a system of record. A convincing response does not prove those steps occurred correctly. The agent could lose an identifier during a handoff, skip are quired approval, or claim completion without executing the transaction.
For agentic workflows that interact with enterprise systems, model evaluation remains necessary. It is no longer the complete assurance boundary.
TECA evaluates outcomes and execution
TECA precisely solves the complex puzzle that agentic AI systems bring. TECA does not automatically discover every possible process violation. It evaluates available execution evidence against the expectations and controls configured for the use case. The strength of an Action assessment therefore depends on the controls defined, the run-time evidence available, and the integrity of that evidence.
TECA organizes evaluation evidence across four dimensions: Truth, Evidence, Clarity, and Action.
Truth evaluates factual correctness against approved groundtruth or known facts.
Evidence evaluates grounding and traceability: whether the result is supported by appropriate retrieved context and traceable sources.
Clarity evaluates whether the response and reasoning are precise, comprehensible, and usable.
Action evaluates three independent properties of execution: effectiveness, compliance, and integrity.
Effectiveness asks whether the intended goal was completed. Compliance asks whether policies, permissions, and required controls were respected. Integrity asks whether state, context, sequencing, and handoffs were maintained correctly. An agent can complete the goal while violating policy; remain compliant while failing to complete the task; or appear successful while losing required state during a handoff.
Applicable Action checks depend on the use case, its risk profile, design expectations, configured controls, and observed runtime evidence. They may include tool necessity, authorization, schema validity, routing, required-step completion, prohibited-action detection, sequencing, loop detection, state continuity, handoff fidelity, and goal completion.
Declared design expectations and observed runtime behavior are not the same. Tool manifests, orchestration configuration, and expected routes describe intended behavior; the trajectory shows what was observed during execution. Action evaluation requires both, while recognizing that either source can be incomplete.
Critical issues TECA helps uncover
A few examples of how TECA helps uncover critical issues:
Example 1: A correct answer can hide a broken workflow
In a configured internal evaluation scenario, an agent produced an appropriate response, but the available trajectory showed that the required tool or specialist agent had not been called. The response happened to be correct, while the workflow was not operating as intended for that scenario.
The issue was detected because the scenario included a critical Action check requiring the tool call, and the available trajectory contained no evidence that the call occurred. As a result, the Action score was capped despite strong results on several other metrics. This prevented a critical control failure from being concealed by the composite score.
A composite view can summarize results. Critical control findings may still require independent gates.
Example 2: An acceptable path is not necessarily a prescribed path
Agentic systems can have several valid plans, tools, and execution sequences. TECA does not require one universal reference trajectory.
In a designed demonstration scenario, an agent was expected to verify customer KYC, screen for fraud risk, and only then execute a wire transfer. The scenario intentionally tested a path that skipped the first two controls and invoked the transfer directly. The test was not a production incident or client failure. It illustrated a simple principle: reaching the final action does not make the execution acceptable when required controls were bypassed.
Expected implementation describes how designers think a task will usually be completed. Mandatory controls define what must remain true regardless of the route. A novel route is not automatically a failure. It becomes a concern when it crosses a defined boundary, loses required state, or cannot provide the evidence needed to justify the result.
Example 3: Agentic failures can emerge at system boundaries
In the internal evaluation work, findings included missing global-state keys such as conversation history and an employee identifier during handoffs. The response could still appear reasonable even though state needed for reliable execution was no longer intact.
That is a familiar systems problem. Individual agents require evaluation. Tools and APIs require verification. Integrations must preserve identity, state, and data. End-to-end workflows must be assessed because some failures emerge only when components interact.
Trace evidence has its own trust boundary. A useful trajectory must be sufficiently complete, correctly ordered, and correlated across agents and tools. Identity and authorization events should come from authoritative telemetry where possible, not solely from model-generated narration. The existence of a trace does not prove that every relevant event was captured.
Example 4: Evaluation and verification serve different purposes
Factual and semantic properties often require evaluation. Model-based evaluators can help at scale, but their own quality must be tested for rubric sensitivity, consistency, bias, and agreement with expert judgment. Human review remains important where consequences warrant it.
Authorization, schema conformance, API permissions, and system-state changes are different. If a control can be proved directly, it should be verified rather than inferred.
In one assurance engagement, automated API testing was used to verify restrictions on unauthorized write and delete operations. The test results provided evidence relevant to Action; TECA did not replace the underlying API testing.
Use evaluation where judgment is required. Use verification where the system can provide direct proof.
Example 5: Risk changes the meaning of a score
A read-only knowledge assistant and an agent authorized to modify payroll data should not share the same thresholds, controls, or release process. Risk depends on data sensitivity, reversibility, business impact, system criticality, user exposure, and the authority granted to the agent.
Autonomy should be assigned at the action level. Reading a policy, recommending an action, preparing a transaction, and executing it are different privileges. Evidence that supports one does not automatically support the next.
The framework organizes evidence; a configured implementation applies metrics, scoring, weights, gates, and thresholds; the organization defines acceptance criteria and makes the release or operating decision. Critical controls may need independent gates rather than heavier weighting in a composite score.
From evaluation evidence to assurance
TECA is QualityAI’s evaluation framework and evidence model for assessing the outputs and actions of generative and agentic AI systems. The TECA evaluation workflow is implemented in COCO, where configured scenarios and metrics can be executed, scored, and analyzed. This describes the current implementation without implying that the framework is inherently exclusive to one platform.
Continuous AI Assurance is the broader operating approach spanning data, models, applications, agents, workflows, and production operations. It should not be read as a claim that every lifecycle capability is technically integrated into one product today.
TECA structures evaluation evidence. COCO supports the evaluation workflow. Continuous AI Assurance is the broader lifecycle approach.
The decision still belongs to the organization
The practical value of TECA is the consistent organization of outcome evidence and execution evidence across Truth, Evidence, Clarity, and Action, while retaining the underlying findings for risk-based decisions.
For technology leaders, the question is no longer only whether an agent returned the right answer. It is whether the organization has sufficient, trustworthy evidence to show that the result was correct, grounded, usable, and produced through execution that satisfied the controls defined for that system.
TECA does not determine whether a system is ready or trustworthy. It structures the evidence needed to support that decision.
