How Accurate Are Your AI Agents?

As AI's role matures in Fraud Risk AML (FRAML) and more broadly in Financial Services, so must our methods of evaluation. A singular focus on an 'accuracy' score was suited for a generation where we had rule-engine based decisioning or binary classifiers for scoring 'fraud vs no-fraud'. Today's multi-agents solutions manage complex workflows and e2e customer journey with multiple usecases and touch-points. For these sophisticated workflows, the old benchmark like accuracy specifically 'ai agent accuracy' is insufficient and often misleading.

The real question isn't about an individual agent's accuracy but the effectiveness of the entire AI-augmented workflow. The conversation must shift from a myopic focus on single agents to a holistic view of the multi-agent systems which are and would be the backbone of most modern organizations. We are no longer just building better risk models; we are building digital colleagues, and evaluating them requires a more nuanced approach.

Old Metrics in a New World: The Ghost of Classifiers

Consider the classic Transaction Fraud classifier. Its job is to label a transaction as 'Fraud' vs 'Not Fraud'. We measure it with a trusty toolkit, the confusion matrix:

• True Negative (TN): Good transactions approved.

• True Positive (TP): Fraudulent transactions blocked.

• False Negative (FN): Fraudulent transactions missed (losses).

• False Positive (FP): Good transactions blocked (friction).

From this, we calculated Accuracy, Precision (TP/(TP+FP)), and Recall (TP/(TP+FN)). These metrics, while useful, are deceptively simple. A model with 99.9% accuracy is a catastrophic failure if the 0.1% it gets wrong are all very high-value transactions or part of a sophisticated fraud ring, not only putting the institution at risk of massive losses, but exposure to loss of trust and PR nightmare.

As FRAML practitioners we understand that overall accuracy is often a vanity metric. We care about recall on the highest-value transactions, correctly handling ambiguous cases, all while keeping customer friction low. The business and operations context matters, more than a single, sterile measure.

The Rise of Multi-Agent Workflows: From Soloist to Symphony

Today, we're not just deploying individual model(s); we're building agentic workflows that manage the entire customer lifecycle. The multi-agent system can perceive, reason, plan, and execute actions using various tools to achieve a goal.

Lets click down into FRAML workflows - onboarding a new consumer and then monitoring their account. In Agentic workflow this translates to:

1. Onboarding Agent: Receives a new application, extracts data, and uses OCR to read uploaded docs e.g. driver's license, proof of address etc.

2. Identity & KYC Agent: Takes the extracted data and uses APIs to check it against 3p identity intelligence databases, OSINT, and global sanctions/PEP lists (like OFAC). And applies you internal policies

3. Risk Assessment Agent: Analyzes the customer's profile, history, and plethora of risk signals and say initial funding source to assign a preliminary risk score.

4. Account 360 Agent: Talks to other agents, applies your reasoning, your policies, SOPs, and makes a comprehensive narrative for each application uniquely.

5. Actioning Agent: Enables the downstream actioning, e.g. API orchestration, note taking, updating SORs to actually create the accounts and trigger a "welcome flow".

6. Continuous Monitoring Sub-Workflow: Orchestrate a multi-agent sub-workflow. It watches the account's transaction stream in real-time, looking for Transaction fraud,  AML red flags (e.g., structuring deposits to stay under reporting thresholds), signs of account takeover fraud, or behavior that deviates significantly from the customer's established profile, flagging activity for review.

How do you measure the "accuracy" of this system? Is it the Onboarding Agent's OCR success rate? The KYC Agent's API call success rate? The Monitoring Agent's fraud detection precision? Trying to measure each agent in isolation is like judging a symphony by testing the tuning of each individual violin before the performance. What really matters is the music that fills the hall and your heart.

The true metric here is the business impact of end-to-end customer lifecycle management. Did the entire workflow correctly vet and onboard a legitimate customer smoothly while successfully identifying a high-risk applicant.

The Human Benchmark: Workforce Augmentation

This brings us to the most critical reframing: AI agents are at the core workforce augmentation. They are digital team members designed for data-intensive, contextual, and judgment-oriented workflow tasks, e.g assisting Compliance Officers, helping Sr. Fraud Analysts, or augmenting entire operations teams.

How do we measure the performance of our best human experts? We don't give our star AML  compliance analyst a simple "accuracy score." We conduct a performance review - We look at their case resolution rate, the complexity of their investigations, their ability to identify novel risks, and their adherence to regulatory standards.

Should Agentic AI workforce be any different? Are we heading toward a future where we conduct peer-reviewed performance evaluation for our AI agents ? :)

Agentic Workforce for KYC/Onboarding: 2025-Q2 Peer-review Performance report:

• Strengths: Consistently achieves 99.9% accuracy on standard US-based identification documents. Excellent speed.

• Areas for Improvement: Struggles with non-standard international IDs. Needs to "upskill" on interpreting proof-of-address documents from different countries.

• Promotion Path: This multi-agent workforce can be promoted to "Senior KYC Specialist" by training it on more complex, cross-border verification logic

• Salary Raise: It deserves a "raise" in the form of more GPU resources

This humorous analogy exposes a deeper truth. Evaluating an advanced AI system requires a qualitative, context-rich assessment, much like evaluating a human employee. We need to measure its contribution to the overall business process.

A New Yardstick: From Today's Scorecard to Tomorrow's Strategic Capability

As Multi-Agent workflow technology matures into a fully integrated digital workforce, the frameworks we use to measure its impact will also evolve. However, we can start with a practical metric centric approach today while building toward a more strategic, forward-looking model for tomorrow.

The "Here and Now": The AI Workflow Scorecard:

An effective way to measure an AI Agentic workforce's value is to track its impact on the core business and operational metrics you already use. The power isn't in inventing new KPIs, but in proving the AI is the direct cause of improvements in the ones your business already runs on.

Proving this attribution requires a disciplined approach. The first two steps are standard practice:

1. Establish the baseline: You would be already tracking these KPIs, If not you have more important things to tackle than reading this article :).

2. Use Controlled Experiments: A/B testing or phased rollouts, is the classic way to isolate the impact. Find the path of least friction to seamlessly layer in Agentic workforce in A/B testing, champion challenger mode.

While these two above are familiar, the third step is what becomes critical and unique for an AI-powered workforce

3. Instrument your AI Agentic Workforce for Auditability and fine grain Attribution: This is the key. Your AI agents must be built to track their own work. Every action, every decision, and every completed task should be logged, creating an undeniable audit trail. This moves you beyond correlation to direct attribution. When your overall cycle time drops by 20%, you shouldn't have to guess why. You should be able to pull a report directly from the AI system that says, "I processed 45,000 tasks this month with an average processing time of 92 seconds, contributing X% of the total workload....". This instrumentation makes the AI's contribution transparent, quantifiable, and irrefutable.

The Strategic Future: The AI Capability Maturity

As your organization progresses, measurement must evolve from "What did the agentic workforce do for us today?" to "How is our enterprise-wide AI capability growing over time?" The goal is to build a compounding strategic asset. The AI Capability Maturity could evaluate your progress across three critical axes:

1. Autonomy Level: Measures the degree to which a process can run as assisting operation analyst to human-in-the-loop to full autonomy for certain flows. on a scale from Level 1 (AI-assisted passive) to Level 5 (Fully Autonomous).

2. Decision Complexity: Asks, "What is the 'decision ceiling' of our AI?" Are you moving from simple classifications to complex risk assessments to even generative narrative to SOP driving policy centric decision recommendation to doing e2e actioning.

3. Learning Rate: Measures the core "intelligence" and its ability to improve on its own. The key metric would be an ‘autonomous’ performance uplift per quarter, benchmarked on historical cases as well as champion-challenger mode for forward looking adjudications. A high learning rate proves you've built a system that gets smarter and delivers compounding returns.

Using this model, say a Chief Risk Officer for FI in lending space can report, "This quarter, we advanced our customer onboarding process to Autonomy Level 4, empowering it to handle Decisions up to a complexity of multi-factor risk assessment. Most impressively, its Learning Rate is delivering a 9% improvement in effectiveness from last quarter."

This would be the language of building a sustainable, strategic advantage for Agentic Workforce - the ultimate measure of success in the age of AI.