· L-M

· Bands Operational → Integrated → Optimised

· MEA-07

ROAI Telemetry

ROAI Telemetry is the live operational surface where Return-on-AI signals from every active use case aggregate into a single executive view. Where the ROAI Matrix Framework (VAL-01) defines the methodology, this Module defines the telemetry pipeline, the visualisation standards, the refresh cadence, and the alert thresholds. It tracks per-use-case ROAI scores, value capture rates, and variance against forecast — making AI value continuously visible to General Counsel, the AI Task Force, and Finance. Without ROAI Telemetry, value measurement collapses back into quarterly retrospectives that the operating cadence has already moved past. Methodology v2026.1.

strategic

Continuous

Initial build 12 weeks; then continuous operation with weekly, monthly, and quarterly review cycles

Methodology v2026.1·Verified 23 May 2026·Reviewed 23 May 2026

Executive Summary

The Legal AI ROAI Dashboard is the operational measurement instrument for tracking AI investment value across four dimensions: Operational Gains (efficiency and throughput), Financial and Client Gains (cost reduction and satisfaction), Capability and Culture Gains (AI literacy and adoption), and Strategic and Reputational Gains (risk management and compliance). Built on the ROAI Matrix (STR-08), the dashboard provides real-time performance monitoring, DPS evidence generation, and data-driven scaling decisions. Dimension 4 maps all risk monitoring to Risk Taxonomy 2026 classes, feeding GOV-03 automatically. AI Bill of Materials compliance tracking ensures only registered tools receive ROAI attribution. Agentic Tier deployments have a dedicated monitoring track with kill-switch logging and GOV-05 integration. Targets: 80%+ AI adoption rate; risk detection accuracy ≥90%; error reduction ≥70%; audit preparation time reduction ≥50%.

Defensibility Evidence Produced

Monthly ROAI performance reports constitute DPS evidence for the Sophistication lens (performance metrics achieved against STR-08 targets) and Adoption lens (deployed use cases with active measurement). Annual Q4 dashboard export is the DPS evidence package submitted to GOV-01 governance review.

Elements:

Evidence frameworkContinuous learning

1. Purpose and Scope

The Legal AI ROAI Dashboard is the central measurement instrument for tracking return on AI investment across legal. It operationalises STR-08 (ROAI Matrix) by converting baselines, pilot results, and live deployment data into a unified, defensible view of AI performance.

The scope covers:

All AI use cases approved via USE-01
Both augmentation and agentic deployments
Four value dimensions: Operational, Financial & Client, Capability & Culture, Strategic & Reputational
All nine canonical Risk Taxonomy 2026 classes

Cadence: continuous data ingestion, daily operational tracking, weekly reporting, monthly ROAI analysis, and quarterly strategic reviews.

2. ROAI Dimensions and KPIs

Dimension 1 — Operational Gains

Focus: Time saved, efficiency, throughput.

Core KPIs:

Time Saved Per Task (20–50% reduction; daily tracking)
Document Processing Speed (60–80% faster; real-time)
Matter Turnaround Time (25–40% reduction; weekly)
Automation Rate (70–90% of eligible tasks; weekly)
Capacity Utilisation (15–30% increase; monthly)

Dimension 2 — Financial and Client Gains

Focus: Cost reduction, revenue, client satisfaction.

Core KPIs:

Cost Per Matter (20–40% reduction; monthly)
Outside Counsel Spend Reduction (15–30%; quarterly)
Client Satisfaction / NPS (85%+ satisfaction; 10+ NPS uplift; quarterly)
Revenue Per Hour (10–20% improvement; monthly, law firms)
Client Response Time (50%+ improvement; weekly)

Dimension 3 — Capability and Culture Gains

Focus: AI literacy, adoption, wellbeing, innovation.

Core KPIs:

AI Adoption Rate (80%+ active; weekly). Shadow AI signal: <60% in any team triggers Class 6 investigation and potential GOV-05 review.
User Satisfaction (≥8/10; quarterly)
Training Effectiveness (90%+ completion; 80%+ competency; quarterly)
Work-Life Balance (10%+ overtime reduction; monthly)
Innovation Index (2–3 new use cases per quarter; ≥50% implemented)

Dimension 4 — Strategic and Reputational Gains

Focus: Compliance, risk reduction, market differentiation.

Mapped KPIs:

Risk Detection Rate — Class 1
Compliance Monitoring — Classes 7 & 4
Error Reduction — Classes 1 & 2
Bias Audit Rate — Class 3
IP Content Review Rate — Class 8
AI BoM Vendor Compliance — Class 5
Audit Readiness — Classes 7 & 9

Targets include ≥90% risk detection accuracy, <5% false positives, ≥95% compliance issue prevention, ≥70% error reduction, and ≥50% audit preparation time reduction.

3. Agentic Tier Monitoring

Autonomous AI agents require a dedicated track:

Real-time Intervention Rate: human overrides; target <5% for stable deployments.
Kill-switch Activation Log: all circuit-breaker events, root-caused by Risk Taxonomy 2026 class, feeding GOV-05.
Autonomous Decision Accuracy: AI-only decisions validated against attorney samples (Class 1).
Scope Creep Detection: tasks outside permission scope; triggers Class 6 Shadow AI flag.
Agentic ROAI Attribution: separate ROAI for agentic vs augmentation tiers.

Agentic anomalies automatically generate GOV-05 incident workflows and GOV-03 risk entries where applicable.

4. Ecosystem and Governance Hooks

The dashboard is embedded in the broader operating system:

STR-08 (ROAI Matrix): Monthly ROAI updates Quadrant 2 (financial) and Quadrant 4 (strategic) performance.
USE-01: Quarterly re-scoring uses dashboard data to expand, optimise, or retire use cases.
USE-02: Phase 3 pilot evaluation pulls ROAI metrics for scale / no-scale decisions.
USE-05: Provides pre-AI baselines for all KPIs.
DPS: Quarterly outputs feed Adoption and Sophistication lenses; annual export is the DPS evidence package.
AI BoM: Only registered tools receive ROAI credit; unregistered tools trigger Class 6 Shadow AI flags.
GOV-01: Governance metrics (adoption, policy adherence, audit trail quality) support annual review.
GOV-03: Risk detection metrics and incidents populate the Risk Register by class.
GOV-05: Anomalies, kill-switch events, and severe metric deviations trigger incident response.
STR-07: AI Task Force reviews monthly performance and signs off quarterly ROAI updates.

5. AI BoM Integration Requirements

Each monitored tool must have:

Confirmed AI BoM entry before any ROAI attribution
Vendor performance metrics tied to BoM records
Contract and renewal alerts surfaced in Dimension 4
New tool onboarding gated by BoM registration

Unregistered tools detected in usage analytics:

Trigger a Class 6 Shadow AI flag
Generate a GOV-03 Risk Register entry
May initiate GOV-05 investigation depending on impact

6. Executive Summary View Design

The executive view provides leadership-ready insight:

ROAI Scorecard: overall monthly ROAI %, YTD trend, comparison to STR-05 projections, RAG status.
Key Metrics Scorecard: top 5 KPIs vs targets with trend arrows and alerts.
Financial Impact Summary: total savings, payback progress, budget variance.
DPS Evidence Package: auto-formatted quarterly outputs for Adoption and Sophistication lenses.
Critical Alerts: system issues, metric deviations, GOV-05 triggers, AI BoM compliance flags.

7. Operational Performance Dashboard

This view focuses on day-to-day efficiency:

Time Savings Visualisation: daily hours saved, task-level drill-down, cumulative savings since AI BoM registration.
Automation Effectiveness: automation rate gauges, volume charts, quality indicators.
Capacity Utilisation: before/after comparisons by practice area and matter type.

8. Financial Impact Dashboard

This view supports finance and client value conversations:

Cost Reduction Analysis: cost per matter trends, outside counsel spend, budget variance.
ROAI Calculation Methodology: transparent inputs and assumptions aligned to STR-08 Quadrant 2.
Investment Recovery Tracking: payback progress vs STR-05, with multi-scenario modelling.

9. Risk and Compliance Dashboard

This view aligns with Risk Taxonomy 2026 and governance:

Risk Detection Performance by Class: accuracy, false positive/negative trends, escalation patterns.
AI BoM Compliance Track: registration status, unregistered tool flags (Class 6), renewal alerts (Class 5).
Agentic Tier Monitor: intervention rate, kill-switch log, scope adherence.
GOV-03 Risk Register Feed: incidents by class, remediation status, repeat-incident alerts.

10. Implementation Phase 1 — Baseline Establishment (Weeks 1–4)

Activities:

Collect 12 months of historical data for all planned KPIs.
Establish statistical baselines and confidence intervals.
Document data collection methodologies and source systems.
Map data flows from DMS, matter management, billing, and AI tool APIs.
Confirm AI BoM entries for all tools; block ROAI attribution until confirmed.

Deliverables:

Baseline dataset and KPI catalogue
Data flow diagrams and source-of-truth registry
Initial ROAI target mapping to STR-08

11. Implementation Phase 2 — Dashboard Development (Weeks 5–8)

Activities:

Configure APIs and connectors to all source systems.
Build the executive ROAI Scorecard aligned to STR-08 targets.
Implement AI BoM compliance track and Class 6 Shadow AI alerts.
Build Agentic Tier monitor (if agentic deployments exist) with GOV-05 triggers.
Configure GOV-03 integration for Dimension 4 risk feeds.

Deliverables:

Working dashboard environment
Configured ROAI calculation engine
Governance and alerting ruleset

12. Implementation Phase 3 — Testing and Validation (Weeks 9–12)

Activities:

Validate ROAI calculations against STR-08 Quadrant 2 standards.
Test GOV-05 alert triggers and AI BoM flag logic.
Conduct UAT with Legal Operations and STR-07 AI Task Force.
Train Legal Operations on DPS evidence export and quarterly review workflows.

Deliverables:

Signed-off ROAI calculation model
UAT report and remediation log
Training materials and runbooks

13. Sample ROAI Calculation

|—|—|—|—|

| Attorney Time (Hours × Blended Rate) | $150,000 | $97,500 | $52,500 |

| Outside Counsel Spend | $45,000 | $31,500 | $13,500 |

| Administrative Overhead | $25,000 | $18,750 | $6,250 |

| Total Monthly Savings | | | $72,250 |

| AI Investment (Monthly) | | $18,000 | — |

| Net Monthly Benefit | | | $54,250 |

| Monthly ROAI | | | 301% |

Formula: ROAI = (Total Savings − AI Investment) ÷ AI Investment × 100, applied only to AI BoM-registered tools.

14. Predictive Analytics and Optimisation

Advanced analytics capabilities:

Adoption Trend Modelling: forecast Shadow AI risk and training needs.
Capacity Planning: anticipate future workload vs capacity using USE-01 pipeline.
ROAI Projection: scenario modelling for STR-05 business case refresh.
Performance Degradation Alerts: early warnings for declining accuracy or adoption, triggering GOV-03 and GOV-05 where needed.

15. Continuous Improvement Loop

Quarterly: USE-01 re-scoring based on dashboard performance.
Monthly: Vendor performance from AI BoM reviewed; underperforming vendors flagged.
Annual: DPS evidence package exported across all four dimensions and submitted into GOV-01 review.

16. Roles, Ownership, and Cadence

Owner: Legal Operations, with AI Task Force (STR-07) and GC oversight.
Cadence:
- Continuous ingestion and monitoring
- Weekly operational and adoption reports
- Monthly ROAI and vendor performance reviews
- Quarterly strategic and DPS reviews

17. Governance and Measurement Metadata

Methodology Version: v2026.1
Risk Class: cross-class (all nine canonical classes)
DPS Lenses: Adoption and Sophistication
Certification Track: Practitioner
Advisory Relevance: Programme Design (months 2–6) and Strategic Retainer (ongoing)
Telemetry: dashboard activations, monthly ROAI scores, DPS exports, GOV-05 trigger counts.

18. Success Metrics and Leading Indicators

Success indicators include:

ROAI meeting or exceeding STR-08 targets across dimensions
High AI adoption with low Shadow AI incidents
Reduced outside counsel spend and improved client satisfaction
Strong audit readiness and regulator-grade documentation

Leading indicators:

Rising automation rates with stable or improved accuracy

Operational Signals

roai-dash.telemetry-uptime

→ Defensibility Posture Statement

Telemetry availability against operating SLA — DE-3 Evidence framework record.

continuous

roai-dash.variance-alerts-actioned

→ Annual Legal AI OS Index

ROAI variance alerts actioned within SLA feeds the Annual Legal AI OS Index Sophistication signal.

monthly

roai-dash.executive-engagement

→ Console

Executive interactions with telemetry surface for Console intelligence substrate.

On change

Recommended Stakeholders

Owner

Head of Legal Operations

Approvers

General Counsel
Finance Partner

Contributors

AI Task Force
Engineering / IT

Informed

Board
Audit Committee
CIO / CISO

Inputs · Outputs

Inputs

· Baseline metrics from USE-05 (Baseline Metrics Capture Guide)
· AI pilot results from USE-02 Phase 3 evaluation
· Approved use case portfolio from USE-01 quarterly re-scoring
· ROAI targets from STR-08 (ROAI Matrix)

Outputs

· ROAI performance reports by dimension (operational, financial, client, strategic)
· DPS Sophistication and Adoption lens evidence packages for GOV-01 review
· STR-08 Quadrant 2 and Quadrant 4 updates
· GOV-03 Risk Register feeds by Risk Taxonomy 2026 class
· GOV-05 alert triggers for anomalous AI performance and Agentic Tier deviations
· Scaling recommendations for USE-01 quarterly use case re-scoring

Framework Crosswalk

NIST AI Risk Management Framework

NIST

ROAI risk and performance metrics map to NIST AI RMF functions for govern, map, measure, and manage, providing quantitative evidence of AI risk controls in legal.

ISO/IEC 42001 AI Management System

ISO

Dashboard telemetry and governance hooks support ISO 42001 requirements for monitoring, measurement, analysis, and evaluation of AI systems in legal operations.

EU AI Act Governance and Monitoring Obligations

European Union

Risk and compliance dashboards, audit readiness metrics, and incident feeds help demonstrate ongoing monitoring and documentation for high-risk AI use cases.

Operational Artefacts

Legal AI ROAI Dashboard Template
asset
Gated

Sequence

Run before

VAL-01
ROAI Matrix Framework
Measure and demonstrate the return on AI investment across four value dimensions — operational, financial, cultural, and strategic — with direct linkage to the DPS Defensibility lens.

Run next

MAT-06
Board AI Reporting Architecture
Provides the structured report framework for presenting AI governance performance, risk posture, and maturity progression to the Board or Governing Partners.

Diagnostic Relevance

Running ROAI Telemetry strengthens the Sophistication lens — expected Band progression: Integrated → Optimised.

Confidence: high

Key Takeaways

Measure AI value across four dimensions: operational, financial, capability, and strategic.
Link ROAI metrics directly to STR-08 targets and STR-05 business case assumptions.
Integrate AI BoM so only registered tools receive ROAI credit and vendor tracking.
Monitor all nine Risk Taxonomy 2026 classes, including Shadow AI and agentic risk.
Provide DPS-ready evidence for Adoption and Sophistication lenses on a fixed cadence.
Enable scaling, optimisation, or retirement decisions for AI use cases via USE-01.
Feed GOV-01, GOV-03, and GOV-05 with structured telemetry and incident signals.

Run this Module

Operational artefacts available to Practitioner Membership members. Methodology v2026.1.

View Membership

Targeting

Audience

GC / CLOLegal OperationsRisk & Compliance

Strengthens

Adoption lensSophistication lens

Module Details

Format: Module
Difficulty: Operational
Pillar: P5
Layer: M · Measurement
Owner: Head of Legal Operations
Access: Practitioner Membership
Certification: Practitioner

Maturity Bands

OperationalIntegratedOptimised

Risk Classes Mitigated

Operated within

In the Ecosystem

P5 · Use Cases →Module Library →Take the Free Baseline Diagnostic →

Related Modules

View Module Library →

VAL-01

ROAI Matrix Framework

Measure and demonstrate the return on AI investment across four value dimensions — operational, financial, cultural, and strategic — with direct linkage to the DPS Defensibility lens.

USE-03

Baseline Metrics Methodology

Establishes pre-implementation performance benchmarks across four measurement dimensions to enable objective ROAI demonstration and AI optimisation for legal departments.

USE-01

Use Case Prioritization Methodology

The strategic use case selection instrument that evaluates AI opportunities across five dimensions including Risk Taxonomy 2026, gates vendor adoption via AI BoM, and connects the deployed portfolio to DPS Adoption lens evidence.

Referenced in Intelligence

Where this Module is anchored

Canonical Vocabulary

Terms this Module anchors

ROAI 4-Quadrant

The canonical value lens for measuring and communicating Return on AI Investment in legal functions. The 4-Quadrant maps outcomes across four dimensions: efficiency, quality, risk reduction, and strategic value. 'ROI' alone is forbidden in AI contexts; 'ROI 4-Quadrant' and 'Value 4-Quadrant' are forbidden synonyms. ROAI (Return on AI Investment) is the canonical term.

Maturity Stack

The canonical five-Band × four-Lens positioning instrument for the Legal AI OS. The Maturity Stack places an organisation at one of five Maturity Bands — Foundational through Defensible — based on composite scores across four Lenses: Adoption, Sophistication, Defensibility, and Autonomy. The Stack is the canonical positioning instrument; the Maturity Grid is the Year-1 field tool that maps into it.

ROAI

The four-quadrant return framework for AI investment in legal functions, comprising productivity value, Defensibility value, institutional value, and category positioning value. Productivity-only cases underweight the investment by a factor of three or four; the full ROAI frame presents the board with the complete case for funding AI at institutional scale.

Where this Module lives

ROAI Telemetry is the operational dashboard layer of the measurement system. It consumes scoring from VAL-01 (ROAI Matrix), baselines from USE-03 (Baseline Metrics), and pilot outcomes from USE-02. It feeds the Board AI Reporting Architecture (MAT-06) and produces DE-3 (Evidence framework) records into the DPS. Without it, ROAI lives in periodic reports rather than in the operating cadence.

Advisory

When this Module sits inside a Programme.

Modules are operated in-house by GC and Legal Operations teams. When the capability transformation is multi-Pillar — or when the regulator timeline tightens — Advanta operates the canonical Module sequence as a Programme.

View Engagement Models Take the Free Baseline Diagnostic

← Back to Module Library