Purpose
The Weighted Vendor Evaluation Scorecard (VEN-01) is the canonical pre-procurement assessment instrument for selecting legal AI vendors. It provides a structured, defensible scoring methodology across five weighted dimensions, with mandatory Pass/Fail gates that operate independently of weighted scores. Completion of this Scorecard is a required input to USE-02 pilot authorisation; no pilot proceeds with a vendor that has not cleared Pass/Fail thresholds.
Operating cadence: Per-engagement — completed for each AI vendor under active procurement consideration.
Owner: Legal Operations, STR-07 AI Task Force, Technology & Procurement.
---
When to Use This Module
- Before any AI vendor selection decision, regardless of contract value
- When evaluating vendors for a specific use case identified in USE-01
- When refreshing an existing vendor relationship (annual re-evaluation)
- When a Shadow AI audit (Risk Taxonomy 2026, Class 6) surfaces an unapproved tool requiring retrospective assessment
---
AI Bill of Materials — Pre-Evaluation Requirement (Metric 0)
Before beginning weighted scoring, confirm the following AI BoM Pre-Check is complete:
| AI BoM Pre-Check | Status |
|—|—|
| Vendor not already registered in AI BoM as declined or decommissioned | Confirm |
| AI BoM slot approved for this use-case category by STR-07 AI Task Force | Confirm |
| Intended use case aligns with a ranked opportunity in USE-01 | Confirm |
If the vendor clears all Pass/Fail criteria and weighted scoring, AI BoM registration is the mandatory post-selection step before any pilot commences. The AI BoM entry must record: vendor name, product version, approved use-case scope, data classification handled, contract DPA reference, and Agentic Tier designation.
---
Section 1: Evaluation Framework
Dimension Weights and Risk Taxonomy 2026 Mapping
| Dimension | Default Weight | Risk Taxonomy 2026 Class(es) |
|—|—|—|
| 1. Technical Performance | 30% | Class 1: Hallucination and accuracy; Class 3: Bias and fairness; Class 9: Operational resilience |
| 2. Governance and Compliance | 25% | Class 2: Privilege and confidentiality; Class 4: Privacy and data protection; Class 7: Regulatory compliance drift; Class 8: IP and licensing |
| 3. Business Viability | 20% | Class 5: Supply chain and vendor dependency |
| 4. Integration and Usability | 15% | Class 9: Operational resilience; Class 6: Shadow AI (low adoption drives Shadow AI) |
| 5. Cost and Commercial | 10% | Class 5: Supply chain and vendor dependency |
Agentic Tier Supplement
If the vendor’s product includes autonomous AI agents, add the following criteria to Dimensions 1 and 2 before scoring:
| Agentic Tier Criterion | Dimension | Status |
|—|—|—|
| Kill-switch and human override capability | Dimension 2 (Governance) | Mandatory |
| Intervention frequency logging | Dimension 1 (Technical) | Mandatory |
| Autonomous action scope documentation | Dimension 2 (Governance) | Mandatory |
| Audit trail for agentic decisions | Dimension 2 (Governance) | Mandatory |
---
Dimension 1: Technical Performance (30%)
| Subcriteria | Weight | Risk Taxonomy Class |
|—|—|—|
| Accuracy and Reliability | 35% | Class 1: Hallucination and accuracy |
| RAG and Knowledge Integration | 25% | Class 1 |
| Performance and Scalability | 25% | Class 9: Operational resilience |
| Model Sophistication | 15% | Class 3: Bias and fairness |
Accuracy targets: Citation accuracy >95%; Hallucination rate <1%; Legal reasoning consistency verified through structured testing.
---
Dimension 2: Governance and Compliance (25%)
Risk Taxonomy 2026 cross-walk for all Governance and Compliance sub-criteria:
| Sub-criterion | Risk Taxonomy 2026 Class | Evaluation Requirement |
|—|—|—|
| Security Certifications (SOC 2 Type II, ISO 27001) | Class 9: Operational resilience | Current certification required; expired = Pass/Fail failure |
| Data Protection — no training on client data | Class 2: Privilege and confidentiality; Class 4: Privacy and data protection | Contractual and technical safeguards; DPA required per DAT-03 |
| Regulatory Alignment (EU AI Act, GDPR, ABA Rules 1.6/1.1/5.3) | Class 7: Regulatory compliance drift | Compliance mapping across all applicable frameworks |
| Auditability and Transparency | Class 1: Hallucination and accuracy | Complete logging of system activities and decisions |
| Bias Detection and Reporting | Class 3: Bias and fairness | Systematic bias monitoring and reporting required |
| IP and Licensing | Class 8: IP and licensing | AI-generated output ownership clearly defined in contract |
GOV-03 Risk Register feed: Governance and Compliance scores below 3.0 in any subcriteria must be logged as GOV-03 Risk Register entries under the corresponding Risk Taxonomy 2026 class before proceeding.