Validating GenAI Tools for Contract Summarisation and Risk Analysis
A global bank wanted to automate contract summarisation and covenant extraction using GenAI, reducing manual effort, improving accuracy, and ensuring consistent risk reporting across legal operations.
Outcomes
60%
Faster Contract Reviews
45%
Efficiency Gain
30%
Standardisation Increase
40%
Cost Savings
Technology Vendors Suited to Evaluation
Business Problem
Contract reviews were slow, error-prone, and inconsistent across teams. Manual processes increased operational risk, while the lack of standardised templates made compliance difficult to track. Limited reporting also hindered visibility into recurring risks and emerging trends.
The bank needed a way to test multiple GenAI tools quickly, safely, and under consistent evaluation criteria.
Challenges
- Time-Consuming Manual Processes: Slow contract reviews delay decisions and strain resources.
- High Error and Risk Exposure: Human errors can lead to missed terms, compliance failures, or financial risks.
- Lack of Standardisation: Inconsistent language and review processes increase risks and inefficiencies.
- Lack of Insights and Reporting: Limited visibility into risks and trends hampers decision-making and compliance.
From Idea to Evidence with NayaOne
Using NayaOne’s platform, the institution conducted a controlled proof of concept (PoC) to evaluate three GenAI vendors side by side.
- Synthetic Contracts: Provided anonymised contracts containing predefined clauses and errors to each vendor for analysis and summarisation.
- Evaluation Metrics: Measured accuracy, compliance alignment, and processing time for each vendor.
- Secure Testing: Vendors were granted sandbox access to synthetic data, bypassing lengthy third-party risk management (TPRM) cycles.
- Fast Execution: The PoC – from first call to completion -was delivered in just three weeks.
- Decision Confidence: Findings guided vendor selection for deeper evaluation in more complex legal and compliance scenarios.
Impact Metrics
PoC Timeline Reduction
3 weeks with NayaOne vs 12 – 18 months traditionally
Time Saved in Vendor Evaluation
1+ year
Decision Quality
The bank gained hard evidence on detection accuracy, speed, and integration fit.
KPIs
- Summarisation Accuracy (%): Correct identification of key clauses and risk terms.
- Processing Time (minutes): Average time per contract from upload to summary.
- Error Rate (%): Frequency of missed or misclassified clauses.
- Compliance Alignment (%): Match rate against internal legal policies.
- TPRM Bypass Time (days saved): Time saved by testing vendors securely without onboarding delays.
Validate GenAI Tools for Legal Operations
Test and benchmark GenAI contract summarisation tools in a secure sandbox to accelerate reviews, reduce risk, and ensure compliance.
