Detecting and Containing Hallucination Risk in GenAI
The bank needed a way to measure and mitigate the risk of hallucinations as it explored generative AI adoption. Hallucinations - outputs that are fluent but factually incorrect - undermine compliance, data integrity, and customer trust in regulated environments. Without a repeatable method to detect, test, and benchmark model reliability, safe deployment at scale would not be possible.
Outcomes
40%
Context Relevancy Improved
20%
Decrease in Hallucinations
25%
Reduction in Manual Review Effort
100%
Metric-Human Agreement
Technology Vendors Suited to Evaluation
Business Problem
Hallucinations pose a significant barrier to the safe adoption of GenAI in highly regulated industries. Incorrect or misleading outputs create reputational, regulatory, and customer duty risks that cannot be tolerated in critical workflows.
Most organisations lack the tooling and processes to reliably detect, measure, and mitigate hallucinations. Without these safeguards, leaders cannot trust GenAI applications to deliver accurate or compliant results.
A secure sandbox environment was required to test emerging hallucination safety frameworks – providing controlled conditions to evaluate model performance, monitor risks, and build the foundations for a reusable safety layer in future deployments.
Challenges
- Hallucination Risk – LLMs frequently generate incorrect, misleading, or irrelevant outputs, creating reputational, regulatory, and customer duty risks.
- Lack of Reliable Metrics – Existing measures (faithfulness, summarisation, context relevance) were immature and did not consistently align with human evaluation.
- High Reliance on Manual Oversight – Detecting hallucinations required significant human review, slowing down experimentation and inflating costs.
- Inconclusive Early Testing – Results from earlier experiments were inconsistent, highlighting the need for calibration at the use-case level before hallucinations can be accurately detected.
- Operational Blind Spots – No standardised process existed to validate hallucination safety across multiple GenAI models or use cases.
- Adoption Risk – Without a robust safety layer, scaling GenAI use cases into production risked embedding errors into critical workflows.
From Idea to Evidence with NayaOne
To move from concept to proof, the sandbox was used to validate a Hallucination Metric and Diagnosis framework designed to detect and measure incorrect outputs from large language models.
- Controlled Environments: Multiple secure sandboxes were created to run experiments on different models without exposing sensitive data or impacting production systems.
- Metric Validation: The framework tested context relevance, faithfulness, answer relevancy, and summarisation metrics against human evaluation to see if hallucinations could be reliably detected.
- Calibration & Stress Testing: Iterative testing explored whether model-specific calibration improved detection accuracy and reduced reliance on human oversight.
- Evidence Gathering: Query-context-response triplets were generated and scored, with performance compared across metrics, human assessments, and baseline RAG pipelines.
Through this approach, the organisation gained evidence of where hallucination safety metrics were effective, where gaps remained, and how these could evolve into a reusable safety layer to support future GenAI adoption at scale.
Impact Metrics
PoC Timeline Reduction
6 weeks with NayaOne vs 18 – 24 months traditionally
Time Saved in Vendor Evaluation
1+ year
Decision Quality
Decision quality improved by providing measurable evidence on model accuracy, calibration needs, and oversight requirements, enabling safer and faster adoption decisions.
KPIs
- Context Relevancy Accuracy – target of ~70% of model responses where the retrieved context was judged relevant by human evaluators.
- Response Quality (3/3 Scores) – proportion of outputs rated fully correct across context, query, and answer alignment.
- Metric–Human Agreement – ability of safety metrics to align with human judgement:
- Correct Directionality: median score for “correct” responses higher than for “incorrect.”
- Adequate Separability: clear statistical gap between correct and incorrect responses.
- Reduction in Human Oversight – evidence that automated safety metrics could reliably reduce the amount of manual evaluation needed for GenAI outputs.
Validate Hallucination Detection Frameworks Before Enterprise Rollout
Validate how GenAI models handle hallucinations in a controlled sandbox, with evidence you can measure, explain, and trust before moving to production.
