rag-reliability-case-study

Overview

The RAG Reliability Case Study is a production-style Retrieval-Augmented Generation (RAG) HR assistant evaluated against a 20-question golden dataset with an independent judge model. It documents real failure modes, bottleneck distributions, and the engineering decisions behind a faithfulness score of 0.911. The repository is organized into several resources. CASE_STUDY.md provides a full narrative: system overview, evaluation methodology, results, and what this demonstrates. methodology.md covers evaluation framework design — why an independent judge, how the benchmark was constructed. The evaluation/ directory contains a headline metrics summary and stripped results table. The failure-analysis/ directory contains per-failure root-cause write-ups for Q16, Q17, and Q20. The architecture/ directory contains a system architecture diagram — hybrid retrieval + evaluation pipeline. The screenshots/ directory contains UI screenshots of the chat interface and demo assets. A demo video is also referenced as RAG.case.study.mp4.