promptfoo-cicd-framework

This project is a structured LLM evaluation framework built on Promptfoo for SecureBank's AI assistants. It provides a repeatable, automated way to test LLM quality, safety, and compliance across every stage of development — from local debugging to CI/CD gates on merge requests and UAT deployments. The framework addresses the problem that LLM outputs are non-deterministic, and standard unit tests cannot catch issues like missing key facts, hallucinated figures, or compliance violations. Promptfoo solves this by running structured test cases with deterministic and AI-judged assertions against one or more LLM providers simultaneously. The pipeline uses GitLab CI/CD to run regression and functional checks on every PR to uat, and a full test suite on every merge to uat. The framework includes capabilities such as LLM evaluation, multi-model comparison, agent testing, RAG validation, NLP quality metrics, hallucination detection, compliance testing, red-team safety, CI/CD integration, A/B prompt testing, multi-format test cases, JSON response validation, and SQL response validation.