dataops-control-plane

The DataOps Control Plane is a standalone DataOps control plane that is independent of CI/CD tools. It connects to GitHub Actions, GitLab CI, Jenkins, and Kubernetes through provider adapters, monitors the data pipeline lifecycle, collects logs, Data Quality reports, and version information, and then supports detection, root cause analysis, and safe incident recovery. GitHub Actions is selected as the first provider for the MVP, but the core architecture is not dependent on GitHub, so GitLab CI or Jenkins can be added without rewriting the Agent, Policy Engine, and incident handling workflow. The product consists of two independent parts: dataops-agent, which runs in CI/CD to execute portable pipelines and send events/logs/reports, and dataops-platform, a self-hosted suite including FastAPI API, Web UI, PostgreSQL, and Elasticsearch. FastAPI serves both the API and HTML/CSS/JavaScript; the project does not add an intermediate Express backend in the MVP. The system aims to standardize pipeline monitoring across multiple CI/CD platforms, automatically collect evidence when data pipelines fail, use Agentic Hybrid RAG to support root cause analysis (RCA), propose or perform controlled actions such as RETRY, QUARANTINE, ROLLBACK, and CREATE_PR, re-verify pipeline and data quality after recovery, and reduce MTTD, MTTR, and the time engineers spend manually reading logs. It is not intended to replace GitHub Actions, GitLab CI, or Jenkins, nor to let LLMs freely operate production, automatically merge code, or directly modify production data in the MVP. The LLM is not run on every commit; the Agent is only triggered when there is an error or anomaly that needs analysis.