CSG Data Agent

An AI agent built for the Cancer Screening Group at Lunit to let internal teams and external stakeholders across the organization query data, metrics, and specifications through natural language. Instead of digging through dashboards or documentation, users ask questions and the agent delegates to the right specialist to find an answer.

I lead this project — scoping requirements, writing the technical plan, defining tasks and timelines, selecting frameworks and models, authoring evaluation guides, implementing the core system, and coordinating with team members on integrations and tooling.

The primary integration target is our internal DAP (Data Analytics Platform) — a tool that applies pre-defined analytical functions over internal databases. By connecting the agent to DAP, users can explore data without writing SQL or knowing the underlying schema. Future integrations include Confluence MCP for internal documentation and wiki search.

The entire system runs on local models only. No data leaves the internal network — no cloud API calls, no external providers. This was a hard constraint from the start given the sensitivity of clinical and product data involved.

Architecture

The agent follows a subagent delegation pattern built on LangChain DeepAgents, which compiles down to a LangGraph graph. An orchestrator sits at the top: it receives the user query, writes a structured TODO plan, then delegates each task to an isolated specialist subagent via a task() tool call.

Each subagent operates in isolation — it only sees its task string, not the full chat history. Subagents write their findings to a virtual shared filesystem (write_file / read_file / ls), which acts as the shared memory layer between the orchestrator and its specialists. The orchestrator synthesizes the final answer from these files once all delegated tasks complete.

Parallel delegation is a first-class concern: when multiple independent sub-tasks exist, the orchestrator emits them all in a single iteration, letting LangGraph execute them concurrently rather than serially.

Evaluation-Driven Development

Development is structured around evaluation from the start. Before building new capabilities, we define ground-truth datasets and scoring criteria first — then implement until evals pass. This keeps the system honest, surfaces failure modes early so we know where to focus, and a proper harness makes those eval runs fast enough to run regularly.

The eval spans multiple parallel tracks, each on its own timeline:

• Capability (v0.1–v0.4): Phased by query complexity. v0.1 covers single-turn fully-specified queries: tool selection accuracy, end-to-end task success, tool call order, and schema compliance (are tool calls and subagent responses well-formed JSON?). v0.2 adds phrasing robustness and consistency rate across rephrased variants. v0.3 adds disambiguation rate, clarification quality, and parameter correctness for multi-turn queries. v0.4 adds error-recovery rate and tool-call ordering on long multi-step chains. Each query runs 3× and scores are averaged for a stability signal.
• LLM Judge (from v0.1): Deterministic checks cover tool selection; the LLM judge handles the semantic answer check. The judge is validated before its scores are trusted: 2–3 annotators independently label 30–50 examples (blind, no judge exposure), inter-annotator agreement is measured first to validate the rubric itself, then judge agreement against the consensus is checked. The rubric covers helpfulness, relevance, accuracy, faithfulness, and level of detail.
• Answer Faithfulness / Hallucinations (v0.2): Once the capability baseline is green, added as a separate check: does the agent's response reflect what the tools actually returned? A tool returning 1,247 while the agent says "about 2,000" is a faithfulness failure, not a tool failure — keeping the two checks separate means the failure points to the right fix.
• Scope (v0.3): Out-of-scope refusal (does the agent decline requests outside its domain?) and over-refusal (does it still answer legitimate queries phrased in borderline ways?). Most out-of-scope filtering is the orchestrator's responsibility, so this track partly validates that boundary too.
• Orchestrator (separate track): A dedicated eval for orchestrator-level decisions — routing accuracy, correct subagent delegation, and recovery when a subagent fails or returns nothing useful. Kept separate from the DAP subagent eval because the two have different failure surfaces and different fix owners.
• Safety (v1.0): Prompt injection and adversarial resistance, built around an explicit threat model — who might attempt injection, where the attack surface is (direct prompts, retrieved docs), worst-case outcomes, and tolerance. Mitigations are chosen from the threat model: system prompt hardening, input sanitization, LLM-as-judge on inputs, or a dedicated classifier. Sequenced after the capability baseline so failures are attributable to one cause, not two.
• Operational (v1.0 + online): Latency (P50/P95), token cost per query, and loop/retry rate. Post-deploy, online evals sample 1–5% of production traffic to catch silent failures, gradual drift, and query patterns the golden dataset never covered — things offline evals structurally miss.

Failures are tracked against a taxonomy — wrong tool, right tool wrong parameters, hallucinated tool, failed to disambiguate, faithfulness drift, scope leak, prompt injection success — so each failure class maps to a specific fix surface rather than "the model got it wrong."

Observability

The agent streams structured events back to the caller: trace events for each agent/tool step, digest events for streaming tokens in the final answer, sources for extracted references, and done on completion. A gating mechanism prevents planning tokens from leaking into the user-facing stream — only the synthesized answer is emitted.

Tracing is currently handled via LangSmith. The plan is to replace it with Langfuse — a self-hosted alternative — to keep observability data on-prem alongside the rest of the stack.

graph TB
    User["👤 User Query\n(CLI / API)"]

    subgraph Core ["🤖 CSG Data Agent"]
        Orch["Orchestrator\nwrite_todos · read_todos · task()"]
        VFS["📁 Virtual Filesystem\nread_file · write_file · ls"]
        Stream["📡 Streaming\ntrace · digest · sources · done"]
    end

    subgraph Specialists ["⚙️ Specialist Subagents (isolated context)"]
        DataAgent["Data Agent\n(DAP Tools)"]
        KnowAgent["Knowledge Agent\n(Confluence MCP)"]
    end

    subgraph Backends ["🔧 Backends"]
        DAP["DAP\nData Analytics Platform"]
        CMCP["Confluence MCP\nInternal Docs"]
        LLM["Local LLMs\nOllama · air-gapped"]
    end

    User --> Orch
    Orch -->|"parallel task() delegation"| DataAgent
    Orch -->|"parallel task() delegation"| KnowAgent
    Orch <-->|"shared memory"| VFS
    DataAgent <-->|"write findings"| VFS
    KnowAgent <-->|"write findings"| VFS
    DataAgent --> DAP
    KnowAgent --> CMCP
    Core --> Stream
    Stream --> User
    Orch --- LLM
    DataAgent --- LLM
    KnowAgent --- LLM

    style Orch fill:#2d3748,stroke:#f59e0b,color:#f59e0b
    style VFS fill:#2d3748,stroke:#9ca3af,color:#e2e8f0
    style Stream fill:#2d3748,stroke:#9ca3af,color:#e2e8f0
    style DataAgent fill:#2d3748,stroke:#9f7aea,color:#e2e8f0
    style KnowAgent fill:#2d3748,stroke:#9f7aea,color:#e2e8f0
    style DAP fill:#2d3748,stroke:#9ca3af,color:#e2e8f0
    style CMCP fill:#2d3748,stroke:#9ca3af,color:#e2e8f0
    style LLM fill:#1a202c,stroke:#4b5563,color:#6b7280
    style Core fill:#1a202c,stroke:#f59e0b,color:#d1d5db,stroke-width:2px
    style Specialists fill:#1a202c,stroke:#9f7aea,color:#d1d5db,stroke-width:2px
    style Backends fill:#1a202c,stroke:#4b5563,color:#6b7280,stroke-width:1px