DocPilot: Interactive Agent Orchestration and Training for Long Visual Document Understanding

DocPilot is a trainable interactive agent for multimodal long-document question answering. It is built around two core capabilities:

• 🔍 Evidence Seeking: Locate relevant evidence across documents, pages, and regions, and continue searching when evidence is insufficient.
• 🧩 Multimodal Understanding: Interpret text, tables, charts, figures, and layouts through fine-grained visual inspection and code-based reasoning.

The framework combines two components:

1. Agent orchestration. A state-conditioned policy composes six document actions for adaptive evidence seeking, multimodal understanding, and iterative refinement.
2. Curriculum-based agentic training. Compact models first learn individual document-solving abilities and then learn to coordinate those abilities from complete interaction trajectories.

📰News

🏗️ Framework

Multimodal Document Environment

DocPilot represents each document through original page images, MinerU-derived structured elements, page thumbnails, and document overviews. Shared document and page identifiers connect these views for coarse-to-fine evidence access.

Agent Orchestration

Six actions form a state-conditioned loop across three mechanisms:

Mechanism	Action	Role
Evidence Seeking	Navigate	Locate candidate pages from overviews and thumbnails.
Evidence Seeking	Extract	Retrieve source-grounded text, table, and visual evidence.
Multimodal Understanding	Focus	Crop, zoom, or isolate local visual regions.
Multimodal Understanding	Solve	Execute code for aggregation, comparison, and numerical reasoning.
Iterative Refinement	Answer	Generate a candidate answer from the evidence state.
Iterative Refinement	Analyze	Verify grounding, identify gaps, and route the next action.

Each observation updates the evidence state. Unsupported answers return to evidence seeking or multimodal reasoning.

Curriculum-Based Agentic Training

DocAgent-Instruct-40K supports a two-stage curriculum:

1. Single-ability fine-tuning learns individual document-solving capabilities from reasoning traces.
2. Multi-ability agentic training learns their adaptive composition from complete interaction trajectories.

🗂️ Repository Structure

docpilot/
├── app/
│   ├── agent/                 ReAct and trained-model agent implementations
│   ├── tool/                  Navigate, Extract, Focus, Solve, Answer, Analyze
│   ├── workflow/              In-memory and MinerU-backed environments
│   ├── prompt/                Action and role prompts
│   ├── cli.py                 Inference and batch-QA entry point
│   ├── llm.py                 OpenAI-compatible LLM client
│   ├── schema.py              State, evidence, observation, and answer schemas
│   └── vllm_protocol.py       Parser for trained action-tag outputs
├── pipelines/
│   ├── document/              Indexing, MinerU execution, and page-object export
│   └── training/              Trace conversion and training-data utilities
├── configs/
│   ├── docpilot.example.yaml  JSON-action agent configuration
│   ├── docpilot-vllm.yaml     Trained action-tag model configuration
│   └── document.example.yaml  Document preprocessing configuration
├── demo/chat/                 React + FastAPI document-analysis interface
├── assets/                    README figures and a small example document graph
├── pyproject.toml
└── requirements.txt

📦 Installation

DocPilot requires Python 3.11 or newer.

python -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
python -m pip install -e .

The core installation includes MinerU for document preprocessing. If only the preprocessed-document agent is needed, install the dependencies appropriate for that runtime from requirements.txt.

⚙️ Model Configuration

The main CLI reads model settings from YAML. Copy one of the example configurations and fill in the endpoint, key, and model name for your deployment:

llm:
  provider: openai
  protocol: json
  model: YOUR_MODEL
  base_url: YOUR_OPENAI_COMPATIBLE_BASE_URL
  api_key: YOUR_API_KEY

• configs/docpilot.example.yaml uses structured JSON actions with an OpenAI-compatible model.
• configs/docpilot-vllm.yaml uses the action-tag protocol emitted by a trained DocPilot model served through vLLM.
• app/llm/.env contains official provider endpoints for the provider-specific helper modules; every credential value is empty.

Before preprocessing documents or starting the interactive demo, download and verify the MinerU models for your runtime.

For a CUDA GPU (recommended):

mineru-kit models download \
  --tier standard \
  --small-backend torch \
  --vlm-engine vllm \
  --source modelscope \
  --verbose

mineru-kit models verify \
  --tier standard \
  --small-backend torch \
  --vlm-engine vllm

For CPU-only environments:

mineru-kit models download \
  --tier standard \
  --small-backend onnx \
  --vlm-engine llama-cpp \
  --source modelscope \
  --verbose

mineru-kit models verify \
  --tier standard \
  --small-backend onnx \
  --vlm-engine llama-cpp

The models are cached under ~/.mineru/models/ and reused by subsequent runs.

🚀 Quick Start

1. Run on the included example nodes

After configuring a model endpoint:

python -m app \
  --config configs/docpilot.example.yaml \
  --nodes-json assets/sample_nodes.json \
  --question "What was the company's FY2015 revenue?" \
  --show-state

2. Preprocess documents with MinerU

Set the dataset paths in configs/document.example.yaml, then run:

python -m pipelines.document \
  --config configs/document.example.yaml \
  run --stages index,mineru-manifest,mineru-run,page-objects

The available stages are:

python -m pipelines.document list-stages

3. Ask a question about one processed document

python -m app \
  --config configs/docpilot.example.yaml \
  --mineru-dir runs/environment/mineru/DATASET/DOC_ID \
  --question "Your document question" \
  --show-state

4. Run a question file

python -m app \
  --config configs/docpilot.example.yaml \
  --question-file /path/to/questions.jsonl \
  --max-samples 10 \
  --num-workers 1

By default, agent logs, result files, and temporary visual evidence are written under runs/logs/, runs/outputs/, and runs/.workspace/.

💬 Interactive Demo

The demo provides conversation history, model and agent selection, document upload, MinerU preprocessing, PDF viewing, action traces, and evidence localization.

Install the backend and frontend dependencies:

python -m pip install -r demo/chat/backend/requirements.txt
cd demo/chat/frontend
npm ci
cd ../../..

Start both services from the project root:

./demo/chat/start.sh

Open http://127.0.0.1:5173. The API listens on http://127.0.0.1:8008.

• Default agent runs the DocPilot pipeline and preprocesses uploaded PDFs with MinerU.
• Chat agent sends selected PDF pages, images, or text directly to a multimodal OpenAI-compatible model.

Runtime uploads, OCR output, workspace state, and conversation history remain local under demo/chat/backend/runs/ and demo/chat/workspace/. See the demo documentation for configuration details.

🤝 Acknowledgement

• Training Framework: ms-swift
• Sources of Training Data:
  • Single-ability: DocSeeker, HotpotQA, FinQA, ChartQAPro, and InfographicVQA
  • Multi-ability: LongDocURL, DUDE, MADQA, and MMDocRAG