# VLM Evaluation Guide

Evaluate Qwen2.5-VL original weights, AR SFT checkpoints, and MDM (dVLM) checkpoints.

## Prerequisites

```bash
bash chengyue_script/dvlm/utils/install_vlmeval_deps.sh
```

## Quick Reference

| Model Type | Architecture | Recommended Tool | Command |
|------------|-------------|------------------|---------|
| **Baseline** (Qwen2.5-VL) | `Qwen2_5_VLForConditionalGeneration` | `run_all_benchmarks.sh` | `bash .../run_all_benchmarks.sh` |
| **AR SFT** | `Fast_dLLM_Qwen2_5_VLForConditionalGeneration` | `run_all_benchmarks.sh` | `MODEL_PATH=... bash .../run_all_benchmarks.sh` |
| **MDM (dVLM)** | `Fast_dLLM_Qwen2_5_VLForConditionalGeneration` | `run_all_benchmarks.sh` | `MODEL_PATH=... USE_MDM=1 bash .../run_all_benchmarks.sh` |

**Unified entry point**: `chengyue_script/dvlm/evaluation/vlmeval/run_all_benchmarks.sh`

- No `MODEL_PATH` → built-in model mode (original Qwen2.5-VL)
- `MODEL_PATH` set → custom model mode (AR or MDM, auto-detected from path/config)

**Alternative** (lightweight, no GPT Judge): `eval_custom_model.py` — supports 6 tasks with exact match only.

---

## 1. Baseline: Original Qwen2.5-VL

### Option A: VLMEvalKit (Recommended — supports GPT Judge)

```bash
# All default benchmarks (14+ tasks)
bash chengyue_script/dvlm/evaluation/vlmeval/run_all_benchmarks.sh

# Specify model size
MODEL=Qwen2.5-VL-7B-Instruct bash chengyue_script/dvlm/evaluation/vlmeval/run_all_benchmarks.sh

# Specific tasks only
TASKS="MathVista_MINI MMMU_DEV_VAL GQA_TestDev_Balanced" \
  bash chengyue_script/dvlm/evaluation/vlmeval/run_all_benchmarks.sh
```

### Option B: lmms-eval (No GPT Judge needed)

```bash
MODEL_PATH=Qwen/Qwen2.5-VL-3B-Instruct \
  bash chengyue_script/dvlm/evaluation/lmms_eval/eval_lmms.sh gqa mmmu textvqa pope
```

---

## 2. AR SFT Checkpoint

### Option A: VLMEvalKit (Recommended)

```bash
MODEL_PATH=output_models/ar_sft_xxx/checkpoint-2000 \
  bash chengyue_script/dvlm/evaluation/vlmeval/run_all_benchmarks.sh

# Specific tasks
MODEL_PATH=output_models/ar_sft_xxx/checkpoint-2000 \
  TASKS="MMMU_DEV_VAL GQA_TestDev_Balanced POPE" \
  bash chengyue_script/dvlm/evaluation/vlmeval/run_all_benchmarks.sh
```

Auto-detection: if path contains `mdm`, `USE_MDM` is set automatically.

### Option B: eval_custom_model.py (Lightweight)

```bash
python3 chengyue_script/dvlm/evaluation/eval_custom_model.py \
  --model_path output_models/ar_sft_xxx/checkpoint-2000 \
  --tasks gqa,pope,textvqa,mmmu,mmbench,mathvista \
  --num_gpus 8
```

---

## 3. MDM (dVLM) Checkpoint

### Option A: VLMEvalKit (Recommended)

```bash
MODEL_PATH=output_models/j63_mdm_xxx/checkpoint-2000 USE_MDM=1 \
  bash chengyue_script/dvlm/evaluation/vlmeval/run_all_benchmarks.sh
```

MDM-specific environment variables:

| Variable | Default | Description |
|----------|---------|-------------|
| `USE_MDM` | `0` | `1` = MDM decoding mode |
| `BLOCK_SIZE` | from config `bd_size` | Block size for decoding |
| `THRESHOLD` | `0.9` | Confidence threshold |
| `MAX_TOKENS` | from config | Max generation tokens |
| `USE_SPECULATIVE` | `0` | `1` = speculative decoding (A/B test) |
| `FORCE_LEFT_TO_RIGHT` | `0` | `1` = L2R sequential, `0` = confidence-based parallel |

### Option B: eval_custom_model.py (Lightweight)

```bash
python3 chengyue_script/dvlm/evaluation/eval_custom_model.py \
  --model_path output_models/j63_mdm_xxx/checkpoint-2000 \
  --tasks gqa,pope,textvqa,mmmu,mmbench,mathvista \
  --num_gpus 8
```

---

## Two Tools Compared

| | `run_all_benchmarks.sh` (VLMEvalKit) | `eval_custom_model.py` |
|---|---|---|
| **Benchmarks** | 14+ (SEEDBench, AI2D, DocVQA, ChartQA, MMMU Pro CoT, ChartMimic, etc.) | 6 (gqa, pope, textvqa, mmmu, mmbench, mathvista) |
| **GPT Judge** | Yes (MathVista, MMMU, ChartMimic) | No (exact match / contains match only) |
| **TPS stats** | Yes (`tokens_per_step_stats.json`) | Yes (`--track_tokens_per_step`) |
| **Third-party** | Design2Code, HumanEval, MBPP (auto-routed) | No |
| **Output** | `MODEL_PATH/eval_vlmeval/` | `--output_dir` |

---

## Supported Benchmarks (VLMEvalKit)

| Benchmark | Dataset Name | Type | Requires Judge |
|-----------|-------------|------|----------------|
| SEEDBench2+ | SEEDBench2_Plus | MCQ | No |
| AI2D | AI2D_TEST | MCQ | No |
| RealWorldQA | RealWorldQA | MCQ | No |
| DocVQA | DocVQA_VAL | VQA | No |
| ChartQA | ChartQA_TEST | VQA | No |
| MathVista | MathVista_MINI | Math reasoning | Yes |
| MMMU | MMMU_DEV_VAL | MCQ | Yes (gpt-4-0125) |
| MMMU Pro CoT | MMMU_Pro_V_COT | MCQ + CoT | Yes (gpt-4-0125) |
| MMBench | MMBench_DEV_EN_V11 | MCQ | Yes (gpt-4-0125) |
| GQA | GQA_TestDev_Balanced | VQA | No |
| TextVQA | TextVQA_VAL | VQA | No |
| POPE | POPE | Yes/No | No |
| ChartMimic | ChartMimic_v2_direct_600 | Chart-to-code | Yes (gpt-4o) |
| ChartMimic | ChartMimic_v2_customized_600 | Chart-to-code | Yes (gpt-4o) |

---

## Common Environment Variables (run_all_benchmarks.sh)

| Variable | Default | Description |
|----------|---------|-------------|
| `MODEL` | `Qwen2.5-VL-3B-Instruct` | VLMEvalKit built-in model name |
| `MODEL_PATH` | (empty) | Custom checkpoint path; set to enable custom mode |
| `PROCESSOR_PATH` | `Qwen/Qwen2.5-VL-3B-Instruct` | Processor for custom models |
| `TASKS` | all default benchmarks | Space-separated task list |
| `OUTPUT_BASE` | auto | Output directory |
| `NUM_GPUS` | auto-detect | Number of GPUs |
| `JUDGE_MODEL` | `gpt-4o-mini-20240718` | GPT Judge model |

---

## Result Aggregation

```bash
# Single model, multi-checkpoint → Excel
python3 chengyue_script/dvlm/evaluation/vlmeval/aggregate_checkpoints.py \
  --model_dir output_models/j63_mdm_xxx

# Multi-model comparison (eval_custom_model results)
python3 chengyue_script/dvlm/evaluation/aggregate_results.py \
  --output_dirs eval_outputs/baseline eval_outputs/ar_sft eval_outputs/mdm_sft \
  --format markdown --output results_summary.md
```

---

## Auto Checkpoint Monitoring

```bash
nohup python3 chengyue_script/dvlm/evaluation/monitor_checkpoints.py \
  --watch_dir output_models \
  --poll_interval 60 \
  > monitor.log 2>&1 &
```

Auto-detects new checkpoints, validates completeness (`trainer_state.json`), infers AR/MDM type, and submits Slurm eval jobs.

Results location:

```
output_models/j63_mdm/
├── checkpoint-500/
├── checkpoint-1000/
└── eval_vlmeval/           # All eval results
    ├── checkpoint-500/
    ├── checkpoint-1000/
    └── eval_summary.xlsx   # Aggregated (from aggregate_checkpoints.py)
```

---

## Directory Structure

```
chengyue_script/dvlm/evaluation/
├── eval_custom_model.py              # Lightweight eval (AR/MDM, 6 tasks, no GPT Judge)
├── aggregate_results.py              # Multi-model result comparison
├── monitor_checkpoints.py            # Auto-monitor + Slurm submission
├── lmms_eval/
│   └── eval_lmms.sh                 # lmms-eval (baseline, no GPT Judge)
└── vlmeval/                          # VLMEvalKit (recommended, unified entry)
    ├── run_all_benchmarks.sh         # ★ Unified eval script (all 3 model types)
    ├── vlmeval_qwen_vl_mdm.py       # MDM model wrapper for VLMEvalKit
    ├── generate_vlmeval_config.py    # Config generator for custom models
    ├── run_vlmeval.py                # VLMEvalKit runner
    ├── aggregate_vlmeval_results.py  # Single model result summary
    └── aggregate_checkpoints.py      # Multi-checkpoint aggregation (Excel)
```

---

## Notes

- **lmms-eval doesn't support custom architectures**: Fast_dLLM models must use `run_all_benchmarks.sh` or `eval_custom_model.py`.
- **Processor loading**: Custom models load processor from base model (`Qwen/Qwen2.5-VL-3B-Instruct`), not from checkpoint.
- **MDM auto-detection**: If `MODEL_PATH` contains `mdm`, `USE_MDM=1` is set automatically.
- **GPT Judge API**: Configured via `AZURE_OPENAI_*` env vars. `run_all_benchmarks.sh` has NVIDIA LLM Proxy defaults built-in.
