# Evaluate MDM Checkpoint

自动找到实验目录并用 sbatch 提交 VLMEvalKit 评测任务。

## Usage Examples

```bash
# 用实验名称
/eval-checkpoint j63_math_8node_bsz256_anneal_bd32_block_causal_lr_1e-6_unfreeze_vision

# 用训练脚本路径
/eval-checkpoint chengyue_script/dvlm/training/multinode/mdm/j63/train_xxx.sh

# 用 output_models 路径
/eval-checkpoint output_models/j63_math_8node_bsz256_anneal_bd32_block_causal_lr_1e-6_unfreeze_vision

# 只评测最后 3 个 checkpoint
/eval-checkpoint j63_math --last 3

# dry-run 模式（只生成脚本不提交）
/eval-checkpoint j63_math --dry-run
```

## Execution Steps

### Step 1: Parse Input & Find Experiment Directory

根据输入类型自动定位实验目录：

| Input Type | Detection | How to Find exp_dir |
|------------|-----------|---------------------|
| Training script (.sh) | ends with `.sh` | Read file, extract `RUN_ID="${RUN_ID:-xxx}"`, then `output_models/{RUN_ID}` |
| output_models path | contains `output_models/` | Use directly |
| Plain string | none of above | Search in `output_models/` for matching directory |

**RUN_ID 提取正则**: `RUN_ID="\$\{RUN_ID:-([^}]+)\}"`

**模糊匹配**: 如果用户输入 `j63_math`，搜索 `output_models/*j63_math*` 找到匹配目录。

### Step 2: Validate & Count Checkpoints

```bash
# 检查目录存在
ls -d {exp_dir}/checkpoint-* | wc -l

# 列出所有 checkpoint
ls -d {exp_dir}/checkpoint-* | sort -V
```

### Step 3: Submit Evaluation Jobs

**直接执行** (不需要用户确认):

```bash
cd /lustre/fs11/portfolios/nvr/projects/nvr_elm_llm/users/chengyuew/workspace/shiyi/LMFlow

# 提交所有 checkpoint 的评测 (12 个标准 benchmark)
bash chengyue_script/dvlm/evaluation/submit_batch_eval.sh \
  {exp_dir} \
  --skip-existing 1 \
  --force-left-to-right 1

# 提交 MMMU_Pro_V_COT 评测 (单独提交，因为耗时较长)
TASKS="MMMU_Pro_V_COT" bash chengyue_script/dvlm/evaluation/submit_batch_eval.sh \
  {exp_dir} \
  --skip-existing 1
```

**可选参数**:
- `--last N`: 只评测最后 N 个 checkpoint
- `--dry-run`: 只生成脚本不提交
- `--configs "MDM=1_SPEC=0 MDM=1_SPEC=1"`: A/B test 两种解码方式

### Step 4: Report Results

提交后输出：
1. 提交的 job 数量和 job IDs
2. 监控命令: `squeue -u chengyuew | grep eval`
3. 查看进度命令

## Output Format

```
## 评测提交: {exp_name}

**实验目录**: {exp_dir}
**Checkpoint 数量**: {N} 个 (checkpoint-50 ~ checkpoint-{max})

### 提交结果
✅ 已提交 {N} 个标准评测任务 (11 benchmarks)
✅ 已提交 {N} 个 MMMU_Pro_V_COT 评测任务

### 监控命令
```bash
# 查看任务状态
squeue -u chengyuew | grep eval

# 查看评测进度
for ckpt in {exp_dir}/eval_vlmeval/checkpoint-*/; do
  name=$(basename "$ckpt")
  csv_count=$(find "$ckpt" -name "*.csv" 2>/dev/null | wc -l)
  echo "$name: $csv_count/12 benchmarks"
done | sort -V
```

### 结果汇总 (评测完成后)
```bash
python chengyue_script/dvlm/evaluation/plot_checkpoint_results.py \
  --input_dir {exp_dir}
```
```

## Default Benchmarks (12 total)

**标准 Benchmarks (11)**:
- AI2D_TEST
- ChartQA_TEST
- DocVQA_VAL
- GQA_TestDev_Balanced
- MMBench_DEV_EN_V11
- MMMU_DEV_VAL
- MathVista_MINI
- POPE
- RealWorldQA
- SEEDBench2_Plus
- TextVQA_VAL

**额外 Benchmark (1)**:
- MMMU_Pro_V_COT (Chain-of-Thought reasoning, 单独提交)

## Key Scripts

| Script | Purpose |
|--------|---------|
| `chengyue_script/dvlm/evaluation/submit_batch_eval.sh` | 批量生成并提交 sbatch jobs |
| `chengyue_script/dvlm/evaluation/plot_checkpoint_results.py` | 汇总结果生成表格和图 |
| `chengyue_script/dvlm/evaluation/vlmeval/run_all_benchmarks.sh` | 单次评测执行脚本 |

## Debugging

- **No checkpoints found**: 检查实验目录路径是否正确
- **Jobs pending**: `squeue -u chengyuew` 查看队列状态
- **Eval failed**: 查看 `{exp_dir}/eval_vlmeval/checkpoint-*/slurm_err.log`
