# /show-eval Cursor Command - Complete Specification

## Part 1: Usage Examples
```bash
# Single experiment - by name
/show-eval sharegpt4v_8node_bsz256_bd8_block_causal_lr_1e-6_unfreeze_vision

# Single experiment - by training script
/show-eval chengyue_script/dvlm/training/multinode/mdm/sharegpt4v/sweep_flexible_constant_and_block_causal/train_sharegpt4v_8node_bsz256_anneal_bd16_block_causal_lr_1e-6_unfreeze_vision.sh

# Single experiment - by output directory
/show-eval third_party/VLMEvalKit/output_models/sharegpt4v_text_8node_bsz256_bd8_block_causal_lr_1e-6_unfreeze_vision

# Multi-experiment comparison
/show-eval exp1 exp2
/show-eval train_bd8.sh train_bd16.sh
/show-eval third_party/VLMEvalKit/output_models/exp1 chengyue_script/.../train_exp2.sh

# Baseline/原版模型 (HuggingFace)
/show-eval Qwen2.5-VL-3B-Instruct
/show-eval Qwen2.5-VL-7B-Instruct
/show-eval qwen 3b baseline
```

## Part 2: Input Parsing

### 2.1 Input Type Detection
| Input Type | Detection | How to Extract exp_name |
|------------|-----------|------------------------|
| Training script | ends with .sh | Read file, grep RUN_ID="${RUN_ID:-...}", extract default value |
| VLMEvalKit output | contains third_party/VLMEvalKit/output_models/ or VLMEvalKit_local/output_models/ | Extract path component after output_models/ |
| Training output | contains output_models/ (not VLMEvalKit) | Extract path component after output_models/ |
| **Baseline model** | matches known HF model names or keywords | Map to canonical model name |
| Plain string | none of above | Use directly as exp_name |

### 2.2 Baseline Model Detection (NEW)

**Known baseline model patterns**:
| User Input | Canonical Name | Result Location |
|------------|----------------|-----------------|
| `Qwen2.5-VL-3B-Instruct`, `qwen 3b`, `qwen2.5 3b baseline` | `Qwen2.5-VL-3B-Instruct` | `VLM_DIR/output_models/Qwen2.5-VL-3B-Instruct/` |
| `Qwen2.5-VL-7B-Instruct`, `qwen 7b`, `qwen2.5 7b baseline` | `Qwen2.5-VL-7B-Instruct` | `VLM_DIR/output_models/Qwen2.5-VL-7B-Instruct/` |
| `Qwen2.5-VL-72B-Instruct`, `qwen 72b` | `Qwen2.5-VL-72B-Instruct` | `VLM_DIR/output_models/Qwen2.5-VL-72B-Instruct/` |

**Detection logic**:
```python
# Keywords that trigger baseline model detection
baseline_keywords = ['qwen', 'baseline', 'original', '原版', 'pretrain', 'instruct']

# If input contains these keywords, try to match to a known model
if any(kw in input.lower() for kw in baseline_keywords):
    # Extract size: 3b, 7b, 72b
    if '3b' in input.lower():
        exp_name = 'Qwen2.5-VL-3B-Instruct'
    elif '7b' in input.lower():
        exp_name = 'Qwen2.5-VL-7B-Instruct'
    elif '72b' in input.lower():
        exp_name = 'Qwen2.5-VL-72B-Instruct'
```

### 2.3 RUN_ID Extraction from Training Script
Training scripts define RUN_ID like:

```bash
RUN_ID="${RUN_ID:-sharegpt4v_8node_bsz256_anneal_bd16_block_causal_lr_1e-6_unfreeze_vision}"
```

Steps:
- Use read_file to read the .sh file
- Find line matching regex: `RUN_ID="\$\{RUN_ID:-([^}]+)\}"`
- Extract group 1 as exp_name

## Part 3: Result Location Logic

### 3.1 VLM_DIR 解析

**VLM_DIR** 统一指向 VLMEvalKit 根目录：
- 优先 `third_party/VLMEvalKit`
- 不存在则使用 `VLMEvalKit_local`

### 3.2 Search Order (check in this order, use first that exists)

**For baseline models (no checkpoint subdirs)**:
1. `VLM_DIR/output_models/{model_name}/{model_name}/*_acc.csv` (individual benchmark CSVs)
2. `VLM_DIR/output_models/{model_name}/summary.md` (if generated)

**For trained models (with checkpoint subdirs)**:
1. `VLM_DIR/output_models/{exp_name}/checkpoint_results.md`
2. `VLM_DIR/output_models/{exp_name}/checkpoint_results.csv`
3. `VLM_DIR/output_models/{exp_name}/eval_vlmeval/checkpoint-*/summary.md`
4. `output_models/{exp_name}/eval_vlmeval/checkpoint-*/summary.md`

Speedup (tokens_per_step) results:
5. `output_models/{exp_name}/eval_vlmeval/checkpoint-*/<TASK>/tokens_per_step_stats.json`

### 3.3 Baseline Model Result Parsing (NEW)

VLMEvalKit outputs individual CSV files per benchmark:
```
VLM_DIR/output_models/Qwen2.5-VL-3B-Instruct/
└── Qwen2.5-VL-3B-Instruct/
    ├── Qwen2.5-VL-3B-Instruct_AI2D_TEST_acc.csv
    ├── Qwen2.5-VL-3B-Instruct_ChartQA_TEST_acc.csv
    ├── Qwen2.5-VL-3B-Instruct_DocVQA_VAL_acc.csv
    └── ...
```

**Parsing logic**:
```bash
# Extract scores from each *_acc.csv
for f in *_acc.csv; do
    task=$(basename "$f" | sed 's/.*_\(.*\)_acc.csv/\1/')
    # CSV format: "split","Overall",...
    # Get Overall score from second column of data row
    score=$(tail -1 "$f" | cut -d',' -f2 | tr -d '"')
done
```

**Score normalization**:
- Some scores are 0-1 (e.g., AI2D: 0.808), multiply by 100
- Some scores are 0-100 (e.g., ChartQA: 94.88), use as-is
- Detection: if score < 1.5, assume 0-1 scale and multiply by 100

### 3.4 If No Results Exist
If no results found for baseline model:

```bash
# Run this to evaluate baseline:
sbatch chengyue_script/dvlm/evaluation/vlmeval/eval_qwen_baseline.sh
```

### 3.5 Common Benchmarks
Common eval tasks include:
- SEEDBench2_Plus, AI2D_TEST, RealWorldQA, DocVQA_VAL, ChartQA_TEST, MathVista_MINI, MMMU_DEV_VAL, MMMU_Pro_V_COT, MMBench_DEV_EN_V11, GQA_TestDev_Balanced, TextVQA_VAL, POPE

## Part 4: File Formats

### 4.1 checkpoint_results.md Format (Aggregated - Trained Models)
```
# Checkpoint Benchmark Results

| Step | AI2D_TEST | ChartQA_TEST | DocVQA | GQA | MMBench | MMMU | MathVista | POPE | RealWorldQA | SEEDBench2_Plus | TextVQA | Avg |
|------|------|------|------|------|------|------|------|------|------|------|------|------|
| 50 | 78.6 | 68.1 | 60.9 | 42.3 | 71.7 | 41.8 | 55.9 | 86.0 | 65.0 | 65.7 | 35.0 | 61.0 |
| 100 | 80.1 | 75.1 | 86.3 | 57.8 | 75.3 | 47.0 | 60.9 | 86.0 | 65.5 | 68.3 | 59.8 | 69.3 |
```

### 4.2 Baseline Model Output Format (NEW)
```
## Baseline: Qwen2.5-VL-3B-Instruct

| Model | AI2D | ChartQA | DocVQA | GQA | MMBench | MMMU | MathVista | POPE | RealWorldQA | SEEDBench2+ | TextVQA | Avg |
|-------|------|---------|--------|-----|---------|------|-----------|------|-------------|-------------|---------|-----|
| Qwen2.5-VL-3B-Instruct | 80.8 | 81.2 | 93.5 | 62.4 | 74.2 | 47.8 | 61.2 | 87.5 | 66.3 | 68.9 | 76.2 | 72.7 |

*Source: VLM_DIR/output_models/Qwen2.5-VL-3B-Instruct/Qwen2.5-VL-3B-Instruct/*_acc.csv*
```

### 4.3 tokens_per_step_stats.json Format
```json
{
  "mean": 1.68,
  "std": 0.52,
  "total_tokens": 12345,
  "total_steps": 7345,
  "tokens_per_step_ratio": 1.68,
  "distribution_pct": {
    "1_token": 77.5,
    "2_tokens": 16.6,
    "3+_tokens": 5.9
  }
}
```

## Part 5: Anti-Hallucination Protocol (CRITICAL)

### Before outputting ANY numbers:
1. MUST use `read_file` tool on actual result files
   - Never guess or infer numbers from memory
   - Never use numbers from previous conversations
   - ALWAYS read the file fresh, even if you think you know the contents

2. File reading order (for each experiment):
   a. First try: `VLM_DIR/output_models/{exp}/checkpoint_results.md`
   b. If not exists: `VLM_DIR/output_models/{exp}/checkpoint_results.csv`
   c. If not exists: Read individual `summary.md` files from eval_vlmeval/
   d. **For baseline**: Read individual `*_acc.csv` files
   e. Speedup: read `tokens_per_step_stats.json` per task
   f. If nothing exists: Say "No results found for {exp_name}" - DO NOT make up data

3. For multi-experiment comparison:
   - Read ALL experiments' result files BEFORE generating any table
   - Do NOT output partial data or mix old/new readings

4. Verification:
   - After outputting table, ALWAYS cite source files
   - Format: `*Source: {file_path}*`

5. If results incomplete:
   - Show `-` for missing values
   - Note which checkpoints/benchmarks are missing
   - NEVER fill in guessed values

## Part 6: Output Formats

### 6.1 Single Experiment Output (Trained Model)
```
## Experiment: sharegpt4v_8node_bsz256_bd8_block_causal_lr_1e-6_unfreeze_vision

| Step | AI2D | ChartQA | DocVQA | GQA | MMBench | MMMU | MathVista | POPE | RealWorldQA | SEEDBench2+ | TextVQA | Avg | TPS |
|------|------|---------|--------|-----|---------|------|-----------|------|-------------|-------------|---------|-----|-----|
| 50 | 78.6 | 68.1 | 60.9 | 42.3 | 71.7 | 41.8 | 55.9 | 86.0 | 65.0 | 65.7 | 35.0 | 61.0 | 1.25 |
| 100 | 80.1 | 75.1 | 86.3 | 57.8 | 75.3 | 47.0 | 60.9 | 86.0 | 65.5 | 68.3 | 59.8 | 69.3 | 1.63 |

**Best Checkpoint**: step 100 (Avg: 69.3, TPS: 1.63)

*Sources:*
- *Scores: VLM_DIR/output_models/.../checkpoint_results.md*
- *Speedup: output_models/.../eval_vlmeval/checkpoint-100/*/tokens_per_step_stats.json*
```

### 6.2 Baseline Model Output (NEW)
```
## Baseline: Qwen2.5-VL-3B-Instruct

| Model | AI2D | ChartQA | DocVQA | GQA | MMBench | MMMU | MathVista | POPE | RealWorldQA | SEEDBench2+ | TextVQA | Avg |
|-------|------|---------|--------|-----|---------|------|-----------|------|-------------|-------------|---------|-----|
| Qwen2.5-VL-3B-Instruct | 80.8 | 81.2 | 93.5 | 62.4 | 74.2 | 47.8 | 61.2 | 87.5 | 66.3 | 68.9 | 76.2 | 72.7 |

*Source: VLM_DIR/output_models/Qwen2.5-VL-3B-Instruct/Qwen2.5-VL-3B-Instruct/*_acc.csv*
```

### 6.3 Multi-Experiment Comparison (Compact xxx/yyy Format)
```
## Comparison: bd8 vs bd16_anneal

**Legend**: bd8 / bd16_anneal

| Step | Avg | TPS |
|------|-----|-----|
| 100 | 69.3/71.1 | 1.63/1.82 |
| 200 | 71.7/73.3 | 1.70/1.85 |
```

### 6.4 Comparison with Baseline (NEW)
```
## Comparison: j63_math_bd32 vs Qwen2.5-VL-3B-Instruct (baseline)

| Benchmark | j63_math_bd32 (best) | Baseline | Δ |
|-----------|---------------------|----------|---|
| AI2D | 80.0 | 80.8 | -0.8 |
| ChartQA | 80.3 | 81.2 | -0.9 |
| DocVQA | 89.8 | 93.5 | -3.7 |
| ... | ... | ... | ... |
| **Avg** | **71.4** | **72.7** | **-1.3** |

*Note: Baseline is single-step AR, trained model uses MDM (bd=32)*
```

### 6.5 Speedup Aggregation Rule
- For a checkpoint, compute **overall TPS** by summing `total_tokens/total_steps` across all task stats files.
- If any task missing, still compute over available tasks and note missing tasks.
- **Baseline models have TPS=1.0** (standard AR decoding)

## Part 7: Reference Scripts

### 7.1 Generate Aggregated Results (Trained Models)
```bash
python chengyue_script/dvlm/evaluation/plot_checkpoint_results.py \
  --input_dir VLM_DIR/output_models/{exp_name}
```

### 7.2 Generate Single Checkpoint Summary
```bash
python chengyue_script/dvlm/evaluation/vlmeval/aggregate_vlmeval_results.py \
  --input_dir {checkpoint_eval_dir} \
  --output summary.md
```

### 7.3 Evaluate Baseline Model (NEW)
```bash
sbatch chengyue_script/dvlm/evaluation/vlmeval/eval_qwen_baseline.sh
```

## Part 8: Known Result Locations Summary
| Location | Type | Description |
|----------|------|-------------|
| VLM_DIR/output_models/{exp}/checkpoint_results.md | Aggregated | All checkpoints in one table |
| VLM_DIR/output_models/{exp}/checkpoint_results.csv | Aggregated | CSV format |
| VLM_DIR/output_models/{exp}/checkpoint_results.png | Plot | Trend visualization |
| VLM_DIR/output_models/{exp}/eval_vlmeval/checkpoint-*/summary.md | Per-ckpt | Individual checkpoint results |
| output_models/{exp}/eval_vlmeval/checkpoint-*/summary.md | Per-ckpt | Alternate location |
| output_models/{exp}/eval_vlmeval/checkpoint-*/<TASK>/tokens_per_step_stats.json | Speedup | Tokens-per-step stats |
| **VLM_DIR/output_models/{model_name}/{model_name}/*_acc.csv** | **Baseline** | **Per-benchmark CSV for HF models** |

## Part 9: Known Baseline Models (NEW)

| Model Name | Size | Status | Notes |
|------------|------|--------|-------|
| Qwen2.5-VL-3B-Instruct | 3B | Evaluating (Job 8185931) | Primary baseline |
| Qwen2.5-VL-7B-Instruct | 7B | Partial results | 5/11 benchmarks |
| Qwen2.5-VL-72B-Instruct | 72B | Not evaluated | Requires multi-node |

--- End Command ---
