# /show-error Cursor Command - Complete Specification

## Part 1: Usage Examples

```bash
# Compare two experiments by name
/show-error exp_bd4 exp_bd8

# Compare with specific checkpoint policy
/show-error exp_bd4 exp_bd8 --checkpoint_policy latest
/show-error exp_bd4 exp_bd8 --checkpoint_policy step:2000

# Compare specific benchmarks only
/show-error exp_bd4 exp_bd8 --benchmarks TextVQA,DocVQA

# Full command with CLI script
python chengyue_script/dvlm/evaluation/error_case_compare.py exp_bd4 exp_bd8
python chengyue_script/dvlm/evaluation/error_case_compare.py exp_bd4 exp_bd8 --checkpoint_policy best --benchmarks TextVQA,DocVQA,ChartQA,GQA
```

## Part 2: Input Parsing

### 2.1 Input Type Detection

| Input Type | Detection | How to Extract exp_name |
|------------|-----------|------------------------|
| Plain string | Default | Use directly as exp_name |

### 2.2 Experiment Discovery (Search Order)

**VLM_DIR 解析**：优先 `third_party/VLMEvalKit`，不存在则用 `VLMEvalKit_local`。

For each `exp_name`, search in this order:

1. `VLM_DIR/output_models/{exp_name}/` - VLMEvalKit evaluation results
2. `output_models/{exp_name}/` - Training output location (fallback)

Within each location, look for:
- `eval_vlmeval/checkpoint-*/` directories
- `checkpoint_results.md` or `checkpoint_results.csv` (aggregated results)

## Part 3: Checkpoint Selection Policy

| Policy | Description |
|--------|-------------|
| `best` (default) | Use aggregated results to find checkpoint with highest Avg |
| `latest` | Use most recent checkpoint by step number |
| `step:<n>` | Use specific step (e.g., `step:2000`) |

## Part 4: Per-Sample Artifact Loading

### 4.1 XLSX Schema

VLMEvalKit per-sample results are stored in `*_results.xlsx` files with columns:

| Column | Description |
|--------|-------------|
| `index` | Sample identifier |
| `question` | Question text |
| `answer` | Ground truth (may be list) |
| `prediction` | Model prediction |
| `eval_score` | Evaluation score (0-1) |
| `image_path` | Path to image |

### 4.2 Benchmark Directories

| Benchmark | Directory Name |
|-----------|---------------|
| TextVQA | `TextVQA_VAL` |
| DocVQA | `DocVQA_VAL` |
| ChartQA | `ChartQA_TEST` |
| GQA | `GQA_TestDev_Balanced` |
| POPE | `POPE` |
| MMMU | `MMMU_DEV_VAL` |
| MMMU Pro CoT | `MMMU_Pro_V_COT` |
| MMBench | `MMBench_DEV_EN_V11` |
| MathVista | `MathVista_MINI` |

## Part 5: Error Pattern Taxonomy (v2)

All pattern counts are computed **by Python** from per-sample artifacts. The following patterns are detected:

### Length / Truncation Patterns

| Pattern | Description | Detection Logic |
|---------|-------------|-----------------|
| `empty_or_null` | Prediction is empty/None | `len(pred) == 0` |
| `severe_truncation` | Pred ≤3 chars, GT much longer | `len(pred) <= 3 and len(gt) > 6` |
| `prefix_truncation` | Pred is prefix of GT | `gt.startswith(pred) and len(pred) < len(gt) * 0.8` |
| `too_short_general` | Pred significantly shorter | `len(pred) < len(gt) * 0.5` |
| `noise_coverage` | Pred contains GT but longer | `gt in pred and len(pred) > len(gt) * 2` |
| `too_long_general` | Pred much longer, no GT | `len(pred) > len(gt) * 2 and gt not in pred` |
| `repetition_loop` | Repeated tokens/phrases | Word repeated ≥3 times |

### Yes/No Patterns

| Pattern | Description | Detection Logic |
|---------|-------------|-----------------|
| `yes_no_flip` | Polarity flipped | GT=yes/no, pred=opposite |
| `yes_no_with_explanation` | Correct yes/no + extra | First word matches but extra words |

### Numeric Patterns

| Pattern | Description | Detection Logic |
|---------|-------------|-----------------|
| `numeric_mismatch` | Numbers differ | Both contain numbers, values differ |
| `unit_mismatch` | Same number, different unit | Overlapping numbers, different units |
| `format_numeric` | Formatting difference | Same value, different format (commas, %) |

### String-Level Patterns

| Pattern | Description | Detection Logic |
|---------|-------------|-----------------|
| `case_punct_only` | Case/punct difference | Matches after normalization |
| `whitespace_only` | Whitespace difference | Matches after collapsing spaces |
| `char_level_typo` | High overlap, not exact | 70%+ char overlap, not identical |
| `substring_partial` | Incomplete match | Pred is substring of GT |

### Semantic Patterns

| Pattern | Description | Detection Logic |
|---------|-------------|-----------------|
| `entity_confusion` | Shared tokens, key entity differs | 20-70% word overlap |
| `completely_wrong` | Completely different | >70% edit distance, <20% word overlap |
| `other` | Uncategorized | Default fallback |

## Part 6: Anti-Hallucination Protocol (CRITICAL)

### Before outputting ANY numbers:

1. **MUST use Python counting** from per-sample XLSX files
   - Never guess or infer pattern counts from memory
   - Never use numbers from previous conversations
   - ALWAYS run `error_case_compare.py` or equivalent analysis

2. **File reading order** (for each experiment):
   - Find checkpoint via policy (best/latest/step:N)
   - Locate `*_results.xlsx` in each benchmark directory
   - Load DataFrame and compute patterns via `classify_error_v2()`

3. **For comparison output**:
   - Read BOTH experiments' XLSX files BEFORE generating any table
   - Do NOT output partial data or mix old/new readings

4. **Verification**:
   - After outputting table, ALWAYS cite source files
   - Format: `*Source: {file_path}*`

5. **If artifacts missing**:
   - Show `-` for missing values
   - Note which experiments/benchmarks are missing
   - NEVER fill in guessed values

### Example of CORRECT behavior:
1. User: `/show-error exp_bd4 exp_bd8`
2. AI: [runs error_case_compare.py or equivalent]
3. AI: [outputs table with EXACT counts from XLSX files]
4. AI: *Sources: VLM_DIR/.../TextVQA_VAL/.../xxx_results.xlsx, ...*

### Example of WRONG behavior (NEVER DO):
- Outputting pattern counts without loading XLSX first
- Saying "based on previous analysis..." without re-running
- Guessing or estimating error patterns
- Using numbers from chat history instead of fresh file reads

## Part 7: Output Formats

### 7.1 Scoreboard

```
## Scoreboard

| Benchmark | exp_bd4 | exp_bd8 | Delta |
|-----------|---------|---------|-------|
| TextVQA   | 72.5    | 68.3    | +4.2  |
| DocVQA    | 85.1    | 82.4    | +2.7  |
| ChartQA   | 76.2    | 74.8    | +1.4  |
| GQA       | 62.3    | 61.9    | +0.4  |
| **Average** | **74.0** | **71.9** | **+2.2** |
```

### 7.2 Error Pattern Breakdown

```
### TextVQA - exp_bd4 (checkpoint-2000)

- Total samples: 5000
- Correct: 3625 (72.5%)
- Incorrect: 1375

| Pattern | Count | % of Errors | Description |
|---------|-------|-------------|-------------|
| completely_wrong | 412 | 30.0% | Completely different from GT |
| noise_coverage | 287 | 20.9% | Pred contains GT but has extra noise |
| too_short_general | 198 | 14.4% | Pred significantly shorter |
| char_level_typo | 156 | 11.3% | High char overlap but not exact |
| numeric_mismatch | 89 | 6.5% | Numbers differ |
| ... | ... | ... | ... |
```

### 7.3 Sample Error Cases

```
### TextVQA - exp_bd4

**Case 1** (Pattern: `noise_coverage`)
- Question: What is the text on the sign?
- GT: `STOP`
- Pred: `The text on the sign says "STOP" in red letters.`
- Score: 0.00

**Case 2** (Pattern: `char_level_typo`)
- Question: What brand is shown?
- GT: `Coca-Cola`
- Pred: `Coca Cola`
- Score: 0.00
```

### 7.4 Source Citation

```
## Sources

*All counts computed from per-sample XLSX files:*
- `VLM_DIR/output_models/exp_bd4/eval_vlmeval/checkpoint-2000/TextVQA_VAL/.../xxx_results.xlsx`
- `VLM_DIR/output_models/exp_bd4/eval_vlmeval/checkpoint-2000/DocVQA_VAL/.../xxx_results.xlsx`
- `VLM_DIR/output_models/exp_bd8/eval_vlmeval/checkpoint-1800/TextVQA_VAL/.../xxx_results.xlsx`
- ...
```

## Part 8: Reference Scripts

### 8.1 Main Analysis Script

```bash
python chengyue_script/dvlm/evaluation/error_case_compare.py expA expB
```

**Location**: `chengyue_script/dvlm/evaluation/error_case_compare.py`

**Outputs**:
- `output/analysis/error_case_compare/{expA}_vs_{expB}/summary.md`
- `output/analysis/error_case_compare/{expA}_vs_{expB}/report.json`

### 8.2 Error Pattern Module

**Location**: `chengyue_script/dvlm/evaluation/error_patterns.py`

**Functions**:
- `classify_error_v2(pred, gt, question, benchmark)` - Classify single error
- `count_patterns(results)` - Count pattern occurrences
- `PATTERN_DESCRIPTIONS` - Pattern documentation dict

### 8.3 Related Scripts

| Script | Purpose |
|--------|---------|
| `analyze_short_answer_qa.py` | BD4 vs BD8 comparison (sample-aligned) |
| `llm_analyze_errors.py` | LLM-assisted error analysis |
| `analyze_docvqa_checkpoint_regressions.py` | Cross-checkpoint regression analysis |

## Part 9: Known Result Locations Summary

| Location | Type | Description |
|----------|------|-------------|
| `VLM_DIR/output_models/{exp}/checkpoint_results.md` | Aggregated | All checkpoints scores |
| `VLM_DIR/output_models/{exp}/eval_vlmeval/checkpoint-*/` | Per-ckpt | Checkpoint eval results |
| `VLM_DIR/output_models/{exp}/eval_vlmeval/checkpoint-*/{Benchmark}/*_results.xlsx` | Per-sample | Raw predictions |
| `output/analysis/error_case_compare/{expA}_vs_{expB}/summary.md` | Analysis | Error pattern report |
| `output/analysis/error_case_compare/{expA}_vs_{expB}/report.json` | Analysis | Machine-readable report |
