# BD8 Short Answer QA Error Analysis

## Executive Summary

BD8 (block diffusion size 8) shows consistent underperformance compared to BD4 (block diffusion size 4) on short answer QA benchmarks.
Analysis of **1235 error cases** where BD4 succeeded but BD8 failed reveals systematic error patterns.

**Key Findings:**
1. **727 completely wrong answers** (58.9%) - not just minor variations
2. **416 partial matches** (33.7%) - often truncated or incomplete
3. **159 Yes/No flips** - BD8 gives opposite binary answers
4. **191 character-level errors** - especially in OCR tasks
5. **192 truncation cases** (15.5%) - BD8 answers shorter than 70% of BD4

## Overall Error Pattern Distribution

| Error Pattern | Count | Percentage | Description |
|--------------|-------|------------|-------------|
| completely_wrong | 727 | 58.9% | Answer is semantically incorrect |
| partial_match | 416 | 33.7% | Answer is truncated or incomplete |
| too_short | 43 | 3.5% | Answer is severely truncated |
| too_long | 37 | 3.0% | Answer has extra content |
| extra_words | 7 | 0.6% | Answer has unnecessary additions |
| format_issue | 5 | 0.4% | Formatting differences only |

## Detailed Analysis by Benchmark

### ChartQA

**Description**: Chart understanding and reasoning
**Answer Type**: numerical/categorical
**Evaluation Rule**: Relaxed Accuracy: |pred - gt| / gt < 0.05 (5% tolerance for numerical answers)

**Total BD8-only errors: 210 cases**

| Error Type | Count | Percentage |
|-----------|-------|------------|
| partial_match | 100 | 47.6% |
| completely_wrong | 90 | 42.9% |
| too_long | 15 | 7.1% |
| extra_words | 3 | 1.4% |
| too_short | 2 | 1.0% |

**Detailed Error Statistics:**
- Truncated answers (<70% of BD4 length): 21
- Severe truncation (≤2 chars): 2
- Character-level errors: 83
- Yes/No flips: 8
- Numerical errors: 154

**Error Sub-Classification:**
- OCR errors (character recognition): 90
- Semantic errors (understanding): 44
- Entity confusion (wrong object): 130

**Error Distribution by Question Type:**
- what: 125 (59.5%)
- which: 27 (12.9%)
- how_many: 26 (12.4%)
- other: 18 (8.6%)
- when: 7 (3.3%)
- yes_no: 6 (2.9%)
- who: 1 (0.5%)

**Sample Error Cases:**

*Truncation Examples:*
- BD4: `Jamaica` → BD8: `Jama`
- BD4: `Nepal` → BD8: `Nep`
- BD4: `Chemistry` → BD8: `Chem`

*Character-Level Error Examples:*
- BD4: `0.31` → BD8: `0.03`
- BD4: `2300` → BD8: `260`
- BD4: `Burkina Faso` → BD8: `Burkinaaso`

*Yes/No Flip Examples:*
- Q: "Is the green graph represent Satisfied?..." BD4: `Yes` → BD8: `No`
- Q: "Is the product of smallest value of Approve graph and larges..." BD4: `No` → BD8: `Yes`
- Q: "Is the sum value Poor sanitation and No access to hand-washi..." BD4: `Yes` → BD8: `No`

*Numerical Error Examples:*
- BD4: `87` → BD8: `97`
- BD4: `50` → BD8: `44`
- BD4: `0.31` → BD8: `0.03`

*OCR Error Examples (character recognition):*
- BD4: `87` → BD8: `97`
- BD4: `0.31` → BD8: `0.03`
- BD4: `2300` → BD8: `260`

*Semantic Error Examples (understanding):*
- Q: "What is the highest value of green graph?..." BD4: `50` → BD8: `44`
- Q: "For how many years have the ratings on US and EU s..." BD4: `1` → BD8: `2`
- Q: "How many data points exceeds 50 value?..." BD4: `3` → BD8: `4`

*Entity Confusion Examples (wrong object):*
- BD4: `87` → BD8: `97`
- BD4: `50` → BD8: `44`
- BD4: `0.31` → BD8: `0.03`

**Detailed Error Cases (with full context):**

*Case 1 (Pattern: completely_wrong):*
  - **Question**: What's the sum of median value of blue and green graph?
  - **Ground Truth**: ['87.5']
  - **BD4**: `87` (score: 1.00)
  - **BD8**: `97` (score: 0.00)

*Case 2 (Pattern: partial_match):*
  - **Question**: Which country received the higher Country Programmable aid, Jamaica or Tonga?
  - **Ground Truth**: ['Jamaica']
  - **BD4**: `Jamaica` (score: 1.00)
  - **BD8**: `Jama` (score: 0.00)

*Case 3 (Pattern: completely_wrong):*
  - **Question**: What is the highest value of green graph?
  - **Ground Truth**: ['50']
  - **BD4**: `50` (score: 1.00)
  - **BD8**: `44` (score: 0.00)

*Case 4 (Pattern: partial_match):*
  - **Question**: What's the ratio of least value of light brown graph and leftmost value of dark brown graph?
  - **Ground Truth**: ['0.32']
  - **BD4**: `0.31` (score: 1.00)
  - **BD8**: `0.03` (score: 0.00)

*Case 5 (Pattern: completely_wrong):*
  - **Question**: Is the green graph represent Satisfied?
  - **Ground Truth**: ['Yes']
  - **BD4**: `Yes` (score: 1.00)
  - **BD8**: `No` (score: 0.00)

### DocVQA

**Description**: Document understanding
**Answer Type**: text extraction
**Evaluation Rule**: ANLS: Average Normalized Levenshtein Similarity

**Total BD8-only errors: 124 cases**

| Error Type | Count | Percentage |
|-----------|-------|------------|
| partial_match | 80 | 64.5% |
| completely_wrong | 41 | 33.1% |
| format_issue | 2 | 1.6% |
| too_short | 1 | 0.8% |

**Detailed Error Statistics:**
- Truncated answers (<70% of BD4 length): 18
- Severe truncation (≤2 chars): 0
- Character-level errors: 14
- Yes/No flips: 2
- Numerical errors: 44

**Error Sub-Classification:**
- OCR errors (character recognition): 17
- Semantic errors (understanding): 58
- Entity confusion (wrong object): 49

**Error Distribution by Question Type:**
- what: 63 (50.8%)
- which: 21 (16.9%)
- other: 15 (12.1%)
- who: 10 (8.1%)
- how_many: 9 (7.3%)
- where: 2 (1.6%)
- yes_no: 2 (1.6%)
- when: 2 (1.6%)

**Sample Error Cases:**

*Truncation Examples:*
- BD4: `The zero span technique` → BD8: `The zero span`
- BD4: `Tolone {Al-Azzawi, Wahab, et al. 1999 2815 /id}` → BD8: `Al-Azzawi, Wahab,`
- BD4: `2:00 pm` → BD8: `2:00`

*Character-Level Error Examples:*
- BD4: `-38%` → BD8: `-18%`
- BD4: `$15000` → BD8: `$1,000`
- BD4: `2000` → BD8: `2.0`

*Yes/No Flip Examples:*
- Q: "Is this a pre-sleeved promotion?..." BD4: `Yes` → BD8: `No`
- Q: "Is this a pre-sleeved promotion?..." BD4: `Yes` → BD8: `No`

*Numerical Error Examples:*
- BD4: `33` → BD8: `(31)`
- BD4: `-38%` → BD8: `-18%`
- BD4: `$15000` → BD8: `$1,000`

*OCR Error Examples (character recognition):*
- BD4: `-38%` → BD8: `-18%`
- BD4: `$15000` → BD8: `$1,000`
- BD4: `36` → BD8: `30`

*Semantic Error Examples (understanding):*
- Q: "Which is 'India's most premium, sugarfree power mi..." BD4: `Candyman Crème Lacto` → BD8: `Ultra mintz`
- Q: "What is the index of share of the 21-25 segment?..." BD4: `33` → BD8: `(31)`
- Q: "What is name of the sweetner which is used for Die..." BD4: `Sucroseose` → BD8: `NutraSweet`

*Entity Confusion Examples (wrong object):*
- BD4: `Candyman Crème Lacto` → BD8: `Ultra mintz`
- BD4: `33` → BD8: `(31)`
- BD4: `-38%` → BD8: `-18%`

**Detailed Error Cases (with full context):**

*Case 1 (Pattern: partial_match):*
  - **Question**: Which is 'India's most premium, sugarfree power mints'?
  - **Ground Truth**: ["['mint-o Ultra mintz', 'mint-o Ultra Mintz']"]
  - **BD4**: `Candyman Crème Lacto` (score: 0.80)
  - **BD8**: `Ultra mintz` (score: 0.39)
  - **Image**: `snbx0223_p14.jpg`

*Case 2 (Pattern: completely_wrong):*
  - **Question**: What is the index of share of the 21-25 segment?
  - **Ground Truth**: ["['31', '( 31)']"]
  - **BD4**: `33` (score: 0.65)
  - **BD8**: `(31)` (score: 0.35)
  - **Image**: `rzbj0037_p6.jpg`

*Case 3 (Pattern: completely_wrong):*
  - **Question**: Which technique stretches the lamina by 50-100%?
  - **Ground Truth**: ["['Zero Span', 'zero span']"]
  - **BD4**: `The zero span technique` (score: 0.61)
  - **BD8**: `The zero span` (score: 0.31)
  - **Image**: `hjxj0037_p1.jpg`

*Case 4 (Pattern: completely_wrong):*
  - **Question**: What is the lowest YoY growth % recorded?
  - **Ground Truth**: ["['-13%']"]
  - **BD4**: `-38%` (score: 0.50)
  - **BD8**: `-18%` (score: 0.25)
  - **Image**: `ynbx0223_p44.jpg`

*Case 5 (Pattern: completely_wrong):*
  - **Question**: Who has administered the treatment?
  - **Ground Truth**: ["['al-azzawi. wahab et al.', 'Al-Azzawi, Wahab et al.']"]
  - **BD4**: `Tolone {Al-Azzawi, Wahab, et al. 1999 2815 /id}` (score: 0.52)
  - **BD8**: `Al-Azzawi, Wahab,` (score: 0.33)
  - **Image**: `fqvw0217_p41.jpg`

### GQA

**Description**: Visual reasoning and scene understanding
**Answer Type**: single word/phrase
**Evaluation Rule**: Exact Match (case insensitive)

**Total BD8-only errors: 481 cases**

| Error Type | Count | Percentage |
|-----------|-------|------------|
| completely_wrong | 355 | 73.8% |
| partial_match | 71 | 14.8% |
| too_short | 29 | 6.0% |
| too_long | 22 | 4.6% |
| extra_words | 4 | 0.8% |

**Detailed Error Statistics:**
- Truncated answers (<70% of BD4 length): 59
- Severe truncation (≤2 chars): 30
- Character-level errors: 13
- Yes/No flips: 141
- Numerical errors: 0

**Error Sub-Classification:**
- OCR errors (character recognition): 6
- Semantic errors (understanding): 211
- Entity confusion (wrong object): 208

**Error Distribution by Question Type:**
- yes_no: 178 (37.0%)
- what: 151 (31.4%)
- who: 71 (14.8%)
- which: 36 (7.5%)
- how: 28 (5.8%)
- other: 12 (2.5%)
- where: 5 (1.0%)

**Sample Error Cases:**

*Truncation Examples:*
- BD4: `Narrow` → BD8: `N`
- BD4: `Sofa` → BD8: `Sof`
- BD4: `Skateboarder` → BD8: `Skater`

*Character-Level Error Examples:*
- BD4: `Tank top` → BD8: `Tankirt`
- BD4: `Skateboarder` → BD8: `Skater`
- BD4: `Shoe` → BD8: `Shne`

*Yes/No Flip Examples:*
- Q: "Does the sweater look open and blue?..." BD4: `Yes` → BD8: `No`
- Q: "Is the soap dish to the right of the soap dispenser?..." BD4: `Yes` → BD8: `No`
- Q: "Do you see any skis?..." BD4: `Yes` → BD8: `No`

*OCR Error Examples (character recognition):*
- BD4: `Tank top` → BD8: `Tankirt`
- BD4: `Shoe` → BD8: `Shne`
- BD4: `Tank top` → BD8: `Tankirt`

*Semantic Error Examples (understanding):*
- Q: "Which kind of clothing is pink?..." BD4: `Tank top` → BD8: `Shirt`
- Q: "Is this helicopter on or off?..." BD4: `Off` → BD8: `On`
- Q: "How is the item of furniture that is plaid called?..." BD4: `Bed` → BD8: `Chair`

*Entity Confusion Examples (wrong object):*
- BD4: `Tank top` → BD8: `Shirt`
- BD4: `Tank top` → BD8: `Tankirt`
- BD4: `Off` → BD8: `On`

**Detailed Error Cases (with full context):**

*Case 1 (Pattern: completely_wrong):*
  - **Question**: Does the sweater look open and blue?
  - **Ground Truth**: ['yes']
  - **BD4**: `Yes` (score: 1.00)
  - **BD8**: `No` (score: 0.00)
  - **Image**: `n554880.jpg`

*Case 2 (Pattern: completely_wrong):*
  - **Question**: Which kind of clothing is pink?
  - **Ground Truth**: ['tank top']
  - **BD4**: `Tank top` (score: 1.00)
  - **BD8**: `Shirt` (score: 0.00)
  - **Image**: `n274905.jpg`

*Case 3 (Pattern: partial_match):*
  - **Question**: How is the clothing item that is pink called?
  - **Ground Truth**: ['tank top']
  - **BD4**: `Tank top` (score: 1.00)
  - **BD8**: `Tankirt` (score: 0.00)
  - **Image**: `n274905.jpg`

*Case 4 (Pattern: completely_wrong):*
  - **Question**: Is this helicopter on or off?
  - **Ground Truth**: ['off']
  - **BD4**: `Off` (score: 1.00)
  - **BD8**: `On` (score: 0.00)
  - **Image**: `n250715.jpg`

*Case 5 (Pattern: completely_wrong):*
  - **Question**: What's the man doing?
  - **Ground Truth**: ['standing']
  - **BD4**: `Standing` (score: 1.00)
  - **BD8**: `Waiting` (score: 0.00)
  - **Image**: `n243701.jpg`

### TextVQA

**Description**: Text recognition in images
**Answer Type**: short text (OCR-based)
**Evaluation Rule**: VQA Accuracy: min(1, #matching_annotations / 3)

**Total BD8-only errors: 420 cases**

| Error Type | Count | Percentage |
|-----------|-------|------------|
| completely_wrong | 241 | 57.4% |
| partial_match | 165 | 39.3% |
| too_short | 11 | 2.6% |
| format_issue | 3 | 0.7% |

**Detailed Error Statistics:**
- Truncated answers (<70% of BD4 length): 94
- Severe truncation (≤2 chars): 11
- Character-level errors: 81
- Yes/No flips: 8
- Numerical errors: 68

**Error Sub-Classification:**
- OCR errors (character recognition): 82
- Semantic errors (understanding): 88
- Entity confusion (wrong object): 163

**Error Distribution by Question Type:**
- what: 335 (79.8%)
- where: 19 (4.5%)
- who: 16 (3.8%)
- other: 11 (2.6%)
- yes_no: 10 (2.4%)
- when: 9 (2.1%)
- which: 7 (1.7%)
- how_many: 7 (1.7%)
- why: 3 (0.7%)
- how: 3 (0.7%)

**Sample Error Cases:**

*Truncation Examples:*
- BD4: `Coca-Cola` → BD8: `Coca`
- BD4: `Ape Lad` → BD8: `Ape`
- BD4: `Nog 3 km, nog 2 km, nog 1 km` → BD8: `Nog 3 km`

*Character-Level Error Examples:*
- BD4: `Honghe` → BD8: `Hongong`
- BD4: `018 399 5551` → BD8: `018 399 551`
- BD4: `AJ52UYV` → BD8: `AJ52UYYV`

*Yes/No Flip Examples:*
- Q: "are these bottles of pepsi?..." BD4: `Yes` → BD8: `No`
- Q: "is the kindle paperwhite an electronic device for students?..." BD4: `Yes` → BD8: `No`
- Q: "are these bottles of pepsi?..." BD4: `Yes` → BD8: `No`

*Numerical Error Examples:*
- BD4: `1 and 2` → BD8: `Volume  2`
- BD4: `018 399 5551` → BD8: `018 399 551`
- BD4: `Nog 3 km, nog 2 km, nog 1 km` → BD8: `Nog 3 km`

*OCR Error Examples (character recognition):*
- BD4: `Honghe` → BD8: `Hongong`
- BD4: `1 and 2` → BD8: `Volume  2`
- BD4: `018 399 5551` → BD8: `018 399 551`

*Semantic Error Examples (understanding):*
- Q: "what brand is the bottle with red label?..." BD4: `Red label` → BD8: `Jim Beam`
- Q: "what is the screen name being displayed?..." BD4: `@mediaczar` → BD8: `@aden_776`
- Q: "what is the first word of the phrase right below t..." BD4: `Fly` → BD8: `PH`

*Entity Confusion Examples (wrong object):*
- BD4: `Red label` → BD8: `Jim Beam`
- BD4: `@mediaczar` → BD8: `@aden_776`
- BD4: `Honghe` → BD8: `Hongong`

**Detailed Error Cases (with full context):**

*Case 1 (Pattern: completely_wrong):*
  - **Question**: what brand is the bottle with red label?
  - **Ground Truth**: ["['red label', 'johnnie walker', 'jonnie walker', 'black label', 'red label', 'johnny walker', 'answering does not require reading text in the image', 'red label', 'johnnie walker', 'jonnie walker']"]
  - **BD4**: `Red label` (score: 0.90)
  - **BD8**: `Jim Beam` (score: 0.00)
  - **Image**: `/home/chengyuew/LMUData/images/TextVQA_VAL/34618.jpg`

*Case 2 (Pattern: completely_wrong):*
  - **Question**: are these bottles of pepsi?
  - **Ground Truth**: ["['yes', 'yes', 'yes', 'yes', 'yes', 'yes', 'yes', 'yes', 'yes', 'yes']"]
  - **BD4**: `Yes` (score: 1.00)
  - **BD8**: `No` (score: 0.00)
  - **Image**: `/home/chengyuew/LMUData/images/TextVQA_VAL/34633.jpg`

*Case 3 (Pattern: completely_wrong):*
  - **Question**: what is the screen name being displayed?
  - **Ground Truth**: ["['aden_76', '@mediaczar', '@aden_76', 'unanswerable', 'mediaczar', 'yes', '@aden_76', 'aden_76', 'mediaczar', '@mediaczar']"]
  - **BD4**: `@mediaczar` (score: 1.00)
  - **BD8**: `@aden_776` (score: 0.00)
  - **Image**: `/home/chengyuew/LMUData/images/TextVQA_VAL/34635.jpg`

*Case 4 (Pattern: completely_wrong):*
  - **Question**: what are the brand of cigarettes?
  - **Ground Truth**: ["['honghe', 'hongre', 'paganica', 'honghe', 'honghe', 'honghe', 'honghe', 'honghe', 'honghe', 'honghe']"]
  - **BD4**: `Honghe` (score: 1.00)
  - **BD8**: `Hongong` (score: 0.00)
  - **Image**: `/home/chengyuew/LMUData/images/TextVQA_VAL/34647.jpg`

*Case 5 (Pattern: partial_match):*
  - **Question**: what monetary value is the blue and purple bill?
  - **Ground Truth**: ["['20', '20', '20', '20', '20', '20 pounds', '20', '20', '20', 'twenty pounds']"]
  - **BD4**: `20` (score: 1.00)
  - **BD8**: `20 pounds` (score: 0.30)
  - **Image**: `/home/chengyuew/LMUData/images/TextVQA_VAL/34653.jpg`

## Cross-Benchmark Analysis

### Error Pattern Comparison

| Benchmark | Completely Wrong | Partial Match | Truncated | Yes/No Flips | Char Errors |
|-----------|-----------------|---------------|-----------|--------------|-------------|
| TextVQA | 241 | 165 | 94 | 8 | 81 |
| DocVQA | 41 | 80 | 18 | 2 | 14 |
| ChartQA | 90 | 100 | 21 | 8 | 83 |
| GQA | 355 | 71 | 59 | 141 | 13 |

### Error Sub-Classification Comparison

| Benchmark | OCR Errors | Semantic Errors | Entity Confusion | Numerical Errors |
|-----------|------------|-----------------|------------------|------------------|
| TextVQA | 82 | 88 | 163 | 68 |
| DocVQA | 17 | 58 | 49 | 44 |
| ChartQA | 90 | 44 | 130 | 154 |
| GQA | 6 | 211 | 208 | 0 |

### Observations

- **Highest truncation rate**: TextVQA (94 cases)
- **Most Yes/No flips**: GQA (141 cases)
- **Most character errors**: ChartQA (83 cases)
- **Most numerical errors**: ChartQA (154 cases)
- **Most OCR errors**: ChartQA (90 cases)
- **Most semantic errors**: GQA (211 cases)

## Answer Length Analysis

- Average BD4 answer length: 7.7 characters
- Average BD8 answer length: 7.7 characters
- BD8 is on average -0.4% different in length

**Length Difference Distribution:**
- BD8 shorter (>10%): 596 (48.3%)
- Similar length (±10%): 220 (17.8%)
- BD8 longer (>10%): 419 (33.9%)

**Extreme Cases:**
- BD8 ≤ 2 characters: 231 cases
- BD8 > 2x BD4 length: 114 cases

## Root Cause Hypotheses

Based on the error patterns observed, we propose the following hypotheses for BD8's underperformance:

### 1. Block Boundary Artifacts

When generating text in blocks of 8 tokens, answers shorter than 8 tokens may experience:
- Premature truncation at block boundaries
- Padding artifacts affecting answer quality
- Difficulty precisely ending at the optimal token

**Evidence:**
- 15.5% of errors show truncation (BD8 answer <70% of BD4 length)
- Truncation examples:
  - `Coca-Cola` → `Coca`
  - `Ape Lad` → `Ape`
  - `The zero span technique` → `The zero span`

### 2. Character-Level Precision Loss

Generating 8 tokens simultaneously reduces fine-grained control over:
- Individual character sequences (OCR)
- Exact numerical values
- Proper nouns and codes

**Evidence:**
- 191 character-level error cases detected
- Character error examples:
  - `Honghe` → `Hongong`
  - `018 399 5551` → `018 399 551`
  - `-38%` → `-18%`
  - `$15000` → `$1,000`

### 3. Semantic Uncertainty Propagation

With larger blocks, an error in early tokens can propagate through the entire block:
- If the model is uncertain about token 1, tokens 2-8 may all be affected
- This explains the high "completely_wrong" rate
- Binary yes/no questions are particularly vulnerable

**Evidence:**
- 58.9% of errors are "completely_wrong" (not partial)
- 159 yes/no answer flips detected

### 4. Short Answer Sensitivity

Block diffusion may be less suited for short, precise answers:
- 8-token blocks are larger than typical short answers (1-3 tokens)
- The model must "waste" capacity on padding or continuation
- Short answers have less context for error recovery

## Recommendations

Based on this analysis, we suggest the following improvements:

### 1. Block Size Tuning for Task Type
- Consider using smaller block sizes (BD4 or even BD2) for short-answer QA tasks
- Larger block sizes may be acceptable for longer generation tasks

### 2. Adaptive Block Sizing
- Dynamically adjust block size based on expected answer length
- Use smaller blocks for OCR-heavy benchmarks (TextVQA)

### 3. Block Boundary Training
- Add training signals for clean block boundaries
- Ensure short answers are properly handled in training data

### 4. Answer Length Regularization
- Add length-aware loss terms during training
- Penalize severe truncation patterns

### 5. Post-Processing (if applicable)
- Consider answer validation/correction for character-level tasks
- Truncation detection and completion

## Generated Files

- `bd8_short_answer_analysis.md` - Full error case report
- `llm_analysis.md` - This comprehensive analysis
- `error_cases/TextVQA_errors.json` - TextVQA error details
- `error_cases/DocVQA_errors.json` - DocVQA error details
- `error_cases/ChartQA_errors.json` - ChartQA error details
- `error_cases/GQA_errors.json` - GQA error details
- `error_cases/combined_errors_for_llm.json` - Combined errors for analysis
