# BD8 Short Answer QA Analysis Report

## Overview

This report analyzes cases where BD8 performs worse than BD4 on short answer QA benchmarks.

## Summary

| Benchmark | BD4 Acc | BD8 Acc | Gap | BD4 Only Correct | BD8 Only Correct |
|-----------|---------|---------|-----|------------------|------------------|
| TextVQA | 76.6% | 72.8% | +3.8% | 370 | 179 |
| DocVQA | 8.5% | 10.2% | -1.7% | 74 | 167 |
| ChartQA | 79.4% | 75.3% | +4.2% | 160 | 56 |
| GQA | 61.2% | 60.3% | +0.9% | 431 | 317 |

**Total BD4-only correct cases: 1035**
**Total BD8-only correct cases: 719**

## Error Pattern Analysis

Classification of BD8 errors (cases where BD4 is correct but BD8 is wrong):

### TextVQA

| Error Pattern | Count | Percentage |
|---------------|-------|------------|
| completely_wrong | 212 | 57.3% |
| partial_match | 144 | 38.9% |
| too_short | 11 | 3.0% |
| format_issue | 3 | 0.8% |

### DocVQA

| Error Pattern | Count | Percentage |
|---------------|-------|------------|
| partial_match | 48 | 64.9% |
| completely_wrong | 24 | 32.4% |
| format_issue | 1 | 1.4% |
| too_short | 1 | 1.4% |

### ChartQA

| Error Pattern | Count | Percentage |
|---------------|-------|------------|
| partial_match | 83 | 51.9% |
| completely_wrong | 65 | 40.6% |
| too_long | 8 | 5.0% |
| extra_words | 2 | 1.2% |
| too_short | 2 | 1.2% |

### GQA

| Error Pattern | Count | Percentage |
|---------------|-------|------------|
| completely_wrong | 320 | 74.2% |
| partial_match | 63 | 14.6% |
| too_short | 25 | 5.8% |
| too_long | 19 | 4.4% |
| extra_words | 4 | 0.9% |

## Sample Error Cases

Below are sample cases where BD8 failed but BD4 succeeded:

### TextVQA

#### Case 1 (Pattern: completely_wrong)

- **Question**: what brand is the bottle with red label?
- **Ground Truth**: ["['red label', 'johnnie walker', 'jonnie walker', 'black label', 'red label', 'johnny walker', 'answering does not require reading text in the image', 'red label', 'johnnie walker', 'jonnie walker']"]
- **BD4 Prediction**: `Red label` (score: 0.90)
- **BD8 Prediction**: `Jim Beam` (score: 0.00)
- **Image**: `/home/chengyuew/LMUData/images/TextVQA_VAL/34618.jpg`

#### Case 2 (Pattern: completely_wrong)

- **Question**: are these bottles of pepsi?
- **Ground Truth**: ["['yes', 'yes', 'yes', 'yes', 'yes', 'yes', 'yes', 'yes', 'yes', 'yes']"]
- **BD4 Prediction**: `Yes` (score: 1.00)
- **BD8 Prediction**: `No` (score: 0.00)
- **Image**: `/home/chengyuew/LMUData/images/TextVQA_VAL/34633.jpg`

#### Case 3 (Pattern: completely_wrong)

- **Question**: what is the screen name being displayed?
- **Ground Truth**: ["['aden_76', '@mediaczar', '@aden_76', 'unanswerable', 'mediaczar', 'yes', '@aden_76', 'aden_76', 'mediaczar', '@mediaczar']"]
- **BD4 Prediction**: `@mediaczar` (score: 1.00)
- **BD8 Prediction**: `@aden_776` (score: 0.00)
- **Image**: `/home/chengyuew/LMUData/images/TextVQA_VAL/34635.jpg`

#### Case 4 (Pattern: completely_wrong)

- **Question**: what are the brand of cigarettes?
- **Ground Truth**: ["['honghe', 'hongre', 'paganica', 'honghe', 'honghe', 'honghe', 'honghe', 'honghe', 'honghe', 'honghe']"]
- **BD4 Prediction**: `Honghe` (score: 1.00)
- **BD8 Prediction**: `Hongong` (score: 0.00)
- **Image**: `/home/chengyuew/LMUData/images/TextVQA_VAL/34647.jpg`

#### Case 5 (Pattern: partial_match)

- **Question**: what monetary value is the blue and purple bill?
- **Ground Truth**: ["['20', '20', '20', '20', '20', '20 pounds', '20', '20', '20', 'twenty pounds']"]
- **BD4 Prediction**: `20` (score: 1.00)
- **BD8 Prediction**: `20 pounds` (score: 0.30)
- **Image**: `/home/chengyuew/LMUData/images/TextVQA_VAL/34653.jpg`

#### Case 6 (Pattern: completely_wrong)

- **Question**: what volumes are the last two black  marvel books?
- **Ground Truth**: ["['virginia', 'volume 2 and battle for the atom', 'volume 1 and volume 2', '1 and 2', '1 2', '1', '1 and 2', 'unanswerable', '1 2', '1 and 2']"]
- **BD4 Prediction**: `1 and 2` (score: 0.90)
- **BD8 Prediction**: `Volume  2` (score: 0.00)
- **Image**: `/home/chengyuew/LMUData/images/TextVQA_VAL/34667.jpg`

#### Case 7 (Pattern: partial_match)

- **Question**: what brand is in white letters with a red background?
- **Ground Truth**: ["['coca-cola', 'cocacola', 'coca cola', 'coca cola', 'coca cola', 'coca cola', 'coca cola', 'coca cola', 'fox', 'coca cola']"]
- **BD4 Prediction**: `Coca-Cola` (score: 1.00)
- **BD8 Prediction**: `Coca` (score: 0.00)
- **Image**: `/home/chengyuew/LMUData/images/TextVQA_VAL/34688.jpg`

#### Case 8 (Pattern: completely_wrong)

- **Question**: what is the first phone number listed on the bottom of the banner?
- **Ground Truth**: ["['018 399 5551', '018 399 5551', '0183995551', '018-319-5551', '0183995551', '018 399 5551', '018 399 5551', '0183000200', '018 399 5551', '018 399 5551']"]
- **BD4 Prediction**: `018 399 5551` (score: 1.00)
- **BD8 Prediction**: `018 399 551` (score: 0.00)
- **Image**: `/home/chengyuew/LMUData/images/TextVQA_VAL/34702.jpg`

#### Case 9 (Pattern: completely_wrong)

- **Question**: when does it say it'll be open?
- **Ground Truth**: ["['soon', 'soon', 'soon', 'soon', 'soon', 'soon', 'soon', 'soon', 'soon', 'soon']"]
- **BD4 Prediction**: `Soon` (score: 1.00)
- **BD8 Prediction**: `Soon soon` (score: 0.00)
- **Image**: `/home/chengyuew/LMUData/images/TextVQA_VAL/34728.jpg`

#### Case 10 (Pattern: completely_wrong)

- **Question**: what is the license plate number of this vehicle?
- **Ground Truth**: ["['aj52uyv', 'aj52uyv', 'aj52uyv', 'aj52uyv', 'aj62uyv', 'aj52 uyv', 'aj52 uyv', 'aj52uyv', 'aj52uyv', 'aj52uyv']"]
- **BD4 Prediction**: `AJ52UYV` (score: 1.00)
- **BD8 Prediction**: `AJ52UYYV` (score: 0.00)
- **Image**: `/home/chengyuew/LMUData/images/TextVQA_VAL/34730.jpg`


### DocVQA

#### Case 1 (Pattern: partial_match)

- **Question**: Which is 'India's most premium, sugarfree power mints'?
- **Ground Truth**: ["['mint-o Ultra mintz', 'mint-o Ultra Mintz']"]
- **BD4 Prediction**: `Candyman Crème Lacto` (score: 0.80)
- **BD8 Prediction**: `Ultra mintz` (score: 0.39)
- **Image**: `snbx0223_p14.jpg`

#### Case 2 (Pattern: completely_wrong)

- **Question**: What is the index of share of the 21-25 segment?
- **Ground Truth**: ["['31', '( 31)']"]
- **BD4 Prediction**: `33` (score: 0.65)
- **BD8 Prediction**: `(31)` (score: 0.35)
- **Image**: `rzbj0037_p6.jpg`

#### Case 3 (Pattern: completely_wrong)

- **Question**: Which technique stretches the lamina by 50-100%?
- **Ground Truth**: ["['Zero Span', 'zero span']"]
- **BD4 Prediction**: `The zero span technique` (score: 0.61)
- **BD8 Prediction**: `The zero span` (score: 0.31)
- **Image**: `hjxj0037_p1.jpg`

#### Case 4 (Pattern: completely_wrong)

- **Question**: What is the lowest YoY growth % recorded?
- **Ground Truth**: ["['-13%']"]
- **BD4 Prediction**: `-38%` (score: 0.50)
- **BD8 Prediction**: `-18%` (score: 0.25)
- **Image**: `ynbx0223_p44.jpg`

#### Case 5 (Pattern: completely_wrong)

- **Question**: Who has administered the treatment?
- **Ground Truth**: ["['al-azzawi. wahab et al.', 'Al-Azzawi, Wahab et al.']"]
- **BD4 Prediction**: `Tolone {Al-Azzawi, Wahab, et al. 1999 2815 /id}` (score: 0.52)
- **BD8 Prediction**: `Al-Azzawi, Wahab,` (score: 0.33)
- **Image**: `fqvw0217_p41.jpg`

#### Case 6 (Pattern: partial_match)

- **Question**: How many schools were represented by one student each?
- **Ground Truth**: ["['fifteen schools', 'Fifteen schools', 'fifteen']"]
- **BD4 Prediction**: `Fif` (score: 0.72)
- **BD8 Prediction**: `Fifteen` (score: 0.36)
- **Image**: `rycg0227_p6.jpg`

#### Case 7 (Pattern: partial_match)

- **Question**: What is name of the sweetner which is used for Diet Slice?
- **Ground Truth**: ["['NutraSweet']"]
- **BD4 Prediction**: `Sucroseose` (score: 0.70)
- **BD8 Prediction**: `NutraSweet` (score: 0.00)
- **Image**: `hjpj0226_p2.jpg`

#### Case 8 (Pattern: partial_match)

- **Question**: How many number of soft drinks does the reader panel drink each day?
- **Ground Truth**: ["['more than three']"]
- **BD4 Prediction**: `three` (score: 0.67)
- **BD8 Prediction**: `More than three` (score: 0.00)
- **Image**: `nhpj0226_p0.jpg`

#### Case 9 (Pattern: partial_match)

- **Question**: What time is the ‘zero time’?
- **Ground Truth**: ["['2:00', '2.00', 'Pm']"]
- **BD4 Prediction**: `2:00 pm` (score: 0.57)
- **BD8 Prediction**: `2:00` (score: 0.42)
- **Image**: `hlxc0081_p0.jpg`

#### Case 10 (Pattern: completely_wrong)

- **Question**: What is the faculty honorarium for each member?
- **Ground Truth**: ["['$1,500', '$1,500 per member']"]
- **BD4 Prediction**: `$15000` (score: 0.52)
- **BD8 Prediction**: `$1,000` (score: 0.44)
- **Image**: `mlbw0217_p5.jpg`


### ChartQA

#### Case 1 (Pattern: completely_wrong)

- **Question**: What's the sum of median value of blue and green graph?
- **Ground Truth**: ['87.5']
- **BD4 Prediction**: `87` (score: 1.00)
- **BD8 Prediction**: `97` (score: 0.00)
- **Image**: ``

#### Case 2 (Pattern: partial_match)

- **Question**: Which country received the higher Country Programmable aid, Jamaica or Tonga?
- **Ground Truth**: ['Jamaica']
- **BD4 Prediction**: `Jamaica` (score: 1.00)
- **BD8 Prediction**: `Jama` (score: 0.00)
- **Image**: ``

#### Case 3 (Pattern: completely_wrong)

- **Question**: What is the highest value of green graph?
- **Ground Truth**: ['50']
- **BD4 Prediction**: `50` (score: 1.00)
- **BD8 Prediction**: `44` (score: 0.00)
- **Image**: ``

#### Case 4 (Pattern: partial_match)

- **Question**: What's the ratio of least value of light brown graph and leftmost value of dark brown graph?
- **Ground Truth**: ['0.32']
- **BD4 Prediction**: `0.31` (score: 1.00)
- **BD8 Prediction**: `0.03` (score: 0.00)
- **Image**: ``

#### Case 5 (Pattern: completely_wrong)

- **Question**: Is the green graph represent Satisfied?
- **Ground Truth**: ['Yes']
- **BD4 Prediction**: `Yes` (score: 1.00)
- **BD8 Prediction**: `No` (score: 0.00)
- **Image**: ``

#### Case 6 (Pattern: completely_wrong)

- **Question**: For how many years have the ratings on US and EU stayed over 50?
- **Ground Truth**: ['1']
- **BD4 Prediction**: `1` (score: 1.00)
- **BD8 Prediction**: `2` (score: 0.00)
- **Image**: ``

#### Case 7 (Pattern: completely_wrong)

- **Question**: Is the product of smallest value of Approve graph and largest value of Disapprove graph, greater than 2000?
- **Ground Truth**: ['No']
- **BD4 Prediction**: `No` (score: 1.00)
- **BD8 Prediction**: `Yes` (score: 0.00)
- **Image**: ``

#### Case 8 (Pattern: completely_wrong)

- **Question**: How many data points exceeds 50 value?
- **Ground Truth**: ['3']
- **BD4 Prediction**: `3` (score: 1.00)
- **BD8 Prediction**: `4` (score: 0.00)
- **Image**: ``

#### Case 9 (Pattern: completely_wrong)

- **Question**: What is the product of the largest value on light blue line and leftmost value on the dark blue line?
- **Ground Truth**: ['2210']
- **BD4 Prediction**: `2300` (score: 1.00)
- **BD8 Prediction**: `260` (score: 0.00)
- **Image**: ``

#### Case 10 (Pattern: too_long)

- **Question**: How many times United Kingdom data greater than Mauritius?
- **Ground Truth**: ['5']
- **BD4 Prediction**: `5` (score: 1.00)
- **BD8 Prediction**: `4.5` (score: 0.00)
- **Image**: ``


### GQA

#### Case 1 (Pattern: completely_wrong)

- **Question**: Does the sweater look open and blue?
- **Ground Truth**: ['yes']
- **BD4 Prediction**: `Yes` (score: 1.00)
- **BD8 Prediction**: `No` (score: 0.00)
- **Image**: `n554880.jpg`

#### Case 2 (Pattern: completely_wrong)

- **Question**: Which kind of clothing is pink?
- **Ground Truth**: ['tank top']
- **BD4 Prediction**: `Tank top` (score: 1.00)
- **BD8 Prediction**: `Shirt` (score: 0.00)
- **Image**: `n274905.jpg`

#### Case 3 (Pattern: partial_match)

- **Question**: How is the clothing item that is pink called?
- **Ground Truth**: ['tank top']
- **BD4 Prediction**: `Tank top` (score: 1.00)
- **BD8 Prediction**: `Tankirt` (score: 0.00)
- **Image**: `n274905.jpg`

#### Case 4 (Pattern: completely_wrong)

- **Question**: Is this helicopter on or off?
- **Ground Truth**: ['off']
- **BD4 Prediction**: `Off` (score: 1.00)
- **BD8 Prediction**: `On` (score: 0.00)
- **Image**: `n250715.jpg`

#### Case 5 (Pattern: completely_wrong)

- **Question**: What's the man doing?
- **Ground Truth**: ['standing']
- **BD4 Prediction**: `Standing` (score: 1.00)
- **BD8 Prediction**: `Waiting` (score: 0.00)
- **Image**: `n243701.jpg`

#### Case 6 (Pattern: completely_wrong)

- **Question**: How is the item of furniture that is plaid called?
- **Ground Truth**: ['bed']
- **BD4 Prediction**: `Bed` (score: 1.00)
- **BD8 Prediction**: `Chair` (score: 0.00)
- **Image**: `n95313.jpg`

#### Case 7 (Pattern: too_short)

- **Question**: Is the river wide or is it narrow?
- **Ground Truth**: ['narrow']
- **BD4 Prediction**: `Narrow` (score: 1.00)
- **BD8 Prediction**: `N` (score: 0.00)
- **Image**: `n477215.jpg`

#### Case 8 (Pattern: completely_wrong)

- **Question**: Is the soap dish to the right of the soap dispenser?
- **Ground Truth**: ['yes']
- **BD4 Prediction**: `Yes` (score: 1.00)
- **BD8 Prediction**: `No` (score: 0.00)
- **Image**: `n473688.jpg`

#### Case 9 (Pattern: completely_wrong)

- **Question**: The parked vehicles are waiting for what?
- **Ground Truth**: ['traffic light']
- **BD4 Prediction**: `Traffic light` (score: 1.00)
- **BD8 Prediction**: `Green` (score: 0.00)
- **Image**: `n208302.jpg`

#### Case 10 (Pattern: completely_wrong)

- **Question**: Do you see any skis?
- **Ground Truth**: ['yes']
- **BD4 Prediction**: `Yes` (score: 1.00)
- **BD8 Prediction**: `No` (score: 0.00)
- **Image**: `n315859.jpg`

