# /export-notebooklm Cursor Command

## Context & Goal
Generate a comprehensive, structured research report (`notebooklm_report.md`) specifically designed for **NotebookLM** ingestion. The goal is to allow NotebookLM to generate high-quality slides, podcasts, and summaries about the Qwen2.5-VL MDM (Masked Diffusion Model) experiments.

NotebookLM thrives on:
1.  **Clear Narratives**: Problem -> Solution -> Results -> Trade-offs.
2.  **Structured Data**: Markdown tables comparing Baseline vs. Variants.
3.  **Explicit Context**: Definitions of specialized terms (TPS, Block Causal).
4.  **"Insights" Sections**: Pre-digested conclusions (e.g., "Why did CoT fail?").
5.  **Complete Scoreboards**: The comprehensive table must include **Avg**.
6.  **Score/TPS Cells**: In the full benchmark matrix, each dataset cell must use `score/TPS` (e.g., `80.8/1.41`).

## Part 1: Data Gathering (Automated)

### 1.1 Identify Experiments
Target the following core experiments (unless overridden):
1.  **Baseline**: `Qwen2.5-VL-3B-Instruct` (Standard AR)
2.  **Exp A (bd8)**: `sharegpt4v_8node_bsz256_anneal_bd8_block_causal_lr_1e-6_unfreeze_vision`
3.  **Exp B (bd16)**: `sharegpt4v_8node_bsz256_anneal_bd16_block_causal_lr_1e-6_unfreeze_vision`
4.  **Exp C (Finetune)**: `finetune_qwen2.5-VL-3B-bd-size-32-4k-always-mask-end-0`

### 1.2 Gather Metrics
For each experiment, extract:
-   **Avg Score**: From `checkpoint_results.md` or `summary.md`.
-   **TPS (Tokens Per Step)**: From `tokens_per_step_stats.json` (Calculate Average).
-   **Reasoning Gap**: Specifically extract `MMMU_Pro_V_COT` scores (Critical for failure analysis).
-   **Speedup Ratio**: Calculate `TPS / 1.0` (Since AR TPS is 1 token/step by definition in MDM context).

## Part 2: Execution Script (Python)

Use this script to **aggregate** metrics and generate the full report. It reads
`checkpoint_results.md` (for scores/Avg) and `tokens_per_step_stats.json` (for TPS),
then writes the full matrix with **`score/TPS` cells**.

```python
import json
from pathlib import Path

VLM_DIR = Path("third_party/VLMEvalKit")
if not VLM_DIR.exists():
    VLM_DIR = Path("VLMEvalKit_local")

BENCHMARKS = [
    "AI2D_TEST", "ChartQA_TEST", "DocVQA_VAL", "GQA_TestDev_Balanced",
    "MMBench_DEV_EN_V11", "MMMU_DEV_VAL", "MMMU_Pro_V_COT",
    "MathVista_MINI", "POPE", "RealWorldQA", "SEEDBench2_Plus", "TextVQA_VAL"
]
SHORT_NAMES = {
    "AI2D_TEST": "AI2D", "ChartQA_TEST": "ChartQA", "DocVQA_VAL": "DocVQA",
    "GQA_TestDev_Balanced": "GQA", "MMBench_DEV_EN_V11": "MMBench",
    "MMMU_DEV_VAL": "MMMU", "MMMU_Pro_V_COT": "MMMU_Pro",
    "MathVista_MINI": "MathVista", "POPE": "POPE", "RealWorldQA": "RealWorld",
    "SEEDBench2_Plus": "SEED", "TextVQA_VAL": "TextVQA"
}

FALLBACK_SCORES = {
    "Baseline (AR)": {
        "AI2D_TEST": 80.8, "ChartQA_TEST": 84.0, "DocVQA_VAL": 93.1,
        "GQA_TestDev_Balanced": 59.0, "MMBench_DEV_EN_V11": 76.9,
        "MMMU_DEV_VAL": 47.3, "MMMU_Pro_V_COT": 26.3,
        "MathVista_MINI": 62.3, "POPE": 86.2, "RealWorldQA": 65.1,
        "SEEDBench2_Plus": 68.6, "TextVQA_VAL": 79.1
    }
}
FALLBACK_AVG = {"Baseline (AR)": 71.9}
FALLBACK_TPS = {"Baseline (AR)": 1.00}

EXPERIMENTS = [
    ("Baseline (AR)", "Qwen2.5-VL-3B-Instruct", True),
    ("MDM (bd8)", "sharegpt4v_8node_bsz256_anneal_bd8_block_causal_lr_1e-6_unfreeze_vision", False),
    ("MDM (bd16)", "sharegpt4v_8node_bsz256_anneal_bd16_block_causal_lr_1e-6_unfreeze_vision", False),
    ("LLM (bd32)", "finetune_qwen2.5-VL-3B-bd-size-32-4k-always-mask-end-0", False),
]

def parse_best_row(lines):
    header, rows = [], []
    for line in lines:
        if "|" not in line:
            continue
        cols = [c.strip() for c in line.split("|") if c.strip()]
        if not cols:
            continue
        if not header and "AI2D" in cols:
            header = cols
            continue
        if header and len(cols) == len(header):
            rows.append({h: cols[i] for i, h in enumerate(header)})
    best, best_avg = {}, -1
    for row in rows:
        try:
            avg = float(row.get("Avg", -1))
        except Exception:
            avg = -1
        if avg > best_avg:
            best_avg, best = avg, row
    return header, best

def get_scores(exp_name, label, is_baseline):
    scores = {k: "-" for k in BENCHMARKS}
    avg = "-"
    if is_baseline:
        path = Path("eval_outputs/vlmeval_all/summary.md")
    else:
        path = VLM_DIR / "output_models" / exp_name / "checkpoint_results.md"

    if path.exists():
        header, best = parse_best_row(path.read_text().splitlines())
        for h in header:
            if h == "Avg":
                try: avg = float(best[h])
                except Exception: pass
            if h in BENCHMARKS:
                try: scores[h] = float(best[h])
                except Exception: pass

    fallback = FALLBACK_SCORES.get(label, {})
    for k, v in fallback.items():
        if scores[k] == "-":
            scores[k] = v
    if avg == "-":
        avg = FALLBACK_AVG.get(label, "-")
    return scores, avg

def get_tps(exp_name, label, is_baseline):
    if is_baseline:
        return FALLBACK_TPS.get(label, 1.0)
    base = Path(f"output_models/{exp_name}/eval_vlmeval")
    total_tokens = total_steps = 0
    if base.exists():
        for f in base.rglob("tokens_per_step_stats.json"):
            try:
                d = json.loads(f.read_text())
                total_tokens += int(d.get("total_tokens", 0))
                total_steps += int(d.get("total_steps", 0))
            except Exception:
                pass
    return round(total_tokens / total_steps, 2) if total_steps else FALLBACK_TPS.get(label, 1.0)

rows = []
for label, exp, is_baseline in EXPERIMENTS:
    scores, avg = get_scores(exp, label, is_baseline)
    tps = get_tps(exp, label, is_baseline)
    rows.append({"label": label, "avg": avg, "tps": tps, "scores": scores})

cols = ["Model", "Avg"] + [SHORT_NAMES[b] for b in BENCHMARKS]
table = ["| " + " | ".join(cols) + " |",
         "| " + " | ".join([":---" if i == 0 else ":---:" for i in range(len(cols))]) + " |"]
for row in rows:
    tps_str = f"{row['tps']:.2f}"
    line = [f"**{row['label']}**", str(row["avg"]) ]
    for b in BENCHMARKS:
        line.append(f"{row['scores'][b]}/{tps_str}")
    table.append("| " + " | ".join(line) + " |")

report = Path("notebooklm_report.md")
text = report.read_text()
start = text.find("## 4. Comprehensive Data Matrix")
if start != -1:
    report.write_text(text[:start]
        + "## 4. Comprehensive Data Matrix (The Hard Numbers)\n\n"
        + "All dataset cells use `score/TPS` formatting (e.g., `80.8/1.41`).\n\n"
        + "\n".join(table) + "\n")
```

## Part 3: Report Template (Markdown)

The output file `notebooklm_report.md` MUST follow this structure:

```markdown
# Research Report: Accelerating Qwen2.5-VL via Masked Diffusion Models (MDM)

## 1. Executive Summary
-   **Objective**: Accelerate VLM inference using Block Causal Masked Diffusion without compromising capability.
-   **Key Result**: The **bd8 (Block Size 8)** configuration offers the best trade-off, achieving **~1.4x speedup** with acceptable performance loss (-X% Avg), whereas larger blocks (bd16) degrade reasoning significantly.
-   **Critical Finding**: Complex Reasoning (MMMU Pro CoT) is the most sensitive metric to diffusion acceleration, showing a ~30% relative drop compared to baseline.

## 2. Experimental Setup
-   **Model**: Qwen2.5-VL-3B-Instruct
-   **Method**: Fast_dLLM (MDM with Block Causal Masking)
-   **Baselines**:
    -   Original Autoregressive (AR)
    -   Standard Finetuning (bd32)

## 3. Comprehensive Results Table

| Model Variant | Block Size | Avg Score | TPS (Speedup) | MMMU Pro CoT (Reasoning) | GQA (Perception) |
| :--- | :---: | :---: | :---: | :---: | :---: |
| **Baseline (AR)** | 1 | **[Baseline_Score]** | 1.00x | **[Baseline_CoT]** | [Baseline_GQA] |
| MDM (bd8) | 8 | [Score] | [TPS]x | [Score] | [Score] |
| MDM (bd16) | 16 | [Score] | [TPS]x | [Score] | [Score] |
| Finetune | 32 | [Score] | [TPS]x | [Score] | [Score] |

## 4. Deep Dive Analysis

### 4.1 The Speed-Reasoning Trade-off
*Insight*: As Block Size increases (8 -> 16 -> 32), TPS improves, but Chain-of-Thought (CoT) capabilities degrade disproportionately.
-   **bd8**: Retains [X]% of reasoning ability.
-   **bd16**: Retains [Y]% of reasoning ability.
-   **Hypothesis**: Block decoding disrupts the sequential dependency required for complex logic steps in MMMU Pro.

### 4.2 Speculative Decoding Impact
*(If data available from A/B tests)*
-   Using Speculative Decoding with MDM [improved/degraded] performance by...

## 5. Failure Analysis: The "MMMU Pro" Case
The baseline Qwen2.5-VL achieves **26.3%** on MMMU Pro CoT.
-   MDM bd8 drops to **18.6%**.
-   This indicates that while *perception* tasks (ChartQA, DocVQA) are robust to diffusion, *reasoning* is fragile.

## 6. Conclusion & Next Steps
-   **Recommendation**: Deploy **bd8** for general-purpose usage (1.4x faster).
-   **Future Work**: Investigate "Hybrid Decoding" (AR for CoT, MDM for description) to fix the MMMU Pro gap.

---
**[PROMPT FOR NOTEBOOKLM]**
*Paste the following into NotebookLM chat to generate your presentation:*

"Act as a Senior Research Scientist at NVIDIA/Alibaba. Based on this report, generate a slide deck outline titled 'Accelerating VLMs: The Trade-off Between Speed and Reasoning'.
Slide 1: Title & Hook (The cost of speed).
Slide 2: The Methodology (Block Causal MDM).
Slide 3: The Scoreboard (Compare bd8 vs Baseline).
Slide 4: The 'Reasoning Tax' (Deep dive into why MMMU Pro scores dropped).
Slide 5: Conclusion: Why bd8 is the 'Sweet Spot'.
Focus on the data in Section 3 and the insights in Section 4."
```

## Part 4: Execution Logic

1.  **Search**: Use `glob_file_search` / `read_file` to find the exact numbers for the best checkpoints identified in chat history.
    -   *Hint*: Use the "Best Checkpoint" steps identified in previous `/show-eval` calls.
2.  **Compile**: Fill in the brackets `[...]` in the template above.
3.  **Write**: Save to `notebooklm_report.md`.
4.  **Notify**: Tell the user the file is ready and to drag it into NotebookLM.
