# DFM_Qwen2.5-VL-3B Evaluation Guide

## Overview

This directory contains scripts for evaluating the DFM_Qwen2.5-VL-3B model. This model is based on Qwen2.5-VL-3B-Instruct and has been fine-tuned on text data using the Block Diffusion (DFM) method.

## File Descriptions

### 1. `eval_dvlm_batch.py`
- **Purpose**: Dedicated evaluation script for DFM_Qwen2VL.
- **Features**: 
  - Supports a configurable tokenizer path.
  - Defaults to `bd_size=128` (consistent with training).
  - Registered as the "dfm_qwen2vl" model type.
  - Fixes hardcoding issues from the original evaluation script.

### 2. `run_batch_eval_dvlm_3b_slurm.sh`
- **Purpose**: SLURM batch evaluation submission script.
- **Default Configuration**:
  - Task: minerva_math
  - Batch size: 16
  - bd_size: 128
  - small_block_size: 16
  - threshold: 0.95
- **Supported Tasks**: minerva_math, gsm8k, mmlu, humaneval, mbpp

### 3. `check_dvlm_training_progress.sh`
- **Purpose**: Checks training progress and available checkpoints.
- **Features**:
  - Lists all saved checkpoints.
  - Displays the last few lines of the training log.
  - Checks the error log.
  - Provides evaluation recommendations.

## Usage Workflow

### Step 1: Check Training Progress

```bash
bash shiyi_scripts/check_dvlm_training_progress.sh
```

Example output:
```
========================================
Checking DFM_Qwen2.5-VL-3B Training Progress
========================================
✅ Output directory: output_models/dvlm_qwen2_5_vl_3b_text_sft

📦 Available Checkpoints:
  ✅ checkpoint-500 (complete)
  ✅ checkpoint-1000 (complete)
  
🎯 Latest Checkpoint: checkpoint-1000
   Path: output_models/dvlm_qwen2_5_vl_3b_text_sft/checkpoint-1000
```

### Step 2: Update Checkpoint Path in Evaluation Script

Edit `run_batch_eval_dvlm_3b_slurm.sh`, update line 21:

```bash
# From
model_path=output_models/dvlm_qwen2_5_vl_3b_text_sft/checkpoint-500

# To (use the latest checkpoint)
model_path=output_models/dvlm_qwen2_5_vl_3b_text_sft/checkpoint-1000
```

### Step 3: (Optional) Adjust Evaluation Parameters

Modify parameters in `run_batch_eval_dvlm_3b_slurm.sh` as needed:

```bash
# Task selection
task=minerva_math  # or gsm8k, mmlu, humaneval, etc.

# Performance parameters
batch_size=16      # Adjust based on GPU memory
threshold=0.95     # Denoising threshold (0.9-0.99)

# Block Diffusion parameters (usually no need to modify)
bd_size=128              # Must match training setting
small_block_size=16      # Sub-block size
use_block_cache=False    # Whether to use block cache
```

### Step 4: Submit Evaluation Job

```bash
sbatch shiyi_scripts/run_batch_eval_dvlm_3b_slurm.sh
```

Check job status:
```bash
squeue -u $USER
```

View real-time logs:
```bash
tail -f workdir/eval_dvlm_3b_out.log
```

## Key Configuration Parameters

### 1. bd_size (Block Diffusion Size)
- **Training config**: 128
- **Evaluation config**: Must match training = 128
- **Purpose**: Basic block size for Block Diffusion

### 2. small_block_size
- **Recommended values**: 16 or 32
- **Purpose**: Sub-block size during inference, affects parallelism and speed
- **Trade-off**: Larger values may be faster but slightly lower accuracy

### 3. threshold
- **Recommended range**: 0.90 - 0.99
- **Purpose**: Confidence threshold during denoising
- **Trade-offs**: 
  - Higher values (0.95-0.99): More conservative, higher quality but slower
  - Lower values (0.90-0.93): More aggressive, faster but potentially lower quality

### 4. use_block_cache
- **Default**: False
- **Purpose**: Cache block-level KV cache
- **Recommendation**: Use False for initial evaluation, try True for performance optimization

## Key Differences from Original Evaluation Script

| Item | Original Script (chengyue_script) | New Script (shiyi_scripts) |
|------|-----------------------------------|---------------------------|
| Evaluation file | eval_mdm_qwen_batch.py | eval_dvlm_batch.py |
| Model type | dfm_qwen | dfm_qwen2vl |
| bd_size | 32 | 128 |
| Tokenizer | Hardcoded to Fast_dLLM_1.5B | Configurable to Qwen2.5-VL-3B |
| small_block_size | 8 | 16 |

## Troubleshooting

### Issue 1: "Model path does not exist"
**Cause**: Checkpoint not yet saved or incorrect path

**Solution**:
```bash
# Check training progress
bash shiyi_scripts/check_dvlm_training_progress.sh

# Update checkpoint number in script
```

### Issue 2: "CUDA out of memory"
**Cause**: batch_size too large

**Solution**:
```bash
# Reduce batch_size
batch_size=8  # or smaller
```

### Issue 3: Evaluation is very slow
**Cause**: threshold too high or not using block cache

**Solution**:
```bash
# Lower threshold
threshold=0.90

# Enable block cache (experimental)
use_block_cache=True
block_cache_refresh_interval=8
```

### Issue 4: Tokenizer loading fails
**Cause**: Network issues or Qwen tokenizer not installed

**Solution**:
```bash
# Method 1: Manually download tokenizer
# Visit https://huggingface.co/Qwen/Qwen2.5-VL-3B-Instruct
# Download locally and update tokenizer_path

# Method 2: Use mirror (if available)
export HF_ENDPOINT=https://hf-mirror.com
```

## Evaluation Results Location

- **Main log**: `workdir/eval_dvlm_3b_out.log`
- **Error log**: `workdir/eval_dvlm_3b_error.log`
- **Detailed results**: `workdir/slurm_log/dvlm_3b_eval_results/`

## Next Steps

### Phase 1: Text-only Evaluation (Current)
- ✅ Evaluate text reasoning (minerva_math, gsm8k)
- ✅ Evaluate code generation (humaneval, mbpp)
- ✅ Evaluate general capabilities (mmlu, ifeval)

### Phase 2: Vision+Text Evaluation (Future)
- ⏳ Activate vision input
- ⏳ Evaluate VQA tasks
- ⏳ Evaluate multimodal reasoning

## References

- **Training script**: `shiyi_scripts/finetune_dvlm_qwen2_5_vl_3B_slurm.sh`
- **Model code**: `src/models/Qwen2_5VL_DFM/`
- **Original evaluation**: `chengyue_script/run_batch_eval_slurm.sh`

---

**Created**: 2025-10-09  
**Last updated**: 2025-10-09
