# DFM_Qwen2.5-VL-3B Training Status Report

**Date**: 2025-10-09  
**Model**: DFM_Qwen2.5-VL-3B  
**Status**: ❌ Training Failed - Bug Fix Required

---

## Issue Summary

The training job is failing with an `AttributeError` when trying to load the Qwen2.5-VL base model.

### Error Details

```python
AttributeError: 'Qwen2_5_VLModel' object has no attribute 'model'
```

**Location**: `src/models/Qwen2_5VL_DFM/modeling_qwen2vl_dfm.py`, line 106

**Problematic code**:
```python
if hasattr(qwen_vl.model, 'visual'):  # ❌ Wrong!
    self.vision_model = qwen_vl.model.visual
```

**Root cause**: The code assumes Qwen2.5-VL has a `.model.visual` structure, but the actual structure is `.visual` (vision tower is directly at the top level, not under a `.model` attribute).

---

## What Was Completed

### ✅ Evaluation Scripts Created

1. **`eval_dvlm_batch.py`**
   - Dedicated evaluation script for DFM_Qwen2VL
   - Fixes tokenizer path issues
   - Properly configured for bd_size=128

2. **`run_batch_eval_dvlm_3b_slurm.sh`**
   - SLURM submission script
   - Pre-configured for minerva_math, gsm8k, etc.
   - Handles checkpoint path management

3. **`check_dvlm_training_progress.sh`**
   - Monitors training progress
   - Lists available checkpoints
   - Displays training logs

4. **`README_DVLM_EVAL.md`**
   - Complete usage guide in English
   - Troubleshooting tips
   - Configuration explanations

---

## Required Fix

### File: `src/models/Qwen2_5VL_DFM/modeling_qwen2vl_dfm.py`

**Lines 105-120** need to be corrected:

#### Current (Broken) Code:
```python
# Extract and freeze vision components (keep in memory)
# Qwen2.5-VL has 'visual' directly at top level (not 'model.visual')
if hasattr(qwen_vl, 'visual'):
    self.vision_model = qwen_vl.visual
    for param in self.vision_model.parameters():
        param.requires_grad = False
    logger.info("Vision tower extracted and frozen")
```

**Issue**: The comment says "directly at top level" but the check was previously looking at `qwen_vl.model.visual` (which was already commented out but the error suggests it's still in the cached version).

#### Fix Required:

The code looks correct in the source file, but the error suggests the **cached version** in `~/.cache/huggingface/modules/transformers_modules/Qwen2_5VL_DFM/` still has the old buggy code.

**Solution**: Clear the HuggingFace cache and restart training.

---

## Action Items

### Priority 1: Fix Training (REQUIRED)

```bash
# Step 1: Clear the cached model code
rm -rf ~/.cache/huggingface/modules/transformers_modules/Qwen2_5VL_DFM/

# Step 2: Verify the source code is correct
cat src/models/Qwen2_5VL_DFM/modeling_qwen2vl_dfm.py | grep -A 5 "hasattr(qwen_vl"

# Step 3: Resubmit training job
sbatch shiyi_scripts/finetune_dvlm_qwen2_5_vl_3B_slurm.sh
```

### Priority 2: Monitor Training

```bash
# Check training status
squeue -u $USER

# Monitor logs
tail -f log/dvlm_qwen2_5_vl_3b_text_sft/train.log

# Check for checkpoints
bash shiyi_scripts/check_dvlm_training_progress.sh
```

### Priority 3: Run Evaluation (After Training Completes)

```bash
# Wait for first checkpoint (checkpoint-500)
# Then update model_path in run_batch_eval_dvlm_3b_slurm.sh
# Submit evaluation
sbatch shiyi_scripts/run_batch_eval_dvlm_3b_slurm.sh
```

---

## Training Configuration Summary

| Parameter | Value | Notes |
|-----------|-------|-------|
| Base Model | Qwen/Qwen2.5-VL-3B-Instruct | Vision + Language model |
| Training Method | Block Diffusion (DFM) | MDM with bd_size=128 |
| Nodes | 4 | SLURM multi-node |
| GPUs per Node | 8 | Total 32 GPUs |
| Batch Size per GPU | 4 | Global batch = 128 |
| Learning Rate | 2e-5 | |
| Epochs | 1 | |
| Save Steps | 500 | First checkpoint at step 500 |
| Data | nemotron_no_thinking | Text-only conversations |
| Components | Text LLM only | Vision frozen (not used) |

---

## Expected Training Timeline

- **First checkpoint**: ~500 steps (depends on dataset size and GPU speed)
- **Total duration**: ~4 hours (SLURM time limit)
- **Checkpoints**: 500, 1000, 1500, ... (every 500 steps)

---

## Next Steps After Fix

1. ✅ Clear HuggingFace cache
2. ✅ Verify source code
3. ⏳ Resubmit training job
4. ⏳ Wait for checkpoint-500
5. ⏳ Run evaluation on checkpoint-500
6. ⏳ Compare with baseline models

---

## Model Architecture Notes

### DFM_Qwen2VL Structure

```
DFM_Qwen2VLForCausalLM
├── vision_model (frozen)           ← From Qwen2.5-VL, not used in Phase 1
├── mm_projector (frozen)           ← From Qwen2.5-VL, not used in Phase 1  
├── model (DFM_QwenModel)          ← Trainable text LLM with Block Diffusion
│   ├── embed_tokens
│   ├── layers[0..27]              ← 28 transformer layers
│   └── norm
└── lm_head                         ← Language modeling head
```

### Training Data Flow

```
Text Input → Tokenize → DFM Block Diffusion Training
                         ↓
                    Add noise to blocks
                         ↓
                    Denoise with transformer
                         ↓
                    Compute MDM loss
```

### Inference Flow (After Training)

```
Text Input → Tokenize → Initialize blocks with MASK
                         ↓
                    Iteratively denoise blocks
                         ↓
                    Output generated text
```

---

**Status**: Waiting for cache clearing and training restart.  
**ETA for first results**: ~1 hour after restart (assuming fix works)

