# Chengyue Scripts

This directory contains scripts and tools for training, evaluating, and running inference on Vision-Language Models (VLMs), specifically focused on **Diffusion VLM (dVLM)** research with Qwen2.5-VL.

## Directory Structure

```
chengyue_script/
├── dvlm/                      # Main dVLM pipeline (recommended)
│   ├── training/              # Training scripts
│   │   ├── single_node/       # Single-node training (DeepSpeed)
│   │   └── multinode/         # Multi-node training (FSDP + Slurm)
│   ├── evaluation/            # Evaluation tools
│   │   ├── lmms_eval/         # lmms-eval wrapper
│   │   └── vlmeval/           # VLMEvalKit wrapper
│   ├── inference/             # Chatbot and inference
│   ├── models/                # Model code (generation functions)
│   └── utils/                 # Utility scripts
├── nemotron/                  # Nemotron-related training scripts
│   ├── stage1/                # Stage 1 training configs
│   └── stage2/                # Stage 2 training configs
├── modeling_dfm.py            # Core DFM model implementation
├── configuration_dfm.py       # DFM model configuration
└── [legacy scripts]           # Older evaluation/training scripts
```

## Quick Start

### Training

**Single-Node MDM Training:**
```bash
bash chengyue_script/dvlm/training/single_node/train_j63_mdm.sh
```

**Multi-Node MDM Training (8 nodes):**
```bash
sbatch chengyue_script/dvlm/training/multinode/mdm/train_j63_mdm_8node.sh
```

**AR Baseline Training:**
```bash
bash chengyue_script/dvlm/training/single_node/train_sharegpt4v_vlm_ar_baseline.sh
```

### Evaluation

**Evaluate Custom Checkpoint (AR or MDM):**
```bash
python chengyue_script/dvlm/evaluation/eval_custom_model.py \
  --model_path output_models/j63_mdm/checkpoint-2000 \
  --tasks gqa,pope,textvqa,mmmu \
  --num_gpus 8
```

**Baseline Evaluation with VLMEvalKit:**
```bash
# VLMEvalKit 目录：优先 third_party/VLMEvalKit，其次 VLMEvalKit_local
VLM_DIR="third_party/VLMEvalKit"
[ -d "${VLM_DIR}" ] || VLM_DIR="VLMEvalKit_local"
cd "${VLM_DIR}"
torchrun --nproc-per-node=8 run.py \
  --model "Qwen2.5-VL-3B-Instruct" \
  --data MathVista_MINI MMMU_DEV_VAL
```

### Inference

**MDM Chatbot:**
```bash
python chengyue_script/dvlm/inference/chatbot_mdm.py \
  --model-name output_models/j63_mdm/checkpoint-2000 \
  --demo
```

## Model Types

| Type | Description | Key Args |
|------|-------------|----------|
| **AR** | Autoregressive baseline | `--mdm 0` |
| **MDM** | Masked Diffusion Model | `--mdm 1 --bd_size 8` |

Both types use `Fast_dLLM_Qwen2_5_VLForConditionalGeneration` as the base architecture.

## Datasets

| Dataset | Backend | Description |
|---------|---------|-------------|
| ShareGPT4V | `custom_multi_modal` | JSON format with image paths |
| VILA J63 | `webdataset_multi_modal` | WebDataset shards |

## Evaluation Tools

| Tool | Best For | GPT Judge Required |
|------|----------|-------------------|
| `eval_custom_model.py` | Fast_dLLM checkpoints | No |
| VLMEvalKit | MathVista, MMMU, MMBench | Yes |
| lmms-eval | GQA, POPE, TextVQA | No |

## Documentation

- [dVLM Overview](dvlm/README.md) - Architecture and concepts
- [Training Guide](dvlm/training/README.md) - Training configurations
- [Evaluation Guide](dvlm/evaluation/README.md) - Evaluation tools and benchmarks
- [Inference Guide](dvlm/inference/README.md) - Running chatbots

## Key Files

| File | Purpose |
|------|---------|
| `modeling_dfm.py` | Core DFM/MDM model implementation |
| `configuration_dfm.py` | Model configuration classes |
| `dvlm/models/generation_functions.py` | MDM sampling functions |
| `dvlm/evaluation/eval_custom_model.py` | Multi-GPU evaluation script |
| `dvlm/evaluation/monitor_checkpoints.py` | Auto-submit Slurm eval jobs |

## Environment

Requires the `shiyi-vila` conda environment:
```bash
conda activate shiyi-vila
```

Key dependencies:
- PyTorch 2.x with CUDA
- Transformers
- Flash Attention 2
- DeepSpeed (single-node) or FSDP (multi-node)

