# ExecuteGPUHandoff (GPU Slurm Node / cursor-agent CLI Skill)

## Goal
On a **GPU Slurm node** (or login node that can submit Slurm jobs), take a GUI-generated **Handoff Summary** and:
- validate environment
- run the A/B workload (or submit via Slurm)
- collect results into a predictable location
- report back to GUI with a structured summary

This skill assumes the GPU-side agent is allowed to run commands and read logs, and that code changes were already merged by the GUI-side workflow.

---

## Inputs (from GUI handoff)
You should receive a handoff containing:
- repo root path
- commit hash
- checkpoint path(s)
- exact commands for A and B
- expected output directories
- debug playbook

If any of these are missing, request only the minimum:
- repo root + checkpoint dir + A/B commands

---

## Guardrails
- Do not “improvise” benchmark names or checkpoint paths.
- Do not modify source code unless explicitly instructed (GPU side should be execution-focused).
- Prefer non-interactive execution (`--yes`, avoid prompts).
- Always capture logs and return them (paths + tail excerpts).

---

## Protocol

### Step 0: Normalize paths and commit
```bash
cd "<REPO_ROOT_ON_GPU>"
git rev-parse HEAD
```
If the commit doesn’t match the handoff, **stop** and report mismatch.

### Step 1: Environment validation (fast-fail)
Run these before any big job:

```bash
source ~/.bashrc
conda activate "<CONDA_ENV>"

python -c "import torch; print(torch.__version__); print(torch.cuda.is_available())"
nvidia-smi

python -c "import vlmeval" 2>/dev/null || true
python -c "import lmflow" 2>/dev/null || true
```

If a benchmark requires a judge:
- verify `AZURE_OPENAI_*` or `OPENAI_API_KEY` (whatever the repo expects)

### Step 2: Data availability check (no downloading surprises)
Without asking the user for dataset paths, detect likely roots:
- `echo $LMUData` / `echo $LMUDATA` if used
- list known dataset directories if documented in repo

If auto-download is expected, ensure the node has outbound access.

### Step 3: Dry-run (smoke test)
Goal: catch import errors or missing dependencies quickly.

Preferred options (pick one):
- run a single tiny task with a small sample limit (if supported)
- or run one benchmark on 1 GPU / short max tokens

If the pipeline has no `--limit`, at least verify the scripts generate configs:

```bash
# Example: generate config only (if supported)
export CFG_PATH=/tmp/vlmeval_config.json
export MODEL_PATH_ABS="<CHECKPOINT_DIR>"
export DATASETS="<ONE_TASK>"
python3 chengyue_script/dvlm/evaluation/vlmeval/generate_vlmeval_config.py
cat /tmp/vlmeval_config.json | head
```

### Step 4: Execute A then B (serial, reproducible)
Run commands exactly as provided in handoff.

Capture outputs:
- stdout/stderr to a log file
- Slurm job IDs if submitted
- output directory tree

If using Slurm:
- submit with `sbatch ...`
- record job id
- poll until done

### Step 5: Collect results
At minimum, return:
- A and B output directories
- result CSV/JSON paths
- a short metric snapshot (top-level scores)
- any latency/speed indicators if present in logs

If the repo has an aggregate script, run it and include the summary markdown path.

### Step 6: Report back to GUI (structured)
Use the template below.

---

## GPU Report-back Template (copy/paste)

```text
## GPU Run Report
- Repo_root:
- Commit:
- Hostname:
- Slurm_job_ids:
- Conda_env:
- GPUs:

## Commands executed
- A (baseline):
  ...
- B (treatment):
  ...

## Outputs
- A_output_dir:
- B_output_dir:
- Logs:
  - ...
- Result files:
  - ...

## Metrics snapshot
- A:
  - ...
- B:
  - ...

## Performance snapshot (if available)
- A time / tokens-per-sec:
- B time / tokens-per-sec:

## Issues (if any)
- Root cause:
- Evidence (log excerpt):
- Suggested fix:
```

---

## Common failure modes (and what to do)

### Missing judge key / .env
- **Symptom**: judge init fails, “requires working OPENAI API”, or `.env` missing warnings
- **Fix**: set the expected env vars (proxy or API key) in the Slurm job environment

### Missing system deps (LaTeX/Chrome/etc.)
- **Symptom**: compilation/renderer errors
- **Fix**: run on nodes with required deps, or load modules/containers as documented

### Dataset not found
- **Symptom**: missing TSV/images under LMUData
- **Fix**: set dataset root env var used by the framework, or prefetch data

### Import errors
- **Symptom**: `ImportError`/`ModuleNotFoundError`
- **Fix**: confirm `PYTHONPATH`, conda env packages, and that repo checkout matches commit




