sirhcm/tinygrad
1
0
Fork 0
forked from tinygrad/tinygrad
Code Pull Requests Activity

notes from digitalocean_mi350x (#17494)

Browse Source 
* notes from digitalocean_mi350x

* cleanup

* revert non-doc changes on digitalocean_mi350x branch
This commit is contained in:
George Hotz
2026-08-11 13:22:11 -07:00
committed by GitHub
parent a8c84ab34e
commit e11df72e0f
1 changed files with 235 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
extra/runbook_digitalocean_mi350x.md
+235
View File
@@ -0,0 +1,235 @@
# Runbook: Llama 3 8B Training on DigitalOcean MI350X
## Machine Specs
- 8x MI350X GPUs (gfx950, device ID 75b0), 288GB VRAM each
- 2TB RAM, 192 CPUs, 2TB disk
- ROCm 7.14 at `/opt/rocm` (NOT `/opt/rocm-7.1.1` like the submission scripts assume)
- Python 3.12
## Phase 1: System Setup
### 1.1 Install packages
```bash
apt-get update
apt-get install -y python3-pip python3-venv git tmux rclone clang
```
### 1.2 Install Python deps
```bash
python3 -m pip install --break-system-packages numpy tqdm wandb tiktoken sentencepiece
```
### 1.3 Install ROCm dev headers
The base image has ROCm runtime but NOT the HIP dev headers. Need:
```bash
apt-get install -y amdrocm-core-dev
```
This installs `hip/hip_runtime.h` at `/opt/rocm/core-7.14/include/hip/hip_runtime.h`.
The symlink `/opt/rocm/include` → `/opt/rocm/core-7.14/include` makes it available at `/opt/rocm/include/hip/hip_runtime.h`.
### 1.4 Configure ROCm comgr
ROCm 7.14 ships comgr 3.3 at `/opt/rocm/lib/libamd_comgr.so`. tinygrad's DLL loader needs explicit env vars to find it (it searches for `libcomgr.so*` by default, not `libamd_comgr.so*`). Set these in the run command:
```bash
export COMGR_PATH=/opt/rocm/lib/libamd_comgr.so
export COMGR_3_PATH=/opt/rocm/lib/libamd_comgr.so
```
Also add ROCm libs to ldconfig so comgr's shared library dependencies resolve:
```bash
cat > /etc/ld.so.conf.d/rocm.conf << 'EOF'
/opt/rocm/lib
/opt/rocm/lib/llvm/lib
/opt/rocm/lib/rocm_sysdeps/lib
EOF
ldconfig
```
### 1.5 Install geohot tmux config
```bash
curl -sL https://raw.githubusercontent.com/geohot/configuration/master/.tmux.conf -o ~/.tmux.conf
```
### 1.6 Reload amdgpu driver
tinygrad's HCQ backend needs `/dev/kfd` which is created by the amdgpu kernel driver.
If the driver was unloaded, reload it:
```bash
modprobe amdgpu
ls /dev/kfd # should exist
```
## Phase 2: Clone tinygrad
```bash
cd /root
git clone https://github.com/tinygrad/tinygrad.git
cd tinygrad
python3 -m pip install --break-system-packages -e .
```
## Phase 3: Download C4 Dataset
The C4 data is on the MLCommons Cloudflare R2 bucket in Megatron-LM indexed format.
```bash
rclone config create mlc-training s3 provider=Cloudflare \
access_key_id=76ea42eadb867e854061a1806220ee1e \
secret_access_key=a53625c4d45e3ca8ac0df8a353ea3a41ffc3292aa25259addd8b7dc5a6ce2936 \
endpoint=c2686074cb2caf5cbaf6d134bdba8b47.r2.cloudflarestorage.com
mkdir -p /root/datasets/c4-8b
rclone copy mlc-training:mlcommons-training-wg-public/llama3_1/datasets/c4/llama3_1_8b/ /root/datasets/c4-8b/ -P
```
Files downloaded (~85GB total, ~6 minutes):
- `c4-train.en_6_text_document.bin` (79 GB)
- `c4-train.en_6_text_document.idx` (870 MB)
- `c4-validation-91205-samples.en_text_document.bin` (159 MB)
- `c4-validation-91205-samples.en_text_document.idx` (1.8 MB)
- `LICENSE.txt`, `NOTICE.txt`
### Symlink for the submission script
The `dev_run.sh` script hardcodes `BASEDIR="/raid/datasets/c4-8b/"`. Symlink:
```bash
mkdir -p /raid/datasets
ln -s /root/datasets/c4-8b /raid/datasets/c4-8b
```
## Phase 4: wandb Login
```bash
wandb login
```
Enter API key from https://wandb.ai/authorize
## Phase 5: Run Training
### 5.1 Smoke test (beam search, 2 layers, fake data)
Always run beam first to validate the pipeline:
```bash
cd /root/tinygrad
COMGR_PATH=/opt/rocm/lib/libamd_comgr.so \
COMGR_3_PATH=/opt/rocm/lib/libamd_comgr.so \
CC=/opt/rocm/core-7.14/lib/llvm/bin/clang \
DEV=AMD:HIP \
ROCM_PATH=/opt/rocm BASEDIR=/root/datasets/c4-8b/ \
bash examples/mlperf/training_submission_v6.0/tinycorp/benchmarks/llama31_8b/implementations/tinybox_8xMI350X/dev_beam.sh
```
### 5.2 Full training run
```bash
cd /root/tinygrad
COMGR_PATH=/opt/rocm/lib/libamd_comgr.so \
COMGR_3_PATH=/opt/rocm/lib/libamd_comgr.so \
CC=/opt/rocm/core-7.14/lib/llvm/bin/clang \
DEV=AMD:HIP \
ROCM_PATH=/opt/rocm BASEDIR=/root/datasets/c4-8b/ \
WANDB=1 \
bash examples/mlperf/training_submission_v6.0/tinycorp/benchmarks/llama31_8b/implementations/tinybox_8xMI350X/dev_run.sh
```
## Environment Variable Reference
| Variable | Value | Why |
|---|---|---|
| `COMGR_PATH` | `/opt/rocm/lib/libamd_comgr.so` | tinygrad's DLL loader needs explicit path to find comgr 3.3 |
| `COMGR_3_PATH` | `/opt/rocm/lib/libamd_comgr.so` | comgr 3.x uses a separate `comgr_3` module with its own path var |
| `CC` | `/opt/rocm/core-7.14/lib/llvm/bin/clang` | System clang doesn't know gfx950; must use ROCm's bundled clang |
| `DEV` | `AMD:HIP` | Force HIPRenderer (comgr-based) over HIPCCRenderer (hipcc subprocess) |
| `ROCM_PATH` | `/opt/rocm` | Script defaults to `/opt/rocm-7.1.1` which doesn't exist |
| `BASEDIR` | `/root/datasets/c4-8b/` | Where C4 dataset was downloaded (script hardcodes `/raid/datasets/c4-8b/`) |
| `WANDB` | `1` | Enable wandb logging (off by default) |
## Architecture
| Component | Source file |
|---|---|
| Model | `examples/mlperf/models/flat_llama.py` — FlatTransformer, FP8 MXFP4 weights, fused QKV, flash attention |
| Trainer | `examples/mlperf/model_train.py` → `train_llama3()` |
| Optimizer | `examples/mlperf/optim.py` — GradAccClipAdamW, master weights, FP8 re-quant |
| LR schedule | `examples/mlperf/lr_schedulers.py` — CosineAnnealingLRWithWarmup |
| Dataloader | `examples/mlperf/dataloader.py` — Megatron-LM indexed bin format |
| ASM GEMM | `extra/gemm/cdna_asm_gemm.py` — gfx950 MFMA assembly, MXFP4 |
| Flash attention | `extra/thunder/amd/fa.py` |
| Fused kernels | `extra/llama_kernels/` — rmsnorm, silu, quantize, fused_ce |
| GPU driver | `tinygrad/runtime/ops_amd.py` — HCQ, direct KFD ioctl |
| Renderer | `tinygrad/renderer/cstyle.py` — HIPRenderer for gfx950 |
| comgr compiler | `tinygrad/runtime/support/compiler_amd.py` — HIPCompiler using comgr 3.3 |
## Troubleshooting
### `'hip/hip_runtime.h' file not found`
Install `amdrocm-core-dev`:
```bash
apt-get install -y amdrocm-core-dev
```
### `'gfx950' is not a recognized processor` + LLVM crash
System clang doesn't know gfx950. Set `CC=/opt/rocm/core-7.14/lib/llvm/bin/clang`.
### `comgr not available: try setting COMGR_PATH?`
Add ROCm libs to ldconfig and set `COMGR_PATH` and `COMGR_3_PATH`:
```bash
# /etc/ld.so.conf.d/rocm.conf should contain /opt/rocm/lib paths
ldconfig
```
### `comgr not available: try setting COMGR_3_PATH?`
comgr 3.x uses a separate module. Set `COMGR_3_PATH=/opt/rocm/lib/libamd_comgr.so` too.
### `FileNotFoundError: '/raid/datasets/c4-8b/...'`
Script hardcodes `BASEDIR`. Either symlink or edit the script:
```bash
mkdir -p /raid/datasets && ln -s /root/datasets/c4-8b /raid/datasets/c4-8b
```
### `No such file or directory: 'clang'`
Install clang: `apt-get install -y clang` (for CPU compilation).
For gfx950 HIP compilation, comgr (not clang) is used — ensure the ROCm 7.14 comgr 3.3 is properly loaded via `COMGR_PATH` and `COMGR_3_PATH`.
## Appendix: KVM Virtualization Observations
### Virtualization detection
```
$ systemd-detect-virt
kvm
$ lspci -nn | grep AMD
83:00.0 ... Device [1002:75b0]
```
CPU flags include `hypervisor`. `dmesg` shows `Hypervisor detected: KVM`.
### PCI device ID
`lspci -v` shows device ID `0x75b0` and subsystem ID `0x75a0`:
```
83:00.0 Processing accelerators: ... Device 75b0
Subsystem: ... Device 75a0
```
tinygrad's `PCIIface` in `ops_amd.py` and `hive_reset.py` did not list `0x75b0`, so the GPU was not found. Adding `0x75b0` to the device ID list in both files fixes the detection.
### amdgpu driver behavior
On first boot, amdgpu loaded and bound to all 8 GPUs. On one boot it failed to initialize:
```
[ 799.780369] amdgpu 0000:83:00.0: Failed to alloc msi vectors
[ 799.781476] amdgpu 0000:83:00.0: sw_init of IP block <vega20_ih> failed -22
[ 799.782724] amdgpu 0000:83:00.0: amdgpu_device_ip_init failed
[ 799.793885] amdgpu 0000:83:00.0: Fatal error during GPU init
```
On a subsequent boot, amdgpu initialized successfully (SMU initialized, VRAM ready). After unbinding all 8 GPUs from amdgpu, `rmmod amdgpu` wedged the module (stuck in "Unloading" state in `/proc/modules`), requiring a full VM reboot.
### `/dev/kfd`
`/dev/kfd` exists when amdgpu is loaded. Opening it returns `OSError: [Errno 22] Invalid argument`.
### VRAM BAR reads all 0xFF
After amdgpu initializes the GPU and is then unbound, reading the VRAM BAR (via `/sys/bus/pci/devices/0000:83:00.0/resource0`) returns all `0xFF` at all offsets — including the discovery table at `vram_size - 64KB`. tinygrad's `AMDev._run_discovery()` fails with `AssertionError: discovery signatures mismatch`.
A PCI reset (`echo 1 > /sys/bus/pci/devices/0000:83:00.0/reset`) did not change the VRAM contents — still all `0xFF`.
VRAM was also all `0xFF` when read via `/dev/mem` at the BAR physical address (`0xa0000000000`).
### VFIO attempt
Bound the GPU to `vfio-pci` with `enable_unsafe_noiommu_mode=1`. The GPU bound successfully and `/dev/vfio/noiommu-0` appeared. Running tinygrad with `VFIO=1` still failed with the same `discovery signatures mismatch` — VRAM BAR still reads all `0xFF`.
### No IOMMU in guest
`dmesg` has no `AMD-Vi` entries. PCI devices have no `iommu_group` symlink.
### No fan control
No `fan*` or `pwm*` hwmon entries exist. Only `temp*`, `power*`, `freq*` are exposed. GPU temps read 56-63°C, power ~265W per GPU.
### Current status: NOT WORKING
tinygrad's `PCIIface` finds the GPU (after adding `0x75b0`) but `AMDev._run_discovery()` fails because the VRAM discovery table reads all `0xFF`. This was observed with the GPU unbound from any driver, after PCI reset, and with VFIO bound.