explicit-dev
2
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 George HotzandGitHub
|
a0a901c8e4 |
faster qwen 3.8 (#17720)
* faster qwen * test fix * dead code * fix gguf issue * pretty nt loads * lil * use warp |
||
|
George HotzandGitHub
|
7e561fcb97 |
mergable fast RDNA3 Qwen 3.8 (#17512)
* mergable fast RDNA3 Qwen 3.6 * AMD * quant 256 multiple * cleanup cast * llm kernels: adapt to Ops.BIND removal Variables are 0-d ALU BUFFERs in the tensor graph and take the ALU PARAM form inside kernels (UOp.variable(param=True)). Add kernel_var helper for the conversion, and keep start_pos in bound form at the graph level so function implicit-input collection and the schedule's binds rename-back line up. * adaptive prefill chunk sizes for recurrent models + iq4xs model entry one TinyJit per static prefill chunk size: capture 128 and 32 at warmup, generate picks the largest that fits the remaining prompt. long prompts prefill 2x faster (555 tok/s on Qwen3.6-27B IQ4_XS) without pushing short prompts through token-by-token decode. * minimize diff: early-return custom attention path, keep master state init * minimize: master _attention with gated fused-scan swap, kernels/amd only, single chunk size - GatedDeltaNetBlock._attention keeps master's symbolic-padding structure; the recurrent scan is swapped for the fused gated_delta_prefill kernel only on RDNA3 with static shapes (fast_scan), everything else uses the old path - all AMD kernel code lives in tinygrad/llm/kernels/amd.py (drop kernels/__init__.py, drop the generic fallback kernel - the old scan covers non-RDNA3) - single prefill chunk size 32; non-RDNA3 recurrent keeps master's chunk_size=1 - the conv+normalize miscompile doesn't trigger with master's window-buffer conv, so the contiguous workaround is dropped * warmup: single code path for fast and old recurrent * drop fast_scan/fast_recurrent flags, inline the RDNA3 gate (cached) * generate: chunk size is always 32, no device gating static chunks for recurrent models everywhere: the fused kernel path on RDNA3, the old scan elsewhere (which is also faster chunked than token-by-token) * warmup: drop redundant _init_state loop (lazy init in the eager step covers it) * symbolic-length prefill with the custom kernels the prefill path is fully symbolic again (master's generate, one prefill graph for every chunk size, no static-tail decode): padded steps are exact no-ops in the scan (beta=0, alpha=exp(0)=1), flash attention positions queries at start_pos instead of valid_kv_len-M, and quant linears pad to the chunk bucket prefill 401 tok: 284 -> 348 tok/s on Qwen3.8-27B IQ4_XS (tail chunks no longer decode token-by-token), decode unchanged at 45 tok/s * cli: default qwen3.6:27b to the fast IQ4_XS quant, add qwen3.8:27b Q4_K_M falls back to slow inline dequant with the custom kernels, IQ4_XS is the fast path. qwen3.8 quants use unsloth's UD (dynamic) naming * warmup: back to master's two-liner plus a cache reset with symbolic prefill, generate([0])'s 1-token chunk captures the symbolic prefill graph that serves every chunk size, and JIT batching on capture measurably doesn't matter with the fused kernels (347.7 tok/s either way) * cli: pin qwen3.8:27b to the pre-UD revision the UD-IQ4_XS replacement mixes in Q3_K tensors (ggml type 11) the loader doesn't support; the pinned revision is byte-identical to the known-good file * warmup: identical to master the leftover cache is self-consistent: get_start_pos only reuses a full strict-prefix match, everything else restarts with a state reset * model: hoist the quantized_attention import to the top level * hoist the GDN query scale out of the branch, restore master dtype.py the scale is the same op in both paths, apply it once after the transpose. the dtype.py diff was a stale pre-SPEC=2 copy, not intentional work * gated_delta_prefill: don't pass the bound start_pos as a call src device-less param buffers in call srcs crash hcq2's _get_enqueue_devs. the var already reaches the graph through the state AFTER chain (conv state store), same as the flash kernels' valid_end * llm: half KV cache with custom flash kernels, drop the int8 quantized cache matches master's new half cache default: no scales, no packing, one less buffer. the store casts to half explicitly (buffer-only half usage misses the renderer's half define). 45.5 tok/s decode, 348.7 tok/s prefill — same as int8 * llm: zero-init the KV cache the int8 path was accidentally protected from uninitialized memory by its zero-initialized scale buffer; with a plain half cache the flash prefill kernel's P*V wmma computes 0*NaN=NaN on masked lanes past the valid region (manifested as garbage tokens at 32k context where the allocator reuses dirty VRAM) * gate that * llm/kernels/amd: reorganize by kernel family, drop the clutter sections: shared helpers, quant linear, flash attention, gated delta prefill. no AxisType.WEAK (default), no ALLOW_DEVICE_USAGE override (unneeded), magic numbers become names (QUANT_SIZES, Q5_K/Q6_K/IQ4_XS), merged wrapper layers (flash_attention_causal_cached folded into flash_attention), one _unbind helper for the bound-var dance * test: universal recurrent reuse assertion, fix lambda lint * 1-token chunks have a static shape: they are decode steps a 1-token chunk routes to the decode graph via the existing dispatch, so warmup and decode-only workloads never build the big symbolic prefill graph: CI benchmark command 12m50s -> 5m29s (master: 6m48s), 220 -> 123 compile jobs also restores the ALLOW_DEVICE_USAGE override in amd_custom_kernels_supported: Device[] asserts inside @function contexts (ALLOW_DEVICE_USAGE=0), and the first gate call can happen there depending on test order * generate: back to plain symbolic binding, the static-1 rule wasn't worth it * custom kernels: Q4_K support (ggml type 12) Q4_K is Q5_K without the high-bit array: same d/dmin/scales layout (so _q5_scales works unchanged), 144-byte blocks, qs at word 4. both the dp4a decode kernel and the WMMA prefill kernel take a ggml_type branch now. Qwen3-8B Q4_K_M: decode 16.5 -> 114.8 tok/s, prefill 69 -> 536 tok/s * raise line count to 26500 (qwen did it) * benchmark qwen3.8 * little updates |