* mergable fast RDNA3 Qwen 3.6
* AMD
* quant 256 multiple
* cleanup cast
* llm kernels: adapt to Ops.BIND removal
Variables are 0-d ALU BUFFERs in the tensor graph and take the ALU PARAM form
inside kernels (UOp.variable(param=True)). Add kernel_var helper for the
conversion, and keep start_pos in bound form at the graph level so function
implicit-input collection and the schedule's binds rename-back line up.
* adaptive prefill chunk sizes for recurrent models + iq4xs model entry
one TinyJit per static prefill chunk size: capture 128 and 32 at warmup,
generate picks the largest that fits the remaining prompt. long prompts
prefill 2x faster (555 tok/s on Qwen3.6-27B IQ4_XS) without pushing short
prompts through token-by-token decode.
* minimize diff: early-return custom attention path, keep master state init
* minimize: master _attention with gated fused-scan swap, kernels/amd only, single chunk size
- GatedDeltaNetBlock._attention keeps master's symbolic-padding structure;
the recurrent scan is swapped for the fused gated_delta_prefill kernel only
on RDNA3 with static shapes (fast_scan), everything else uses the old path
- all AMD kernel code lives in tinygrad/llm/kernels/amd.py (drop kernels/__init__.py,
drop the generic fallback kernel - the old scan covers non-RDNA3)
- single prefill chunk size 32; non-RDNA3 recurrent keeps master's chunk_size=1
- the conv+normalize miscompile doesn't trigger with master's window-buffer conv,
so the contiguous workaround is dropped
* warmup: single code path for fast and old recurrent
* drop fast_scan/fast_recurrent flags, inline the RDNA3 gate (cached)
* generate: chunk size is always 32, no device gating
static chunks for recurrent models everywhere: the fused kernel path on RDNA3,
the old scan elsewhere (which is also faster chunked than token-by-token)
* warmup: drop redundant _init_state loop (lazy init in the eager step covers it)
* symbolic-length prefill with the custom kernels
the prefill path is fully symbolic again (master's generate, one prefill graph
for every chunk size, no static-tail decode): padded steps are exact no-ops in
the scan (beta=0, alpha=exp(0)=1), flash attention positions queries at
start_pos instead of valid_kv_len-M, and quant linears pad to the chunk bucket
prefill 401 tok: 284 -> 348 tok/s on Qwen3.8-27B IQ4_XS (tail chunks no longer
decode token-by-token), decode unchanged at 45 tok/s
* cli: default qwen3.6:27b to the fast IQ4_XS quant, add qwen3.8:27b
Q4_K_M falls back to slow inline dequant with the custom kernels, IQ4_XS is
the fast path. qwen3.8 quants use unsloth's UD (dynamic) naming
* warmup: back to master's two-liner plus a cache reset
with symbolic prefill, generate([0])'s 1-token chunk captures the symbolic
prefill graph that serves every chunk size, and JIT batching on capture
measurably doesn't matter with the fused kernels (347.7 tok/s either way)
* cli: pin qwen3.8:27b to the pre-UD revision
the UD-IQ4_XS replacement mixes in Q3_K tensors (ggml type 11) the loader
doesn't support; the pinned revision is byte-identical to the known-good file
* warmup: identical to master
the leftover cache is self-consistent: get_start_pos only reuses a full
strict-prefix match, everything else restarts with a state reset
* model: hoist the quantized_attention import to the top level
* hoist the GDN query scale out of the branch, restore master dtype.py
the scale is the same op in both paths, apply it once after the transpose.
the dtype.py diff was a stale pre-SPEC=2 copy, not intentional work
* gated_delta_prefill: don't pass the bound start_pos as a call src
device-less param buffers in call srcs crash hcq2's _get_enqueue_devs. the
var already reaches the graph through the state AFTER chain (conv state
store), same as the flash kernels' valid_end
* llm: half KV cache with custom flash kernels, drop the int8 quantized cache
matches master's new half cache default: no scales, no packing, one less
buffer. the store casts to half explicitly (buffer-only half usage misses
the renderer's half define). 45.5 tok/s decode, 348.7 tok/s prefill —
same as int8
* llm: zero-init the KV cache
the int8 path was accidentally protected from uninitialized memory by its
zero-initialized scale buffer; with a plain half cache the flash prefill
kernel's P*V wmma computes 0*NaN=NaN on masked lanes past the valid region
(manifested as garbage tokens at 32k context where the allocator reuses
dirty VRAM)
* gate that
* llm/kernels/amd: reorganize by kernel family, drop the clutter
sections: shared helpers, quant linear, flash attention, gated delta prefill.
no AxisType.WEAK (default), no ALLOW_DEVICE_USAGE override (unneeded), magic
numbers become names (QUANT_SIZES, Q5_K/Q6_K/IQ4_XS), merged wrapper layers
(flash_attention_causal_cached folded into flash_attention), one _unbind
helper for the bound-var dance
* test: universal recurrent reuse assertion, fix lambda lint
* 1-token chunks have a static shape: they are decode steps
a 1-token chunk routes to the decode graph via the existing dispatch, so
warmup and decode-only workloads never build the big symbolic prefill graph:
CI benchmark command 12m50s -> 5m29s (master: 6m48s), 220 -> 123 compile jobs
also restores the ALLOW_DEVICE_USAGE override in amd_custom_kernels_supported:
Device[] asserts inside @function contexts (ALLOW_DEVICE_USAGE=0), and the
first gate call can happen there depending on test order
* generate: back to plain symbolic binding, the static-1 rule wasn't worth it
* custom kernels: Q4_K support (ggml type 12)
Q4_K is Q5_K without the high-bit array: same d/dmin/scales layout (so
_q5_scales works unchanged), 144-byte blocks, qs at word 4. both the dp4a
decode kernel and the WMMA prefill kernel take a ggml_type branch now.
Qwen3-8B Q4_K_M: decode 16.5 -> 114.8 tok/s, prefill 69 -> 536 tok/s
* raise line count to 26500 (qwen did it)
* benchmark qwen3.8
* little updates
* improvements to the llm server
* eval script
* eval llm
* better eval gets 58.71
* cleanups
* add temperature, but multinomial is absurdly slow
* claude is so smart
* lint
* remove slop
* no more stop
* remove llm generate always reset start_pos
by itself seems like a bug, also added a test to repro forward_jit.reset() issue
* issue is jit graph, so revert that test
* qwen model is mostly working
* add Q4_K quantization support to GGUF parser, add qwen3:1.7b model
- Add Q4_K (type 12) dequantization in nn/state.py
- Add qwen3:1.7b model using Q4_K_M quantization (smaller than Q8_0)
- Make bos_token_id optional for models like Qwen3 that don't have it
- Fix line length issues and add preset parameter to SimpleTokenizer
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <[email protected]>
* smaller diff
* test dequant
* half split
* better
* simple tok
* mock token
* polish
* better
* fix
* replace
---------
Co-authored-by: Claude Opus 4.5 <[email protected]>
* llm: add created/model fields, non-streaming support, and tests
- Add `created` timestamp and `model` fields to response (required by OpenAI spec)
- Add non-streaming mode support for /v1/chat/completions
- Add `send_data` helper to HTTPRequestHandler for responses with Content-Length
- Refactor viz/serve.py to use send_data
- Add integration tests using real OpenAI client
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <[email protected]>
* add openai to testing
* toml
* Remove 'openai' from dependencies
Removed 'openai' from the dependencies list.
* bump cache
---------
Co-authored-by: Claude Opus 4.5 <[email protected]>
