* delete Ops.FUNCTION/GETTUPLE/TUPLE: call outputs are AFTER on RETURNED placeholders
value-producing calls: the body is a plain parametric program that stores outputs
into output PARAMs (slots after the input PARAMs). the RETURNED placeholders are
inputs to the call, bound to the output PARAMs positionally wherever the call is
resolved, and callers AFTER on them like normal buffers. gradient flows through
the generic AFTER rule; everything is just Ops.CALL.
* RETURNED identity is its placement in the call srcs, not a nonce
slot=-1 merging collapses duplicate-signature outputs into one uop (t+1,t+2 grads
and multi-grad backward calls dedupe wrongly), and skipping the uop cache breaks
schedule_cache (stale linear hits since structural keys assume interning). instead
the RETURNED's placement (output index among call srcs) is its identity: identical
call constructions merge deterministically, positions never collide.
* resolve RETURNED afters in the tensor graph like values (master parity with gettuple)
- remove the CONTIGUOUS wrap of tagged call-output afters, it forced call outputs
(e.g. local shard amax) into their own buffer/kernel instead of inlining
- inline RETURNED afters at transform time via returned_after_finalize, dissolving
to values for consumers; calls with bound-variable or unresolved UNSHARD args
keep the schedule-time resolution path
- allow movement ops (flat-storage views) in kernel graph value positions in the spec
- port embedding backward + extra/llama_kernels (local_abs_max, rmsnorm) to the new API
* use SINK, not GROUP, for gradient value containers
spec.py only blesses GROUP of stores/groups/loops; the gradient value bundles
(the forward values, root_grad seeds, and the after->call gradient edge) are
plain value containers, and SINK-of-values is already in the spec.
also fix extra/llama_kernels/rmsnorm: returned_outputs is a property
* CALL is positional: RETURNS work in any src position, convention lives in call_outputs
- all resolution paths (gradient, precompile transform, binding) locate RETURNEDs
by identity, not by "last srcs"; only call_outputs builds the args-first layout
- grad_fxn padding aligns grads with the call's actual src positions
- add test_two_return/precompiled
* source-compat shim for maketuple/gettuple so foreign code built before the redesign keeps working
UOp.maketuple returns a _LegacyTupleValues holder; .call builds call_outputs;
CALL.gettuple(i) is returned_outputs[i]. the produced graphs are identical to
the new-api versions, so nn/extra/mlperf code is reverted to upstream text
* simplify function.py call construction + drop the resolved-call cache
- function.py: single and tuple returns both build the call through call_outputs
- tensor.py: resolve_function is deterministic and interned, the global cache was unneeded
* bind zero-offset views of flat storage to the storage instead of padding them
call args need offset 0 and enough length, not views: flat_storage collapses the
zero-offset contiguous view chain to the sized storage base, so resolved call args
are storage-bare like master (no PAD/SHRINK chains in the kernel graph)
* spec.py: drop stray rebase-collision edits, keep only the RETURNED changes
* test_multitensor: revert to master, the gettuple shim covers it
* materialize all tagged RETURNED afters into real buffers
call outputs need real storage regardless of whether they are finals of the current
realize: deferred/stateful outputs (the fp8 grad-amax mailbox) are consumed by later
realize steps as call args, where a resolved value would have no ranges
* call input buffers: wrap RETURNED-based afters, not real-buffer afters
precompiled call input binding kept any AFTER unwrapped; an AFTER on a RETURNED
placeholder has no storage behind it, so its value leaked into the kernel graph with
no consumer able to register ranges (llama3 8B fp8 mailbox pipeline crash).
materialize afters whose base has no buffer identity instead.
this was the fix matching master for the REDUCE-has-no-ranges crash and restores
the llama-kernels amax kernel count
* call slots are src positions, always; never rearrange
one upstream cause behind the three P1 findings: the raw CALL machinery binds
positionally (resolve_function params, gradient padding) but a second args-first
convention crept in where RETURNEDs get moved to trailing slots. position is
identity now:
- transform_precompiled_call keeps RETURNEDs' original src positions: outs take
their places, other args become input buffers; no slot renumbering
- implicit gradients are emitted aligned to original src positions (None at
RETURNED positions)
- flat_storage drops the hand-rolled contiguity analysis: reshape itself is the
flat-prefix check (it raises ValueError); strided views materialize first
* nits on call slot positions; regression tests for interspersed RETURNED
- flat_storage back to pad_to().reshape() (reshape keeps movement views, it is not a contiguity check)
- input_buffer checks has_buffer_identity(after_ok=True)
- TestArgOrder: interspersed RETURNED (plain + precompiled transform), its gradient,
padded and strided function inputs
* device fixes
* TestArgOrder: padded regression uses zero-start padded/shrunk view
* TestArgOrder: clone to force buffer identity in padded/strided regression tests
* slim: revert prepare formatting, drop reverted-bug tests, restore viz guards, clean comments, mirror returned on param
* gut transform_precompiled_call, delete returned_after_finalize
the transform keeps master's shape; the prepare-stage resolve_AFTER rule already
inlines plain call outputs, and materialization is owned by the input-buffer rule
(afters on real buffers bind, afters on RETURNEDs contiguous)
* update spec for returned
* transform_precompiled_call: inline the input-buffer rule, drop sorted() (body stores are already slot-ordered)
* drop dead RETURNED-era rules: prepare's after-shell strip (leftover from returned_after_finalize, which is gone), redundant pattern-covered SINK check, defensive slot-sorts (bodies are slot-ordered by construction)
* drop final_tags: final outputs of value calls materialize at sink construction
The set of finals is already known precisely (the big_sink's srcs), so track
nothing: wrap each final AFTER-on-RETURNED in CONTIGUOUS right after numbering.
Precompiled calls are excluded - transform_precompiled_call in the flatten
pass gives their outputs real buffers, and wrapping before that transform
leaves a stale tag that breaks the output copy.
* drop unused default_dtype import
* delete Ops.FUNCTION: value-producing calls are just CALLs with a TUPLE body
a CALL with a TUPLE body (always void dtype) is value-producing and gradient-able,
outputs are extracted with GETTUPLE like before; all other CALLs are opaque.
TUPLE/GETTUPLE are untouched
* match the TUPLE body in the UPat, not in the rewrite
* remove FUNCTION from tinyspec (regen pdf) and viz
* fix and spec
* move shape into arg for param/buffer
* no param_from_shape
* drop gratuitous syntax changes
* image is a in-graph view, folded into the param arg at render; drop dead multi param sharding
* view_as helper, simpler resolve_function, spec update
* spec: param/buffer are flat storage, no shape input
* image dims live in the param arg from transform_to_image; tighten kernel graph spec
* kernel graph spec: only RESHAPE/SHRINK over storage values, not all movement
* kernel graph: call args are storage, not views (pm_no_view_args); assert in spec
* strip views at the kernel graph level (pm_no_views), move into rangeify
* touchups
* UOp.param accepts single sint as shape (int,)
Change UOp.param signature from shape:tuple[sint, ...]|None to shape:tuple[sint, ...]|sint|None.
A single sint means 1D shape (sint,). Update all callers from (n,) to n syntax.
No param_from_shape — multi-dim shapes stay as tuples.
* use single int syntax in hcq2 copy_with_kernel
* mergable fast RDNA3 Qwen 3.6
* AMD
* quant 256 multiple
* cleanup cast
* llm kernels: adapt to Ops.BIND removal
Variables are 0-d ALU BUFFERs in the tensor graph and take the ALU PARAM form
inside kernels (UOp.variable(param=True)). Add kernel_var helper for the
conversion, and keep start_pos in bound form at the graph level so function
implicit-input collection and the schedule's binds rename-back line up.
* adaptive prefill chunk sizes for recurrent models + iq4xs model entry
one TinyJit per static prefill chunk size: capture 128 and 32 at warmup,
generate picks the largest that fits the remaining prompt. long prompts
prefill 2x faster (555 tok/s on Qwen3.6-27B IQ4_XS) without pushing short
prompts through token-by-token decode.
* minimize diff: early-return custom attention path, keep master state init
* minimize: master _attention with gated fused-scan swap, kernels/amd only, single chunk size
- GatedDeltaNetBlock._attention keeps master's symbolic-padding structure;
the recurrent scan is swapped for the fused gated_delta_prefill kernel only
on RDNA3 with static shapes (fast_scan), everything else uses the old path
- all AMD kernel code lives in tinygrad/llm/kernels/amd.py (drop kernels/__init__.py,
drop the generic fallback kernel - the old scan covers non-RDNA3)
- single prefill chunk size 32; non-RDNA3 recurrent keeps master's chunk_size=1
- the conv+normalize miscompile doesn't trigger with master's window-buffer conv,
so the contiguous workaround is dropped
* warmup: single code path for fast and old recurrent
* drop fast_scan/fast_recurrent flags, inline the RDNA3 gate (cached)
* generate: chunk size is always 32, no device gating
static chunks for recurrent models everywhere: the fused kernel path on RDNA3,
the old scan elsewhere (which is also faster chunked than token-by-token)
* warmup: drop redundant _init_state loop (lazy init in the eager step covers it)
* symbolic-length prefill with the custom kernels
the prefill path is fully symbolic again (master's generate, one prefill graph
for every chunk size, no static-tail decode): padded steps are exact no-ops in
the scan (beta=0, alpha=exp(0)=1), flash attention positions queries at
start_pos instead of valid_kv_len-M, and quant linears pad to the chunk bucket
prefill 401 tok: 284 -> 348 tok/s on Qwen3.8-27B IQ4_XS (tail chunks no longer
decode token-by-token), decode unchanged at 45 tok/s
* cli: default qwen3.6:27b to the fast IQ4_XS quant, add qwen3.8:27b
Q4_K_M falls back to slow inline dequant with the custom kernels, IQ4_XS is
the fast path. qwen3.8 quants use unsloth's UD (dynamic) naming
* warmup: back to master's two-liner plus a cache reset
with symbolic prefill, generate([0])'s 1-token chunk captures the symbolic
prefill graph that serves every chunk size, and JIT batching on capture
measurably doesn't matter with the fused kernels (347.7 tok/s either way)
* cli: pin qwen3.8:27b to the pre-UD revision
the UD-IQ4_XS replacement mixes in Q3_K tensors (ggml type 11) the loader
doesn't support; the pinned revision is byte-identical to the known-good file
* warmup: identical to master
the leftover cache is self-consistent: get_start_pos only reuses a full
strict-prefix match, everything else restarts with a state reset
* model: hoist the quantized_attention import to the top level
* hoist the GDN query scale out of the branch, restore master dtype.py
the scale is the same op in both paths, apply it once after the transpose.
the dtype.py diff was a stale pre-SPEC=2 copy, not intentional work
* gated_delta_prefill: don't pass the bound start_pos as a call src
device-less param buffers in call srcs crash hcq2's _get_enqueue_devs. the
var already reaches the graph through the state AFTER chain (conv state
store), same as the flash kernels' valid_end
* llm: half KV cache with custom flash kernels, drop the int8 quantized cache
matches master's new half cache default: no scales, no packing, one less
buffer. the store casts to half explicitly (buffer-only half usage misses
the renderer's half define). 45.5 tok/s decode, 348.7 tok/s prefill —
same as int8
* llm: zero-init the KV cache
the int8 path was accidentally protected from uninitialized memory by its
zero-initialized scale buffer; with a plain half cache the flash prefill
kernel's P*V wmma computes 0*NaN=NaN on masked lanes past the valid region
(manifested as garbage tokens at 32k context where the allocator reuses
dirty VRAM)
* gate that
* llm/kernels/amd: reorganize by kernel family, drop the clutter
sections: shared helpers, quant linear, flash attention, gated delta prefill.
no AxisType.WEAK (default), no ALLOW_DEVICE_USAGE override (unneeded), magic
numbers become names (QUANT_SIZES, Q5_K/Q6_K/IQ4_XS), merged wrapper layers
(flash_attention_causal_cached folded into flash_attention), one _unbind
helper for the bound-var dance
* test: universal recurrent reuse assertion, fix lambda lint
* 1-token chunks have a static shape: they are decode steps
a 1-token chunk routes to the decode graph via the existing dispatch, so
warmup and decode-only workloads never build the big symbolic prefill graph:
CI benchmark command 12m50s -> 5m29s (master: 6m48s), 220 -> 123 compile jobs
also restores the ALLOW_DEVICE_USAGE override in amd_custom_kernels_supported:
Device[] asserts inside @function contexts (ALLOW_DEVICE_USAGE=0), and the
first gate call can happen there depending on test order
* generate: back to plain symbolic binding, the static-1 rule wasn't worth it
* custom kernels: Q4_K support (ggml type 12)
Q4_K is Q5_K without the high-bit array: same d/dmin/scales layout (so
_q5_scales works unchanged), 144-byte blocks, qs at word 4. both the dp4a
decode kernel and the WMMA prefill kernel take a ggml_type branch now.
Qwen3-8B Q4_K_M: decode 16.5 -> 114.8 tok/s, prefill 69 -> 536 tok/s
* raise line count to 26500 (qwen did it)
* benchmark qwen3.8
* little updates
* cpu hcq2
* temp
* slop
* test with backpressure
* x
* x
* x
* x
* x
* x
* Dx
* save reverts
* um?
* x
* x
* call from py
* x?
* x
* submitters gone
* x
* x
* z
* Dx
* Dx
* x
* x
* fixes
* repl
* x
* f
* for now keep hcqbuffer
geohot
sirhcmandGitHub