tinygrad

9,700 Commits 336 Branches 15 Tags

3 Commits

Include renames

Author SHA1 Message Date

Author	SHA1	Message	Date
chenyuandGitHub	90c3ed17c5	move cast to before softmax in attention (#9213 ) * move cast to before softmax in attention saved some memory because exp (which is used for backward) are done in half. training bert seems fine and can fit BS=78 now (from 66) * test	2025-02-24 17:24:59 -05:00
chenyuandGitHub	ff3f2a9c1a	Revert "move attention upcast (#7830 )" (#7903 ) This reverts commit `c07daf40e7`.	2024-11-25 18:59:51 -05:00
chenyuandGitHub	c07daf40e7	move attention upcast (#7830 ) still upcast before softmax, but faster because intermediate buffer can be stored in half (as long as qk is within half range).	2024-11-22 17:10:51 -05:00

chenyuandGitHub

90c3ed17c5

move cast to before softmax in attention (#9213 )

* move cast to before softmax in attention

saved some memory because exp (which is used for backward) are done in half. training bert seems fine and can fit BS=78 now (from 66)

* test

2025-02-24 17:24:59 -05:00

chenyuandGitHub

ff3f2a9c1a

Revert "move attention upcast (#7830 )" (#7903 )

This reverts commit c07daf40e7.

2024-11-25 18:59:51 -05:00

chenyuandGitHub

c07daf40e7

move attention upcast (#7830 )

still upcast before softmax, but faster because intermediate buffer can be stored in half (as long as qk is within half range).

2024-11-22 17:10:51 -05:00