geohot/tinygrad
1
0
Fork 0
forked from tinygrad/tinygrad
Code Pull Requests Actions Activity
3,637 Commits 4 Branches 15 Tags
e0fd41bb463ef9d9df3e281e770ee4fc23bb56e5
Commit Graph
3637 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
David Hou e0fd41bb46 manually shard running mean and running var 2024-02-29 13:41:27 -08:00
David HouandGitHub 06e07950f6 Merge branch 'master' into bn_sync_weights 2024-02-23 13:08:16 -08:00
Carson RadtkeandGitHub 15df9406d6 fix exec_alu(UnaryOps.SQRT, <...>, (0,)) + add test (#3487) 
* fix exec_alu(UnaryOps.SQRT, <...>, (0,)) + add test

* sqrt(0) != nan

* fix tabs
 2024-02-23 18:28:00 +01:00
nimlgenandGitHub 52567da07f jit grapher simplified (#3478) 2024-02-23 16:20:16 +01:00
George HotzandGitHub 2113e1eb63 move all reduces to the end in lazy (#3475) 
* move all reduces to the end in lazy

* apply as reshape, not permute
 2024-02-23 15:49:11 +01:00
David HouandGitHub 5cfcc2a8d7 support MLB reshaping on-axis for evenly sharded (#3484) 
* support MLB reshaping on-axis for evenly sharded

* update test

* not -> !=
 2024-02-23 07:51:36 -05:00
chenyuandGitHub 358a24eae6 symbolic use mod for rmod and use floordiv for rfloordiv (#3485) 2024-02-23 01:05:13 -05:00
David Hou 0b5f6aa56d break out MLB assign and reshape changes 2024-02-22 20:29:50 -08:00
David Hou 4f31d69db0 add backprop test for UnsyncedBatchNorm 2024-02-22 18:20:41 -08:00
David Hou 99536555e4 pass num_devices to UnsyncedBatchNorm in test, allow UnsyncedBatchNorm to be used with LB 2024-02-22 14:51:16 -08:00
David Hou 48564f0c47 don't import batchnorm from hlb_cifar in test_multitensor 2024-02-22 12:30:54 -08:00
nimlgenandGitHub 6d048a0c0b cache collector optimizations are allowed only for kernel operations (#3476) 2024-02-22 12:26:57 +01:00
George HotzandGitHub 7698781389 Revert "wmma: add CUDA tensor core (#3464)" (#3474) 
This reverts commit e9cef13f0b.
 2024-02-22 11:58:16 +01:00
Francis LamandGitHub e9cef13f0b wmma: add CUDA tensor core (#3464) 2024-02-22 11:57:08 +01:00
David Hou 6cac3c99e2 argfix axis 2024-02-21 23:34:06 -08:00
David Hou 5e8b6f8d10 E501 2024-02-21 23:33:35 -08:00
David Hou fb3db11b8e test mlb assign change axis 2024-02-21 23:30:20 -08:00
David Hou a9c47285dd multitensor reshape tests 2024-02-21 23:15:45 -08:00
wozeparrotandGitHub 57678012e1 Upload correct benchmark artifact (#3471) 
* fix: correct filename

* fix: why is this .py?
 2024-02-22 01:14:16 -05:00
David Hou 5af6fb67ad UnsyncedBatchNorm with synced trainable weights for hlb cifar 2024-02-21 22:07:14 -08:00
chenyuandGitHub ab40c0cf93 clean up long lines in symbolic (#3469) 2024-02-21 21:57:44 -05:00
chenyuandGitHub 7c0fc40123 enable test IMAGE=2 PYTHON=1 python3 test/test_ops.py TestOps.test_simple_conv2d (#3468) 2024-02-21 18:30:12 -05:00
chenyuandGitHub 77d2a4c12a regenerate kernel dataset after reduce arg to axis change (#3467) 
```
./extra/optimization/generate_dataset.sh
gzip /tmp/sops
mv /tmp/sops.gz extra/datasets/
```
 2024-02-21 18:16:13 -05:00
David HouGitHubGeorge Hotz
 		f513c37e64 support same uidx in multiple shape positions (#3205) 
* support same uidx in multiple shape positions

* rename var

* update comment

* add contiguous index check to global_store too

* update comment

* small change

* is this better?

* smh

* smaller change?

* get rid of more changes

* get rid of more changes

* is this even making anything better

* comment

* fix test

---------

Co-authored-by: George Hotz <[email protected]>
 2024-02-21 19:37:03 +01:00
chenyuandGitHub 1eb24af63b fix softmax and log_softmax for 0d tensor (#3463) 
matched torch to take axis \in [-1, 0] and used axis=None internally
 2024-02-21 11:30:30 -05:00
George HotzandGitHub 871ba73e65 _reduce_op is axis based now (#3462) 
* _reduce_op is axis based now

* axis_

* update lin failures

* disable that

* fix shape
 2024-02-21 16:36:31 +01:00
George HotzandGitHub 22a90cbb15 change frontend reduce API to use axis (#3460) 
* change frontend API to axis

* switch lazy to also take axis input
 2024-02-21 12:26:17 +01:00
chenyuandGitHub 6c1063ba39 add mypy --strict-equality to pre-commit (#3458) 
matched ci mypy behavior
 2024-02-21 03:41:05 -05:00
chenyuandGitHub 02683a8659 gate the cast before movements in lazy (#3452) 
it made gpt2 slower (2ms -> 2.5ms on 3090, 7ms -> 8ms on M1 Max with BEAM=2).
disabled it in gpt2 benchmark before understanding the full issue
 2024-02-20 09:36:22 -05:00
chenyuandGitHub 0d326a48b8 fix LtNode simplification when lhs and rhs contain same variables (#3451) 
* fix LtNode simplification when lhs and rhs contain same variables

`(Variable("a", 1, 5) < Variable("a", 1, 5))` should eval to `NumNode(0)`

* fix with less perf impact
 2024-02-20 09:06:55 -05:00
George HotzandGitHub 1b6e890ef2 uops flop counter (#3373) 
* factor out winograd functions

* test counter

* uops flop counter

* more correct

* ish

* correct

* cleanup

* tests for uops flop counter

* tests still fail

* fix symbolic uops flop cnt

* fix symbolic uops flop cnt

* hmm, it's an alu

* uops alu resolve

* relax that
 2024-02-20 09:36:30 +01:00
Patrick TsaiGitHubPatrick Tsai
 		9dd64b1f5f Fix python cast uint/int overflow (#3448) 
* Fix numpy uint/int overflow

* lol

* Works

* Update

* Move overflow test to float64/float32

* One line

* Update

* One more

---------

Co-authored-by: Patrick Tsai <[email protected]>
 2024-02-20 09:20:43 +01:00
qazalandGitHub 7864fb69d1 delete MovementOps (#3434) 
* delete MovementOps

* keep extra/to_movement_ops.py
 2024-02-19 23:21:44 +01:00
nimlgenandGitHub 015d414786 fix gpu page fault by ensuring code memory persistence during execution (#3435) 
* fix pf for exec image memory

* no new noqa: E501
 2024-02-19 13:40:53 +01:00
Daniel YehGitHubChen-Chen Yeh
 		0a4029c519 fix path to models folder (#3442) 
Co-authored-by: Chen-Chen Yeh <[email protected]>
 2024-02-19 13:35:57 +01:00
Patrick TsaiGitHubPatrick Tsai
 		ac9d94a068 Cast correctly in python emulator (dtype tests pass) (#3446) 
* Cast correctly in python emulator

* Update test yml and fix lint

* make ruff pass

* mypy passes

---------

Co-authored-by: Patrick Tsai <[email protected]>
 2024-02-19 13:34:02 +01:00
chenyuandGitHub ddec76e9c4 remove unused LtNode.__floordiv__ (#3445) 2024-02-18 22:12:54 -05:00
chenyuandGitHub 86efdf0b34 remove create_rednode (#3444) 
handle Node collapsing into NumNode similar to OpNode
 2024-02-18 21:08:19 -05:00
chenyuandGitHub 2da734920e use __getnewargs__ to fix unpickling Variable (#3441) 
it's recommended to use __getnewargs__ to update the args of classes that use __new__ when unpickling.
It's preferred because it does not change the __new__ behavior.
 2024-02-18 10:28:37 -05:00
nimlgenandGitHub 5647148937 fix hip invalid ordinal (#3440) 2024-02-18 08:31:44 -05:00
chenyuandGitHub 8c0e85fdaf limit symbolic substitute var_vals to have NumNode or Variable (#3438) 
this can greatly reduce the posiible output types of substitute
 2024-02-18 01:29:44 -05:00
geohot 6b4f734dc1 hotfix: better copy stats 2024-02-16 16:52:39 +01:00
geohot c7fda10aa0 hotfix: disk doesn't sync 2024-02-16 16:46:48 +01:00
chenyuandGitHub 230fc33d5b limit sint to be Union[int, Variable, MulNode, SumNode] (#3430) 
* limit sint to be Union[int, Variable, MulNode, SumNode]

these are the only allowed nodes in a Tensor shape

* stride can be sint
 2024-02-16 10:05:46 -05:00
George HotzandGitHub fe97a85014 the compiler is a driver (#3427) 2024-02-16 10:18:09 +01:00
zkuandGitHub 2d702ca073 If feasible, do not truncate float64 down to float32 in cstyle renderer (#3420) 
* do not truncate float64 precision

* use l suffix to try avoid overload confusion

* long line, ruff bloats the function otherwise

* fmt

* remove long double suffix (l), it's sufficient to have the float32 (f) suffix to avoid function overload ambigouity; add test showcasing rtol=1e-12 precision increase, the test fails without the renderer changes

* use more reasonable test values, same as test_int_to_float_unary_func

* disable test for CUDACPU, does not support half and segfaults on some operations per dtypes_alu test

* disable test for HIP, renderer does not support f64 precision

* do not use noqa E501, break up condition
 2024-02-16 10:08:59 +01:00
chenyuandGitHub 30f26279c5 add back "CPU" in test_onnx_backend supports_device (#3426) 
the onnx tests were all skipped.
 2024-02-16 00:49:30 -05:00
xarkesGitHubchenyu
 		28a8b72024 Remove Interpreted device & remaining CPU/TORCH ref (#3423) 
* Remove Interpreted device & remaining CPU/TORCH ref

* Oops

* supports_device was useful

* Fix doc wording

---------

Co-authored-by: chenyu <[email protected]>
 2024-02-16 00:30:21 -05:00
chenyuandGitHub 6efa68f97b remove use of TORCH in pre-commit (#3424) 
it's silently using DEFAULT after removing TORCH
 2024-02-15 19:38:37 -05:00
geohotstanandGitHub 5eb4c902f6 correct division dtype casting (#3405) 
* 新年快乐

* fix: exclude floordiv onnx tests

* fix: less weird if statements in div

* 龙年大吉

* fix: tempfix onnx div

* fix: use reference impl for div
 2024-02-15 19:34:40 -05:00