geohot/tinygrad
1
0
Fork 0
forked from tinygrad/tinygrad
Code Pull Requests Actions Activity
2,648 Commits 4 Branches 15 Tags
0d2b3a9d33afe18fc7b73a33db2820c15e31db13
Commit Graph
427 Commits
Author SHA1 Message Date
20kdcandGitHub 967a88a505 examples/waifu2x: Cleanup waifu2x vgg7 model format (now uses safetensors) (#2082) 2023-10-18 09:20:11 -07:00
wozeparrotandGitHub 4d1e59abfd fix: only when distributed (#2102) 2023-10-17 20:09:04 -07:00
Sean D'SouzaandGitHub 999c95ea29 fix: hlb cifar types (#2099) 2023-10-17 19:23:50 -07:00
geohot 9b1c3cd9ca hlb_cifar: support EVAL_STEPS=1000, print when dataset is shuffled 2023-10-18 01:11:08 +00:00
George HotzandGitHub c36d306606 KOPT is over, BEAM is upstream (#2071) 
* create cache for q learning

* make linter happy

* global beam

* where it belongs

* bugfix

* ditch the kopt, use the beam

* faster lin and DEBUG=2 okay

* remove kopt, move search to features
 2023-10-16 09:46:03 -07:00
Ahmed HarmoucheandGitHub 0d3410d93f Stable diffusion: Make guidance modifiable (#2077) 2023-10-15 14:36:43 -07:00
George HotzandGitHub 49bcfec383 0s in the action space (#2070) 
* 0s in the action space

* simpler

* skip duplicate actions
 2023-10-14 11:22:48 -07:00
mmmkkaaayyandGitHub 91168a28c4 whisper: make file transcription work, add basic CI test (#2042) 2023-10-13 17:13:35 -07:00
George HotzandGitHub 6f1810af2d with unroll, the action space goes from 161 -> 127 (#2060) 
* with unroll, the action space goes from 161 -> 127

* more reliable instrumentation

* beam search is so op

* beam bugfix
 2023-10-12 20:52:23 -07:00
Yixiang GaoandGitHub 3187962476 CIFAR HALF mode (#2041) 
* load weights in fp16

* add dtype option in nn

* fix test

* no need for dtype in nn

* add option to load weights in FP16, but NaN

* change loss scaler

* cast to float32 for norm layer

* add a todo for the forward pass padding

* fix transform
 2023-10-12 10:19:51 -07:00
George HotzandGitHub c5edb3c374 train value net, improve API, add BCE (#2047) 
* api cleanups, BCE losses

* valuenet

* fixup examples

* learning okay

* add valuenet runner

* net improvements

* net improvements

* 40% win rate
 2023-10-12 07:56:38 -07:00
George HotzandGitHub 41bfeb2c1e start work on auto opt (#2034) 
* start work on auto opt

* lin failure

* not beating hcopt

* greedy

* timing is fast

* codegen.search

* greedy search in handcode_opt

* track running gflops

* clean up those files

* no failure
 2023-10-11 12:54:53 -07:00
chenyuandGitHub e2b83f1b42 Variable.bind newer (#2017) 
* Variable.bind attempt 2

* ShapeTracker.unbind

* fix llama

* fix types

* test case

* View.vars cleanup

* include mask in symbolic source

* mask can be sint

* st.unbind in bufferops

* assert ast contain free Variable only

* cleanup

* conservative unbinding reduce op arg

* move reduceop unbind

* fix llama JIT arg behavior
 2023-10-10 10:03:01 -07:00
Ahmed HarmoucheandGitHub e27fedfc7b Fix stable diffusion output error on WebGPU (#2032) 
* Fix stable diffusion on WebGPU

* Remove hack, numpy cast only on webgpu

* No-copy numpy cast
 2023-10-10 06:40:51 -07:00
chenyuandGitHub 25555c836f llama default to JIT only if device supports JIT (#2028) 2023-10-09 17:26:02 -07:00
George HotzandGitHub 16ca8410f8 op logger + replay (#2021) 
* logops

* fix dtype printing

* needs inf

* ops dataset

* minor improvements

* 12k kernels

* opt can compile

* graph flops
 2023-10-08 15:10:18 -07:00
mmmkkaaayyGitHubmmmkkaaayy
 		af6e2f31ca whisper: cast model output token to int32 (#2013) 
Co-authored-by: mmmkkaaayy <[email protected]>
 2023-10-08 05:56:22 -07:00
geohot 44ed94ef5c use the device abstraction in handcode_resnet50_opt 2023-10-07 13:22:20 -07:00
George HotzandGitHub 121f7aa8c5 Schedule item (#2012) 
* ScheduleItem

* put var_vals in the schedule

* fix tests, wow that proliferated quickly

* not ready to be in the schedule
 2023-10-07 08:59:25 -07:00
George HotzandGitHub f54959e5cd move print tree into graph (#2003) 
* move print tree into graph

* add winograd profiling test

* change pre-commit to run ruff first
 2023-10-07 04:39:21 -07:00
Ahmed HarmoucheandGitHub 2114dc13d1 Allow multi-input model export (#1995) 
* Allow multi-input model export

* Add model export unit test

* Fix efficientnet compilation

* Only run model export test on JIT supported devices

* Skip export model test if not EXPORT_SUPPORTED_DEVICE
 2023-10-07 04:13:34 -07:00
chenyuandGitHub 05be57f57f Fix llama with empty prompt (#1997) 
* fix llama with one token prompt

* llama is all_jitted
 2023-10-06 06:48:07 -07:00
chenyuandGitHub da2b3e55f4 simpler llama - don't shrink twice (#1981) 2023-10-05 14:31:46 -07:00
chenyuandGitHub c99fa58dd2 simplify gpt2 example (#1973) 
* simplify gpt2 example

* kernel_jitted_count and jit tests

* Revert "kernel_jitted_count and jit tests"

This reverts commit 31a3c26dd0.

* all_jitted test in test_real_world
 2023-10-05 07:09:29 -07:00
nimlgenandGitHub 2ea1dd3e87 no process() in Linearizer (#1966) 
* no process() in Linearizer

* more process() clean up
 2023-10-04 07:18:42 -07:00
Daniel RiegeandGitHub 579cabf668 Fix examples/train_efficientnet (#1947) 
* added missing colon

* bug fixes for cifar10 dataset loading
needed a reshape to work with conv layers and resolve fetched tensor to numpy since further code expects numpy array
 2023-10-02 02:23:38 -07:00
George HotzandGitHub 90326dbdc3 resnet50 hand coded optimization (#1945) 
* resnet50 hand coded opt

* hand optimize one kernel

* opt in both places to fix test
 2023-09-29 09:34:51 -07:00
George HotzandGitHub 4ff35e2b97 better resnet eval (#1943) 2023-09-29 05:40:25 -07:00
George HotzandGitHub 48c8d130ae simpler GPT2 (#1941) 
* don't realize in gpt2

* simpler gpt2
 2023-09-29 04:41:09 -07:00
Yixiang GaoandGitHub 094d3d71be with Tensor.train() (#1935) 
* add with.train

* remove the rest TODOs

* fix pyflake

* fix pyflake error

* fix mypy
 2023-09-28 18:02:31 -07:00
George HotzandGitHub adab724caa schedule2, keep the tests working with small changes (#1932) 
* lazy cleanups

* ast functions take in LazyOps

* op instead of self.op

* _base for mops

* fix contiguous

* start schedule

* test_schedule

* fix openpilot

* more tests

* bugfix and test skip

* work

* make sure things get freed

* fix zerosized tensors

* fix failing test

* fix ceil and friends

* fix openpilot

* disable training

* disable test collectives
 2023-09-28 09:14:43 -07:00
Dat D. NguyenandGitHub ae9529e678 chore: remove redundant noise in stable diffusion example (#1910) 2023-09-24 21:33:45 +08:00
Gijs KoningandGitHub b8ff20ffe4 Gpt2 (#1896) 
* small helps

* got something working

* faster?

* faster yes

* cleanup

* cleanup

* cleanup

* Fix non jit

* Fix fp16 and some cleanup

* Fix fp16 and some cleanup

* cleanup

* similar to master

* cleanup
 2023-09-22 20:14:47 +08:00
Yixiang GaoandGitHub cb5d6576cb cifar step time 65ms while stay above 94% (#1888) 
* change reduceop heruistics

* add model ema and jit hack

* add ema eval

* have to create a duplicate eval function for jit

* remove manual seed

* 94% achieveable with normal eval

* ema is outputting the same results as normal

* fix ema bug

* ema achieves 94% with fix seed

* multigpu tested

* constant fold decay, fix jit, adjust message for multigpu

* pull SpeedyResNet out of train_cifar()
 2023-09-21 11:19:32 +08:00
nimlgenandGitHub 4c31dfafb3 add seed to gpt-2 (#1869) 2023-09-15 17:34:14 -04:00
segf00ltandGitHub 9e8c1dbf34 patch to remove hack from stable_diffusion.py (#1814) 
* patch to remove hack from stable_diffusion.py

* sorry linter

* realize after assign?

* float16 broken in llvmlite use float64 for now

* int32

* idiot forgot to change test array dtype
 2023-09-08 09:26:50 -07:00
chenyuandGitHub ebcda8a714 Move var_vals from ShapeTracker to LazyBuffer (#1819) 2023-09-08 09:25:10 -07:00
geohot 722823dee1 stable diffusion: force fp16 free 2023-09-06 15:11:05 -07:00
Yixiang GaoandGitHub 22cf15e9d0 convert function into tinygrad (#1803) 2023-09-06 14:41:26 -07:00
Pavol RusnakandGitHub 52a92bf95d use class Foo: instead of class Foo(): (#1797) 
* use class Foo: instead of class Foo():

* add ruff linter, copy settings from .flake8 to ruff.toml
 2023-09-06 12:20:25 -07:00
badccandGitHub fd25792c8b Ensure freqs as type float32 in freqs_cis (#1798) 2023-09-06 10:24:15 -07:00
geohot f67638b27a delete broken DDPG example 2023-09-06 08:01:12 -07:00
Francis LamandGitHub 0379b64ac4 add seed option to stable_diffusion (#1784) 
useful for testing correctness of model runs
 2023-09-05 19:45:15 -07:00
George HotzandGitHub fb1cc6bf4b llama jit is default, print tok/sec (#1774) 
* llama jit is default, print tok/sec

* jit not default in CI
 2023-09-05 10:12:16 -07:00
Yixiang GaoGitHubGeorge Hotz
 		66a6bbd029 codellama (#1702) 
* add codellama with pre-downloaded weights

* add rope_theta, fix param

* fix test

* add 7B-Python

* add 7B-Instruct

* replace single quotes with doulbe

---------

Co-authored-by: George Hotz <[email protected]>
 2023-09-02 08:45:12 -07:00
chenyuandGitHub a2745819f6 faster gpt2 jit path and gpt2 in test_real_world (#1738) 2023-09-02 08:39:12 -07:00
geohotstanandGitHub 94b1257f5e Changed DEVICE to Device.DEFAULT in deep_determinist_policy_gradient (#1715) 
* added device in optim and deep

* oops forgot to del print code

* use Device.DEFAULT instead

* removed device
 2023-08-31 07:08:51 -07:00
nimlgenandGitHub b5cf274da3 remove memory peak for quantized llama (#1720) 2023-08-30 16:32:30 -04:00
chenyuandGitHub e4eb5d55c7 critical realize for unjitted llama (#1718) 2023-08-30 14:52:32 -04:00
George Hotz cd7ceed914 gpt2: print total instead of sync time 2023-08-30 10:59:42 -07:00