sirhcm/tinygrad
1
0
Fork 0
forked from tinygrad/tinygrad
Code Pull Requests 2 Actions Activity
494 Commits 336 Branches 15 Tags
ece07a3d128519633186ca76b0aabbfb0df4dbcc
Commit Graph
82 Commits
Author SHA1 Message Date
Marcel BischoffandGitHub 42b4761025 transformer >99.98% test accuracy in ~30s (#230) 
* transformer

* BS might divide len(Y_test)

* outoput when accuracy is high

* more readeable

* fixed loss in serious_mnist for new API
 2021-01-02 07:45:09 -08:00
geohot f9170505b3 if you like your transformers twice as slow, use the GPU 2020-12-29 17:14:23 -05:00
geohot 3f8e137b6f extra/transformer 2020-12-29 14:14:00 -05:00
geohot bcb3ceeca3 set training in functions 2020-12-28 22:45:46 -05:00
geohot 51bf164b72 dropout, training 2020-12-28 22:12:23 -05:00
geohot 7b8fee038d it works! forgot the sqrt 2020-12-28 16:23:52 -05:00
geohot 1faf05ef67 ahh, it's better if i don't train the embedding 2020-12-28 16:07:02 -05:00
geohot c3832e1bde hmm, fix layernorm to not be batchnorm and it breaks 2020-12-28 13:06:21 -05:00
geohot 2e89e75dcb layernorm fixes transformer instability 2020-12-28 12:58:15 -05:00
geohot 593233b668 log and exp are first class ops 2020-12-28 10:00:30 -05:00
Marcel BischoffandGitHub ffff98db78 Evaluation in Transformers (#218) 
* 2serious

* load/save

* fixing GPU

* added DEBUG

* needs BatchNorm or doesn't learn anything

* old file not needed

* added conv biases

* added extra/training.py and checkpoint

* assert in test only

* save

* padding

* num_classes

* checkpoint

* checkpoints for padding

* training was broken

* merge

* rotation augmentation

* more aug

* needs testing

* streamline augment, augment is fast thus bicubic

* tidying up

* transformer eval
 2020-12-28 09:24:51 -05:00
geohot 65b07d2f4f fix onehot embed 2020-12-27 18:50:38 -05:00
geohot d864e1c71a transformer is training 2020-12-27 18:46:32 -05:00
geohot a361ef6861 fixup training loop 2020-12-27 18:35:56 -05:00
geohot f15bec6dbc make multidot work on CPU 2020-12-27 17:25:37 -05:00
geohot 131e04c90c cpu only decorator 2020-12-27 17:18:55 -05:00
geohot 2f1b2c0a3b add transpose, start on transformer 2020-12-27 16:59:12 -05:00
LiamandGitHub 075cf4bc02 Update examples to new API (#205) 2020-12-19 09:04:12 -08:00
geohot e5f2055776 ane refactor, fix examples 2020-12-18 09:19:06 -08:00
iainwoGitHubIain Wong
 		56d44637f3 fixed pylint, formatted python files iwth cblack on localhost (#204) 
* fixed pylint, formatted python files iwth cblack on localhost

* Revert "fixed pylint, formatted python files iwth cblack on localhost"

This reverts commit 07e2b88466.

* dedented 4-spaces added linter

Co-authored-by: Iain Wong <[email protected]>
 2020-12-17 14:37:31 -08:00
Marcel BischoffandGitHub da72a0eed4 Big MNIST model with PIL augmentation and load/save (#160) 
* 2serious

* load/save

* fixing GPU

* added DEBUG

* needs BatchNorm or doesn't learn anything

* old file not needed

* added conv biases

* added extra/training.py and checkpoint

* assert in test only

* save

* padding

* num_classes

* checkpoint

* checkpoints for padding

* training was broken

* merge

* rotation augmentation

* more aug

* needs testing

* streamline augment, augment is fast thus bicubic

* tidying up
 2020-12-13 20:45:55 -08:00
NeuralLinkandGitHub 1a1c63a08b Gan is real...Look what tiny just generated! (#192) 
* mode collapse solved

* info add

* delete unnecessary imports

* readme
 2020-12-13 20:23:12 -08:00
Marcel BischoffandGitHub 6785614239 tinygrad.utils to extra.utils fix in mnist_gan (#190) 2020-12-12 20:52:36 -08:00
NeuralLinkandGitHub d901ef6b23 🎉 effort to generate mnist data using GAN with tinygrad. [WIP] (#166) 
* 🎉 effort to generate mnist data with tinygrad.

* dropout added

* working gan

* minor bug fixes

* more bug fixes

* todo reg l2

* detach

* logsoftmax twice
 2020-12-12 17:58:04 -08:00
geohot a5aced8d47 30 MEGAReLUs. we need to lose 12 lines 2020-12-12 17:07:34 -08:00
geohot bc5df477de readme and .ane() 2020-12-12 16:15:38 -08:00
George HotzandGitHub da873cd556 Single ReLU in ANE (#188) 
* aneworks

* cleanup
 2020-12-12 16:11:34 -08:00
geohot 1d10559d1d tinygrad.utils -> extra.utils 2020-12-12 15:26:07 -08:00
DauletandGitHub c7e95ddb21 Add diamond model test (#181) 
* add backward pass test for diamond model

* fix train_efficientnet example
 2020-12-11 09:21:36 -08:00
Marcel BischoffandGitHub d204f09316 some progress on batchnorms (draft) (#147) 
* no of categories for efficientnet

* need layer_init_uniforn

* merge fail

* merge fail

* batchnorms

* needs work

* needs work how determine training

* pow

* needs work

* reshape was needed

* sum with axis

* sum with axis and tests

* broken

* works again

* clean up

* Update test_ops.py

* using sum

* don't always update running_stats

* space

* self

* default return running_stats

* passes test

* need to use mean

* merge

* testing

* fixing pow

* test_ops had a line dropped

* undo pow

* rebase
 2020-12-09 22:14:27 -08:00
geohot 9a64d13b94 add conv biases and max pool 2020-12-09 08:01:20 -08:00
geohot 99fa65f057 enable batchnorm in serious mnist 2020-12-09 03:29:40 -08:00
geohot c225e62dd2 touchups 2020-12-09 02:52:28 -08:00
geohot 0e02f394ee serious_mnist 2020-12-08 21:43:05 -08:00
geohot 00312b8ad1 batchnorm work 2020-12-06 14:40:07 -08:00
geohot 102e6356e9 replace layer_init_uniform with .uniform 2020-12-06 13:44:31 -08:00
geohot 521098cc2f se optional, track time better 2020-12-06 12:29:42 -08:00
geohot 609d11e699 trainer works with CIFAR 2020-12-06 12:20:14 -08:00
geohot 80a9c777ba requires grad, optim in train enet 2020-12-06 11:10:30 -08:00
geohot c66c27d22e get parameters 2020-12-06 10:45:04 -08:00
geohot 51daaa43d4 fix memory leaks, add gc test 2020-12-06 10:34:40 -08:00
geohot b8deb36e56 train BS=16 for 32 steps 2020-12-04 10:00:32 -08:00
adamritterGitHubholonomicjl
 		5797e63d9b Train efficientnet should respect NUM environment variable (#122) 
Co-authored-by: holonomicjl <[email protected]>
 2020-11-16 20:02:31 -08:00
geohot 2ffb8de1ea move efficientnet to extra 2020-11-16 08:08:07 -08:00
dustcollector12andGitHub 6f033ea30a enable local images for efficientnet.py (#116) 2020-11-13 07:00:12 -08:00
Marcel BischoffandGitHub a3989f9e18 Supporting .png files in efficientnet (#102) 
* to make it work locally

* definitely not working

* Conv2D GPU passes some of the tests

* Conv2D GPU passes more of the tests

* passes some tests and mnist

* removed unecessary code

* Conv2D Backpass works

* wrong test_ops.py

* white space + test backward

* ereased useless code

* removed default argument

* long lines

* works also with 4 channel .png files

* commenting out

* track
 2020-11-10 20:06:24 -08:00
geohot db755fa103 promote swish to a tensor ops 2020-11-10 15:48:11 -08:00
geohot 55012d21bb debug in backward pass too 2020-11-10 01:19:52 -08:00
geohot c76a20b4be 4s and 7s work 2020-11-10 00:54:17 -08:00
geohot 8b23033fa9 support all the enet sizes 2020-11-09 18:04:16 -08:00