nimlgen/tinygrad
1
0
Fork 0
forked from tinygrad/tinygrad
Code Pull Requests Activity
8,135 Commits 334 Branches 15 Tags
3af7a08a06afe76e5fb727ec71981c94da660c39
Commit Graph
184 Commits
Author SHA1 Message Date
chenyuandGitHub 4992958dae update bert beam params (#9423) 
BEAM_MIN_PROGRESS=5 for setup speed
 2025-03-12 13:00:41 -04:00
chenyuandGitHub 22fc0a2e36 bert sum acc in half (#9412) 
also BS=96
 2025-03-11 23:03:15 -04:00
chenyuandGitHub 01e8b60911 acc_dtype -> dtype (#9402) 
matched numpy and torch
 2025-03-10 16:05:30 -04:00
chenyuandGitHub 2af129c078 bert corealize multiple outputs (#9359) 
1% faster step
 2025-03-05 10:58:37 -05:00
chenyuandGitHub ad72269f08 bert put eval copy and getting lr in jit (#9350) 2025-03-04 20:57:03 -05:00
chenyuandGitHub 9eb45eb629 add a flag to skip bert train (#9349) 2025-03-04 17:13:00 -05:00
qazalandGitHub 845814f396 revert buffer_view change (#9311) 
* Revert "BUFFER_VIEW is a node in the kernel graph + delete ViewOp (#9298)"

This reverts commit 3210b656b6.

* Revert "substitute ast from kernel op [pr] (#9293)"

This reverts commit 5a9c788ae6.
 2025-03-01 11:00:12 +01:00
qazalandGitHub 3210b656b6 BUFFER_VIEW is a node in the kernel graph + delete ViewOp (#9298) 2025-02-28 12:15:04 +02:00
George HotzandGitHub 3f4eb9006a test for device mismatch [pr] (#9250) 
* test for device mismatch [pr]

* fix bert
 2025-02-26 13:06:33 +08:00
chenyuandGitHub 979e84f30e RESET_STEP in bert setup and beam (#9248) 
running dev_beam migh OOM without it but runs fine in real run.
 2025-02-25 19:15:10 -05:00
chenyuandGitHub 6610ad58ab hotfix bert no shard with only one device (#9243) 
`LLVM=1 BERT_SIZE="tiny" DEFAULT_FLOAT=HALF BENCHMARK=5 MODEL="bert" python3 examples/mlperf/model_train.py` runs for me with this. it should not failed with single device shard though
 2025-02-25 09:05:11 -05:00
chenyuandGitHub 8c7be428e5 update bert BS to 78 (#9236) 
fits 78 now. about 215 tflops on green
 2025-02-24 22:47:35 -05:00
chenyuandGitHub 2e7c2780a9 CLANG -> CPU (#9189) 2025-02-20 18:03:09 -05:00
chenyuandGitHub 3b37cc898b add bert tiny config (#9177) 
set with BERT_SIZE=tiny. easier to study embedding and fusion
 2025-02-19 14:57:03 -05:00
chenyuandGitHub 975c318dbc bert use int32 for input ids (#9173) 
original data was int32 for these. float might have caused precision issues
 2025-02-19 08:17:27 -05:00
chenyuandGitHub ff05bff221 put bert data shard inside jit (#9160) 
python time 45ms -> 9ms, it was spending time to schedule the shard

also init bert data on CLANG since it's from numpy, so we don't create the tensor on default device then shard into GPUS
 2025-02-18 10:36:54 -05:00
chenyuandGitHub 5dc1257ce0 clean up bert fake data iterator [pr] (#9145) 
reuse the same get_data_bert path in setup and real run
 2025-02-17 20:03:38 -05:00
chenyuandGitHub 81597ddd96 increase lr for bert (#9098) 
had one run that converged better https://wandb.ai/chenyuxyz/MLPerf-BERT/runs/u66tv2hh/overview
 2025-02-14 19:10:35 -05:00
chenyuandGitHub b58e7b1898 zero out the weight in bert init run (#9076) 
`DEFAULT_FLOAT=HALF BENCHMARK=10 BS=66 EVAL_BS=6 GPUS=6 MODEL=bert python3 examples/mlperf/model_train.py` no longer oom. I think the buffer of random init weights caused the oom.
 2025-02-14 08:40:41 -05:00
chenyuandGitHub 9e91898941 bert eval at the end of training (#9070) 
always eval at the last epoch
 2025-02-13 16:29:44 -05:00
chenyuandGitHub 7b5ac2c15e free_intermediates in bert (#9040) 
also re-enable dropout and update EVAL_BS
 2025-02-12 10:00:39 -05:00
chenyuandGitHub a092b6395d Tuple -> tuple, List -> list [pr] (#8936) 2025-02-06 14:21:19 -05:00
chenyuandGitHub c7ca7959e6 set DISABLE_DROPOUT=1 in bert script for now (#8799) 2025-01-29 10:51:29 -05:00
chenyuandGitHub c99ae81f63 update default resnet LOSS_SCALER to 256 [pr] (#8774) 2025-01-27 16:59:05 -05:00
chenyuandGitHub af65331b76 update beam params for bert green [pr] (#8726) 
increase BEAM_UPCAST_MAX and BEAM_LOCAL_MAX to default and matched red. 3% faster step
 2025-01-22 22:00:05 -05:00
chenyuandGitHub 9a9079118e envvar BERT_LAYERS [pr] (#8709) 
default is 24 for large
 2025-01-21 22:49:19 -05:00
chenyuandGitHub 9f6d545a16 bert log global_norm in training step [pr] (#8708) 
* bert log global_norm in training step [pr]

and minor cleanups

* .item()
 2025-01-21 20:36:27 -05:00
chenyuandGitHub 1e283c33d3 remove realize in bert model init [pr] (#8707) 2025-01-21 14:11:03 -05:00
chenyuandGitHub 930728c069 bert BS 72->66 [pr] (#8621) 
72 does not fit now
 2025-01-14 18:41:41 -05:00
chenyuandGitHub 994944920b simpler batch_load_train_bert [pr] (#8582) 
don't think that buffer is really beneficial. 5% faster data_time and 1ms faster per step.
https://wandb.ai/chenyuxyz/MLPerf-BERT/runs/69c9lx8y/overview
 2025-01-12 20:25:05 -05:00
chenyuandGitHub def90b22f6 EVAL_BS=36 for bert [pr] (#8576) 
3X faster eval compared to BS=6.
green https://wandb.ai/chenyuxyz/MLPerf-BERT/runs/ka5p5sm9/overview
red https://wandb.ai/chenyuxyz/MLPerf-BERT/runs/a7maxsxd/overview
 2025-01-12 09:43:56 -05:00
chenyuandGitHub 64a917b7eb remove LAZYCACHE ContextVar [pr] (#8175) 
also removed from resnet latest script
 2024-12-11 22:02:52 -05:00
chenyuandGitHub 3e2430f822 use tqdm tqdm in mlperf training (#7929) 
issue in benchmark dashboard logging, revert back to tqdm tqdm for now
 2024-11-27 21:57:05 -05:00
qazalandGitHub 9828277c03 view doesn't have buffer, fix the tests [pr] (#7841) 
* view doesn't have buffer, fix the tests [pr]

* need assigns
 2024-11-22 20:41:55 +08:00
Francis LataGitHubchenyu
 		90eff347e2 tinytqdm write support (#6359) 
* add write support

* add test

* update test case to compare write outputs

* assert final write output

* flush when using write

* update write logic

* Revert "update write logic"

This reverts commit 5e0e611b46.

---------

Co-authored-by: chenyu <[email protected]>
 2024-10-16 14:51:41 -04:00
chenyuandGitHub ed1ed9e4ff bert use BS=72 (#7015) 
memory 131 -> 138
green tflops 201 -> 209
red tflops 160 -> 169
 2024-10-12 09:41:56 -04:00
chenyuandGitHub 36056e0760 update mlperf systems and copy 4.1 to 5.0 (#7004) 2024-10-11 16:20:34 -04:00
chenyuandGitHub 0e42662f2a log seed at the right place for bert (#7000) 2024-10-11 10:39:40 -04:00
nimlgenandGitHub 5496a36536 update red mlperf bert readme (#6969) 2024-10-11 13:08:06 +03:00
chenyuandGitHub b5546912e2 10% more TRAIN_STEPS for bert (#6971) 
got two very close run, adding more steps for buffer
 2024-10-09 19:21:43 -04:00
chenyuandGitHub 35cf48659b limit beam param for bert on green (#6966) 
seems to mitigate the crash
 2024-10-09 11:48:18 -04:00
chenyuandGitHub 1ff2c98f8a fix logfile name for bert red (#6952) 2024-10-08 05:37:52 -04:00
chenyuandGitHub a78c96273a update bert epoch logging (#6940) 
* update bert epoch logging

epoch for bert is simply number of examples seen (which is used for RCP check)

* update total steps too

* more changes
 2024-10-08 00:34:06 -04:00
chenyuandGitHub 102dfe5510 back to 2**10 for bert loss scaler (#6934) 
getting 2 NaN for this, revert back to 2**10
 2024-10-07 10:17:21 -04:00
chenyuandGitHub 0cf815a93a bert use BS=66 and update hparams (#6932) 
with dropout memory improvement, we can fit BS=66 now. revert back to the hparams in #5891 too
 2024-10-07 05:08:27 -04:00
chenyuandGitHub 718b959349 log epoch start and stop for bert (#6912) 2024-10-06 06:39:46 -04:00
chenyuandGitHub 16c1fa4208 use BEAM=3 for red box bert runs (#6904) 
BEAM=4 slightly exceeded 30 minutes setup
 2024-10-05 09:21:12 -04:00
chenyuandGitHub 0e706227a2 add seed to bert result log filename (#6903) 
* add seed to bert result log filename

* different name for different benchmark
 2024-10-05 09:15:24 -04:00
chenyuandGitHub 7391376528 update bert hparams (#6876) 
4h32m with this https://wandb.ai/chenyuxyz/MLPerf-BERT/runs/q99frv1l/overview.

loss scaler 2**13->2**10. matched the closest submission, no nan for ~10 runs.

increased lr and total step a bit.

`PARALLEL=0` after setup, same as resnet.
 2024-10-04 00:39:06 -04:00
chenyuandGitHub 5f77217772 bert default CKPT to 0 (#6840) 
not required
 2024-10-01 21:55:56 -04:00