Elias WahlandGitHub 4a114756f6 New BERT dataloader (#5881) ...

* One file == One topic

* update test

* new dataloader

* update train script

* get index is faster

2024-08-02 15:12:23 -04:00

..

create_pretraining_data.py

Wikipedia preprocessing script (#4229)

2024-04-23 10:28:01 -04:00

external_test_preprocessing_part.py

New BERT dataloader (#5881)

2024-08-02 15:12:23 -04:00

pick_eval_samples.py

Wikipedia preprocessing script (#4229)

2024-04-23 10:28:01 -04:00

tokenization.py

Wikipedia preprocessing script (#4229)

2024-04-23 10:28:01 -04:00