Sparse Distillation: Speeding Up Text Classification by Using Bigger Models

Qinyuan Ye, Madian Khabsa, Mike Lewis, Sinong Wang, Xiang Ren, Aaron Jaech