We introduce BERT-NAR-BERT (BnB) – a pre-trained non-autoregressive sequence-to-sequence model, which employs BERT as the backbone for the encoder and decoder for natural language understanding and generation tasks. During the pre-training and fine-tuning with BERT-NAR-BERT, two challenging aspects are considered by adopting the length classification and connectionist temporal classification models to control the output length of BnB. We evaluate it using a standard natural language understanding benchmark GLUE and three generation tasks – abstractive summarization, question generation, and machine translation. Our results show substantial improvements in inference speed (on average 10x faster) with only little deficiency in output quality when compared to our direct autoregressive baseline BERT2BERT model. Our code is publicly released on GitHub (https://github.com/aistairc/BERT-NAR-BERT) under the Apache 2.0 License.We introduce BERT-NAR-BERT (BnB) – a pre-trained non-autoregressive sequence-to-sequence model, which employs BERT as the backbone for the encoder and decoder for natural language understanding and generation tasks. During the pre-training and fine-tuning with BERT-NAR-BERT, two challenging aspects are considered by adopting the length classification and connectionist temporal classification models to control the output length of BnB. We evaluate it using a standard natural language understanding benchmark GLUE and three generation tasks – abstractive summarization, question generation, and machine translation. Our results show substantial improvements in inference speed (on average 10x faster) with only little deficiency in output quality when compared to our direct autoregressive baseline BERT2BERT model. Our code is publicly released on GitHub (https://github.com/aistairc/BERT-NAR-BERT) under the Apache 2.0 License. Leer más
Related posts:
Relative cost of ginning cotton using saw, conventional roller, and high‐speed roller gins in the United States
In data: finding importers is largest hurdle for wine exporters to Japan
Runtime and Design Time Completeness Checking of Dangerous Android App Permissions Against GDPR
Model-averaging as an accurate approach for ex-post economic optimum nitrogen rate estimation
A Graph Neural Network for EEG-Based Emotion Recognition With Contrastive Learning and Generative Adversarial Neural Network Data Augmentation
2.1 W/mm Output Power Density at 10 GHz for H-Terminated Diamond MOSFETs With (111)-Oriented Surface