​​Long Text Generation by Modeling Sentence-Level and Discours | Data Science by ODS.ai 🦜

​​Long Text Generation by Modeling Sentence-Level and Discourse-Level Coherence

Modern NLP models still struggle with generating long and coherent texts, especially for open-ended dialogues such as story generation. The authors of the paper suggest a new model architecture HINT (a generation model equipped with HIgh-level representations for loNg Text generation) with two pre-training objectives to improve the language generation models: predicting inter-sentence semantic similarity and distinguishing between normal and shuffled sentence orders. Experiments and ablation studies show that these improvements result in more coherent texts than state-of-the-art baselines.


Paper: https://arxiv.org/abs/2105.08963

Code: https://github.com/thu-coai/HINT

A detailed unofficial overview of the paper: https://andlukyane.com/blog/paper-review-hint

#deeplearning #nlp #nlg #pretraining