Master Thesis Wei Ding

Exploring the Possibilities of Applying Transfer Learning Methods for Natural Language Processing in Software Development

Nowadays, we have a growing number of mature applications in the field of natural language processing (NLP), especially natural language understanding (NLU) and generation (NLG), like chatbots or auto-generated reports. Such applications relieve users from repeatable works and assist them in achieving high-demanding yields. We have different programming languages in the software development domain, which require deep understanding by human beings. Suppose we can apply methods for natural language processing in the software development world. In that case, we could help both programmers with good programming skills and project managers or data scientists, who need to understand code but do not have a strong programming background, to do their job more convenient, by generating documents to make the code easier to read and understand, generating code difference descriptions to compare and evaluate codes quickly, or generating code structure or dependencies to make programming more manageable.

In recent years, transfer learning is becoming quite successful. This machine learning method pre-trains a model first on a large amount of unlabeled data with an unsupervised task, then fine-tunes the same model on smaller labeled datasets. In this thesis, we examined the effect of transfer learning for tasks in the software development domain. We compared transfer learning with single-task learning and multi-task learning on thirteen tasks involving nine programming languages. We used the transformer encoder-decoder architecture to develop different sizes of models with these training strategies. We call these models CodeTrans. Our CodeTrans models outperform all the state-of-the-art models for all the tasks.

The pre-trained models generated by transfer learning could be applied to the tasks in the software development domain. Fine-tuning these models on new tasks would save a lot of training steps and time. Therefore, we published our CodeTrans pre-trained and fine-tuned models online so that everyone can use these models to generate text for relevant tasks or to fine-tune new tasks freely.