Our error analysis shows that BERTRAM is sensitive to the quality of examples it is shown, and that performance can be improved even further by manually selecting a gold set of contexts for each idiom. Our results, and subsequent ranking at first place in the pre-train setting and second place in the fine-tune setting, show that BERTRAM can learn high-quality word embeddings for idioms and that this leads to better performance on downstream tasks. BERTRAM is used due to its high performance on rare words, which share many properties with idioms such as non-compositionality and being rare examples of component pieces. We build our system by augmenting BERT models for each language with single token embeddings learnt using BERTRAM. Results for the baseline system are in italics. Aggregated results for all languages in bold. 3.5 Fine-tune Setting Figure 2: Overall and Idiom STS Only Spearman Rank on the development set whilst training on the Idiom STS data Settingįinal Spearman Rank (SR) scores of the system on the test set, split into idiom Semantic Text Similarity (STS), general STS, and all datasets. Due to a lack of dev split data for Galician we use the result from the Portuguese model as they are trained on the same data. The results (shown in figure 1) show that 45 epochs are optimal for Portuguese and 35 for English. Idiomatic and Semantically similar should be given an STS score of 1, and be given the same score when compared to the literal use.Įvaluating the models on the dev split, we investigate the optimal number of epochs for the English and Portuguese models. 2 Background Usageīlockchains, fundamentally, are banking because what they’re doing is allowing the transaction of value across networks … they’re doing it in an orthogonally different way," he said Wednesday in what may be his swan song in public office.īlockchains, fundamentally, are banking because what they’re doing is allowing the transaction of value across networks … they’re doing it in an orthogonally different way," he said Wednesday in what may be his bird song in public office.īlockchains, fundamentally, are banking because what they’re doing is allowing the transaction of value across networks … they’re doing it in an orthogonally different way," he said Wednesday in what may be his final performance in public office.Įxample sentences for the Idiomatic STS data. 1 1 1 The code for creating the embeddings and the modified baseline system code can be found on GitHub. ![]() This improved performance gained an overall spearman rank score of 0.6402 and first place on the pre-train setting, and an overall spearman rank score of 0.6504 and second place on the fine-tune setting. Evaluating our model on the task shows that externally trained idiom embeddings significantly increase the performance on STS data containing idioms while maintaining high performance on general STS data.
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |