Of the 8,000 examples, Fasttext only misclassified 43. I was, quite frankly, astonished at its performance out of the box. I ran this over the validation set to get a sense of how well the model did. It’s that simple to identify which language an arbitrary string is. Today we’re only going to use its language prediction capabilities. This library has tons of amazing stuff in it. To do that we turn to the excellent Fasttext library from Facebook. Naturally, the first step toward normalizing any language to English is to identify what our unknown language is. Taking a look at the training data we see that there are about 220K English* example texts labeled in each of six categories.Įxample of the validation data 🕵️♀️ Identify the Language 🕵️♀️ Then use our translation transformer to convert all other texts to English and make our predictions using the English model. We can train a simple model on the English training set. It has a training set of over 223k comments labeled as toxic or not in English and 8k comments from other languages in a validation set. The Jigsaw Multilingual Toxic Comment Classification challenge from Kaggle is perfect for this. To explore how effective this approach is I needed a dataset of small text spans in many different languages. The very talented Chema Bescós has been kind enough to translate this article into Spanish which you can find here if English is not your first language :). If you build a model in just English your performance will suffer, but if you can normalize all the text to one language you’ll probably do better. This is useful because sometimes you’ll be working in a domain where there is textual data from many different languages. I almost feel bad making this tutorial because building a translation system is just about as simple as copying the documentation from the transformers library.Īnyway, in this tutorial, we’ll make a transformer that will automatically detect the language used in text and translate it into English. HuggingFace recently incorporated over 1,000 translation models from the University of Helsinki into their transformer model zoo and they are good.
Now the models are so much better and the tooling around these models leagues better as well. All that was to translate one language to one other language.
It was a ton of work from processing the data to designing and implementing the model architecture. I remember when I built my first seq2seq translation system back in 2015.
0 kommentar(er)
