From statistical counts to semantic and contextual embeddings, this article will teach you how to effectively transform unprocessed text into numerical features that machine learning models can utilize.
Among the subjects we will discuss are:
- How to use and why TF-IDF is still a reliable statistical baseline.
- Meaning beyond keywords is captured by averaging GloVe word embeddings.
- How context-aware representations are provided by transformer-based embeddings.
Overview
One basic drawback of machine learning models that frequently irritates novices in natural language processing (NLP) is their inability to read. A logistic regression or neural network will fail instantly if you feed it a raw email, a customer review, or a legal contract. Algorithms are mathematical operations on equations that depend on numerical input to work. They are able to comprehend vectors but not words.
A key procedure that fills this gap is feature engineering for text. It is the process of converting the qualitative subtleties of spoken language into numerical lists that a machine can understand. The success of a model is frequently determined by this translation layer. A simple algorithm provided rich, representative features will outperform a sophisticated algorithm fed poorly designed features.
The last few decades have seen a substantial change in the field. From straightforward counting systems that view documents as collections of unconnected words to intricate deep learning architectures that comprehend a word’s meaning depending on its surrounding words, it has come a long way.
The statistical underpinnings of TF-IDF, the semantic averaging of GloVe vectors, and the most advanced contextual embeddings offered by transformers are the three different approaches to this problem that are covered in this article.
1. TF-IDF Vectorization: The Statistical Basis
Counting them is the simplest approach to convert words to numbers. This was the norm for many years. A technique called “bag of words” allows you to easily count the instances of a term in a document. Raw counts, however, have a serious drawback. Grammatically obligatory but semantically empty articles and prepositions like “the,” “is,” “and,” and “of” are the most common words in practically any English text. These often occurring words will overpower your data if you depend solely on raw counts, overpowering the uncommon, specialized terms that truly give the document its meaning.
We employ term frequency–inverse document frequency (TF-IDF) to solve this. This method evaluates terms based on their rarity throughout the dataset as well as how frequently they occur in a particular document. It is a statistical balancing act that rewards unique words and penalizes common ones.
phrase frequency (TF), the first section, quantifies how often a phrase appears in a document. Inverse document frequency (IDF), the second component, gauges a term’s significance. The logarithm of the total number of documents divided by the number of documents containing the particular term yields the IDF score.
When the word “data” appears in every single document in your dataset, its IDF score becomes close to zero, which effectively eliminates it. On the other hand, the word “hallucination” has an extremely high IDF score if it appears in just one document. The outcome of multiplying TF by IDF is a feature vector that identifies the individual characteristics that set one document apart from another.
2. Averaged Word Embeddings (GloVe) For Meaning Capture
Although TF-IDF is effective at matching keywords, it lacks semantic understanding. Because the words “good” and “excellent” have distinct spellings, it treats them as entirely separate mathematical properties. It is unaware that their meanings are almost identical. We turn to word embeddings to solve this.
Words are transferred to vectors of real numbers using a process called word embedding. The fundamental idea is that mathematical representations of words with comparable meanings ought to be similar. The distance between the vectors for “king” and “queen” in this vector space is comparable to that between “man” and “woman.”
GloVe (global vectors for word representation), created by Stanford academics, is one of the most widely used pre-trained embedding sets. Their findings and datasets are available on the official Stanford GloVe project page. Billions of words from Wikipedia and Common Crawl data were used to train these vectors. The model determines the semantic association between words by looking at how frequently they appear together (co-occurrence).
We have a tiny obstacle to overcome before we can use this for feature engineering. Although our data often comprises of sentences or paragraphs, GloVe offers a vector for a single word. Finding the mean of the vectors of the words in a sentence is a popular and useful method for representing the entire sentence. You check up each word’s vector in a ten-word sentence and average them all. As a result, the entire sentence’s “average meaning” is represented by a single vector.
3. Transformer-Based Embeddings For Contextual Intelligence
Although the averaging approach outlined above was a significant advancement, it came with a new drawback: it disregards context and order. Because “The dog bit the man” and “The man bit the dog” contain exactly the same words, they produce exactly the same vector when averaged. Additionally, whether you are sitting on a “river bank” or visiting a “financial bank,” the word “bank” has the same static GloVe vector.
We use transformers, more especially BERT (Bidirectional Encoder Representations from Transformers) models, to solve this. Transformers use a technique known as “self-attention” to read the full sequence at once rather than sequentially from left to right. This enables the model to comprehend that a word’s meaning is determined by the words that surround it.
It is not always necessary to train a model from scratch when feature engineering with a transformer. As a feature extractor, we instead employ a pre-trained model. After feeding the model our text, we take the output out of the last hidden layer. In particular, models such as BERT prepend a unique token known as the [CLS] (classification) token to each sentence. The aggregate comprehension of the entire sequence is intended to be stored in the vector representation of this particular token once it has passed through the layers.
At the moment, this is regarded as the gold standard for text representation. You can study the Hugging Face Transformers library documentation, which has made these models available to Python developers, or read the groundbreaking work on this design, “Attention Is All You Need.”
In Conclusion
From the basic to the complex, we have explored the field of text feature engineering. We started with TF-IDF, a statistical technique that is excellent at matching keywords and is still quite useful for spam filtering or basic document retrieval. We then switched to averaged word embeddings, like GloVe, which gave models semantic meaning and enabled them to comprehend analogies and synonyms. Lastly, we looked at transformer-based embeddings, which provide deep, context-aware representations that support today’s most sophisticated AI applications.
There is only the appropriate technique for your restrictions; there is no one “best” technique among these three. TF-IDF doesn’t require a lot of hardware and is quick and interpretable. Transformers are the most accurate, but they also use a lot of memory and processing power. It is your responsibility as a data scientist or engineer to balance these trade-offs in order to provide the best possible solution for your particular issue.