Natural language processing (NLP) is the area of AI that lets computers read, interpret and produce human language. Search engines, spell checkers, translation apps, email filters and chatbots all rely on it. Language is difficult for computers because it is ambiguous, full of context and constantly changing, so NLP combines linguistics, statistics and machine learning.
Why language is hard for machines
Consider these sentences.
- "I saw her duck." Did she lower her head or is there a bird?
- "The bank was crowded." River bank or financial bank?
- "Great, another meeting." Is that really positive?
Humans use context and world knowledge to resolve such cases. NLP systems must learn to do the same from data.
The classic NLP pipeline
Traditional systems processed text in stages.
- Tokenisation: split text into words or sub-words.
- Normalisation: lower-casing, removing punctuation, fixing spelling.
- Stop-word removal: dropping very common words such as "the" and "of" for some tasks.
- Stemming or lemmatisation: reducing words to a base form ("running" to "run").
- Part-of-speech tagging: labelling nouns, verbs and so on.
- Parsing: analysing grammatical structure.
- Named entity recognition (NER): finding people, places, organisations, dates and amounts.
Modern deep-learning models often learn many of these steps implicitly, but the vocabulary is still useful.
Representing words as numbers
Computers need numbers. Approaches have evolved.
- Bag of words: count how many times each word appears. Simple, but loses word order.
- TF-IDF: weights words so that those frequent in one document but rare overall stand out.
- Word embeddings: dense vectors where similar words are close together (Word2Vec, GloVe).
- Contextual embeddings: vectors that depend on the sentence, so "bank" gets different values in "river bank" and "bank loan". Transformer models produce these.
Common NLP tasks
| Task | What it does | Example |
|---|---|---|
| Text classification | Assign a category | Spam vs not spam |
| Sentiment analysis | Detect opinion | Positive, negative, neutral reviews |
| Named entity recognition | Extract entities | Find names and dates in contracts |
| Machine translation | Convert languages | Nepali to English |
| Summarisation | Shorten text | News digest |
| Question answering | Answer from text | Customer FAQ bot |
| Information extraction | Pull structured data | Invoice fields |
| Text generation | Write new text | Draft emails |
From rules to transformers
NLP has gone through three broad eras.
- Rule-based (1950s to 1980s): hand-written grammar rules and dictionaries.
- Statistical (1990s to 2000s): probabilistic models learned from corpora.
- Neural (2010s onward): recurrent networks, then transformers and large language models, which understand context far better and can be adapted to many tasks.
Today most advanced NLP starts with a pretrained transformer and fine-tunes it, or simply prompts a large language model.
Example: sentiment analysis in Python
With the Hugging Face transformers library you can classify sentiment in a few lines:
from transformers import pipeline
classifier = pipeline("sentiment-analysis") # downloads a default English model
reviews = [
"The staff were friendly and the room was spotless.",
"Delivery took forever and the box was damaged.",
]
for text, result in zip(reviews, classifier(reviews)):
print(f"{result['label']:8} {result['score']:.2f} {text}")For production, choose and evaluate a model that fits your language and domain, because a general English model may perform poorly on Nepali or on mixed-language text.
Challenges specific to Nepali and similar languages
- Less training data than English, so models may be weaker.
- Different scripts and romanised writing: people often type Nepali in English letters ("ramro cha"), which many models handle poorly.
- Code-mixing: sentences blend Nepali and English.
- Few ready-made evaluation datasets for tasks like sentiment or NER.
Collecting your own labelled examples from your domain is often the most effective improvement.
Where NLP shows up in business software
Even small organisations meet NLP quickly. A support inbox can be sorted by topic and urgency automatically. Free-text feedback forms can be summarised into themes. Product catalogues can be searched by meaning, so a customer typing "warm winter jacket" finds items titled "fleece coat". Names, phone numbers and amounts can be pulled out of messages and pasted into billing or ERP records, which saves manual typing and reduces mistakes. In each case, the best approach is to start with one narrow task, measure it on real messages, and keep a person in the loop for unclear cases.
Evaluating NLP systems
Pick metrics that match the task: accuracy, precision, recall and F1 for classification; BLEU or human review for translation; and always read a sample of real outputs. Numbers alone can hide embarrassing failures.
Frequently Asked Questions
Is NLP the same as a large language model?
No. NLP is the whole field. Large language models are one powerful technique within it.
Do I need linguistics knowledge?
Basic understanding helps, but you can build useful NLP applications with libraries and pretrained models.
What programming languages are used?
Python dominates thanks to libraries such as spaCy, NLTK and Hugging Face Transformers.
Can NLP understand sarcasm?
Sometimes, but it remains difficult. Sarcasm depends on tone and context that text alone may not reveal.
Conclusion
NLP turns messy human language into data computers can use. From tokenisation to transformers, the field has moved from hand-written rules to models that learn from vast text collections. Start with a clear task, a pretrained model and honest evaluation.