OCR Explained: How AI Reads Text From Images and PDFs

Optical character recognition (OCR) is the technology that reads text from images: scanned pages, photos of signs, bank cheques, ID cards or screenshots. The result is machine-readable text you can search, copy and process. OCR has existed for decades, but deep learning has made it far more robust, especially for messy real-world images.

How OCR works

Modern OCR typically has four stages.

1. Image preprocessing

Raw images are cleaned to make text stand out:

Converting to grayscale
Removing noise and shadows
Correcting skew (rotating tilted pages)
Adjusting contrast and binarisation (turning the image into black text on white background)
Resizing so characters have enough pixels, commonly around 30 pixels in height or about 300 DPI for scans

Good preprocessing often matters more than the choice of OCR engine.

2. Text detection

The system locates regions that contain text: lines, words or characters. In natural scene images (like street signs), neural networks detect text boxes in varying orientations and fonts.

3. Recognition

Each detected region is converted to characters. Older engines segmented individual characters and matched shapes. Modern engines such as Tesseract 4 and later use recurrent or transformer-based neural networks that read whole lines at once, which handles joined letters and varied fonts better.

4. Post-processing

Dictionaries, language models and layout rules fix errors, for example distinguishing the digit 0 from the letter O using context. Systems may also reconstruct reading order, tables and paragraphs.

What affects accuracy

Image quality: blur, low resolution, glare, shadows and compression artefacts.
Fonts and handwriting: printed text in common fonts is easiest; handwriting is much harder.
Layout: multiple columns, tables, forms and background patterns complicate reading.
Language and script: engines need trained data for the script. Devanagari (used for Nepali) is supported by several engines, but accuracy may be lower than for English, especially with conjunct characters and poor scans.
Document type: receipts, IDs and invoices often need template-aware extraction.

A Python example with Tesseract

Install the Tesseract engine on your computer, then the Python wrappers.

pip install pytesseract pillow opencv-python

import cv2
import pytesseract

img = cv2.imread("scan.jpg")
gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
gray = cv2.resize(gray, None, fx=1.5, fy=1.5, interpolation=cv2.INTER_CUBIC)
_, binary = cv2.threshold(gray, 0, 255, cv2.THRESH_BINARY + cv2.THRESH_OTSU)

# lang="eng" for English; use "nep" (Nepali) or "eng+nep" if the language data is installed
text = pytesseract.image_to_string(binary, lang="eng", config="--psm 6")
print(text)

The --psm option tells Tesseract how to interpret layout. For example, 6 assumes a single uniform block of text, while 4 is for single columns of varying sizes. Trying different modes can improve results.

Getting positions and confidence

OCR can return the location and confidence of each word, useful for validation:

data = pytesseract.image_to_data(binary, output_type=pytesseract.Output.DICT)
for word, conf in zip(data["text"], data["conf"]):
    if word.strip() and float(conf) < 60:
        print("Check this word:", word, conf)

Flagging low-confidence words lets a person verify only the uncertain parts.

Tips for better results

Scan at 300 DPI or take photos in good, even light, directly from above.
Avoid shadows and glare. Use a flat surface and indirect light.
Keep the page straight or deskew it.
Crop to the relevant area, such as the field you need.
Use the right language data and combine languages only when necessary.
Validate with rules. For example, check that a date has a valid format or that totals add up.
Keep a human review step for important documents.

Beyond plain OCR: document understanding

Many business needs go beyond reading characters. Extracting the invoice number, date and total from a bill requires understanding layout and field meaning. Approaches include:

Template-based extraction for fixed forms
Layout-aware models that use both text and position
Vision-language models that can answer questions about a document image
Combining OCR with a language model to convert messy text into structured JSON

When building ERP and billing tools, the practical pattern that works well is: OCR, then rules for easy fields, then a model for hard cases, with human confirmation before data enters the accounting records.

Privacy

Documents often contain sensitive personal and financial data. If you use a cloud OCR service, check its data handling terms, or run an open-source engine on your own server.

Frequently Asked Questions

Can OCR read handwriting?

Some engines can read neat handwriting, but accuracy is lower and varies widely. Specialised handwriting recognition models perform better.

Does OCR work for Nepali?

Yes, with language models trained for Devanagari, though quality depends heavily on scan quality and font. Always test on your own documents.

Why does my OCR output contain strange characters?

Likely causes are a blurry image, wrong language setting, low resolution or a decorative font. Preprocessing and language selection usually help.

Is OCR the same as AI?

OCR is a task. Modern OCR uses AI methods such as neural networks to achieve high accuracy.

Conclusion

OCR turns images of text into data through preprocessing, detection, recognition and cleanup. Better images, correct language settings and validation produce better results, and for important documents, a quick human check remains wise.