AI for Accessibility: Captions, Screen Readers and Image Descriptions

Billions of people live with disabilities that affect how they see, hear, speak, move or process information. Accessible technology lets them take part in education, work and daily life on equal terms. AI is expanding what is possible: automatic captions, descriptions of images, speech interfaces and simplified text. It is not a magic fix, but used thoughtfully, it can remove real barriers.

Where AI helps

For people who are deaf or hard of hearing

• Automatic captions for videos, meetings and phone calls using speech recognition.
• Live transcription apps on phones that show what is being said in real time.
• Sound recognition alerts for doorbells, alarms or crying babies.
• Sign language research: recognition and generation of sign language is advancing but remains limited; it is not a replacement for human interpreters.

For people who are blind or have low vision

• Image descriptions (alt text): vision-language models can describe photos, charts and screenshots.
• Scene and object recognition apps that read labels, currency notes, colours and signs.
• OCR that reads printed text aloud from a camera.
• Screen readers (software that reads interfaces aloud) combined with AI summaries of complicated pages.
• Navigation aids that describe surroundings.

For people with speech or motor disabilities

• Voice control to operate devices hands-free.
• Speech recognition tuned to atypical speech, with personalised models for people with conditions that affect speech.
• Predictive text and eye-tracking communication boards that speed up typing.
• Voice banking and synthetic voices that preserve a person's own voice.

For people with cognitive or learning differences

• Text simplification and summarisation to plain language.
• Text-to-speech with highlighting for dyslexia and reading support.
• Reminders, task breakdowns and planning assistants.
• Translation for people who use a second language.

Examples in everyday tools

• Video platforms generate captions automatically, in multiple languages.
• Phones describe images or read text in the camera view.
• Meeting software provides live captions and transcripts.
• Browsers offer reading modes and read-aloud features.
• Messaging apps convert voice notes to text.

Benefits beyond disability

Accessibility features help everyone. Captions help in noisy places or when sound must be off. Voice control helps when hands are busy. Simplified language helps non-native speakers. This is sometimes called the curb-cut effect: features designed for some benefit all.

Limitations and risks

• Accuracy: automatic captions can mishear names, technical terms and accents. Errors can change meaning. For important content (courses, legal, medical), review or use professional captioners.
• Poor alt text: AI-generated descriptions can be generic ("a person standing"), wrong or miss what is important for context. Good alt text depends on the purpose of the image.
• Language coverage: support for Nepali and many other languages is weaker than for English.
• Bias: speech recognition often works worse for certain accents, genders or speech patterns, and vision models may misdescribe people.
• Privacy: cameras and microphones capture sensitive information. Understand where data goes.
• Over-reliance: replacing human support or accessible design with AI patches is not enough.
• Inclusion in design: tools built without disabled users' involvement often miss real needs.

Accessible design still comes first

AI should supplement, not replace, accessibility practices:

1. Write meaningful alt text for informative images; leave decorative images empty.
2. Use semantic HTML: headings, lists, labels and landmarks that screen readers understand.
3. Ensure sufficient colour contrast and do not rely on colour alone.
4. Support keyboard navigation and visible focus.
5. Provide captions and transcripts for audio and video.
6. Allow text resizing and responsive layouts.
7. Avoid flashing content that can trigger seizures.
8. Test with real users and with assistive technologies.
9. Follow standards such as the Web Content Accessibility Guidelines (WCAG).

A practical example: drafting alt text with AI

AI can draft alt text that a person then checks and edits. A prompt for a vision-capable model:

import base64
import anthropic

client = anthropic.Anthropic()

def draft_alt_text(image_path: str, context: str) -> str:
    with open(image_path, "rb") as f:
        data = base64.standard_b64encode(f.read()).decode("utf-8")
    reply = client.messages.create(
        model="claude-sonnet-5-5",
        max_tokens=150,
        messages=[{
            "role": "user",
            "content": [
                {"type": "image", "source": {"type": "base64", "media_type": "image/jpeg", "data": data}},
                {"type": "text", "text": (
                    f"Write concise alt text (under 125 characters) for this image on a page about: {context}. "
                    "Describe what is important for understanding the page. Do not start with 'Image of'. "
                    "If the image is purely decorative, reply exactly: DECORATIVE.")},
            ],
        }],
    )
    return reply.content[0].text.strip()

print(draft_alt_text("trek.jpg", "a trekking guide for the Annapurna region"))

Review every result. Check that names, text in the image and key details are right, and that the description fits the page's purpose.

Adding captions to your own videos

A common workflow is to generate captions with speech recognition, then correct them. Open-source tools such as Whisper can produce subtitle files (SRT or VTT) that video players can use. Always proofread, especially names and numbers, and provide a transcript where possible.

Building inclusive AI products

• Involve people with disabilities in research, design and testing, and compensate them fairly.
• Offer multiple input and output options (voice, text, touch, keyboard).
• Allow users to correct and personalise recognition.
• Be transparent about accuracy and limitations.
• Protect privacy, especially for audio and video.
• Support local languages and dialects.
• Provide a human alternative, such as live interpreters or phone support, for critical services.

Local context

In Nepal, accessibility needs are significant, while resources for assistive technology and local-language tools are limited. Open-source speech and text-to-speech models, community-driven datasets and affordable smartphone-based apps can help, but they need local data, testing and partnership with disability organisations.

Frequently Asked Questions

Can AI replace sign language interpreters?

No. Current technology cannot match the nuance, accuracy and cultural role of human interpreters.

Are automatic captions good enough for compliance?

Often they need review. Accessibility regulations usually expect accurate captions, so correct automatic ones, particularly for important content.

Can AI write good alt text?

It can draft helpful descriptions, but humans should verify them and add context.

Where can I learn more?

Study WCAG guidelines, try assistive technologies yourself and listen to disabled users' experiences.

Conclusion

AI can widen access through captions, image descriptions, voice interfaces and simplified text. Its value is greatest when combined with accessible design, human review and the involvement of people who use assistive technology.