Voice interfaces turn spoken words into text and written text back into speech. These two technologies, speech-to-text (STT), also called automatic speech recognition (ASR), and text-to-speech (TTS), power voice assistants, captions, call-centre tools, accessibility features and audiobooks. Here is how each one works.
Speech-to-text: from sound waves to words
Step 1: capturing audio
A microphone converts sound into an electrical signal, which is sampled many thousands of times per second to produce a digital waveform. A common speech sampling rate is 16,000 samples per second.
Step 2: extracting features
Raw waveforms are hard for models to use directly, so the audio is converted into a spectrogram, a picture showing which frequencies are present over time. A popular variant is the mel spectrogram, which scales frequencies roughly the way human hearing does.
Step 3: the acoustic and language model
Modern systems use deep neural networks, often transformers, that map the sequence of spectrogram frames to a sequence of characters, word pieces or words. The model learns both how sounds map to letters and which word sequences are likely, so it can choose "recognise speech" rather than "wreck a nice beach" from similar sounds.
Step 4: decoding and cleanup
The system picks the most probable transcript, then adds punctuation, capitalisation and sometimes speaker labels and timestamps.
What makes speech recognition hard?
- Accents and dialects: a model trained mostly on one accent may perform worse on others.
- Background noise and echo: busy streets and poor microphones reduce accuracy.
- Overlapping speakers: separating voices is difficult.
- Domain vocabulary: names, technical terms and place names can be misrecognised.
- Code-switching: mixing Nepali and English in one sentence challenges many systems.
- Low-resource languages: less training data usually means lower accuracy.
Accuracy is often measured by word error rate (WER): the percentage of words that are substituted, deleted or inserted compared with a reference transcript.
Text-to-speech: from words to sound
TTS also works in stages.
- Text analysis: the system normalises text (turning "Rs. 2,500" into "two thousand five hundred rupees"), splits sentences and decides pronunciation, often through a grapheme-to-phoneme step.
- Prosody prediction: it chooses rhythm, stress, pitch and pauses so speech sounds natural instead of flat.
- Acoustic model: a neural network converts the phonetic and prosody information into a spectrogram.
- Vocoder: another network turns the spectrogram into an actual audio waveform.
Older systems stitched together recorded fragments of speech. Neural TTS generates the waveform itself, which sounds smoother and allows many voices and styles from one model.
Voice cloning and ethics
Some TTS systems can imitate a specific person's voice from a short sample. That is useful for accessibility (for example, people who have lost their voice) and creative work, but it also enables scams and impersonation. Use cloning only with clear consent, and be cautious about unexpected voice messages asking for money or secrets.
Everyday uses
- Voice typing and dictation
- Automatic captions and meeting transcripts
- Voice assistants and smart speakers
- Screen readers and reading aids
- Navigation instructions and announcements
- Audio versions of articles and e-learning content
- Customer-service automation
Python examples
Speech-to-text with the open-source Whisper model (install openai-whisper and ffmpeg first):
import whisper
model = whisper.load_model("small") # larger models are more accurate but slower
result = model.transcribe("meeting.mp3") # language is detected automatically
print(result["text"])Offline text-to-speech with pyttsx3:
import pyttsx3
engine = pyttsx3.init()
engine.setProperty("rate", 160) # words per minute
engine.say("Welcome to our store. Your order is ready.")
engine.runAndWait()Cloud TTS and STT services generally offer higher quality and more languages than offline libraries, but they require internet access, may cost money and send audio to a provider, which matters for privacy.
Tips for better results
- Use a good microphone placed close to the speaker.
- Record in a quiet room and avoid echo.
- Provide custom vocabulary or hints for names and terms when the service supports it.
- Always review transcripts of important recordings before relying on them.
- For TTS, write text the way it should be spoken and spell out unusual abbreviations.
Frequently Asked Questions
How accurate is speech-to-text?
In clean audio with common languages, modern systems can be very accurate. With noise, accents or specialised terms, errors increase, so review important transcripts.
Can AI speak Nepali?
Support for Nepali is improving but varies by service. Test the specific tool with your own sample sentences before building a product around it.
Is it safe to send recordings to cloud services?
It depends on the provider's policy and your data. Avoid uploading sensitive recordings unless you have agreed terms or use an on-device or self-hosted model.
What is the difference between ASR and voice recognition?
ASR converts speech to text. Voice (speaker) recognition identifies who is speaking. They are different tasks.
Conclusion
Speech-to-text turns audio into words using spectrograms and neural networks, and text-to-speech reverses the process with prosody modelling and vocoders. Understanding the steps helps you choose tools, improve accuracy and use voice AI responsibly.