When your phone unlocks with your face, suggests the next word or removes noise from a call, the AI often runs directly on the device. This is edge AI: running machine-learning models close to where data is created, on phones, cameras, cars, wearables or small computers, instead of sending everything to distant cloud servers.
Edge versus cloud
| Cloud AI | Edge AI | |
|---|---|---|
| Where it runs | Remote data centres | On the device or a nearby gateway |
| Latency | Network delay | Very low |
| Privacy | Data is sent out | Data can stay on device |
| Connectivity | Needs internet | Works offline |
| Compute power | Very high | Limited |
| Model size | Large | Small, optimised |
| Cost | Pay per use or server | Hardware cost, little running cost |
Real systems often mix both: a small model on the device handles fast, private tasks, and harder requests go to the cloud.
Why run on the device?
- Speed: no round trip to a server. Essential for camera effects, voice wake words and safety systems.
- Privacy: photos, voice and health data can be processed without leaving the phone.
- Offline use: works in areas with poor or expensive internet, which is common in rural regions.
- Lower bandwidth and server cost: you send only results, not raw video.
- Reliability: continues to work during network outages.
- Energy efficiency for specialised chips compared with sending data constantly.
Typical edge AI applications
- Face unlock and photo enhancement
- Keyboard suggestions and on-device voice typing
- Wake-word detection for assistants
- Real-time translation and live captions
- Object detection in security cameras and drones
- Predictive maintenance on factory machines
- Health monitoring with wearables
- Smart agriculture sensors
- Driver monitoring and assistance systems
- QR and text scanning in apps
Hardware for the edge
- Mobile processors with neural engines or NPUs: specialised units that accelerate neural-network maths at low power.
- GPUs on phones and single-board computers.
- Microcontrollers: tiny chips running TinyML models with kilobytes of memory.
- Accelerators: devices like Google's Coral edge TPU or NVIDIA Jetson modules for cameras and robots.
Making models fit
Cloud-sized models are too big for most devices, so developers shrink them.
- Quantisation: convert weights from 32-bit floats to 8-bit integers or smaller, cutting size and speeding up inference.
- Pruning: remove weights that contribute little.
- Knowledge distillation: train a small student model to mimic a large teacher.
- Efficient architectures: designs such as MobileNet and EfficientNet are built for mobile.
- Operator fusion and hardware-specific compilers to run efficiently on the target chip.
Example: converting a model for mobile with TensorFlow Lite
import tensorflow as tf
# a small Keras model for illustration
model = tf.keras.applications.MobileNetV2(weights="imagenet", input_shape=(224, 224, 3))
converter = tf.lite.TFLiteConverter.from_keras_model(model)
converter.optimizations = [tf.lite.Optimize.DEFAULT] # dynamic-range quantisation
tflite_model = converter.convert()
with open("mobilenet_v2.tflite", "wb") as f:
f.write(tflite_model)
print(f"Model size: {len(tflite_model) / 1e6:.1f} MB")You then bundle the .tflite file in an Android or iOS app and run it with the TensorFlow Lite interpreter. Other options include Core ML on Apple devices, ONNX Runtime for cross-platform deployment, and PyTorch's mobile and ExecuTorch runtimes.
Using on-device AI in Flutter
Flutter apps can use plugins that wrap platform ML libraries, for example for text recognition, barcode scanning, face detection or custom TensorFlow Lite models. On-device processing is particularly helpful for apps used in places with limited connectivity, such as attendance, scanning and field data collection tools.
Challenges
- Limited memory and compute: smaller models may be less accurate.
- Battery and heat: continuous inference drains power.
- Fragmentation: many phone models and chips behave differently, so you must test widely.
- Updating models: you need a plan to ship new versions to devices.
- Debugging and monitoring are harder than on a central server.
- Security: a model stored on a device can be copied or tampered with.
Federated learning
Federated learning trains a shared model across many devices without collecting raw data centrally. Each device trains on its local data and sends only model updates, which are then combined. It can improve privacy, for example in keyboard prediction, though it is complex to implement and not a perfect privacy guarantee.
When to choose edge AI
Choose edge when low latency, privacy or offline operation is essential and the task can be done by a compact model. Choose cloud when you need large models, frequent updates or heavy computation. A hybrid design often gives the best experience.
Frequently Asked Questions
Is edge AI the same as on-device AI?
On-device AI is a type of edge AI. Edge can also include nearby gateways and local servers.
Do edge models need internet?
Not for inference, though you may need internet to download or update them.
Can large language models run on phones?
Small, heavily optimised models can, and this is improving, but they are far smaller than the biggest cloud models.
Is on-device processing always private?
It keeps raw data local, but apps may still send results or analytics elsewhere. Check the app's privacy policy.
Conclusion
Edge AI brings intelligence to the place where data is created, giving speed, privacy and offline capability. By shrinking models and using the right runtime and hardware, you can build useful features that work in a pocket-sized device.