On-device AI
In short
On-device AI is artificial intelligence that runs directly on your phone, laptop or other gadget rather than in a remote data centre. It can work offline, respond quickly and keep personal data on the device.
- Also called
- On-device machine learning, On-device generative AI
- Topic
- Using AI tools
- Level
- Beginner
- First described
- Running neural networks on phones needed smaller designs. Google's MobileNets paper of April 2017 introduced efficient models "for mobile and embedded vision applications"5.
Phones and laptops now carry chips designed for AI maths, often called neural processing units (NPUs). Microsoft built its Copilot+ PCs, launched in May 2024, around chips capable of more than 40 trillion operations per second so that AI features could run locally1.
On Android, Google's Gemini Nano model runs inside a system service called AICore. Google says this lets apps offer generative AI "without needing a network connection or sending data to the cloud"2, though speed depends on the phone's hardware2.
Apple Intelligence uses an on-device model of about 3 billion parameters3. To fit a phone's memory, Apple compresses it to under 4 bits per weight on average3. Apple's Foundation Models framework lets app developers call that on-device model directly4.
Most products are hybrids. Apple sends harder requests to larger models on its Private Cloud Compute servers34, and Microsoft pairs small local models with large models in its Azure cloud1. So "on-device" describes where a particular feature runs, not a promise that nothing ever leaves your phone.
Example
Microsoft's Live Captions on Copilot+ PCs translates audio from more than 40 languages into English subtitles, one of the features Microsoft says runs locally on the device1.
Why it matters
It decides whether your messages, photos and voice leave your device when you use AI features. It also means some AI tools keep working on a train with no signal.
Questions people ask
Is on-device AI more private?
Usually, because the data is processed on the device. Google says on-device processing keeps sensitive data on the phone2. But many products mix local and cloud processing, so check which features send data to servers3.
Do I need a new phone for on-device AI?
Often, yes. Apple's on-device model only runs on devices that support Apple Intelligence4, and Microsoft's Copilot+ features were built for PCs with a dedicated NPU1.
Is on-device AI the same as a local model?
They overlap. On-device AI usually means models built into a phone or PC by its maker, such as Gemini Nano2. A local model usually means one you download and run yourself.
Related terms
Tools that use it
- Apple Writing Tools (Write with Siri)Free proofreading and rewriting built into recent iPhones, iPads and Macs, now run through Siri.
- Siri AIApple's rebuilt Siri, which holds conversations, answers from the web and acts in apps on iPhone, iPad and Mac.
Sources
- Microsoft, Introducing Copilot+ PCs (Official Microsoft Blog)
- Android Developers, Gemini Nano
- Gunter et al. (2024), Apple Intelligence Foundation Language Models, arXiv:2407.21075
- Apple Developer Documentation, Foundation Models framework
- Howard et al. (2017), MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications, arXiv:1704.04861