‘AI features’ now appear on phones, laptops and cameras without specifying where the computation happens. That detail determines what the feature can do, what it costs you, whether it works without a signal, and who else sees your input.
The short version
- On-device: private, works offline, no marginal cost, sharply limited by memory and power.
- Cloud: far more capable, needs a connection, costs the vendor money per request, and your input leaves the device.
- Most shipping products are hybrid and rarely tell you which path a given request took.
- The honest question is not which is better, but which one just ran.
The constraint is memory, then power
A model’s parameters have to fit in memory to run. A phone has a few gigabytes available to applications; a data-centre accelerator has far more, and many can be combined. That gap is the reason the largest models cannot run locally at all, regardless of how fast the phone’s processor is.
Techniques such as quantisation — storing parameters at lower precision — shrink models substantially, with some loss of quality. They narrow the gap; they do not close it. The second constraint is energy: sustained inference drains a battery and generates heat, so phones throttle it.
What each side is genuinely good at
On-device wins
- Privacy. Input never leaves the hardware, which is the only guarantee that does not depend on a policy document.
- Latency. No network round trip. For live tasks — transcription, translation, camera effects — this is decisive.
- Availability. Works on a plane, underground, abroad.
- Cost. Zero marginal cost per request, which is why vendors prefer it wherever quality allows.
Cloud wins
- Capability. Larger models reason better, handle longer context and cover more languages and domains.
- Freshness. Updated centrally and immediately, without waiting for device updates.
- Uniformity. Everyone gets the same behaviour regardless of how old their hardware is.
On-device is not a cheaper cloud. It is a different set of capabilities with a different failure mode — and the marketing rarely distinguishes them.
Hybrid, and the transparency problem
In practice most products route: simple requests run locally, harder ones go to a server. This is sensible engineering. It is also opaque, because the same interface handles both and the user is seldom told which path was taken.
The consequence is that you cannot reason about privacy from the feature’s description alone. A summarisation feature that runs locally on a short note may silently send a long document to a server. Some vendors publish this routing behaviour; many do not.
Questions worth asking of any AI feature
- Does it work in airplane mode? That is the cheapest test of where it runs.
- Does the documentation state when data leaves the device, and what happens to it?
- Is input retained, and is it used for training? Is there an opt-out, and is it on by default?
- Is there an enterprise or regional variant with different handling? If so, the consumer default is the weaker one.
These are the same questions worth asking about the benchmark claims that accompany such features — see how to read an AI benchmark. They also connect to a broader architectural argument we make in local-first software.
Where this is heading
Device memory is growing, dedicated neural accelerators are now standard, and small models keep improving faster than most people expected. More work will move on-device, because it is cheaper for vendors as well as better for users.
But the frontier moves too. The largest models will remain in data centres for the foreseeable future, so the split is not a transitional state — it is the steady state. The thing to insist on is not one or the other, but disclosure of which you are getting.
Frequently asked questions
How can I tell if a feature runs locally?
Enable airplane mode and try it. If it works, it ran on the device. If it fails or degrades, some or all of it was remote.
Does on-device AI drain the battery?
Sustained inference is demanding, yes. Short, occasional tasks are negligible; continuous processing such as live video effects is not, which is why devices limit how long such features run.
Is on-device automatically private?
The computation is. Whether the application also uploads the input, the output or telemetry is a separate decision, and only the privacy documentation or a network trace will tell you.
