An on-device model is three pieces in a trench coat. A sensor produces a raw signal. A compressed network turns that signal into a prediction. An app decides what to do with the prediction, and how to show it. Each piece has its own failure mode, and the failure modes are where the design decisions hide.
The system, at a high level
On-device machine learning means the model runs where the data is created, not in a data center. The model is trained offline, usually on a GPU cluster, then compressed and converted for the target hardware. That conversion step is not cosmetic. It rewrites the arithmetic so the network can run on a phone's neural accelerator, a DSP, or the CPU cores that are already there. The phone does not get the same model the cluster trained. It gets a smaller, faster, more brittle cousin. The trade is latency, privacy, and battery against accuracy and flexibility. A cloud model can be swapped without an app update. An on-device model is frozen into the build. That constraint shapes everything downstream, from how often the app ships to how gracefully it handles inputs the training set never contained.
Sensor, model, app
The sensor layer is physical. A camera frame arrives as photons on a small sensor, gets demosaiced, white-balanced, and cropped before any model sees it. A microphone signal is filtered and windowed. An accelerometer reports three axes at a fixed rate. These steps are unglamorous and they dominate real-world accuracy. A model trained on clean crops will underperform on a frame that is slightly blurred or underexposed. The model layer is arithmetic. Convolutions, matrix multiplies, activation functions, all quantized to eight-bit integers or lower. Quantization is where accuracy quietly leaks. A network that scores well in floating point can lose several points after conversion, and the loss is not uniform across classes. The app layer is judgment. It decides when to run inference, how to smooth results over time, and what to do when confidence is low. A pose tracker that redraws a skeleton every frame will jitter. One that smooths too aggressively will lag behind a fast movement. Neither layer is wrong. They are negotiating.
Edge cases, where it gets interesting
The edge cases are not exotic. They are Tuesday. A phone in a pocket warms up, and the thermal governor throttles the accelerator, so inference slows by half. The app must decide whether to drop frames or drop accuracy. A user with darker skin tone and a wrist-based optical sensor gets a weaker signal, because the physics of reflected light does not care about fairness. A model trained mostly on one demographic will fail quietly on another, and the failure will look like user error. Then there is the update problem. Federated learning and model patching let a deployed model improve, but they also introduce a new surface: a bad patch can degrade every device at once. The on-device stack is not a smaller version of the cloud stack. It is a different stack with different bottlenecks, and the bottlenecks move depending on the device, the weather, and the person holding it.
What breaks, and why that is useful
What breaks is the promise of seamlessness. The marketing word assumes the layers disappear. They do not. They stack, and each one leaves a fingerprint on the output. Knowing where the fingerprints are is how you debug a product that feels wrong without crashing. If a face detector misses a face in dim light, the sensor layer is the first suspect. If it finds faces that are not there, the model layer is. If it finds the right face but the box jitters, the app layer is. That mapping is not a trick. It is the whole job. The sensor is honest; the wrist, the pocket, and the room are not always great places to look. The model is honest; the training set is not always representative. The app is honest; the user is not always patient. Naming the layer that failed is how you fix the right thing instead of the nearest thing.
FAQ
Why does my phone get hot and slow down when I use the camera?
Because the neural accelerator is doing real work, and the thermal governor throttles it when the phone gets warm. A pocket is a bad heatsink. The app has to choose between dropping frames and dropping accuracy, and most choose frames.
Can I trust the accuracy numbers in the spec sheet?
Only if they were measured after quantization, on hardware like yours, with inputs like yours. A floating-point score from the training cluster is a starting point, not a promise. The conversion step leaks accuracy, and it does not leak evenly.
What is the first thing to check when a model misbehaves in the field?
Name the layer. Dim light and motion blur point to the sensor. False positives point to the model. Jitter and lag point to the app. Fixing the right layer is faster than retraining the wrong one.
