From PyTorch to Shipping Local AI on Android

Why run AI locally and why it’s hard on Android

Running AI models directly on Android devices offers benefits like low-latency interactions, offline functionality, and enhanced privacy, but achieving consistent performance across the fragmented Android ecosystem is a significant hurdle. A recent developer reported issues with a pose-detection app, experiencing sluggish performance and crashes on devices beyond the initial test set.

The core problem is the vast diversity of Android hardware, where performance varies dramatically based on CPU, GPU, and NPU capabilities, runtime support, and driver implementations. This leads to unpredictable behavior, frustrated users, and often, the abandonment of on-device AI features, costing developers time and potentially valuable functionality.

Key Insights

• Performance Variability: Android devices exhibit significant performance differences even with the same model due to varying hardware and software configurations.
• Quantization Benefits: Reducing model precision (e.g., to INT8) can drastically reduce latency and memory usage, particularly on resource-constrained devices.
• Embedl Hub: A platform designed to compile, optimize, benchmark, and analyze on-device AI models, offering a device cloud for testing and performance comparison.

Working Example

embedl-hub compile \
--model /path/to/mobilenet_v2.onnx

embedl-hub quantize \
--model /path/to/mobilenet_v2.tflite \
--data /path/to/dataset \
--num-samples 100

embedl-hub benchmark \
--model /path/to/mobilenet_v2.quantized.tflite \
--device "Samsung Galaxy S24"

Practical Applications

• Fitness Apps: Real-time pose estimation for exercise tracking and form correction, leveraging on-device processing for privacy and responsiveness.
• Pitfall: Failing to test across a wide range of devices can lead to poor user experiences and negative reviews, ultimately resulting in feature removal.

References:

• https://dev.to/embedl-hub/from-pytorch-to-shipping-local-ai-on-android-6g9