AivexaNewsSearch
AI news for builders and product teamsChecked every hour

Google ML Drift Accelerates On-Device AI Inference across Diverse GPUs

Collected Oct 25, 2026

Google has released ML Drift, an open-source GPU compute engine for on-device AI and machine learning inference, published under the Apache 2.0 license and reported by InfoQ's Sergio De Simone. ML Drift is the core GPU acceleration engine underneath LiteRT, Google's runtime for running models locally, and Google is now exposing it so developers can build their own graphics and inference runtimes. The engine targets a broad set of workloads, from conventional computer vision, audio and depth processing through to generative AI.

The central problem ML Drift addresses is hardware diversity. Applications running on edge devices land on a wide range of GPU architectures, driver versions and low-level APIs, and developers generally cannot know in advance which specific hardware a build will execute on. ML Drift sits between the model and that fragmented landscape, supporting OpenGL ES, OpenCL, Metal and WebGPU so the same inference code can reach many kinds of devices without per-vendor plumbing. Google frames the goal as a new foundation for both portability and performance, describing it as delivering stability for traditional ML alongside peak performance for advanced generative AI.

Compared with the legacy TensorFlow Lite GPU delegate, ML Drift adds several technical changes. It uses unified shaders through tensor virtualization, supports extensible custom operations, and adds five-dimensional tensor support. Google has also built hardware-specific optimizations for GPUs from Arm, Intel and Qualcomm Technologies, aimed at maximizing hardware utilization. On desktop, ML Drift extends inference support to Windows and Linux through Dawn, Chromium's WebGPU implementation, while keeping a native Metal backend on macOS. Dawn gives a single API model on top of WebGPU's hardware abstraction layer, so the engine avoids talking directly to platform-specific graphics APIs such as DirectX and Vulkan.

Those design choices are the substance of the release for engineers. Tensor virtualization, as the name suggests, decouples the logical tensor the model describes from the physical memory allocation behind it. A likely trade-off is that this indirection buys flexibility and lower fragmentation during live inference, but requires careful management to avoid overhead in the hottest loops. Native five-dimensional tensors remove a common workaround in which spatial or transformer-style models are reshaped into 4D layouts with padding and bookkeeping. Extensible custom operations let teams add kernels Google did not ship, which matters for newer model architectures that would otherwise stall until a vendor update arrives.

The performance evidence Google offers covers both benchmark runs and shipped products. On Gemma models running on developer-grade workstation hardware, Google reports 12% lower memory overhead than other frameworks, naming llama.cpp and MLX LM as comparisons. The company also says ML Drift already runs on millions of devices through Chrome, YouTube Shorts, Photos and Meet. In YouTube Shorts, segmentation-based effects on Android and iOS see up to a 40% reduction in average frame latency. In Google Photos, computational photography and segmentation pipelines speed up by as much as two seconds. Adobe Lightroom and Photoshop gain up to 30% improvement on selected AI photo-editing features including Select Subject, Select Sky and Adaptive Portrait. Snapchat records a 30% reduction in model latency for ML-powered face and style-generation effects on Android, which its teams use to justify running larger diffusion models.

The release also changes the support picture for existing TensorFlow Lite users. Google will stop updating the legacy TFLite GPU delegate, and recommends migrating to the LiteRT ML Drift GPU accelerator for backward compatibility with current models. The surface is split by audience: application developers can use the LiteRT ML Drift accelerator, while GPU and system engineers can drop to a lower-level C++ API to construct custom GpuModel compute graphs and write their own shaders.

Reaction from practitioners has been pointed. AI/ML engineer Pavithra Agraharam, writing on LinkedIn, called native 5D tensor support without layout workarounds a major win for spatial and transformer models on edge GPUs, and said separating logical tensors from physical memory allocations should cut memory fragmentation during live inference. Matthew Charles Vladislav Busel, founder and sole developer of Tensorust, asked how it compares with Core ML on recent iPhones for latency and memory, since that is what many teams ship against today. Google has not published comparative figures against Core ML. That gap is worth noting: for teams whose audience is concentrated on Apple hardware, the native path remains the incumbent until independent numbers appear.

The memory-overhead figure is the most transferable claim for cross-platform developers, because it concerns the class of frameworks teams actually choose between rather than Google's own apps. A 12% reduction is meaningful where model weights and activations compete with the rest of an app for a fixed memory budget, though the measurement is on workstation-class hardware and may not translate directly to phones with tighter thermal and bandwidth constraints. Similarly, the app-level gains are vendor-reported and tied to specific features, so they indicate what is achievable rather than a guarantee for arbitrary models.

Why it matters: teams running on-device inference across mixed Android, iOS and desktop hardware now have an Apache 2.0 engine with a C++ layer for custom work, and the move signals that the TFLite GPU delegate is on a maintenance path that will push projects toward LiteRT. Mobile and edge developers planning model deployment should treat the migration question as near-term rather than optional, and should validate the reported memory and latency gains on their own target devices before committing. For product teams, the ability to hit low-level GPU code without writing per-vendor backends may shorten the path from a prototype model to a shipped feature, though the lack of published comparisons against Core ML is a likely deciding factor for Apple-heavy audiences.

Read at InfoQ · AI, ML & Data Engineering

Based on reporting from the original publisher. Visit the source for full context and later updates.

Publisher excerpt

Google has released ML Drift, an open-source, cross-platform GPU compute engine designed for on-device AI and machine learning inference. Licensed under Apache 2.0, it abstracts the complexities of different GPU hardware and low-level APIs, supporting OpenGL ES, OpenCL, Metal, and WebGPU. By Sergio De Simone