AivexaNewsSearch
AI news for builders and product teamsChecked every hour

How to Rewrite Apache Spark UDFs for GPU Acceleration with AI Agents

Collected Oct 13, 2026

NVIDIA has published a set of agent skills, cuDF skills for Apache Spark, aimed at removing one of the persistent bottlenecks in GPU-accelerated Spark workloads: user-defined functions. The company describes the approach in a developer blog post that also reports a production conversion at a major consumer technology company.

User-defined functions, or UDFs, are custom routines that extend Spark's built-in functionality. They are popular for bespoke business logic, but they sit outside Spark's declarative SQL and DataFrame interface. That interface is precisely what lets the cuDF plugin for Apache Spark map query operations onto GPUs transparently. A UDF contains arbitrary code the engine cannot reason about, so an accelerated pipeline must convert columnar data into rows, fall back to the CPU to run the function row by row, then convert the results back to columnar form. The cost is twofold: the round-trip data transfer itself, and the compute speedup that is forfeited because the function never runs on the GPU.

NVIDIA lays out three rewrite routes, ordered by complexity and intended use case. The simplest is replacing the UDF with an equivalent Spark SQL expression, which makes the logic visible in Spark's expression tree so the cuDF plugin can substitute its GPU implementation at planning time. When that is not enough, a UDF can implement the RapidsUDF interface from the cuDF plugin, whose contract requires a single method, evaluateColumnar. The plugin recognizes such classes and hands them a GPU-resident ColumnVector, which the function processes in parallel using the Java API for cuDF. For logic that cannot be composed efficiently from existing cuDF operations, a RapidsUDF can call a custom CUDA kernel through the Java Native Interface.

The blog uses a Celsius-to-Fahrenheit Scala UDF to illustrate the first two paths, showing the original Function1 implementation, a SQL replacement that keeps data on the GPU, and a RapidsUDF version that builds the arithmetic from cuDF scalars and column operations.

Rewriting UDFs is effective but laborious, requiring careful API selection, semantic validation and performance measurement, and columnar APIs can be unfamiliar to engineers without vectorized programming experience. That is where agents come in. With what NVIDIA calls proper guardrails, an agent can search unfamiliar APIs, produce candidate rewrites, and iterate while executable tests and benchmarks decide whether each change is acceptable.

The cuDF skills for Apache Spark break the work into four focused feedback loops: testing, conversion, benchmarking and optimization. Each skill defines both the work to perform and the evidence required before moving to the next stage. A project template supplies cuDF- and Spark-specific scaffolding, including test and benchmark harnesses, encoded routines for output verification, and build configuration, so the agent spends its effort on the UDF rather than rebuilding infrastructure. Advisory guidance covers GPU programming practices such as memory management and configuration, while the agent remains free to explore different implementations and pick one based on measured correctness and performance.

The headline benchmark comes from a major consumer technology company that used the workflow to convert more than 30 production UDFs. The rewrites preserved the behavior of the original CPU functions while removing fallbacks from otherwise GPU-accelerated pipelines. For one UDF, operator time dropped from 274.55 to 12.78 hours on a 600-billion-row workload, a 22x speedup. According to NVIDIA, that UDF affected more than 3,000 jobs. NVIDIA says agent assistance makes such gains attainable in hours rather than days or weeks, and that most rewrites are achievable within minutes.

This suggests the practical value is less about raw GPU throughput, which cuDF already delivers for declarative Spark SQL, and more about closing the fallback gap that UDFs open in pipelines that are otherwise fully accelerated. A single UDF can force repeated host-device transfers and serialize execution, so eliminating it may matter disproportionately to end-to-end time. A likely trade-off is that RapidsUDF and CUDA rewrites introduce GPU-specific code into codebases, which raises questions about portability and long-term maintenance even when the Java API path hides much of the complexity. Semantic equivalence is another risk the post itself flags: agent rewrites are only as trustworthy as the verification tests and benchmark harnesses around them, which is consistent with the skills' evidence-gated design.

The skills are available on GitHub and can be installed directly or through the npx skill installer using the command NVIDIA provides. NVIDIA also points to a Data + AI Summit session on coding with AI agents for Spark UDF GPU acceleration and to the cuDF plugin for Apache Spark getting-started guide.

Why it matters: Teams running Spark on GPUs now have a documented, partly automated path to remove UDF fallbacks that quietly undermine acceleration, with a measured 22x improvement on a very large workload as the headline result. Developers should expect to invest in test and benchmark harnesses first, since the workflow relies on them to accept or reject each agent-generated rewrite, and to weigh the maintainability of GPU-specific UDF code against the speedup. The broader inference is that agent-assisted migration tooling is becoming a standard way to modernize existing data pipelines rather than a novelty.

Read at NVIDIA Developer Blog

Based on reporting from the original publisher. Visit the source for full context and later updates.

Publisher excerpt

Enterprise applications have become increasingly demanding. Agentic AI has compounded the need for vast amounts of enterprise data with frequent, complex...