AivexaNewsSearch
AI news for builders and product teamsChecked every hour

Tracing Agent Harness Behavior with NVIDIA NeMo Relay

Collected Sep 30, 2026

NVIDIA Developer Blog published a tutorial describing how to use execution traces from NVIDIA NeMo Relay to understand agent behavior and judge whether harness changes improve task outcomes.

NeMo Relay is described as a common observability layer that captures ordered lifecycle events and structured trajectories for agent runs. Hermes Agent includes native NeMo Relay integration, producing ATOF event streams, ATIF step-by-step trajectories, and OpenTelemetry spans with OpenInference labels for inspection in tools such as Arize Phoenix.

The tutorial walks through two examples. In the first, a terminal-tool task runs a script in an isolated Docker container; the runner verifies the exact output VALUE=42 alongside completed model scopes, token usage, and zero tool errors. One verified run reported 74 events, two completed LLM scopes, 7,239 prompt tokens, 96 completion tokens, 7,335 total tokens, one tool call, and zero tool errors, with a three-step ATIF trajectory.

In the second, a multi-tool research task requires Hermes to read a travel record, search the web, verify a conference on its official site, write a report, and return the name COLT 2026, with model and tool spans sent to Phoenix. The post notes traces can contain prompts, model responses, tool arguments and results, and file paths, and should be reviewed before sharing.

A Hermes ToolPerf case study compared pinned baseline and fixed revisions across nine tasks, three runs per model per arm, totaling 108 runs on August 6. Sonnet finished 24 of 27 runs on baseline and 23 of 27 on fixes, with unchanged turns, tool calls, and result data. Qwen Coder completed three more tasks, rising from 19 of 27 to 22 of 27, while mean LLM calls rose from 3.8 to 4.9, tool calls from 2.8 to 3.9, tool-result data from 16 KB to 33 KB, and duration from 27 s to 42 s. One blocked-command task went from 33% to 100%, and a case-insensitive search task moved from 3.3 turns to 9.3. A hidden-file search stayed at 0 to 33% on both arms and models.

Read at NVIDIA Developer Blog

Based on reporting from the original publisher. Visit the source for full context and later updates.

Publisher excerpt

An agent can finish a task and still take an inefficient path. A failed search can trigger another search. A truncated file read can lead to a command fetching...