Build agent memory with NVIDIA NeMo Agent Toolkit and Amazon S3 Vectors

An AWS Machine Learning Blog post describes implementing Amazon S3 Vectors as the persistent memory layer within the NVIDIA NeMo Agent Toolkit (NAT), deployed on Amazon Elastic Kubernetes Service (Amazon EKS). The post follows an earlier one on memory architecture and moves to implementation, using a multi-agent investment research use case as its running example.

NAT is described as an open source framework for building, profiling, and optimizing AI agents that is framework-agnostic, working with Strands Agents, LangChain, LlamaIndex, CrewAI, and custom implementations. Its memory subsystem stores and retrieves conversation history, user preferences, and long-term knowledge across agent invocations, and is extensible through custom memory providers. Key components named are the MemoryEditor interface with add_items(), search(), and remove_items(); the MemoryItem data model; the MemoryBaseConfig Pydantic class; the auto_memory_agent workflow wrapper; and provider discovery via the _type field in YAML config. Built-in providers listed are Mem0, MemMachine, Redis, and Zep.

The implementation is presented in three steps: create the S3 Vectors infrastructure, implement a custom MemoryEditor plugin, and configure the agent workflow. The example index uses 1024 dimensions to match Amazon Titan Text Embeddings V2, with a cosine distance metric and the content field marked as non-filterable metadata. Prerequisites include an AWS account, an existing Amazon EKS cluster, NVIDIA NeMo Agent Toolkit installed (tested with version 1.6), Python 3.11 or 3.12, an embedding model, and kubectl and Docker.

The post also covers responsible AI and data handling, advising a retention policy, avoiding personally identifiable information in memory metadata, and scoping access through per-tenant indexes and least-privilege IAM policies. It describes multi-agent memory coordination via agent_id and team_id metadata filters, memory consolidation of episodic into semantic memories, and deployment on Amazon EKS using containerized NAT agents run with nat serve, a Dockerfile, Kubernetes Deployment and HorizontalPodAutoscaler manifests, and IAM Roles for Service Accounts. It notes Amazon Bedrock AgentCore as a managed alternative.