Recreating a 70-year love story frame by frame

Google DeepMind and Primordial Soup collaborated on a documentary short called "Love, Rendered" that uses artificial intelligence to recreate a cherished, unrecorded memory for an elderly couple. The film follows Burt and Ethelle Shatz, who have been married for over 70 years, as they navigate Burt's cognitive decline. Among his fading memories is the day they met at a student co-op in Cleveland. Because no photograph or film of that day exists, it lives only in their minds.

The film was directed by Academy Award-nominated filmmaker Liz Garbus, who produced it with Dan Cogan and Darren Aronofsky. It was created through a collaboration between Google DeepMind and Primordial Soup, Aronofsky's creative venture. Michael Chang, a Google DeepMind engineer, served as the film's technical lead.

The technical approach had two parts. First, engineers used generative models to restore black-and-white photos of Burt and Ethelle from their youth, keeping subsequent recreations faithful to the couple. Second, they used performance capture models to map the couple's present-day micro-mannerisms, such as the tilt of Burt's head, a hesitation in his speech, and the crinkle around his eyes, onto their younger likenesses. Ethelle participated as an active co-creator, correcting details like the curve of a staircase and the shape of a shoe heel.

The creative team's interest in memory resilience led them to reminiscence therapy, the clinical practice of using sensory cues like songs and old photographs to stimulate memories. People can try related technology: Google says users can upload a photo to the Gemini app and ask it to restore and colorize the image while preserving the appearance, expression and pose of the people in it.

Why it matters: The Gemini app makes photo restoration and colorization available to anyone with an old family photograph, while the film demonstrates how image restoration and performance capture were combined to reconstruct a moment that was never recorded.