Perception and Interaction Module (PIM)

Gold definitionUpdated Apr 2, 2026

Definition

The Perception and Interaction Module (PIM) is a component within the InteractAvatar framework designed to generate text-aligned interaction motions for talking avatars engaged in Grounded Human-Object Interaction (GHOI). It leverages environmental perception via detection to enable realistic, controlled interactions.

At a glance

Executive summary

The Perception and Interaction Module (PIM) helps create realistic talking avatars that can interact with objects based on text commands. It does this by first understanding the environment and then generating motions that match the text, solving a key challenge in generating high-quality interactive videos.

TL;DR

PIM is a module that helps virtual avatars understand their surroundings and perform actions with objects based on text instructions, making interactive videos more realistic.

Key points

Leverages detection for environmental perception to generate text-aligned interaction motions.
Addresses the challenge of extending talking avatar generation to grounded human-object interaction (GHOI) and mitigates the control-quality dilemma.
Used by researchers and engineers in video generation, virtual reality, and interactive AI for creating realistic virtual characters.
Unlike methods that struggle with environmental perception or control-quality in GHOI, PIM explicitly decouples perception and planning for better control.
Focuses on enhancing realism and control in generative AI for complex human-object interactions in virtual environments.

Use cases

Virtual Assistants: Creating AI assistants that can not only talk but also physically interact with virtual objects in a user's environment, e.g., a virtual assistant demonstrating how to use a software feature by interacting with virtual buttons.
Interactive Gaming Characters: Developing game NPCs (Non-Player Characters) that can realistically pick up, use, or manipulate objects in the game world based on dynamic game events or player commands.
Content Creation for Animation: Automating the generation of complex character animations involving object interaction for films, advertisements, or educational content, reducing manual animation effort.
Robotics Simulation: Training and testing robotic control policies in highly realistic virtual environments where human-like avatars interact with objects, providing diverse and complex scenarios.

Also known as

PIM