Lies We Can See Joint verbal and non-verbal deception by VLM agents in embodied social interactions

The blind spot

Deception follows the action space

Chatbots could deceive only in conversation: a false claim in the transcript. Digital agents (e.g., code agents) now hold real permissions and act on our behalf, and deception has kept pace: faked test results, quietly disabled oversight. Each time agents gained a new way to act, deception followed.

A "body" is the next action space. There, deception targets not the record but other agents' eyes. No evaluation today watches that channel. So we built a world where it can be watched, counted, and scored.

Abstract

Strategic deception by LLM and VLM agents has emerged as a central AI alignment and safety concern. Social-deduction games, where each player holds a hidden role and communicates with others to deduce identities, serve as the canonical testbed. Existing testbeds, however, are text-only and run on a single fixed agent configuration, missing the non-verbal sensorimotor channels that are central to deception taxonomies and leaving it ambiguous whether an observed behavior reflects the underlying model or the surrounding harness. We introduce MineAmongUs, a 3D multimodal Among Us sandbox where imposter agents must deceive crewmates through joint verbal and non-verbal action. We also propose ARIA, a configurable VLM-agent harness that exposes five ablation axes, and an atom- and arc-level annotation scheme grounded in deception taxonomies and operationalized at scale by an LLM-as-a-Judge reaching near-human atom-labeling agreement. Empirical results show that VLM agents pursue imposter wins through joint verbal and non-verbal deception, with non-verbal channels emerging as the more decisive winning contributors across both harness ablation and cross-VLM evaluation.

The game

A game that rewards deception by design

MineAmongUs is Among Us, played inside Minecraft: a social deduction game (or hidden-role game) of eight agents on one shared map, six crewmates against two imposters hiding among them. A match alternates between two phases until one side wins.

Roles are hidden: crewmates never learn who the imposters are, while the two imposters know each other and can win only by deceiving everyone else.

The sandbox

The same rules, now embodied

MineAmongUs realizes the game in a 3D embodied world, and the agent is inside it. At every step it receives an egocentric RGB frame rendered from its own bot, alongside scoreboard signals, chat history and its own position. It acts by dispatching a program to that bot: pathfind, attack, flee, report, speak, vote.

The agent harness

ARIA: is it the model, or the harness?

When an agent deceives, which part deserves the credit: the VLM itself, or the memory, planning and prompts wrapped around it? A single fixed agent can never tell you. ARIA (Ablation-Ready Imposter/crewmate Agent) makes every one of those components switchable, so a behavior can be traced back to the component that produced it.

Memory

window a rolling buffer of recent steps · semantic per-player beliefs kept up to date by a VLM

Planning

reactive picks what to do one step at a time · hierarchical sets a long-term strategy and plans each step under it

Reflection & skill

off no reflection · meeting reviews each meeting afterwards and saves useful strategies for later

Prompt style

minimal game rules and role only · deterministic adds explicit deception tactics with worked examples

State representation

ego only what the agent currently sees · privileged every player's position given as text coordinates, even out of sight.

LLM-as-a-Judge

Scoring the lie, not just the win

Win rate says who won; it does not say how. From expert annotation of 48 gameplay logs we derive 23 atoms, the smallest scorable deceptive acts, grouped into six families across the verbal and non-verbal channels; an arc chains atoms into one multi-step deception. A judge model (Qwen3.6-27B) then takes over the labeling at scale: it reads each match's log and marks every atom.

Atom taxonomy

Agreement: human–human Cohen's κ = 0.792 against human–LLM κ = 0.709.

Experiments · RQ1

Fix the model, vary the harness

RQ1: "With the VLM backbone held fixed, how does each cognitive component of the harness change imposter win rate (WR), and which deception strategies correlate with winning?"

With state settled, the ablation grid crosses the four remaining axes (memory, planning, reflection & skill, prompt style), two settings each: 2⁴ = 16 imposter configurations. Two backbones (GPT-4.1-mini, Qwen3.6-27B), two crewmate settings, three repetitions: 16 × 2 × 2 × 3 = 192 matches.

Both roles run the same backbone (self-play). Each cell aggregates 16 imposter configurations × 3 repetitions = 48 matches.

Per-atom statistics over all 192 games: raw count and point-biserial correlation r_pb between per-game atom count and the imposter-win indicator. Highlighted rows: the kill-execution loop and post-kill non-reporting.

Experiments · RQ2

Fix the harness, vary the model

RQ2: "With the harness held fixed, how do twelve VLM backbones compare as imposters and crewmates, and which deception strategies separate winners from losers?"

Every VLM plays imposter against every VLM as crewmate, itself included: 12 × 12 = 144 matchups. We run this under four harness settings carried over from RQ1 (two crewmate configurations, each against a matching imposter and against the RQ1-best imposter), twice each: 144 × 4 × 2 = 1,152 matches.

Atoms per game, averaged within each group; × is the top-3 to worst-3 ratio.

Per-model cluster shares (% of each VLM’s atom budget) and cross-model Pearson r against imposter WR (N = 6). Bold marks each top-3 VLM’s dominant channel.

Main Takeaway

Can an agent lie with its body? Yes, it can, and it decides who wins.

Across 1,344 matches, VLM agents pursued the imposter objective through joint verbal and non-verbal deception, and in both experiments the behaviors most associated with winning were non-verbal. Agents lie with their bodies, and that is where the games were decided.

The limitation to keep in mind: this is deception "inside a game", under a spatial scaffold (privileged state) that current VLMs still need. Part of what wins the game is simply playing it well, and playing this game well is not the same as being good at deception in general.

Even so, the takeaway is the new axis itself: deception that agents carry out with words and actions together can now be made to happen, recorded and counted in one controlled place. Our sandbox, harness and annotation scheme are open for future alignment research to adapt and extend.

Citation

BibTeX

@misc{mineamongus2026,
  title         = {Lies We Can See: Joint Verbal and Non-Verbal Deception by VLM Agents in Embodied Social Interactions},
  author        = {Ahn, Jaewoo and Kim, Junseo and Kim, Hyunseo and Yun, Heeseung and Son, Jaehyeon and Kira, Zsolt and Kim, Gunhee},
  year          = {2026},
  eprint        = {2608.30428},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CL},
  url           = {https://arxiv.org/abs/2608.30428}
}