01 / Key message
Token by Token, Compromised: Backdoor Vulnerabilities in Unified Autoregressive Models
Backdoor attacks that use ordinary text triggers to jointly manipulate image and language generation in unified autoregressive models.
Cross-modal trigger path
02 / Method
How it works
Bind a trigger
Associate a common word or subtle character with a chosen multimodal behavior.
Compromise the model
Inject the association through poisoned training data or a direct parameter update.
Activate across modalities
The trigger redirects both image-token and text-token continuations at inference time.
03 / Abstract
Abstract
Unified autoregressive models generate text and images through shared parameters and token vocabularies, creating attack surfaces that cross modality boundaries. Token by Token Backdoor Attack (ToBAC) studies these vulnerabilities through both data poisoning and direct model modification. Seemingly ordinary triggers, including common words or subtle characters, can redirect visual generation while also changing language behavior. Experiments on Liquid and Janus-Pro show that multimodal backdoors can remain unobtrusive at input time yet reliably activate brand promotion, ideological influence, or other targeted outputs.
04 / Contributions
What this adds
- 01
Multimodal threat model
Establishes backdoor vulnerabilities for unified autoregressive systems that share a model and vocabulary across text and image tokens.
- 02
Two attack settings
Introduces data-poisoning and model-poisoning variants that cover both limited-access and direct-access adversaries.
- 03
Cross-modal control
Shows that a single innocuous trigger can coordinate targeted changes in visual generation and language behavior.
05 / Citation
Citation
Braun, T., Grebe, J. H., Shakibania, H., Rohrbach, A., & Rohrbach, M. (2026). Token by Token, Compromised: Backdoor Vulnerabilities in Unified Autoregressive Models. arXiv:2605.19227.
BibTeX
@misc{braun2026token,
title = {Token by Token, Compromised: Backdoor Vulnerabilities in Unified Autoregressive Models},
author = {Tobias Braun and Jonas Henry Grebe and Hossein Shakibania and Anna Rohrbach and Marcus Rohrbach},
year = {2026},
eprint = {2605.19227},
archivePrefix = {arXiv},
primaryClass = {cs.CR},
url = {https://arxiv.org/abs/2605.19227}
}