Artificial Intelligence1 min read

Previous-Token Prediction Based LLM Inversion for Prompt Reconstruction

By · Published by Everything Blog

In short

The mechanism functions by training an explicit inverse model from scratch on synthetic data created directly by the target language model. Instead of predicting the subsequent token, this specialized model mirrors traditional forward generation by predicting preceding tokens to uncover the initial input. The system also supports generative sampling to yield diverse prompt variations that reliably trigger comparable responses from the target model. Furthermore, the approach demonstrates robust transferability by reconstructing prompts from text originating from multiple different language models.

Key points

  • Training an inverse model to predict previous tokens resolves the complex many-to-many re…: Training an inverse model to predict previous tokens resolves the complex many-to-many relationship of standard auto-regressive generation.
  • The inversion architecture supports sampling to create multiple diverse prompt variations…: The inversion architecture supports sampling to create multiple diverse prompt variations that produce similar outputs on the target model.
  • The system demonstrates strong transferability by successfully reconstructing inputs from…: The system demonstrates strong transferability by successfully reconstructing inputs from texts produced by entirely different language models.

Sources checked by our analysis pipeline: 1

Computer Science > Computation and Language

[Submitted on 31 Jul 2026]

Title:PTP: Previous-Token Prediction based LLM Inversion for Near-Exact Prompt Reconstruction

View PDF HTML (experimental)Abstract:Large language models (LLMs) generate text by auto-regressively sampling the next token. This inherently leads to a many-to-many mapping between prompts and responses, complicating the task of inferring prompts from observed outputs. Prior work on LLM inversion frames prompt recovery as a semantic reconstruction task. They rely on fine-tuning pretrained sequence-to-sequence models on large external datasets--and requiring access to model weights or logits--to generate semantically plausible prompts. In contrast, we present a functional approach to inverting a given LLM in a black-box setting, without auxiliary aids. We train an explicit inverse language model entirely from scratch on data synthetically generated from the target LLM itself. Analogous to forward next-token prediction, our inverse model is trained using previous-token prediction, establishing a generative link between the forward and inverse processes that enables faithful prompt reconstruction. Moreover, it naturally supports diverse prompt reconstructions through sampling, whereby all such prompts induce similar responses under the forward, target LLM. Our approach…

Artificial Intelligence