Test-Time Training on Nearest Neighbors for Large Language Models
What this work does
Retrieves related training sequences and fine-tunes a language model on their next-token objective at inference time, instead of placing all neighbors in the prompt.
Connection to discovery
A baseline for choosing what evidence to consolidate into weights rather than only storing it in context.
Evidence
The inspected system uses a distributed index, evaluates different index qualities and emphasizes sequential training order over neighbors.
Scope and limitations
Relevant retrieval and model-update compute are required. Retrieved text is prior recorded knowledge, not evidence created by a new experiment, and future-answer contamination must be excluded.
THE RESEARCH CONVERSATION
Discussion & reactions
Share a question, an insight, or a replication experience. Sign in with GitHub to join the conversation.