Dynamic Evaluation of Transformer Language Models
What this work does
Updates a Transformer-XL on already observed sequence segments before predicting later segments, treating weights as a short-term adaptation memory.
Connection to discovery
Establishes the causal ordering needed for test-time learning from newly acquired evidence without training on future targets.
Evidence
Character-level cross-entropy and word-level perplexity comparisons measure the effect of SGD or RMS-style online updates.
Scope and limitations
Language-model likelihood improvement is not scientific discovery. Reset policy, tuning data and allowed observation order are part of the evaluation protocol.
THE RESEARCH CONVERSATION
Discussion & reactions
Share a question, an insight, or a replication experience. Sign in with GitHub to join the conversation.