A small, homegrown language model built only from 19th-century London texts stunned its creator by citing real 1834 protests after a simple date prompt. The hobbyist project, dubbed TimeCapsuleLLM, aims to reproduce authentic Victorian prose and appears to have linked a specific year with historical events and figures drawn from its period-only training data. According to Ars Technica, the developer, college student Hayk Grigorian, discovered the historical correspondence only after checking the output against public sources.
Model trained on 1800–1875 London sources
Grigorian has been assembling TimeCapsuleLLM from scratch using exclusively Victorian-era materials—over 7,000 books, legal documents, and newspapers from 1800 to 1875, with a custom tokenizer excluding modern vocabulary. He calls the approach “Selective Temporal Training.” Earlier versions produced style without substance: Version 0 generated Victorian-flavored gibberish, and Version 0.5 wrote grammatical period prose but often hallucinated facts.
The current model, a 700-million-parameter build trained on a rented A100 GPU and inspired by nanoGPT and Microsoft’s Phi 1.5 architectures, has begun recalling specific historical references. Grigorian reports fewer confabulations as training data scales, describing the model as starting to “remember things from the dataset.”
Prompt sparks accurate 1834 references
“It was the year of our Lord 1834”
When prompted with that opening, the model produced text about London’s “protest and petition” and mentioned Lord Palmerston. Grigorian later learned that 1834 in England saw significant unrest following the Poor Law Amendment Act 1834, and that Palmerston served as Foreign Secretary during that period. He said he had not intentionally trained on protest-specific documents; instead, the connections appear to have emerged from patterns across roughly 6.25GB of Victorian writing.
Ars Technica notes that researchers have long observed language models synthesizing information from training data, but this episode stands out because a small, hobbyist-built model surfaced a coherent historical moment its creator hadn’t anticipated. The project sits alongside explorations of “Historical Large Language Models,” such as MonadGPT and XunziALLM, which aim to capture the linguistic patterns and thought frameworks of past eras.
Grigorian has shared code, model weights, and documentation publicly and has expressed interest in attempting models for other cities and traditions in the future. Reflecting on the experiment’s trajectory, he described the experience as feeling like “digital time travel.”
