Our paper Pre-training Limited Memory Language Models with Internal and External Knowledge is accepted at ICLR 2026!