Research Papers

REALM: Retrieval-Augmented Language Model Pre-Training

Bakes retrieval directly into pre-training, so the model learns to lean on external documents.

REALM matters because it shows retrieval is not a bolt-on. Models trained this way develop different habits — they defer to sources more readily.

The habit difference

Retrieval-native models cite sources more often, hedge less, and are more sensitive to source formatting than models where retrieval is added after training.

The practical takeaway: the more retrieval-native a system is, the more your source formatting affects whether you get quoted.