Five Papers That Shaped Modern AI (Start Here)
If you want to understand why AI works the way it does today, you don't need to read a hundred papers. You need to read about five, in order, and understand what each one changed. Everything since is, to a first approximation, a refinement or a scaling-up of ideas these introduced.
This is that list — the five we'd hand a strong engineer joining the field. Not the five "best" papers by citation count, but the five that each moved the ground under everyone's feet. Read in sequence, they tell one continuous story: how we got from machine translation to agents.
Architecture → scale → reasoning → alignment → efficient training. Three of the five landed within a few months of each other in early 2022.
1. Attention Is All You Need (2017)
Vaswani et al. · June 2017 · arXiv:1706.03762
Why it mattered: it invented the Transformer, the architecture underneath every large model since.
Before this, sequence models were recurrent — they processed tokens one at a time, which made them slow to train and forgetful over long inputs. The Transformer threw out recurrence entirely and replaced it with self-attention: a mechanism that lets every token look directly at every other token in one step. That single change did two things. It let models learn long-range relationships that recurrence kept losing, and — because attention is parallelizable — it let training scale across hardware in a way recurrence never could. The paper's title was a swagger, and it earned it. Almost nothing that came after is possible without it; when people say "an LLM," they mean a very large Transformer.
2. Language Models are Few-Shot Learners — GPT-3 (2020)
Brown et al. · May 2020 · arXiv:2005.14165
Why it mattered: it showed that scale alone unlocks a new capability — learning from the prompt, with no training.
GPT-3 took the Transformer and made it enormous: 175 billion parameters trained on a large slice of the internet. The headline wasn't the size, though — it was in-context learning. Give the model a few examples of a task in the prompt, and it performs the task, with no fine-tuning and no weight updates. This reframed the entire practice of using models. You stopped training a model per task and started prompting one general model, and "prompt engineering" became a discipline overnight. It's also the paper that made scale the field's central bet: make the model bigger, and qualitatively new abilities appear.
3. Chain-of-Thought Prompting (2022)
Wei et al. · January 2022 · arXiv:2201.11903
Why it mattered: it showed that asking a model to show its work unlocks reasoning it otherwise can't do.
Large models were oddly bad at multi-step problems — arithmetic, logic, word problems — even as they aced other tasks. The fix turned out to be almost embarrassingly simple: prompt the model to produce intermediate reasoning steps before the final answer, and accuracy on hard problems jumps. "Let's think step by step" became a genuine capability unlock, not a party trick. More deeply, it revealed that a lot of a model's competence is latent — present in the weights but only accessible if you give it room to compute. Every "reasoning model" that followed is a descendant of this observation.
4. Training Language Models to Follow Instructions with Human Feedback — InstructGPT (2022)
Ouyang et al. · March 2022 · arXiv:2203.02155
Why it mattered: it turned raw, unruly language models into helpful assistants — the step that made the technology usable by everyone.
A raw pretrained model predicts likely text; it does not, by default, do what you ask. InstructGPT introduced RLHF — reinforcement learning from human feedback — to close that gap: humans rank model outputs, a reward model learns those preferences, and the language model is tuned to produce what humans prefer. The result was startling: a much smaller RLHF-tuned model was preferred over the giant base model, because it was actually trying to be helpful. This is the paper that made the assistant paradigm work, and it's the direct methodological ancestor of ChatGPT and every chat model since. Alignment stopped being a safety footnote and became the productization step.
5. Training Compute-Optimal Large Language Models — Chinchilla (2022)
Hoffmann et al. · March 2022 · arXiv:2203.15556
Why it mattered: it corrected how the whole field spends its compute — and proved most big models were trained wrong.
The prevailing wisdom after GPT-3 was "bigger is better," so labs raced to add parameters. Chinchilla showed that for a fixed compute budget, parameters and training data should scale roughly together — and that the giant models of the day were badly undertrained, starved of data relative to their size. A smaller model trained on more tokens beat a larger one trained on fewer. This reshaped training strategy across the industry: data quality and quantity became first-class, and "compute-optimal" entered the vocabulary. It's why the strongest models aren't simply the biggest ones.
Where the story goes next
Read those five and you have the spine: an architecture that scales (1), the discovery that scale creates capability (2), a way to summon reasoning from it (3), a way to make it helpful (4), and a way to train it efficiently (5).
Everything we work on sits one step past the end of this list. Once models could reason and follow instructions, the obvious next move was to let them act — to call tools, observe results, and loop — which is the agent paradigm and the line of work (ReAct, Toolformer, and what followed) that this blog mostly lives in. If you want the sequel to this reading list, that's where it starts. But start here. These five are the grammar; the rest is composition.
Written by CatalEx Engineering. We build the AI operating layer for AI-native companies — one platform to build, deploy, and run AI agents in production. More at catalex.co.