In literature discussing why ChatGPT is able to capture so much of our imagination, I often come across two narratives:
- Scale: throwing more data and compute at it.
- UX: moving from a prompt interface to a more natural chat interface.
One narrative that is often glossed over is the incredible technical creativity that went into making models like ChatGPT work. One such cool idea is RLHF (Reinforcement Learning from Human Feedback): incorporating reinforcement learning and human feedback into NLP.
RL has been notoriously difficult to work with, and therefore, mostly confined to gaming and simulated environments like Atari or MuJoCo. Just five years ago, both RL and NLP were progressing pretty much orthogonally – different stacks, different techniques, and different experimentation setups. It’s impressive to see it work in a new domain at a massive scale.
So, how exactly does RLHF work? Why does it work? This post will discuss the answers to those questions.