Ilya: Predicting the Next Token is Understanding the World

A 44-second clip of Ilya Sutskever explaining AI's next-word prediction to NVIDIA's Jensen Huang has sparked Reddit discussions on AI understanding and reasoning.

A 44-second video clip from a year ago, where Ilya Sutskever explains to Jensen Huang that "AI predicting the next word can lead to real understanding," has sparked discussions on Reddit recently.

In the clip, Ilya uses an analogy while answering Jensen’s question:

"I'd like to take a small detour and offer an analogy to explain why more accurate next-word prediction can lead to deeper understanding—true understanding. Imagine you're reading a detective novel. The story is full of diverse characters, events, and intricate plots that leave you puzzled. Then, on the last page, the detective gathers all the clues and people and says, 'I will now reveal the identity of the criminal, and this person’s name is…'"

This snippet comes from a "fireside chat" between NVIDIA founder and CEO Jensen Huang and OpenAI co-founder and chief scientist Ilya Sutskever at NVIDIA's GTC conference on March 23, 2023.

At the time, GPT-4 had just been released the day before, and Jensen praised its breakthroughs while asking Ilya, "What is the main difference between ChatGPT and GPT-4? What led to GPT-4’s improvements in these areas?"

The "detective novel" analogy ignited online discussions about AI’s understanding, the distinction between understanding and prediction, and whether AI truly possesses cognition.

In the full video, Ilya explains that while large neural networks predicting the next word in various online texts seem to be learning statistical correlations in the text, they are actually learning a model of the world.

Ilya uses the detective novel analogy to explore the reasons behind GPT-4's breakthroughs, his views on Scaling Law, AI’s potential and limitations, and the importance of AI reliability and safety.

In the year and a half since this fireside chat, Ilya’s philosophy and technical convictions have remained consistent, even reflected in his startup, SSI.

Some of his views still hold valuable insights into the current direction of AI development.