73 Years Ago, Shannon Planted the Seed for Large Model Development
Did Shannon Predict the Future? Discover the Roots of LLMs
Tracing AI's roots back to its founding fathers.
Is the principle behind today's booming Large Language Models (LLM) proposed by Claude Shannon?
Today, Princeton University Professor Sebastian Seung shared this perspective:
In 1951, Claude Shannon, working at Bell Labs in Murray Hill, New Jersey, posed the problem of predicting the next word, laying the seed for current LLMs.
Many seemingly cutting-edge concepts were actually proposed decades ago, even in new fields like computer science. This notion and the mention of Murray Hill sparked discussions, with Turing Award winner and Meta Chief Scientist Yann LeCun noting that influential work originated from Murray Hill, Florham Park, and Princeton.
LeCun listed several notable research outcomes:
Hopfield Networks (by Hopfield, who worked at both Bell Labs and Princeton University)
ConvNets
Boosting/Adaboost
Non-negative Matrix Factorization
Support Vector Machines (SVM) and Kernel Methods
Structured Prediction
Computational Learning Theory/VC Theory
So, how did Shannon contribute to today's path toward AGI with his early work?
Keep reading with a 7-day free trial
Subscribe to AI Disruption to keep reading this post and get 7 days of free access to the full post archives.