If there’s even a small chance of a really big thing happening, should you do it?
“The need to be first or to perish” is what set in motion the explosion of consumer use of artificial intelligence. That argument raised billions of dollars, set off one of the largest infrastructure investments in American history and set off a global arms race.
The company that set off the arms race is OpenAI, and its cofounder and public face Sam Altman. Altman wielded this argument widely: If OpenAI doesn’t race toward the possibility of superintelligence than an existing incumbent like Google will, or China will. But no one else, nowhere else, could have done this. At least not now.
Or so the argument goes in Empire of AI: Dreams and Nightmares in Sam Altman’s OpenAI published this year and written by journalist Karen Hao.
One Chinese researcher told Hao that no one would have been funded outside Silicon Valley with over $1 billion without a clear purpose, so we created the risk and the solution. As she writes: “everything OpenAI did was the opposite of inevitable.”
Altman shares a birthday with the legendary physicist Robert Oppenheimer (“father of the atomic bomb”). Altman loves the comparison of OpenAI to the Manhattan project, though Hao notes that “he never seemed to add that Oppenheimer spent the second half of his life plagued by regret and campaigning against the spread of his own creation.”
At an industry event in December 2024, Altman’s cofounder and one-time chief scientist Ilya Sutskever said that “we have but one internet,” in referring to the source material that the AI industry has already primarily digested. Having consumed all of us, they seek more, Hao argues, in preferring the “empire” metaphor of the AI industry. The book is exhaustive and critical. A very worthwhile read for those following the industry, even though it goes into even greater detail on internal politics than I needed. It reads as authoritative.
Below I share my notes for future reference.
My notes:
- The fall 2023 Altman ousting wasn’t trivial but represented: how do we govern artificial intelligence?
- She prefers the empire metaphor, of extraction and exploration of labor
- [[Immediately clear the question is whether if OpenAI didn’t make as big a push would someone else have?]]
- Musk viewed Deepmind founder Hassabis as his rival
- Sam Altman’s dad went to Penn
- Paul Graham and Peter Thiel were Altman’s primary mentors
- Altman’s addition to the YC questionnaire: describe a time you hacked a system to your advantage
- Thiel’s monopoly strategy: competition is for losers lecture at Altman’s Stanford course; comparing to disc drive manufacturing in the 1980s:, many competing companies drove innovation that was good for consumers but not for the wealth building of entrepreneurs
- Altman’s United Slate: (1) prosperity from technology, (2) economic fairness, (3) personal liberty
- Geoffrey Hinton a mentor for Ilya Sutskever
- Greg Brockman less beholden to scientific skepticism of AGI
- Open AI was supposed to be open as anti Google deep mind
- “As we get closer to building AI it will make sense to start being less open” Sutskever wrote — Musk, Altman agreed, acknowledging before launch that the “open” goal was for after it reached its goals, but to complete in science it couldn’t always be open
- Before December 2015 $1b launch, Sutskever was in a bidding war with Google and the $2m plus salary grew so much he felt open AI was needed as counterbalance
- Timnit Gebru
- Likely apocryphal story of the NASA janitor who said his job was putting a man on the moo
- Arthur C Clarke famous three laws in failure of imagination:
- When a distinguished but elderly scientist states that something is possible, he is almost certainly right. When he states that something is impossible, he is very probably wrong.
- The only way of discovering the limits of the possible is to venture a little way past them into the impossible.
- Any sufficiently advanced technology is indistinguishable from magic
- Pro Publica’s groundbreaking 2016 “machine bias” reporting series
- Deborah Raji wrote a paper in response to Amodei’s influential 2016 warning of alignment being separate from economics issues — her paper argued they were together and the commercial pursuits were already happening: “in reality, for AI systems to even be built, there is very often a hidden human cost”
- Google’s Deepmind winning Go in 2017 made Musk nervous
- Compute is based on three things: processing power of individual chip (how many calculations per second); total number of chips snd how long they run
- Sutskever’s Alexnet grad school paper in 2012 (with his advisor Geoffrey Hinton) is the start of AI revolution
- The need of compute was so big they didn’t have enough money to reach there
- Musk advised that Tesla was the right home for OpenAI but Brockman and Sutskever chose Altman as their preferred leader —Musk left and only ever put in $45m of the $130m of the original $1b opening boast
- In April 2018, open AI had a new charter and its new AGI definition: “ highly autonomous systems that outperform humans at most economically valuable work”
- A GPT 2 demo was enough for Bill Gates to approve the Microsoft investment
- Like Moore’s Law, there’s OpenAI Law that they want to follow maniacally
- Author’s MIT tech Review story from Feb 2020, started Musk tweets and internal staff email form Sam Altman about disconnect between commitment and reality
- Power and Progress book by Daren Acemoglu: all technologies start with a rallying cry of broad benefits
- John McCarthy changed from “automata studies” to “artificial intelligence” as a branding exercise, which NYT reporter Cade Metz calls the field’s original sin, as the term has generated both excessive hype and unwarranted fears about the technology.
- McCarthy’s 2004 “what is artificial intelligence” document admitted that without a human definition of intelligence it’s difficult to recreate it — so we use human benchmarks
- Jenna Burrell: the goals of AI research are “ever-receding horizon of the future”
- Early AI research: symbolists (rules, Minsky) and connectionists (learning, Rosenblatt, and later Geoffrey Hinton at CMU in 1980s, who rebranded his backpropogation into deep learning, which led to popularizing neural networks) but neural networks needed computing power and more data
- ELIZA was a symbolist example using rules, later so was IBM Watson
- 2012-2022 laid foundation for ChatGPT breakthrough
- Google and big data holders with generalist ambition preferred connectionist over symbolist — that commercial interests led the current win
- 2014: surveillance capitalism
- A 2023 paper by Ria Kalluri and others showed AI research used detached language: text l, rather than personal emails and creative works; objects rather than photos of people and detection rather than surveillance
- At May 2023 in Rwanda AI conference, author’s Wall Street Journal badge was recognized by researchers because of WSJ as a data set not as people’s work
- The Costs of Connection book in 2019 by Nick Coulldry on data colonialism
- 2015, Uber famously hired 40 out of 100 researchers from CMU in AI , symbolic of shift from university to industry as the costs for big advanced AI systems grew
- Gary Marcus’s 2020 book Rebooting AI (a Hinton nemesis!): neurosymbolic AI that combines connectionist and symbolist — his March 2022 essay deep learning is hitting a wall — right before DALL E 2 was released to great fanfare
- “Probable and accurate are not the same thing” for pro ballistic LLM — connectionism became dominant but didn’t have to be
- It’s interesting she mentions many examples of image generators getting race wrong (cleaning people are black and Hispanic, engineers and doctors in Africa are men — [[though author doesn’t mention when Google got it wrong the other way of black founding fathers]]
- “This is the empire’s logic: the perpetuation of the empire rests as much on rewarding those with power and privilege as it does exploiting and depriving those, often far away and hidden from view, without them”
- Sutskever most credited with OpenAI’s scaling ethos
- When Sutskever said in 2022 that AI could be slightly conscious, a researcher responded on twitter that in the same sense a wheat field is slightly pasta
- August 2017 “Attention is all you need” paper Google on introducing transformer (wider and more text than the short-range pattern analysis that spawned things like early iPhone listed predictive text capabilities on text; Google wanted for search and translate but Sutskever saw it for scalable neural network
- Alec Radford chose instead of translation like Google to use a data set of books and the transformer architecture to try predictive text — and it appeared to pick up nuance in English language
- Amodei and OpenAI: reinforcement learning through human feedback
- Open AI law grown to scaling laws
- “Pure language” versus “grounding” hypothesis in physical world
- One Chinese researcher said no one would have been funded outside Silicon Valley at $1b without a clear purpose — we created the risk and the solution . Her reporting: “everything ApenAI did was the opposite of inevitable” — Sam Altman and its founding team created this
- OpenAI had no policy or qualms about changing from GPT 2 into much wider data sources like common crawl and transcribing YouTube videos while Google wouldn’t with its own YouTube because of ifs standards
- A change from controlling the inputs (data sources) to
- Abeba Birhane: data swamps and hate scaling laws
- The 2018 “Gender Shades” paper, co-authored by Joy Buolamwini and Timnit Gebru, is highly important because it provided concrete, empirical evidence of racial and gender bias in commercial AI systems
- Algorithms of Oppression: How Search Engines Reinforce Racism is a 2018 book by Safiya Umoja
- Gebru’s stochastic parrots Google paper with Bender got her fired (author reported on it here). Jeff Dean continued to criticize the paper especially its reliance on Strubell’s environmental cost estimates
- Anthropic team leaving OpenAI (in late 2020) meant less resistance to commercialization
- AI use of “red-teaming” has co opted a rigorous cybersecurity term for much more lax approach
- June 2021 launch of GitHub Copilot was part of OpenAI negotiation —the Microsoft team now wanted its own commercial product to get credit
- Altman: Worldcoin; investments in Retro Biodciebces and Helion energy — and via his YC stake did have a stake in OpenAI though he undersells that
- It was Appen’s Ryan Kolln who told author about the paradigm shift from controlling inputs to controlling outputs
- Billy Perigo’s Time story on Kenya outsourced content moderation at Sama, a social enterprise that was struggling after its founders cancer death
- The 2019 book Ghost Work: How to Stop Silicon Valley from Building a New Global Underclass by Mary L. Gray and Siddharth Suri exposes the hidden, low-paid human labor that powers AI and online services, introducing this human labor of annotation
- The Fairwork project at the University of Oxford is a major international action-research initiative that evaluates and ranks the working conditions of digital labor platforms across the globe
- Scale AI’s Alexander Wang: pulling out of Venezuela and then Kenya when they said enough users in those countries were trying to scam the company out of money, and the company shifted to reflect where customers wanted them — going up market — and the author notes this technology “will devalue the labor of everyone else”
- William MacAskill’s (his book) early 2013 effective altruism paper on Earn to Give argues that it’s more efficient to earn in a morally ambiguous job and then donate to more effective charities than to just work at an ineffective charity
- Three big causes identified in MacAskill’s 2018 TED talk that were big in scale, tractable and relatively neglected: global health like malaria nets; abolishing factory farming and existential risk
- EA vs e/acc (ee-ack) or the doomers and boomers of AI
- Was the test and answers just in the training data? From peer reviewed to “PR reviewed science”
- Backed by a false rumor that Anthropic was developing a chatbot, OpenAI pushed to launch ChatGPT (November 2022) using its 3.5 version, which had been out for months and not that difference from version that was out for two years but the interface matters. The team planned for an upper limit of 100k users, it hit 1m in five days and 100m in 2 months — the fastest ever to reach that number
- It had awaken google
- Atacameño (Lickanantay) Indigenous communities in Chile’s Atacama region are actively protesting lithium mining, citing severe impacts on water resources, land subsidence (sinking), and cultural heritage
- Water crisis near Google data centers in Chile and Uruguay
- A lot of global north and south mentions
- May 2023 Altman before Congress giving policy that was distant from OpenAI: (1) create a licensing agency, (2) AI safety standards and (3) independent audits
- The Frontier Model Forum is an industry-supported non-profit focused on addressing significant risks to public safety and national security
- Sara Hooker among researchers that do not think compute is the right threshold for frontier
- IBM VP Christina Montgomery: require AI models to disclose training data
- Policies in 2024 split between open vs closed; techno nationalism vs border less science
- Resnet was developed by Chinese researchers in Microsoft Beijing office
- Sept 2023 Raji was long academic testifying to Congress that wasn’t paid by industry to the Doomer community — and present at Chuck Schumer’s AI Insights forum
- CMU grad and Polish scientist Jakub Pachocki oversaw the development of OpenAI’s ChatGPT 4
- Altman shares a birthday with the legendary physicist Robert Oppenheimer (“father of the atomic bomb” ), Altman loves the Manhattan project metaphor, though the author notes that “he never seemed to add that Oppenheimer spent the second half of his life plagued by regret and campaigning against the spread of his own creation”
- The legendary “near zero” chance of trinity nuclear test (July 1945) killing everyone— same with AIs
- Sam Altman owned the OpenAI startup fund without fully disclosing to his board, one of many issues that got him removed
- A September 2023 New York Magazine profile of Sam Altman, titled “Sam Altman Is the Oppenheimer of Our Age,” details a strained relationship between Sam and his sister Annie
- Helen Toner and her “costly signals” paper about state conveying to the public their AI regulation goals
- Murati and Sutskecer among the executives who spoke to Helen Toner and the rest of the independent board members
- Employees called the Altman ouster “the blip”
- Scarlett Johansson voice was used (inspired?) by an early 2024 OpenAI voice assistant, even after she had declined to offer her voice
- Author argues: Sam Altman will tell you what you want to hear; Musk tells you what will surprise you
- Of 7,000 languages, half are endangered, a third of online presence, 2% on Google translate and 0.2% are supported by GPT 4 above 80% accuracy
- M?ori AI language: consent, reciprocity, and Mauro sovereignty “data is the last frontier of colonization ” said inventor Mahelona
- 310 hours of language recordings donated as opposed to the 680k hours ripped from the web by OpenAI for whisper; the M?ori language model used open source Mozilla Foundation DeepSpeech and developed a specialized model that hit 86% accuracy. Te Hiku used just 2 GPUs – small and task specific
- “The critiques that I lay out in this book… Are not by any means meant to dismiss AI in its entirety. What I reject is the dangerous notion that broad benefit from AI can only be derived from – indeed will ever emerge from – a vision for the technology that requires the complete capitulation of our privacy, our agency and our worth, including the value of our labor and art, toward an ultimately imperial centralization project”
- Timnit Gebru: DAIR has 7 pillars
- Tech worker Community Africa
- Ria Kalluri at Queer in Ai workshop aT 2019 NeurIPS argued that the right question about Ai is whether it consolidates or redistributes power
- Forget the doomer worry and think immediate term harm
- Joseph Weizenbaum (of ELIZA), once a technology is sufficiently explained “its magic crumbles away”