Home Creators Posts Import Register Favorites Logout

Content

"In China, which rivals the US in AI talent, no team of researchers and engineers, no matter how impressive, would get $1 billion, let alone ten times more, to develop a massively expensive technology without an articulated vision of exactly what it would look like and what it would be good for...Not even in Silicon Valley did other companies and investors move until after ChatGPT to funnel unqualified sums into scaling...It was specifically OpenAI, with its billionaire origins, unique ideological bent, and Altman's singular drive, network, and fundraising talent, that created a ripe combination for its particular vision to emerge and take over...In other words, everything OpenAI did was the opposite of inevitable; the explosive global costs of its massive deep learning models, and the perilous race it sparked across the industry to scale such models to planetary limits, could only have ever arisen from the one place it actually did."

so I already was aware of the tech industry's obsession with scale, but I didn't realize the extent to which this was basically OpenAI's whole strategy. It almost feels too easy (and in a way it is, considering all the now obvious problems associated with their quantity over quality approach to data sets/resources required for infinitely increasing compute ), to the point of it being comical to imagine one company trying to hide their 'strategy' from the others, when said 'strategy' is just use MORE. Like I know we make fun of 'line go up' logic, but the more I learn about the inner workings of these companies the more literal a foundational principle that seems to be. I think the irony got a little too much for me at the point where, in a conflict over the API's safety issues vs potential revenue, the resolution was 'if we ignore safety now we'll have EVEN MORE money to invest in safety later!' Has the same energy of ignoring environmental impacts of developing possible future AGI bc it can maybe one day solve climate change.

~~~

A few little observations:

"Working with high-profile institutions would help foster more cooperation between industry and academia for addressing AI safety risks"

This bit had me thinking about the Genesis Mission, which I talked about briefly in the last office hours (Angela Collier has a long video about it ) that's requiring researcher's to partner with AI industry in order to receive federal funding---so far I'd be thinking of this as a way to siphon public funds off to private enterprises, but reading this section made me wonder if it's also trying to siphon off the prestige/credibility of research institutions.

~~~

"Big AI was increasingly going the way of Big Tobacco...distorting and censoring critical scholarship against the interests of the public to escape scrutiny."

The entirety of "Science in Captivity" was really hitting for me because of the research I'm doing rn for my next video. For anyone who was in the Agnotology book club and read the essay on manufacturing doubt about the tobacco industry/SDI/climate change, did the tactics google used to discredit Gebru/Strubell's research feel familiar to you too? I wasn't planning on talking about AI in this and the script is already too long but damn it feels like it's worth a mention now.

~~~

"If GPT-2 fell into the hands of terrorists, dictators, or clickbait farms...the model could be used for notorious purposes."

i just think it's funny that "clickbait farms" are included on the same list of threats as terrorists and dictators. technology's made for a weird world

Comments

Jacques Vincelette

Hyman G. Rickover, an admiral in the US Navy, known as the “father of the nuclear navy” for his work building the world’s first nuclear-powered submarines. It was a quote Altman had had painted on the office walls in the early days of OpenAI: "I believe it is the duty of each of us to act as if the fate of the world depended on [them]. Admittedly, one [person] by [themself] cannot do the job. However, one [person] can make a difference. Each of us is obligated to bring [their] individual and independent capacities to bear upon a wide range of human concerns. It is with this conviction that we squarely confront our duty to prosperity. We must live for the future of the human race, not of our own comfort or success." This resonates with Amentum/Bwtx peptalks taking over Canadian Nuclear Laboratories. A game of follow the leader at the edge of the cliff. Privatization of the federal lab was to break free of government no-profit constraints and help foster more cooperation between industry and academia. Pushing natural gas, nuclear energy, and synthetic fuels as environmentally friendly, We'll have EVEN MORE money to invest in safety later!'

Jean Marrero

Yayy i caught up. Its tough but good practice to continuously read. Lots of power dynamics in these chapters. A particular passage caught my eye: "The discussions revealed how quickly measures of intelligence could veer into disturbing assessments of which groups of people had superior or inferior intelligence." I mean yeah, dont get me wrong, with how people are deciding to support politicians based on ads and no critical thinking (FYI: Go Abdul! Fuck the Mega-Donor Ads tipping the scales for Haley, Michigan go vote!). HOWEVER, we dont solve that by glorifying ones intellect but by making education more accessible and affordable. Its easy to think we are top of the world without acknowledging the social structure that led to our access to knowledge and training. 👏This 👏 is 👏 why 👏 STEM folk 👏 need humanities and social sciences. As for the last chapter, its such a common trope on companies penalizing employees for trying to do the right thing 🙂‍↔️🙂‍↔️. FML There is no justice, there is no God. Some people will just always get away with it at the end and thats fucking depressing, but good thing we got enough distractions to not think about it too much!

AJ Biffl

Yes! I absolutely was screaming "agnotology" during Science in Captivity. Hiding the internal findings, both Google and OpenAI shutting off accountability research entirely, not publishing externally or at conferences, but then also Dean and Patterson trying to discredit Strubell, saying things are more "nuanced" or they didn't use the right numbers or they don't understand how Neural whatever works and it's only a one-time cost, and OpenAI and Microsoft agreeing to release technical details during this moment to try to save face (even reps from Congress briefly getting involved, and that amounting to nothing). They mentioned this briefly in AI Snake Oil, the pretty sudden change in research culture from "they wanted their discoveries to contribute to human knowledge and not just to a company's bottom line," to suddenly everything behind closed doors. And also about the reproducibility crisis (2018 investigation that found that *none* of the 400 papers they surveyed contained enough details to be reproducible by an independent researcher). But the account here is vivid. What a catalyst. What a rotten mess. Also, lol at "we could write our strategy on a grain of rice"

phil e the theyby

The "pure language" hypothesis described on page 129 seems patently ridiculous to me. Thinking that, because language is the primary way we *communicate*, that all *knowledge* can be documented in *text* requires us to believe that 1) language is ONLY text [meaning things like tone and volume are irrelevant to verbal communication], 2) there are no experiences that aren't best represented in text format. But the so-obvious-it's-cliche example of learning to ride a bike directly and immediately contradicts this; you cannot competently ride a bike immediately just from reading text, because the learning it requires is physical and kinesthetic, not linguistic. In "Life in Code," Ellen Ullman does a great job of explaining how much of our lives and our consciousness are centered around our physical bodies doing things that machines don't do (eating, sleeping, waste elimination). In a set of chapters focused on the actions of individual people ("Scale of Ambition" a bit less so), I couldn't help but see them as individual actors playing out the predetermined dance whose steps were choreographed by much larger systems. The actors involved correctly identified that monopoly is the most effective path to power, control, and agency within our system, and are acting accordingly. They aren't devoid of humanity (I think there is genuine concern about the future of humanity), but they only know how to process that through their limited ideological systems, not realizing that they're interpreting these things through established cultural lenses that also meet their own emotional needs. One of my favorite parts of AI Snake Oil was when the authors were discussing how tech oligarchs painting the risk of AGI as a real, imminent threat isn't necessarily guided by "objective analysis" of the situation, but because that being true would make them (the tech oligarchs) *some of the most important people in human history*. If AGI is right around the corner and they're determining the fate of it, they're not just a new class of robber baron capitalists; they're literally the saviors of all of humanity. "Science in Captivity" really triggered my cynicism in a way that made me very sad, particularly here: "Like many others, I had come to see her ethical AI team as a bastion of critical accountability research, a hopeful sign that companies were developing a capacity for self-reflection" (168) My cynical inner voice kicked in: "*eyeroll* How naive. Corporations will always do the thing they're designed for, which is consolidate power. They can't self-reflect." But I hate this reaction, not because I think the analysis is inaccurate, but because I shut myself off to feeling grief in response to someone who believed in something losing that faith. That is genuinely tragic, and I don't want to intellectualize that away. I want us to understand our systems and their limitations, and to work to be as effective as possible in that. Gebru isn't dumb; she's brilliant and is an absolute hero for doing everything in her power to prove to Google how harmful this all is (and would continue to be). The fact that Google didn't honor her message (which they were never going to) doesn't mean that she did the wrong thing; she showed their internal contradictions to the world, to anyone who would listen, and allowed the people who actually WOULD listen to find each other and rally together. The story of Gebru and Black in AI is my favorite part in this book so far, because it's a reminder that wherever there's oppression and systemic violence, there are people resisting. So I'm glad that Hao is telling their story (or at least a piece of it).

Justin Lee

I definitely agree with you on how ridiculous pure language is, something cannot be captured in words and the idea of systemic incentives can warp seemingly noble impulses

AlexTheTutor

Tangential observations: China has been storing oil reserves for years now and currently have the largest oil reserves in the world. This is the only reason why global oil prices are not currently at some crazy level of $200 per barrel; they can draw down on their oil reserves and stabilize global markets. In the language of medicine, China has been a great economic buffer due to recent acidic global policies… and stands to benefit greatly from it in the short-term. In the same way, China isn’t investing multi-billions in AI for no reason... China has shown its ability to plan for the long-term. The US (with its quarterly reports of publicly traded companies and private equity) always demand greater and greater results in the present moment. Basic science research (with no plans for immediate profits) is impossible in this kind of system, and as a result 10k plus American scientists are planning to leave or have already left to practice science in other countries. So, where would that leave advances in AI? It takes a lot of technique knowledge and sophisticated leadership to know how to advance AI and to actually advance it for the betterment of society. There is a video game: Dragon Quest, specifically Dragon Quest 8, in which at some point in the game, there is a casino. You can bet and win a lot of tokens, and trade these tokens for the best equipment in the game. The game is by chance, and so it is designed for players to lose. But, this is just a video game, and it is easy enough to reset the game as many times as you want, until you win the jackpot and get the best equipment and all the money you will ever need. If only real life had a game like this, how easy it would be to make as much money. Anyway, I wonder how AI will be used to evolve Polymarket…

AJ Biffl

That bit about grief really hits. I've been a cynic for so long, so used to passing by objectively horrifying things while scrolling or reading. It's easier to try to retreat from the horrors, to become numb, than to continue feeling them and continue fighting. You're right, Gebru is nothing short of a hero for continuing despite the colossal forces trying to silence her.

AJ Biffl

I'm also flip-flopping on the pure language thing, because on the one hand language is a thing that can grow to include new things, new experiences, new ideas. The pure language hypothesis seems to only need language to be "complete" (in a grander sense of completeness akin to Turing complete, but for expressing information), not to be the most efficient way to encode information, nor even a perfect way to encode information. So yes there are experiences and feelings that don't translate directly into words, but there are also poets

phil e the theyby

The "pure language" thing apparently is a bit of a trigger point for me, so apologies if I'm too intense here :P My interpretation of the "pure language" hypothesis is that language, essentially in the form of logical statements, is all that's needed to transmit information (although like you said, it might not be a particularly efficient way of doing so). But I think most things that LOOK like transmitting information via language actually involve several other processes. I think that understanding poetry relies not only on the phonetics of language, rhythm, and meter (none of which are strictly contained within the text) but also on triggering reactions because of their connection to our lived experience, which serves as an external referent. An LLM can, by virtue of imitating existing poetry, use some of that language and potentially leech off of some of the experience referenced by poets. But because they have no access to the direct experience, they can't ever embody it nearly as well as an actual poet would be able to. I think that, if text CAN be used to express all experience (which I personally doubt), it can only do so in the sense that "a million monkeys tapping at typewriters can write any book." Is it possible? Sure. But it's so insanely inefficient and requires so much wasted energy to do something that other methods can do much more effectively. So from my perspective, the "throw more compute at it" strategy was bound to run into the limits of physical reality FAR before it reached its explicit goal of AGI. I realize that's veering away from the "is it possible" question that you were talking about and towards the "is it practical" question; I guess I only bring that up because it seems like the direction of the industry has been to make a firm claim that the "pure language" hypothesis is correct (despite there clearly being deep philosophical debate around it), AND that it is practical and wise to act on the basis of the pure language hypothesis. "AI Snake Oil" talked about some of these limits around predictive AI (essentially, insufficient data actually exists to create accurate predictions for most of the things we care about), but "Empire of AI" is painting a picture that even with the entire internet, there's insufficient data to create AI systems that can imitate most economically valued human labor. That said, the next chapters talk a bit about the big tech companies realizing those limitations and shifting strategy to adjust for them, so I'm curious to see how it plays out from an actually researched perspective. But from my own layperson's perspective, the results of the last couple of years of AI research and development have been...not much over what ChatGPT did from its first launch.

AJ Biffl

Ok yeah definitely hard agree that it's not practical or realistic to actually bring about something deeper than regurgitation by training with language alone (and I was shocked by how early in development they were already scraping the bottom of the barrel to find training data) Like I said I keep flip-flopping. Like on one hand I have a deep conviction that given the time and energy and thought, a person could communicate any idea or feeling to another person with words alone, text alone. But how much of that requires connecting to past experiences or feelings? My understanding of a word is tangled up in my memories and experiences, and in this hypothetical, the words that effectively communicate from person A to person B might not be effective in communicating the same idea to person C. But then maybe is it the case that an LLM builds up its own subjective understanding of those words during its training? It doesn't build up an objective field of meaning because such a thing doesn't exist, but it associates meanings together into a larger map, and it builds that map based on its training. That map is its version of experience, a small sliver of the reality we experience, but is it enough to connect an incoming stream of text from anonymous usernames to real things?

Laura Robinson

There seems to be a real tension between the rhetoric of AI having all human knowledge and how these models only have a consistent point of view insofar as the humans designing them weigh certain ideas as valuable. I don't think that the people pushing these things have a notion of knowledge being anything but explanations that fit perfectly/the best with material reality and don't have a notion that no ideas completely solve the problems in its domain and that human concerns are a part of theory evaluation. Having access to all human narratives and finding the one that most engaging to the user feels like a big element of the cases of AI psychosis that have been trickling out, over and above just sycophancy. But also deciding which ideas are the ones that the bot is trained to maintain is an ideological project for which a lack of transparency is concerning.

Some Ed

I think when someone did a thing that turned out to be negative and apologizes for it profusely, and somebody who wasn't them says, "Well, it was probably inevitable", most of the time that turns out to be more or less trustworthy. When the person or people perpetrating a thing tell you "here's this thing we're doing that was inevitable anyway", that feels very different.