Home Creators Posts Import Register Favorites Logout

Content

"After analyzing one thousand years of technology history, the authors conclude that technologies are not inevitable. The ability to advance them is driven by a collective belief that they are worth advancing. The irony is that for this very reason, new technologies rarely default to bringing widespread prosperity...Those who successfully rally for technology's creation are those who have the power and resources to do the rallying. As they turn their ideas into reality, the vision they impose---of what technology is and whom it can benefit---is thus the vision of a narrow elite, imbued with all their blind spots and self-serving philosophies. Only through cataclysmic shifts in society or powerful organized resistance can a technology transform from enriching the few to lifting the many."

The aforementioned authors here are Daron Acemoglu and Simon Johnson, who wrote "Power and Progress: Our Thousand-Year Struggle Over Technology and Prosperity". next book club pick, anyone?

~~~
something I tend to think about AI is that deep learning itself is a really impressive and (in some cases, not for literally everything ever) useful development in software, but the biggest tech companies are just deploying it in the worst ways. I still think that's true to an extent, but I think these chapters somewhat took the air out of that feeling for me. e.g., yes there are definitely useful applications in fields like climate change and healthcare, but companies like OpenAI aren't just not doing that, their dominance as AGI-motivated above all else is actively disincentivizing it:

"the main challenge of working in [climate change and healthcare] was not technical limitations. It was quite the opposite: persuading talented scientists to focus on problems that necessitated rather simple machine learning solutions, instead of the latest cutting-edge techniques that satisfied their ambitions and looked better on a research resume."

this seems like a clear example of what AI could be, but Altman et al are forcing it not to be. booo

another example of this: for anyone who read AI Snake Oil, did "Dreams of Modernity" also make you think of the part about AI research being increasingly dedicated to reaching higher and higher benchmarks at the expense of catching up the understanding of how AI systems work the way they do? basically the authors attributed a lot of the issues emergent in modern AI as a result of these systems being built not on "alchemy" rather than any kind of scrutable logic. they had made reference to that being a result of the influence of industry, but I think did a lot to flesh the nature/extent of that influence for me:

"[Early on, ] Companies quickly came to dominate over governments and foundations as the biggest funders of AI research and were soon setting the research agenda based on advancements that could also produce short term-profitability... [Today,] The major AI powers are no longer setting the agenda so much as bending an entire discipline to their will."

obviously it's bad that this is happening insofar as it's producing the exploitative manner in which the big AI companies are operating, but it also just makes me sad to see the space for curiosity and creativity shrink (at all, let alone in service of such bs).

it reminds me of the vibe of my old grad department---it was an astrophysics department in the bay area, and like this chapter mentioned, we had a bunch of professors who had either come from tech or were simultaneously working in tech, and most students would also go on to work in tech. this relationship had a significant influence on the culture/values of the department in a way that, to me, felt like it was crushing a lot of actual intellectual curiosity; everything had to be some machine learning model using whatever massive data set, and you did better science by having a better ml model and a bigger data set---it felt like such a novelty to see research that relied on something like a clever way of anticipating some kind of novel signal based on astronomy phenomenology and basic physics, instead of just bootstrapping your way to some statistically significant correlations.

on that note, not to simp for AI bros or anything, but using mannequin challenge videos as data for processing 3D space was kinda brilliant

Comments

Mx. Alba

These chapters confirmed what I had already surmised. Have you ever heard of the Australian bloke who won a French Scrabble tournament by memorizing the entire French dictionary of accepted Scrabble words? He didn't speak any French, neither before nor after memorizing the dictionary, but he won the tournament. Generative AI is much the same. It's a genuinely impressive party trick, but there's no substance. And the way they're scaling the technology is just brute forcing it beyond our economic capabilities. Moore's Law didn't state that the amount of transistors per chip would double every 18 months... It included commercial viability. Heposited that cutting edge technology could and probably would be able to far exceed his "law" - but not in a commercially viable way. The "OpenAI Law", scaling "compute" way faster than Moore's Law, is *possible*, but not commercially viable. They're pumping trillions of dollars into brute force data centers just for the privilege of "getting there" first... And where is "there"? Well, where we would have eventually gotten anyway in a few years without burning trillions of dollars. And burning, it is. All that hardware will be obsolete in a year. It feels like they're just trying to brute force their way to intelligence, without actually putting any intelligence into it... Seems like a dead end to me. A mad stampede toward the edge of a cliff. And on another topic: "AGI, if ever reached, will solve climate change, enable affordable health care, provide equitable education." YOU DON'T NEED AGI FOR THAT. We can solve climate change by not burning fossil fuels anymore. Affordable healthcare and equitable education is shown in quite a few countries already, not just the usual suspects in Europe, but also in Cuba, or at least, until the latest flurry of sanctions by the US, and China. If all those trillions of dollars that are now dumped into AGI were instead invested in green energy, universal healthcare and public education, wouldn't that make more sense?

Jacques Vincelette

Confusing "Law" with "business strategy" and "marketing ploy", no wonder layers' courts and cops entangle in organized crime. p115: " It is fueled and abetted by the culture of AI research that views consuming as much data as possible as its moral responsibility." p118: "This is the empire’s logic: The perpetuation of the empire rests as much on rewarding those with power and privilege as it does on exploiting and depriving those, often far away and hidden from view, without them often far away and hidden from view, without them."

phil e the theyby

Karen Hao really lays the hubris behind this process bare in these chapters, and the religious fervor of the pursuit of AGI is laid bare. The first time Brockman’s question about climate change and equitable healthcare is laid out on page 76, I wrote “Holy shit these are LAUGHABLE. The solutions are VERY obvious; the problem is political will.” But all of the AGI devotees treat every problem as a technological problem. But even if the god of AGI came down and gave us the answer, what if the answer was…”start listening to people who are most harmed by these policies and give them decision-making power”? Another L take by Brockman was: “Our mission is to ensure that AGI benefits all of humanity. And the way we want to do that is: Build AGI and distribute its economic benefits.” (79) So like…you don’t know how to build AGI. The economic benefits for this thing you don’t know how to build are entirely theoretical at this point. And this is what he puts his faith in? I’m starting to suspect that the intense fervor these men have around AGI is driven by their genuine, accurate realization that the world is fucked up, coupled with the inability to entertain ways of looking at those problems outside of the frame they’ve been given. And they are literally socially handicapped in their ability to do so; understanding the real solutions to these problems requires acknowledging the ways they and everyone they love are complicit, if not active agents, in creating the harm, violence, and environmental destruction all around us. As the consequences of capitalism amplify, the need for solutions becomes more urgent, and they turn with intense fanaticism towards the “solution” that doesn’t require them to acknowledge their role at all: a magical computer god that will come up with easy, comfortable answers to all of these problems. I hadn’t considered Dr. Fatima’s point about how AI has been pushed towards all the worst uses of AI and away from more tangible, practical benefits, but that point really helps tie together for me this disconnect between “potentially useful technology” and “world-ending nightmare.” Just this recognition that the structural incentives push people away from the actually practical and beneficial uses of AI is helpful to ground me in our current AI landscape.

Jae

Dr Fatima's note about the effects of industry on academia definitely aligns with my experience. The homogeny of thought in academia is problematic before any outside forces get involved, so AI companies influencing research *really* exacerbates the issue. My background is in computer science, I was fascinated with AI long before it was synonymous with neural networks. I've always found NNs to be too simple; the hypothesis is that brains are just a bunch of connected neurons that fire when they get the right signal, and produce some result. Can the complexities of the human brain, flesh & blood & chemical signals, the entire field of neuroscience, be reduced to a rudimentary analogue? I don't buy it, I don't think LLMs will EVER be able to think like a human. Maybe if they took the research in an entirely different direction, made an attempt to define intelligence, and integrated neurobiology research into their approaches, but that isn't what we are seeing. Reading this book I keep asking myself: is the goal actually to create intelligence? Their approaches are questionable, their intentions are questionable, their disregard for life is sickening, and they barely have a concept of a plan beyond throwing all of earth's resources at the problem.

Laura Robinson

To put it succinctly, I got Tom Lehrer's "Wehrner von Braun" stuck in my head from reading this.

Dr Jennifer Burgess

These two chapters expanded on some of the field's history given in AI Snake Oil, with important detail about all the politics and arguments going on with the people involved. I share the frustration about paying attention to output over understanding, a really common requirement for grant funding in medicine and neuroscience now is that you "show how it will benefit patients", well what if we need to research an aspect of the human body to advance our understanding that will hopefully cumulatively lead to a solution but won't in itself yet? Arg.

Mx. Alba

In the mid 1990s, the idea of AI also drew me to computer science. In hindsight, I'm really happy that I went into practical IT, instead of into research. I wouldn't have wanted to have become an AI tech ... Sib? No bro. 🤭

Laura Robinson

I don't know if thinking mechanistically like a human is quite what they're going for, especially since the future they present is that LLM-based AI is going to become smarter and more objective than humans. ChatGPT-2 and "Attention Is All You Need" seemed to suggest that AIs could "understand" and use language in a way that just required neural nets and that result was so surprising that it created a whole research paradigm and I think that the belief that this could be made into a super intelligence is, or at least was, genuine. But this is also a crowd that has a very simplistic view of intelligence wherein facts meet logic and truth comes out and only lack of data and human error are barriers. Getting humanity out of the way can only help from that standpoint.

Black EnviroNerd

I went to an Ivy League for undergrad and the people in this book remind me so much of some of the individuals I met while I was there. Specifically this part reminded me of one of my friends at the time, “He was sheepish to admit that he also read books for fun -science fiction epics like The Three-Body Problem trilogy by Chinese writer Liu Cixin and Issac Asimov’s Foundation series.” My friend was a brilliant computer science major and exceptionally deep thinker. He worked in a super demanding, hypercompetitive (and in my opinion toxic) lab as well that really seemed to erode his self worth. Anyways he had a passion for novels and I remember there was a night where he was describing how much a particular passage from a book he was reading moved him. Afterwards though he went on and talked about how it's a waste of time to read novels and that his buddies in his major and the lab made fun of him for it. He was genuinely morally conflicted over reading fiction as opposed to programming or reading a non fiction book. I think recalling my relationship with him and reading of these characters really got me thinking of how Silicon Valley and our culture at large is really grooming our talented, oftentimes neuro-spicy young men specifically into this quasi religious framework that Dr. Fatima talked about in the Technocracy video. I myself struggled to leave STEM work and education in favor of my interests in policy for years because of how much my self worth was tied to my intelligence and how I thought that intelligence was only valuable in STEM settings. Emotional intelligence, the ability to think about political structures, understanding my own experiences as a black man and intersectionality were all things I didn’t really understand the value in until after college. Thus far one of my biggest takeaways is how Karen Hao is really illustrating how the confluence of great man theory, masculinity, and technocracy is really shaping these (mostly) men’s delusions.

phil e the theyby

Similarly to Laura, I don’t think the goal is to really create “intelligence” in any way that we conventionally think of that term. While they describe AGI as something that will be “unimaginably smart” (obviously using a definition of “smart” that is both very monodimensional AND very vague), it seems like the descriptions of it almost immediately go to something that sounds more like a god than a mortal being. The way it’s described (from what I’ve seen) essentially treats it as an entity that can effortlessly solve any problems (because it can improve itself and make itself smarter, instantly! Infinitely!). So yeah, they describe it as creating “intelligence,” but the thing they’re describing sounds very disconnected from any definition of intelligence that can be operationalized or made tangible.

Jacques Vincelette

Last week on July 9, Canadian Nuclear Laboratories launched a new Cultural campaign on "Nuclear Professionalism": no tolerance for "cartoon fiction" Homer Simpson, Fallout, or Journal of Atomic Scientists. My comment on interesting timing with the 71st anniversary of the Einstein Russell Manifesto was censured within the hour.

Jacques Vincelette

"Outside both offices, the same two-year period would see the pandemic further ravage what had already become the epicenter of the tech industry’s gentrification. Long-standing Latino businesses would shutter. Violent crime would jump. A line of tents and discarded trash would spring up steps away from the Pioneer Building as homelessness reached crisis levels." Whether or not Covid-19 was developed with AI assisted CRISPR, intersectionality extends well beyond wokism.

Jacques Vincelette

Sutskever chimed in: “fundamentally the bottleneck is that you have a large number of humans and they don’t communicate as fast, they don’t work as fast, they have a lot of incentive problems. Imagine it’s a large computer network of intelligent computers (...); they all communicate results between them extremely fast.” Brings me back to Cybernetics and Stafford Beer: "The purpose of a system is what it does."

Jacques Vincelette

70th anniversary if you count yearly intervals since 1955, but we make space for creative accounting in service of cultural hedgemoney.

Jacques Vincelette

On Wednesday, July 15, US Secretary of War rolled out gender affirming hormone therapy across troops. Resonates well with the US Secretary of States /National Security Advisor's call for the liberalization of Cuba.

AJ Biffl

Power and Progress is definitely on my reading list now! One really interesting thing in chapter 3 was Brockman's favorite sci-fi series (or at least the ones Hao relates, which might have been on purpose): Three Body Problem and the Foundation Series. Those both suffer from being primarily driven by smart-guy-big-idea-saves-the-day plots, and little else. They're both borderline misanthropic in their presentation. There are minimal interpersonal relationships, lone-wolf or fearless-leader protagonists, and their bright, shiny, good ideas save the day and bring the plot forward etc., and there's barely any character depth or philosophical depth. Early (timeline) Foundation books are literally centered around single guy and his massive throbbing idea which drives the rest of the series, and the rest of the series jumps around in time to various new protagonists with no connection to the older, they have a good idea, and then the plot moves on (and in fact the whole premise of the series is that these bright ideas that save the day are historical inevitability and that technology is always winning). The Three Body Problem trilogy also basically introduces a new protagonist every book out of nowhere, and there are almost no meaningful relationships between characters at all (the Netflix show (or at least season 1) was a major improvement in that it actually had real characters and had them know each other, instead of a new one popping up out of a hat as needed). It is also chock full of big-idea-saves-day, this-one-guy-is-so-important, etc. There's also a legitimately horrifying plot arc in the second book about a relationship with a major power imbalance (like, a truly colossal power imbalance it is unbelievable) that is literally never addressed or come to head or have any consequences. I should mention some qualifications because I actually do like and recommend both series, with those caveats. The Foundation books have some gems of characters, and there are some genuinely very COOL ideas at play in both series, but out of any two sci-fi series to reference, Brockman picked probably the two most technocracy-fetish and anti-human-community sci-fi books possible. Like I literally looked over my bookshelf and there's not a single one that fits that vibe more perfectly.

phil e the theyby

This is FASCINATING; thanks so much for sharing! It really speaks to how our connection to art (especially the art that impacts us the most) ties so deeply into our values.

Dr. Fatima

having lived in the bay when the pandemic started, this was too real :/

Jean Marrero

The novel aspect of chap 1 and 2, i felt kinda disappeared at 4. I understood 3 to be more of "we can use this for good!" and I cab resonant with that idealizing but I feel this is the problem of "STEM only". A good understanding of psychology or how power influences or even politics should inform on benchmarks if the goal is to do good. But like usual, absolute power corrupts absolutely and this whole AI empire thing is no different. We see it with data centers and the greed to fuck with neighbors. Was late and did not go in depth this week but i hope you show up on time lol.

Jacques Vincelette

Here in Deep River we played zombie apocalypse in the woods. My daughter had a rainbow ray to turn me into a loving zombie. Simpler games, looking back.

Mx. Alba

Well what they're calling intelligence, I call an impressive party trick. Like that guy who just memorized the dictionary was able to place correct words on the Scrabble board, these LLMs are just word prediction on steroids.

Anthony Perez Eisenbarth

I have a lot of catching to do with a lot of projects and this book club. (My personal life has been a mess lately what with burying my late grandmother and my dog. But no rest for the wicked.) Hao’s book is heavy on the interpersonal aspect of the AI industry, much more so than AI Snake Oil. Which is kind of cool, I guess, some people prefer that kind of thing. A lot of other points she makes are in the same realm as AI Snake Oil. The Brockman material in "Nerve Center" is a good example of this. Brockman cannot name a historical case of a technology's benefits being successfully redistributed and instead free-associates through fire, cars, and utilities before landing nowhere. But the confidence that "the trajectory is already there," that OpenAI's task is merely to take hold of the reins the initial conditions of something already in motion, does not appear to rest on anyone having actually worked out where that trajectory leads. Hao reads the moment mostly through Brockman's own psychology, his need for recognition, his tunnel vision, and I think that gives Brockman too much credit. Chapter 3 gives us real insight into Brockman’s psychology: the teenage Turing obsession, the need for recognition, the workaholism, the marriage staged in front of the lab’s logo. All of that reads as though it explains something about OpenAI's trajectory. I really appreciated the cotton gin analogy that Hao presents at the start of Chapter 4, and I think it's the right example to open on. The labor intensification with the cotton gin machine and slavery is happening analogously with generative AI and wage labor now. A UC Berkeley Haas study published this year followed forty workers at a two-hundred-person tech company for eight months and found that generative AI didn't free up anyone's time. Workers took on broader task loads than before, work started going into lunch, evenings, and people ended up juggling more threads of work at once, mostly without anyone telling them to. Separately, a Workday survey of over three thousand employees found that even though most workers save a few hours a week using AI tools, close to 40 percent of that saved time gets eaten right back up fixing the tool's own mistakes. One other thing that I like is how direct Hao is that deep learning did not become so prevalent because the underlying scientific dispute (whether neural networks can actually store information and reason) got settled. She says that dispute is still open. It won because it was cheap to commercialize, needed no domain expertise, and rewarded whoever already held the most data. Subsequently, funding and careers were reorganized around where the money went, a kind of enshittification of academia. “Even as the need for alternatives has grown ever more urgent, the diversity of ideas in AI research has only collapsed further. Students are dropping out of their PhDs to go straight to industry. Senior academics are facing a crisis of how to continue pushing the bounds of the field without joining a deep-pocketed company. More and more researchers have turned their focus not just to deep learning but to large language models exclusively. The major AI powers are no longer setting the agenda so much as bending an entire discipline to their will.” Fatima’s comment about machine learning in physics is very interesting. I kept thinking about how closely it maps onto the history of economics as a discipline. Economics has its own version of this capture. "Mainstream" economics, of which Daron Acemoglu is an exemplar of, now functions as a synonym for what used to be called neoclassical or orthodox, while the older heterodox traditions (institutionalist, post-Keynesian, Marxian) have been pushed to the margins for most of a century. The irony is that some of the field's flagship journals were themselves founded within that tradition. The Review of Economic Studies is the clearest case, launched in the 1930s by a group that included Paul Sweezy and Abba Lerner, both working well within a socialist framework, Sweezy himself an outright Marxist. The American Economic Association's founders, Richard Ely chief among them, came out of the German historical school and had genuine sympathy for labor and reform politics. Over the following decades that lineage was steadily displaced, until the journals descended from it became reliable outlets for findings than essentially shill free market conclusions. Not to mention the reverence for complex mathematical models for the sake doing “real” analysis but I digress. There is real irony in Acemoglu being the one supplying the "narrow elite" framework in his book. Power and Progress argues that technology’s direction gets set by whoever has the resources to do the rallying, and that the rest of us inherit their "blind spots" in the technology's wake. That is a precise description of what happened to his own discipline. The postwar consolidation of economics around neoclassical orthodoxy was not the natural maturing of a science any more than OpenAI's scaling doctrine is for the next great technology.

Piochi

I love your insights! I think that the issue is AI is the same issue with anything that is developed into a capitalist system. The goal is never "What do people need" but "How can this be profitable or at the very least, sustain itself within our system", the only need these tech bros are fufilling is their own, and then they gaslight themselves and the world to think it will be good actually. (I kinda think that so far, they truly believe they are good people) That's a problem I've observed while studying in design, we all have this reflex to start with the solution and then find the problem it can solve. But the part that is important (I personally loved that part) was to understand the users' need, and their reality. It's not a really complicated step, but it takes the one ressource we never want to invest, which is time. You need to both observe the user and ask them question to get at the core of what exactly is the need (And I don't believe even people always know what their own needs are, it takes time). And sometimes, the solutions isn't a product, it's a service, is a ritual, it's the extraction of something etc. A good design is one that is used and is useful, now we create stuff that are addictive because we don't take the time to create good designs. I do think there's value to explore technologies on their own and play with them even if there's isnt a need because it can open possibilities we haven't considered, but that's what research is for and I think it shouldn't be a business but a common.