Home
Creators
Search Recent Random
Posts
Search DMs Popular Hash Lookup Tags Random
Importer
Import FAQ
Register Favorites Keys Review DMs Logout
Community
Telegram ThePornDude
Home Creators Posts Import Register Favorites Logout

Decoding Academia 18: AI Edition (Audio) (Patreon)

Published:
2023-04-21 23:38:04
Edited:
2023-06-12 05:39:20
Imported:
2026-06
Tags:

Downloads

Some files exceed the archive size limit and weren't saved: 1 file. Please note these limits are in place to keep this site running long term, without costing a fortune. You can favorite the creator though, certain milestones increase the limit.

Content

*WARNING*

This is a severely waffley episode. Matt and I talk about the paper but we end up getting into the weeds in some fairly obscure and likely rather idiosyncratic debates.

This might sound like what we normally do but I am providing this warning for a reason. This is a loooong episode with a looooong discussion. Matt is very interested in AI and I have various opinions some of which are well thought out and others... less so.

If you are interested in the topic this might be your jam and this is useful preparatory work for the upcoming AI episode but otherwise you might not find this discussion particularly entertaining.

For those interested in the paper, it is a rather technical paper called 'Attention is all you need' (Vaswani et al., 2017) that Matt recommended because it is one of the breakthrough papers that helped to create LLMs. That said I think the paper will be a bit of a struggle for anyone not up to speed with the technical details of AI research. It was hard for me to follow at least. Nonetheless, it is attached to the video post anyone who is curious. The discussion ends up spiralling away from the paper in any case.

Audio version should be available above.

Comments

matthew parsons

Sounds perfect accompaniment to a morning of domestic chores

Kyle Wilson

Bring on the full waffle!

Barrett

I rather enjoyed that one, I'm not sure I now know what the transformer is (although I've heard about it being the key piece of architecture several times over the past month without figuring it out so not matt's fault). I appreciated the debate on the broader topic though because it remained grounded and focused on some of the important, but nuanced details. Which seems rare in the current public AI discourse, but maybe that's just the contrast between the wonky academic approach versus the typical riding the waves of dopamine of a novel topic into fantastical guruesque riffs on skynet (or recently, patriotAI). I found the discussion of agency and functional pragmatism really useful. I'm not convinced they aren't, but I wonder if Matt's AI experiments are as interesting as he seems to think (take that in the most charitable way possible though). The what is a photon/how is a photon or similarly the simpson's sign joke "Sneed's Feed & Seed (formerly Chuck's)" are examples, as I imagine the topology of both, even if it isn't the exact same nouns, are repeated a lot in the training material. e.g., "Nobody Asks "How is X?" has its own Know your meme page. Sneed's has several Simpson's fan/trivia pages explaining it and I have to imagine the Sneed tokens are pretty rare. I guess my point is i'd expect it to pick up grammatical constructs like the "everyone asks what nobody asks how is X" and notice that they occur with a wide range of X tokens, and it probably doesn't represent a whole lot of 'emergence'. That said, and I think similar points were made toward the end of the episode, if it already has captured enough natural language that it can explain the joke and there are only so many joke constructions, then maybe it does functionally "understand the joke" for all intents and purposes? thanks and looking forward to the AI episode!

Martin Unland Elorrieta

I understand Chris's intention, but saying that the only thing GPT is doing is calculating probabilities for the next word is just like saying "the only thing a computer is doing is calculating the intensity of each pixel on the display screen"...

brianshmrian

Thanks for picking this paper. I've tried to glance at it a few times ever since ChatGPT came out last year. While I'm a computer programmer, I've never worked on neural networks or really any AI so it doesn't make much sense to me. The best accessible explanation on the LLM transformer architecture that I've seen is from Stephen Wolfram: https://writings.stephenwolfram.com/2023/02/what-is-chatgpt-doing-and-why-does-it-work/ In terms of whether it really understands anything, I'm with Matt. Why does it matter? If it produces useful output then that's all that really matters. Well... unless it's somehow conscious and suffering like the rest of us.

Kirsten Greed

"How many letters are there in the alphabet?" Maybe I got this :-)

Kirsten Greed

I wish this link was public so I could post it into my online university machine learning course chat!

Kirsten Greed

The video link that is... Although, I wonder if the whole Decoding Academia series could be accessible via institutional logins.

Kirsten Greed

I emailed Monash's library cc decodingthegurus@gmail.com to see if they could figure something out.

Alexis

This was great — well worth signing up to the patreon for! Now, the bad news: sorry, Chris and Matt, I think you may need to call in… the philosophers of mind 💀💀💀 Not to say Chris and Matt’s perspectives are not fascinating; I just think so many of the deep issues here are unavoidably philosophical. As a mere surfer of the discourse, may I recommend this https://podcasts.apple.com/gb/podcast/space-time-mind/id847581558?i=1000570039158 And make a plea for either Alex Kiefer or the great Pete Mandik as a guest on DTG. I look forward to continuing my patronage; you guys are awesome.