Home
Creators
Search Recent Random
Posts
Search DMs Popular Hash Lookup Tags Random
Importer
Import FAQ
Register Favorites Keys Review DMs Logout
Community
Telegram ThePornDude
Home Creators Posts Import Register Favorites Logout

Computerphile Video: More about GPT-2 (Patreon)

Published:
2019-07-17 14:16:51
Imported:
2026-06
Tags:
haven't archived this post yet. have a subscription? use the importer!

Content

The latest in what's turning out to be a whole series of videos about GPT-2, because I guess I talked about it for ages. This one is about the various tasks they apply the model to in the paper, and how they managed to take a system that's designed to just generate text by continuing from a prompt, and tricked it into doing lots of different language tasks without any fine-tuning or modification.

https://www.youtube.com/watch?v=p-6F4rhRYLQ

Files

Previews only

More GPT-2, the 'writer' of Unicorn AI - Computerphile

More examples of how GPT-2 pays attention to things. Rob Miles https://www.facebook.com/computerphile https://twitter.com/computer_phile This video was filmed and edited by Sean Riley. Computer Science at the University of Nottingham: https://bit.ly/nottscomputer Computerphile is a sister project to Brady Haran's Numberphile. More at http://www.bradyharan.com

Comments

Richárd Nagyfi

what if we trained GPT-2 on scientific papers, primed it with question-answer pairs on a subject, asked an unanswered question and let GPT-2 generate more text?

robertskmiles
>>24919655

That would be a fun experiment. My guess is that it would give surface-level-plausible answers, perhaps even correct ones for very simple, commonly known questions. But my intuition is that what GPT-2 does is 'babble', it's the equivalent of a human just saying the first word that comes to mind. It doesn't have anything that could represent the relevant types of knowledge needed to actually answer scientific questions, so the answers to unknown questions would be either gibberish or 'the obvious first wrong guess'. Compare it with Tshitoyan's recent work on unsupervised learning from scientific papers, which uses word embeddings, which are actually able to represent some of the required types of concepts, like "A is to B as X is to Y".

Anonymous

Very well explained as usual, Robert! Thank you for a much needed clear voice on these often oversimplified topics. Your videos lift the overall level of public discourse on GPT-2 immensely. Here’s a GPT-2 completion of my above praise: “I also like having the option of just watching it without any commentary - this time I decided to check your other works as well. Robert Wiblin: Yeah, and this is my new favorite video on the topic. It really nails it. It explains everything so well. So yeah, thanks, and thanks for watching this one in particular, because if we get more views like these, we'll definitely be putting up some more of these and we'll try and get to some of the more difficult stuff as well. You know, this is a topic that's definitely pretty popular on YouTube in part because it is hard to explain without visuals.” :D

robertskmiles
>>24944630

Thank you! Wow that's amazing, I actually read this comment while on the way back from Robert Wiblin's house. It does sound like the kind of thing he might say, especially the bit about visuals, since his main output is blog posts and podcasts :)