Home Creators Posts Import Register Favorites Logout
haven't archived this post yet. have a subscription? use the importer!

Content

Same video as yesterday's post (fixed up a bit), but now it is public, so feel free to help share it and all that.   Thanks!

Files

Previews only

Gradient descent, how neural networks learn | Deep learning, part 2

Subscribe for more (part 3 will be on backprop): http://3b1b.co/subscribe Thanks to everybody supporting on Patreon. https://www.patreon.com/3blue1brown http://3b1b.co/nn2-thanks For any early stage ML startup founders, Amplify Partners would love to hear from you via 3blue1brown@amplifypartners.com To learn more, I highly recommend the book by Michael Nielsen http://neuralnetworksanddeeplearning.com/ The book walks through the code behind the example in these videos, which you can find here: https://goo.gl/kCJDJm Also check out Chris Olah's blog: http://colah.github.io/ His post on Neural networks and topology is particular beautiful, but honestly all of the stuff there is great. And if you like that, you'll *love* the publications at distill: https://distill.pub/ For more videos, Welch Labs also has some great series on machine learning: https://youtu.be/i8D90DkCLhI https://youtu.be/bxe2T-V8XRs "But I've already voraciously consumed Nielsen's, Olah's and Welch's works", I hear you say. Well well, look at you then. That being the case, I might recommend that you continue on with the book "Deep Learning" by Goodfellow, Bengio, and Courville. Thanks to Lisha Li (@lishali88) for her contributions at the end, and for letting me pick her brain so much about the material. Here are the articles she referenced at the end: https://arxiv.org/abs/1611.03530 https://arxiv.org/abs/1706.05394 https://arxiv.org/abs/1412.0233 Music by Vincent Rubinetti: https://soundcloud.com/vincerubinetti/ ------------------ 3blue1brown is a channel about animating math, in all senses of the word animate. And you know the drill with YouTube, if you want to stay posted on new videos, subscribe, and click the bell to receive notifications (if you're into that). If you are new to this channel and want to see more, a good place to start is this playlist: http://3b1b.co/recommended Various social media stuffs: Website: https://www.3blue1brown.com Twitter: https://twitter.com/3Blue1Brown Patreon: https://patreon.com/3blue1brown Facebook: https://www.facebook.com/3blue1brown Reddit: https://www.reddit.com/r/3Blue1Brown

Comments

Anonymous

Just amazing videos... Thank you!

Gavin Olson

My first thought during the discussion of what happens when a random input is put into the system was "Why not add an 11th 'None of the Above' output state". The part about what happens when you train a network with mislabeled data made a good counterpoint, though: the network would be identifying the specific "not a digit" test data, not the more abstract concept of "not a digit" itself.

Alexey Badalov

Wish this and the previous NN video were 60fps.

Phil Moyer

Gavin - I'm currently working on a related problem, namely, application of ANNs to intrusion detection. In this case, the NN recognizes "normal" patterns, but you want it to alert on "non-normal" patterns. Thus, it has to generate a confidence in it's categorization and generate metadata when the confidence is sufficently low.

Jesse Thompson

That might be one valid way to try to express a certain output belief, but I'd imagine that just turning off all the output neurons ought to express the same belief just as effectively, right?

Anonymous

Please consider doing a video on transforms like Laplace and Fourier.

Benjamin BairMoshe

I wanted to say, that I did something very similar at work intuitively with a much simpler problem. Maybe you could use this as an example in another video? We have a virtual environment that allows the user to chose a background color giving us 3 values (only 3 dimensions) for RGB. Based on the user's selection we wanted to automatically decide if the text on the page should be either black or white (1 output). So we found various values where the preference changed from black to white in order to develop a formula. We developed a cost function and adjusted our variables manually. In effect I got a very hands on approach at manually developing a very simple neural network.

Anonymous

I love your videos, guys! And it inspire me.... I’d like to work more my math... But now I am reviewing a Scrip theater play to a Theater in São Paulo... 😊 It’s difficult to conciliate both... But since I watch your video I could see that is possible to do both 🙂👍

Anonymous

HI! Fantastic videos - thank you so much! I am currently diving into data science and your videos really help me to understand the math behind. I am not a natural within the math. field. :) When will the next video about neural networks come out? Best regards, N.

Anonymous

Any other optimization methods used? Simplex? Data dependent?