Home Creators Posts Import Register Favorites Logout
haven't archived this post yet. have a subscription? use the importer!

Content

Hey folks, just making a Patreon post for the "Backpropagation calculus" video, based on the outcome of this poll.

If you'd prefer this not to be one of the videos you support, not a worry, just send me a direct message on Patreon and there's a notion of a "refund" for specific paid posts.

Files

Previews only

Backpropagation calculus | Appendix to deep learning chapter 3

This one is a bit more symbol heavy, and that's actually the point. The goal here is to represent in somewhat more formal terms the intuition for how backpropagation works in part 3 of the series, hopefully providing some connection between that video and other texts/code that you come across later. Find the full playlist at http://3b1b.co/neural-networks Thanks to everyone supporting on Patreon. http://3b1b.co/nn3-thanks http://3b1b.co/support For more on backpropagation: http://neuralnetworksanddeeplearning.com/chap2.html https://github.com/mnielsen/neural-networks-and-deep-learning http://colah.github.io/posts/2015-08-Backprop/ Music by Vincent Rubinetti: https://soundcloud.com/vincerubinetti/ ------------------ 3blue1brown is a channel about animating math, in all senses of the word animate. And you know the drill with YouTube, if you want to stay posted on new videos, subscribe, and click the bell to receive notifications (if you're into that): http://3b1b.co/subscribe If you are new to this channel and want to see more, a good place to start is this playlist: http://3b1b.co/recommended Various social media stuffs: Website: https://www.3blue1brown.com Twitter: https://twitter.com/3Blue1Brown Patreon: https://patreon.com/3blue1brown Facebook: https://www.facebook.com/3blue1brown Reddit: https://www.reddit.com/r/3Blue1Brown

Comments

Anonymous

Excellent!!! It's really helpful!

Anonymous

You're talking completely over my head on this one so I think your beginning disclaimer should be: "If you don't have a very good handle on calculus via my calculus track, then this video is going to annoy you". My eyes went full-glazed over at the 5 minute 30 mark because none of the equations are clicking. The one part of the whole video where I felt I got a small foothold on understanding is at 7:25 where you're talking about the sum of squared differences between "alj" and "yj". I have a little bit of experience with this because I thoroughly understand RMSE, as described very clearly here: <a href="https://stackoverflow.com/questions/17197492/root-mean-square-error-in-python" rel="nofollow noopener" target="_blank">https://stackoverflow.com/questions/17197492/root-mean-square-error-in-python</a> The minus gives you the distance-cost, the squared throws it all into the positive, the sum totals up all the costs. I think you're doing the listeners a dis-service when you're reading the equations verbatim, like when you say "Sum" of "ylj" and "yj" squared. We can read the equation, because you make it pop out as you say it. I'd prefer you say: "The sum of "What the network produced" minus "The correct targets". Rather than reading out the letters you've selected, give them a little mnemonic so we know what kind of data the "YLJ" holds. I think a bottom up approach might be better, where you begun at the cost function, because that's where the first concrete numbers are found. Start at the beginning, which is the cost, then percolate out towards what we need to do, plot a tangent to the curve and roll downhill, fiddling the weight. It's all still pretty fuzzy. It hasn't clicked yet. Still, good job, I like it, but a lot of listeners are going to scratch their heads and say: "Sounds real cool, but I don't get it".

Anonymous

you made me, a 12 year old kid, understand calculus. here, one month later, i'm teaching myself some multivariable calculus at 13. thanks.

Anonymous

Thank you so much for your wonderful video. I wonder if you are also interested in explaining back propagation in tensor notation, which I would like to get familiar with. (Actually, I wish I could learn tensor in general, but could not find a good material to study. So, if you could just let me know some good textbooks, it would also be much appreciated. I study some graduate level statistics and machine learning classes, but I did not have good undergrad math prerequisites.)

3blue1brown

I highly recommend Michael Nielsen's (free) book on the topic. You'll find that it aligns with these videos, and he talks more about the notation to make this algorithm more compact, and how it looks when you implement it yourself.

Anonymous

Thanks for the reference