Home Creators Posts Import Register Favorites Logout

Content

Hi All,

I hope you're having a great start to 2022. I took the month of January off to have a vacation with my family in Australia. We've enjoyed it so much here that we're staying a little longer. But I am getting back to work with the first video for this year being the second part of the analog computing video (linked is a low res draft). 

I would love to get your thoughts on this video. I'm sure some of you are better versed in neural networks and computing than I am so please let me know any errors you spot. All suggestions, comments, and questions are welcome - as are title/thumbnail ideas. I will do my best to incorporate your feedback before launch. (there is a missing animation or two so you may have to use your imagination for now).

Once again I really appreciate all of your support and helpful suggestions.

All the best,

Derek

Files

Analog Part 2

Comments

Pavlo Pravdiukov

Can't call the war a good start of 2022, but I'm sure you know about it. Great video! Using analog computing for artificial NNs seems logical, my friend worked on analog "real" NNs, that more closely mimic real neurons. I'm sure many people are unaware how different ANN and real neurons. Might be a good theme for the next episode ;)

Gerald Cecil

Informative and fun to watch, as usual. To fix in edit: your "brown board" comparison of analog vs digital multiplication was sketched with too low contrast to see clearly.

Kimberly Green

There is a famous work of art known as "The Great Wave Off Kanagawa". And there's a vintage poster that riffs off of it, converting the right side of the tidal wave into a line grid version. (I own a copy if it.) It is meant to symbolize the shift from an analog world to a digital world. A fun thumbnail might be a further riff on that theme. An analog-to digital-back to analog image of some kind. Maybe the face of Derek.

Didier de Gery

Very interesting and informative. It gives the impression that this technology may not have gotten out of the lab. In the late 70s, as a student, I visited GE nuclear in San Jose. They used neural networks to model power plants. There were inputs and outputs they could measure and internal processes that they could not. By tweeking the analog components to match the parts of the process they could measure, they were able to estimate in real time what was going on in the reactor. Adding something like this may make your video too long or be a distraction to your main premiss, but it can also show that these technologies have practical applications even when they are not at their peek. Thanks for making great videos

katgod

Great video as usual. 2 comments, what Gerald Cecil said and a little info. on the analog computer at the beginning assuming it can be purchased.

Andrew Alvarez

Awesome video! I just have a couple of notes: 9:00 - Minor point, but I was a little confused by what you said about showing the images as 60x64 pixels, since the images themselves are still displayed as 30x32 pixels and you’re just showing 4 corresponding images in a 2x2 grid. 9:45 - You should name-drop 3blue1brown’s backprop vids here too! Though admittedly he’s more mainstream and you probably wanted to give recognition to a strictly more underrated channel, which I’d respect. 16:40 - animation error, though you’re probably already aware 17:47 - The ADC results at the bottom don’t make sense, at least not as a simple matrix multiplication. The sums are [8.1 7.9 9.8] and the ADCs show [341 473 291]. Even if you’re multiplying each result by some ADC conversion factor, each ADC factor would have to be different for the results shown. Is there an error in the math, or is there some nonlinearity in the conversion that’s not mentioned/shown? 20:10 - It would be interesting if you name-dropped Neuromorphic Computing here, since that’s the design of computers built specifically to mimic real neurons. I know IBM, Intel, HP and Stanford have all done research on neuromorphic hardware, and at least Stanford’s is described as “Mixed Analog-Digital”. HP’s research talks about “memristors”, which also seem pretty cool, and analog in nature.

Chris Stone

-The "brown board", is too hard to see. -At one point you mention Backpropagation, might be nice to just mention that there are other methods of learning -- though I agree with not describing them here. -You mention the first AI Winter without mentioning the Second directly, only indirectly. -The activation function discussed in the video is all/mostly linear. Non-linear are getting a lot of research right now. Human neurons are non-linear, not that emulation is necessarily the best progression.

Edward Jones

As always you communicate complex ideas in way that is clear to the general viewer while not oversimplifying. I've just got a couple of observations that might be interesting. In the summary saying that the brain is fully analogue is perhaps a little misleading. The action potentials that biological neurons use to communicate are effectively digital events, there's either action potential (spike) or there isn't. This links in to what was said at 19:07 where the fellow from Mythic explains how their chips use analogue in the compute but digital representations to communicate values in order to avoid signal degradation when running networks with many layers. Other analogue neuromorphic (brain-inspired) chips (see BrainScaleS, NeuroGrid) use this mixed signal (analogue *and* digital) approach too, paying the energy and hardware overhead cost of digital only when it is "worth it", usually when values are being communicated, but doing local computations in analogue to save energy. Though out of the scope of this video, there's a number of different neuromorphic hardware projects, see SpiNNaker and Loihi, that are all digital exploit the flexibility and reprogrammability of low-power digital processing elements with fast communication between these elements to, amongst other things, try and work out how much detail we need in computational neuroscience models to be able to model the brain well and to do useful things; what is the best mapping between a neuron and transistor/groups of transistors?

Arian Prabowo

4:47, double check if the threshold is really called a "bias". Or maybe it is only called "bias" by some people. I think of it as an activation function in form of a step function. 12:03, top-5 accuracy is a more common metric than top-5 error rate. 14:50, some shaky ground right here. What you said is right, but for the wrong reasons. In a sense, we can trade away some precisions without losing much performance for example: https://pytorch.org/blog/accelerating-training-on-nvidia-gpus-with-pytorch-automatic-mixed-precision/ However, reproducibility is still very important, especially in the face of fairness. Imagine someone failing a credit check / going to prison just because of literally rounding errors. The 96% vs 98% is also kinda important, because from the Bayesian perspective, the goal is not simply to get low error, but to accurately model the underlying distribution as well as uncertainty in the data. The term parameters captures both the weights, as well as the bias, and a bunch of other stuff beyond weight and biases that the network has to learn. Maybe it is too confusing and unfamiliar for general audience, but that's the right term. 20:08, This is a very shaky ground, I would like to se some reference. First of all, ANN is an extremely rough analogy to how biological brain work. For instance, biological brain don't learn via backprop. Secondly, both digital and analog are modelling the exact same process. One is not a better model than the other. An equally good / bad digital / analog computer will be comparable in their performance. Digital is limited by the "size", analog is limited by the signal to noise ratio, they are different tools, but they are doing the exact same thing. The fact that our brain is analog is completely irrelevant, we just need a digital computer with big enough memory such that the precision is higher than the signal to noise ratio in the biological brain. But most importantly, action potential is digital. Yes the information is encoded in frequency, instead of the strength of the action potential, and frequency is analog-ish, but nothing is stopping digital computer from modelling that. In fact, we have exactly that right now, it is called spiking neural network https://en.wikipedia.org/wiki/Spiking_neural_network . If you have any more questions about AI / ANN / deep learning, don't hesitate to contact me, either about this video or in general.

Riaan Jonker

The only comment I have is about the blackout at 16:37 to 17:17. As you continued explaining how the electrons flow through the cells, I found it confusing, as I lost track of the conversation (My mind drifted away from listening, to wondering why the eye's ain't seeing something) until I rolled back to listen again.

Liran Katzir

Slight inaccuracy in referencing the bias term. While in the video the formula presents $weights * activation > bias$, in the Machine learning literature, $weights * activation + bias > 0$.

Shawn Van Ness

Nitpick -- nobody cares if the analog board can run Word. The question is, can it run Mario?

Shawn Van Ness

Seriously, it's a pretty good introductory video.. I have a background in feed-forward NNs back in the 90s, so most of the history and context, rings true to me. I wasn't aware of the work being done to build "analog chips". That was most fascinating. You mentioned (correctly) that most of the energy cost (and clock time) goes into training... washing through thousands/millions of training-set images, over and over.. the actual real-time operation of classifying 1 image, is fairly fast. (Again you correctly highlight memory bus latency as being more important than cpu "speed".) I'd be curious to see some hard numbers, comparing both latency and energy consumption, of the analog chips vs a modern multi-core CPU or GPU..

Dustin Brothers

That’s a nice scope! Holy smokes!! 🤣👍🏼

John Cox

My guess is that this part was still in production, but a "Still in production" static image would have been nice.

John Cox

First, you've done a lot of research. This much is clear. Well done! Ever heard of photonics (light-based circuitry)? This will be the holy grail of non-quantum processor technology, I think. https://cacm.acm.org/magazines/2021/9/255027-photonic-processors-light-the-way/fulltext https://lighton.ai/photonic-computing-for-ai/ Feedback: I can see that this is an optimistic video about the possibilities of analog computers for AI, but I'm going to try to put the brakes on the enthusiasm for a bit. Sorry! For the sake of thoroughness, please also consider what AI is _not_. Despite the visions of very optimistic futurists hoping that sheer willpower will bring true artificial intelligence into existence, current AI technology bears little resemblance to human intelligence unless we set a very low bar for what "intelligence" means. Current AI models bear a much closer resemblance to gut reactions, to "instinct" (which is why I personally prefer to think of current "AI" as "Artificial Instinct"), but this hasn't stopped imaginations from running wild with the term. The problem got bad enough that AI researchers had to create a new term, "general purpose AI", to differentiate it from the distorted original term. Current AI models are incapable of predicting outcome of an action (other than that they were trained in predicting classification probability or "correct"/"not correct", depending on the problem). There is currently no ability to train them on cultural consequences of their decisions and loop through different possibilities to arrive at the most favorable, as many people apparently think they do. There is also no "I don't know" (not even an option). There is no ability to consider input that doesn't match the expected format (remember how facebook face tagging was always a fixed size box? this is why; as a math function the input for a given model must always be the exact same size). EDIT: Made a mistake about variable input size. Humans can't do this either. If humans have a fixed number of neurons for their senses, then a fixed input size would be a perfect model. I was thinking of "variable input size", but it isn't variable at all. It's rather a vast input array + ability to efficiently learn to ignore select inputs. (If I may be a nerd) Perhaps we should consider using Mass Effect's term "Virtual Intelligence" instead of AI, as if to indicate the _appearance_ of intelligence ("virtual") without actually being truly so. @4:48 "But if it's less than that, the neuron doesn't fire." Minor confusion: The timing of the voiceover doesn't line up with the animation. When you say, "doesn't fire", that's about when the animation _does_ fire the neuron. @11:00 - later The discussion about the vision learning problem reminds me of the major handicap of the current mathematical models: (1) inability to precisely carve an object out of the background along lines of contrast (as Photoshop seems to be able to do with ease with in-house computations), and (2) inability to handle inputs of varying size. Without the ability to consider objects on their own, isolated from the background, it will continue to take absurd quantities of data just to screen out the noise. Consider how a child is able to learn what a cow, a dog, a car, and a tree look like just from some simple cartoons in a child's book, and then they are able to see one in person and make the connection. Current computer vision models cannot come anywhere remotely close to this level of training efficiency. You alluded to this problem in the video: in order to become better with current technology, the amount of data is enormous. It has become almost stupidly unwieldy. The next major advancement in AI (my opinion) won't be throwing more data and computing power at it, but by taking steps towards human levels of efficiency (however that works). @14:50 "Best of all, neural networks don't need the precision of digital computers..." This is good, but indirect, tie-in to the "nonrepeatable" downside of analog computers mentioned earlier. Suggest making it blatantly direct, such as by mentioning, "remember how analog computers have trouble with repeatability?" If you read this far, thank you! I like how this series is shaping up.

Arian Prabowo

> There is also no "I don't know" (not even an option). There is no ability to consider input that doesn't match the expected format (remember how facebook face tagging was always a fixed size box? this is why; as a math function the input for a given model must always be the exact same size). How would you characterize anomaly detection?

Dan Babitch

There are a couple very high level problems with AI in general. One, there is no generally agreed definition of "intelligence." So what is it exactly we are trying to do? Is this like art, I can't define it but I know it when I see it? :-)

Dan Babitch

(Sorry, I hit RETURN by mistake.) Two, Note that monster mega size training sets are pretty common now. So how's it that a 2 year old child identify cats with 100% accuracy after being presented with exactly ONE example? It seems we understand some things about how our brains work, but not other things.

Charles Twardy

I’ve been waiting for this since seeing party I! Nicely done and the NN stuff is pretty solid. I do agree w/a previous commenter that 3B1B deserves a shout there —exceptionally well done. I also agree with whoever wrote that the video seems too quick to gloss the brain as analog at the end, after earlier telling us about action potentials. Perhaps say a little more about “digital spike but also…. “. Power consumption is really the key story here yes? We can always speed the digital by sufficient power. And not sure how to cover reproducibility here: some kinds of reproducibility are really important — and AI has (some) replication issues. But computational reproducibility is a very limited kind of replication and usually we want something more generalizable. So while analog edge cases may flip flop, that’s not necessarily worse than digital versions consistently falling 1% above a threshold of illusory precision. Finally the optimistic ending seems too close to the naïve highlighted news article near the start. Or maybe I’m just burned by AI winters. Side story for another time: physics-savvy biologists studying neural motor circuits in the 70s built analog computers to simulate them. Later, digital implementations of the differential equations became more convenient.

John Cox

I made a mistake. In hindsight, writing my comment at 9pm without peer review caused me to overlook a perspective. If humans have a fixed number of neurons for their senses, then a fixed input size would be a perfect model. I was thinking of "variable input size", but this was actually a vast input array + ability to efficiently learn to ignore select inputs. To answer your question though, I'll quote people more familiar with the process: "Anomaly detection is the process of finding outlier values in a series of data. That process assumes you have data that falls within a certain understood range (based on historical data, for example), and that occasional values outside that range happen fairly infrequently." https://c3.ai/glossary/artificial-intelligence/anomaly-detection/

Selina McMahon

Sorry - it's a while since I did elementary electronics. To multiply two numbers: You are combining I(1) + (I(2). When the combined current is passed through a resistor, I'm losing how V=((I(1)+I(2))R is a multiplication of I(1) plus (i(2). Surely it's still an addition? As I say, it's a long time since I did rudimentary electronics and I'm a tad old, rusty, falling apart at the seams...etc.

65square.com

We’re using facial analysis for logging people into our new social media app 65square. We published a free app that guesses age/gender and it’s accurate right now with adults to about 3-4 years. Our dataset was very small. As a nonprofit we’re committed to this technology because it offers a good balance between protecting people (especially kids), verification and balancing privacy interests. Great to see you working on this story Derek!

65square.com

Mythic AI- great resource for us to look into! Thanks Title suggestion: Your Brain is This? Thumbnail: Analog chip held someway to show it’s scale against a cartoon (bloody / gory) brain/spinal cord that’s been ripped right from a corpse that’s fallen over in the background

Eran W.

There are many deep learning videos out there, I like how you differentiate yours by focusing on the analog solution (which is new to me). The most important part of the video is the example 05:00 to 07:00, this even with prior knowledge i had to review twice... then at around 07:00 you mention the fact the the algo. will "converge as long as you can map the dots of the 2 categories into groups separated by a straight line", this sentence is only clear to me because of prior AI knowledge, maybe you would like to create an animation for this sentence...

Eran W.

oops too late :)

65square.com

We paid you at your top tier for several months and you don’t add us in the credits? That’s called a scam bud

Sheldon Carpenter

Very interesting video. I really like the way you take very complex ideas and make them far more understandable. Neural networks are still fairly hard to comprehend, but you do a good job of creating a pathway to understanding.Still, this will be a little tougher video for your more casual crowd. I wonder if the wave in self-driving cars will take advantage of this (if they haven't already). It seems like a no-brainer. A very enjoyable and informative video.

Chris Mullin

Good draft video. The jump to self driving seemed very abrupt. Drawing on cardboard didn't work (presumably that will be updated with a white board?). Suggested title: have digital computers met their match?

Chris Mullin

Also, it wasn't clear how digitizing the signals between layers of processing would get around the noise in the processing itself.