Andrej Karpathy: Deep Dive into LLMs Like ChatGPT [video]

[vimgrinder]

So much respect for this guy. He is like Neo of the matrix, bridging the gap between humans and machines. I have so far learned the following for free from his repos/videos:

1. minGPT, nanoGPT (transformers)

2. NLP (make more series)

3. tokenizers (his youtube)

4. RNN (from his blog)

There are many domains which don't have a karpathy and we don't hear about them. So glad we have this guy to spread his intuitions on ML.

[DoingIsLearning]

There are also different styles of teaching and learning. Karpathy always like to start from first principles and increment the building blocks.

Whereas for example Jeremy Howard's style resonates a lot more with how I enjoy learning, very much a "let's build it" and then tinker around to gain intuition on how things inside the box are working.

I see the benefit in both approaches and perhaps Karpathy is more methodical and robust. But I just find Howard's top-down style a lot easier to stay motivated with when I am learning on my own time.

[bamboozled]

"Neo of the matrix", what great analogy! Made my day and gave me a good laugh, thanks.

He is for sure a cool guy.

[Yajirobe]

5. How to solve a Rubick’s cube

[levocardia]

I tell all my friends that Andrej was the best instructor I had in grad school, even though I didn't even go to Stanford--I just watched his CS321n videos on YouTube. Really thrilled that he's still making videos.

[ipsum2]

He's made more than 5 videos covering basically the same topic, of transformer architecture and training. Wonder whats different about this one?

[karpathy]

My YouTube videos fall into two tracks:

1. technical track (all the GPT repro series)

2. general audience track

For (2), I had a 1hr video from 1 year ago, but I didn't actually expect that video to be some kind of authoritative introduction to LLMs. The history is that I was invited to give an LLM talk (to general audience), prepared some random slides for a day, gave the talk, and then re-recorded the talk in my hotel room later in a single take, and that become the video. It was quite random and haphazard. So I wanted to loop back around more formally and do a more comprehensive intro to LLMs for general audience; Something I could for example give to my parents, or a friend who uses ChatGPT all the time and is interested in it, but doesn't have the technical background to go through my videos in (1). That's this video.

[ks2048]

From the description:

    I have one "Intro to LLMs" video already from ~year ago, but that is just a re-recording of a random talk, so I wanted to loop around and do a lot more comprehensive version.

I think he has videos on building GPT2 from scratch, but this seems more high-level.

[hustwindmaple1]

When he drops a vid, you don't ask questions. You watch first and then ask questions :)

[thomassmith65]

Among other things, this video is recent enough to discuss DeepSeek R1.

[sota_pop]

Really love his “let’s build” series - I end up picking up cool Python tricks along the process, even in addition to the higher level content.

[bicepjai]

I still remember how to back propagate using python lists which was part of CS231n project by @karpathy. Amazing thing is I did not goto Stanford.

[Dinux]

Thanks Andrej. I have a pretty good understanding of how LLMs work and how they are trained, but alot of my friends don't. These videos/talks give them 'some' idea.

[arvinsim]

It frustrates me that I can't focus on these long-form videos when they are likely much better than the soundbite sized video counterparts.

[fransje26]

Focus is cultivated.

If you want to start getting your focus back under control, give meditation a try. It's a gentle tool to will help you get you understand how your attention works, and, with training, will give you back the control you need.

[browningstreet]

Serious suggestion: stand while you watch the video (Youtube on a TV helps here), use a pomodoro timer to do it in bits.

I also find I can watch a good bit of a complicated/technical video while walking on a treadmill/stairmaster at the gym. Just enough noise of other people, and the determination to do 45-60 minutes on a fitness machine replaces the determination required to get through a video. Doing it 3 days in a row feeds the motivation cycle in me. After day 1 I'm itching to get back to the gym and get back to the video to do another "episode".

[apetrov]

you don't have to do it in one go, i usually do pen and paper, 45m-1h of video time (~2h wall time) and often redo the same exercise a couple of months later

[brianjking]

Karpathy is too good to us. These videos are so incredibly valuable, thank you for sharing.

[stuckkeys]

I sat through the entire thing...my cheeks fell asleep but well Worth it. Thak you Andrej!

[behnamoh]

I'm a simple man, I see a Karpathy video, I click, I watch, I enjoy. :)

[demarq]

I wish there were another way to distribute video. Content disappears from youtube eventually, for silly reasons.

I think this is important content, the more people know how ai works under the hood the more empowered society will be.

[IncreasePosts]

You can just make a torrent of the video. It will then survive as long as you/others are willing to seed it.

[m_ppp]

Do you think videos disappearing is the biggest problem with YouTube from a distribution perspective?

[layer8]

If accessible knowledge about how LLMs work ever disappears, it won’t be due to YouTube.

[Dinux]

YouTube is known for not deleting videos, and so far the never have (with some obvious exceptions)

[tmp111111]

Andrej, I like you much more now than when you were at Tesla. You have been adding real value to my life and many others. Thank you.

[orand]

What is the relevance of Tesla to liking someone?

[lewiscarson]

I wouldn’t be so harsh on him, Karpathy really knows what he’s at when it comes to this stuff. I was sold when I read his recipe for training neural nets.

[melodyogonna]

What is your problem with Tesla autopilot?