From a tiny computer to modern CPUs and GPUs
A course built as a chain of problems. Start with a machine that can run one instruction at a time from sixteen bytes of memory. Then, one lesson at a time, add exactly the mechanism needed to get past the machine's latest limitation, and say what it costs.
The goal: look at an unfamiliar CPU or GPU block diagram and explain why each major part exists, trace how a program uses it, and predict what limits performance. You need basic programming experience. No electronics, assembly or architecture background is assumed, and nothing has to be installed: every explorer runs in these pages.
Every lesson answers the same questions
Each lesson opens with a four-part card, so you always know why you're learning something before you learn how it works:
Then comes an explanation with diagrams, a worked trace, an interactive explorer, Predict questions whose answers stay folded until you've committed to one, and a checkpoint to explain in your own words.
The course map
Build the smallest useful computer
- Bits and meaning
- Gates, muxes, adders, the ALU
- State and the clock
- A stored-program machine (NIB-8)
Registers, programs & execution contexts
- Register machines and RISC-V
- One ISA, different hardware
- Calls, stacks, traps, thread contexts
Make one CPU core faster
- Pipelining
- Hazards and forwarding
- Caches
- Virtual memory and the TLB
- Branch prediction
- Superscalar width
- Renaming and out-of-order
- Precise retirement, memory order
Add parallelism
- Multicore and cache coherence
- Memory ordering, synchronization
- Hardware multithreading (SMT)
- SIMD, vectors and tails
Understand GPUs
- SIMT and the programming model
- Occupancy, scheduling, divergence
- GPU memory traffic
- Tiling and synchronization
- Roofline and measurement
Connect the modern system
- Matrix hardware and number formats
- Heterogeneous SoCs and shared memory
- Packaging, HBM, NUMA
- Multiple GPUs and collectives
Read real architectures
- Case studies of current CPUs, GPUs and accelerators
- Capstone: explain performance from first principles
The chain so far
Units 1–4 read as one continuous argument, where each mechanism answers the previous machine's bottleneck:
Companion: Stanford CS149 lectures
Self-contained study pages built from Stanford's Parallel Computing course, one per lecture. They look at the same hardware from a programmer's angle: what each feature demands from your code.
Why parallelism? Why efficiency?
- Speedup, communication and load balance
- ILP limits and the power wall
- Memory latency, stalls and caches
A modern multi-core processor
- Multi-core, SIMD and hardware multithreading
- Divergence, latency hiding, throughput
- Kaby Lake core and V100 SM, with peak-rate arithmetic
Latency, bandwidth, and SPMD
- Latency vs throughput, Little's law
- Bandwidth-bound execution
- ISPC: abstraction vs implementation
GPU architecture and CUDA
- CUDA's thread hierarchy and memory model
- V100 SM, warps and SIMT
- Block scheduling and deadlock
Efficiently evaluating DNNs
- Convolution layers and cheaper topologies
- im2col, implicit GEMM and blocking
- Fusion, fused attention, tensor cores
Reference
A standalone map that ties the units and lectures together.
How to study with these notes
- Treat lesson numbers as dependencies, not deadlines. Spend one session reading and a separate one predicting and experimenting. Pipelining hazards, out-of-order execution, and (later) memory ordering and GPU tiling deserve more than one pass.
- Predict before you click. Every explorer is more useful if you've written down what you expect first. When you're wrong, that's the lesson.
- Keep a one-paragraph notebook entry per lesson:
The old limitation was . We added . The new state is . On the worked example, changes. The cost is . The idea fails to help when . I can test that by .
Move on when you can explain the mechanism without its diagram and solve a changed version of the example. If you can name the parts but can't trace the behaviour, do one more small trace before adding complexity.
Ground rules these notes follow
| Rule | Why |
|---|---|
| Separate the instruction contract from any one implementation | Most of this course is hidden machinery that must never change a program's results. |
| Label teaching machines, invented timings and ideal bounds | NIB-8, the 5-stage model and the tiny cache are for tracing, not specifications of real chips. |
| Keep latency, throughput, bandwidth, capacity and energy apart | Most performance confusion comes from blurring two of them. |
| Always ask: whose state is it, is it private or shared, and when does it become visible? | This question runs from function calls to GPU shared memory. |
| Name the cost of every mechanism | Nothing in hardware is free. Area, power and complexity are always in the trade. |
Want the one-page, big-picture tour of every stage from gates to datacenter GPUs first? See the Gates to GPUs overview. These notes go slower and deeper.