Stanford CS149: Parallel Computing
Self-contained study pages, one per lecture of Stanford's CS149 (Fall 2023, taught by Kayvon Fatahalian and Kunle Olukotun). Each follows its lecture's argument, explains every concept from the ground up, and adds diagrams, worked examples and explorers you can manipulate.
These pages complement the main course notes. CS149 looks at hardware from a software engineer's point of view: what each architectural feature demands from your program. Where a lecture relies on background from the course notes, the page recaps it and links to the relevant unit.
Why parallelism? Why efficiency?
- Speedup, efficiency, and the three classroom demos
- What a program and a processor are
- ILP and superscalar execution, and their limits
- The power wall and the end of the free lunch
- Parallel and specialized hardware everywhere
- Memory: latency, stalls, caches, the hierarchy
A modern multi-core processor
- Review: instructions, ILP, superscalar, caches
- Idea 1: multi-core, and expressing parallelism
- Idea 2: SIMD, divergence, explicit vs implicit
- Memory stalls, caches, prefetching
- Idea 3: hardware multithreading and latency hiding
- Real chips: Kaby Lake core, V100 SM
- Every combination: what saturates the chip?
Latency, bandwidth, and the SPMD abstraction
- Latency vs throughput: highway, laundry, pipes
- Memory bandwidth and Little's law
- Bandwidth-bound execution, and why bandwidth is the critical resource
- Abstraction vs implementation
- ISPC: gangs, programIndex, uniform, foreach
- Interleaved vs blocked, undefined programs, reductions
Not added yet
- Parallel programming basics, work distribution and scheduling, …
Stan-CS149/ and ask for a companion page.GPU architecture and CUDA programming
- From 3D graphics to GPGPU and compute mode
- CUDA: grids, blocks, threads; host and device
- Memory spaces, shared memory and __syncthreads
- Inside a V100 SM: sub-cores, warps, SIMT
- The thread-block scheduler, step by step
- Scheduling rules, deadlock, persistent threads
Not added yet
- Data-parallel thinking, distributed computing, …
Stan-CS149/ and ask for a companion page.Efficiently evaluating DNNs
- Neurons, fully connected and convolution layers
- Cheaper topologies: MobileNet's depthwise separable convolutions
- Convolution as GEMM: im2col and implicit GEMM
- Blocking matrix multiply for the memory hierarchy
- Winograd, operator fusion, fused attention
- Tensor cores and DNN accelerators
CS149 materials are © Stanford University and the course instructors. These companion pages are independent study notes: the explanations, diagrams and interactive models are original, and they reference the lectures' structure and examples for learning purposes. See the course site for the official slides and videos.