A modern multi-core processor

Start with ADD r0, r1: read two registers, add their values, and keep the answer in r0. Three kinds of hardware take part. Orange fetch/decode finds and interprets the instruction; the yellow execution unit computes; the slate execution context supplies and keeps the thread's registers and next-instruction address. The same colors recur in every diagram.

Who reads and writes what

For the add, fetch/decode receives the PC and instruction bits; the execution unit receives the decoded operation and register values; the context receives the result. The table follows those handoffs. In this simplified view, the context and memory retain state between instructions. Real cores also have temporary pipeline storage, which we will introduce after the toy example.

Start simple: a machine you can program

Now try a processor small enough to trace by hand. TINY-4 has four registers, four kinds of instruction, and no data memory. Each instruction occupies one byte. Within one clock, it moves through four conceptual phases:

• Fetch (fetch/decode): the PC selects one byte of instruction memory.
• Decode (fetch/decode, reading the context): the byte is cut into fields, and the registers it names are read.
• Execute (execution unit): the ALU adds, subtracts, passes a constant through, or tests a register for zero.
• Write back (execution context): at the clock's rising edge, the result register and the PC are updated together.

The first two bits are the opcode, which selects SET, ADD, SUB, or JNZ. The next two select one of the four registers; the final four encode a small constant, a source register with two unused bits, or a jump address. The registers start at zero and hold 8-bit values from 0 to 255; arithmetic wraps around, so 255 + 1 becomes 0. There is no HALT instruction: the simulator stops at an empty instruction-memory slot. Start with “add two numbers” and follow the PC, instruction bits, register inputs, and result through one clock.

The life cycle, phase by phase

TINY-4 has no data memory. To understand a load or store, add a path to a data cache. The next simulator breaks an instruction into seven conceptual steps. They are a guide to the data flow in this teaching core, not a rule that every real instruction takes seven clocks or executes those steps strictly one after another. An add skips memory access; a store skips register write back.

1. Fetch. The PC, held in the execution context, is the address of the next instruction. Fetch/decode sends it to instruction memory (in practice, the instruction cache) and gets back the instruction's bits.
2. Decode. The bits are split into fields and turned into control signals: which operation the ALU should perform, which registers supply its inputs, which register receives the result, and whether memory is involved.
3. Read operands. The register names from decode select entries in the context's register file, which supplies their current values to the ALU's inputs. The context is read, not changed.
4. Execute. The ALU performs the operation. For arithmetic, the output is the answer. For a load or store, the output is an address: base plus offset.
5. Memory (loads and stores). The address goes to the data cache. A load waits for the value; a store sends its value to memory. A miss can extend the wait over many clocks.
6. Write back (instructions with a register result). The ALU result, or a loaded value, is saved in the destination register. A store instead changes memory and has no register result.
7. Advance the PC. The PC takes the next instruction's address. It advances by four bytes in this fixed-width example, by one byte in TINY-4, or to a branch target when a branch is taken.

Use the next explorer to follow a load through all seven steps. The four-instruction program reads x[2] = 3, multiplies twice to make 9 and then 27, and stores y[2] = 27. Registers r1 and r2 begin at 2, the array index. Step first through the load; then use “Next instruction” to see how its result becomes the first multiply's input. Finally compare an L1 hit with a DRAM miss.

Staying in sync: what the clock does

The clock explains when a computed value becomes stored state. The ALU and much of fetch/decode are combinational logic: as their inputs change, their outputs can change during the clock period. Registers in the execution context are storage elements; they capture settled inputs at a clock edge. The clock period must give the relevant logic enough time to settle before that edge.

TINY-4 is deliberately simple: its fetch, decode, execute, and update fit within one clock. The load simulator relaxes that assumption because even an L1 hit takes several cycles; it holds the thread while memory responds. A pipelined core, which we will see next, adds temporary registers between stages so several instructions can be in progress at once.

Across these models, two correctness rules survive:

• Dependencies are honoured. The second mul reads r1, which the first mul wrote. It must see the new value, 9, not the old one. In the simple core this is automatic, because the first instruction's write back happens at a clock edge before the second instruction reads.
• Visible behavior is preserved. The core may overlap or reorder internal work, but the program must observe results allowed by its instruction and memory rules. The simplified examples here match one-at-a-time execution.

Keeping every box busy: overlapping the phases

In the one-instruction-at-a-time picture, much of the hardware waits while another box works. Pipelining overlaps instructions by placing temporary registers between stages: while one instruction executes, the next can decode and another can fetch. The timeline below uses a different timing model from TINY-4: each instruction has five one-clock stages, F, D, X, M, and W. Doing four instructions serially takes 4 × 5 = 20 clocks; overlapping the stages, with one dependency stall, takes 9.

The drawn timeline gives the load a one-clock M stage to isolate the pipeline idea. A real miss to DRAM can add hundreds of cycles. The first multiply needs that load's result, and the later instructions depend on the multiply, so this short program has no independent work to cover the wait. That is the opening for the rest of the lecture: use other work while one chain is blocked.

Why this matters for the rest of the lecture

Once you see the three boxes as separate resources with a clear contract between them, every design in this lecture is just a different answer to one question: which box do you duplicate?

We can now compress the long trace into one picture. A compiler produces an instruction stream—loads, arithmetic, branches, and stores. The PC selects the next instruction; fetch/decode interprets it; the execution unit works with values held in registers or returned by memory. In the simplest model, one stream uses one context and one execution path:

The next question is whether this one stream contains independent instructions that the core could overlap. That is the last kind of parallelism we can find without splitting the program into threads. Unit 1 builds a version of the simple machine from gates.

In the load-and-cube trace, the first multiply had to wait for the load. Other programs have instructions that can run together. In the explorer, a = 3 and b = 5 are independent; c = a × b must wait for both. An arrow in the dependency graph marks such a “needs this result” relationship. Safe overlap of independent instructions from one stream is instruction-level parallelism (ILP). Try issue width 1, then 2, then unlimited; compare which clocks still contain only one instruction.

A superscalar core can start several instructions in a clock when it has enough execution units and ready inputs. More advanced cores use out-of-order control to find an independent later instruction while an earlier one waits. This diagram considers one thread and one execution context; its overlap comes from within that thread's instruction stream.

The graph also shows the limit: extra issue slots cannot shorten a dependency chain. How much width pays off depends on the program. For the array calculation coming next, the abundant independence is mostly between elements, rather than between nearby instructions for one element.

The load in Section 0 named x[2], but the program did not specify where hardware should find it. A program sees an address space: byte locations numbered by addresses. A cache holds recently used data close to the core. A hit finds the requested data there; a miss looks in a slower cache level or DRAM. For a correctly synchronized program, this changes access time, not the memory result promised by the machine.

Caches move data in fixed-size cache lines, rather than fetching just the named byte. A request for address 0x0 in the toy cache below brings in its four-byte line, addresses 0x0–0x3. A later request for 0x1 can then hit without another trip to memory.

Try it: a two-line cache

The explorer has a 16-byte array and a cache that holds either two or four lines of four bytes each. It is fully associative: any line can occupy any slot. When full, it evicts the least recently used (LRU) line. Step through “locality demo” first and watch which requests hit, then read the array twice with both capacities.

The hierarchy, and why it matters

Real machines stack cache levels: small, fast L1 near a core; larger L2; often a shared L3; then DRAM. A load checks near levels first and goes farther only on misses. The chart gives approximate total access latency for each destination on an Intel Kaby Lake desktop CPU at 4 GHz; the times are alternatives, not numbers to add together.

For the rest of the lecture, keep one job in mind: given an array of N numbers, approximate sin(x[i]) for each element and store it in y[i]. The input angle is in radians. One way to approximate sine is to add a fixed number of terms from its Taylor series:

sin(x) = x − x³/3! + x⁵/5! − x⁷/7! + …

The code starts value at x, then adds −x³/6, then +x⁵/120, and so on. numer, denom, and sign prepare each next term without recomputing powers and factorials from scratch. A finite term count makes this an approximation; for large inputs, practical sine implementations also reduce the angle's range. Here the term count is fixed so every array element follows the same loop structure.

As written, one instruction stream works through the array in index order. A superscalar core may overlap some instructions within an iteration, but it cannot make the running sum of one element independent of itself. The abundant parallel work is across different elements. We will use that fact three ways.

One way to use a chip's transistor budget is to make a core more elaborate: add out-of-order control, branch prediction, and caches to accelerate one stream. Another is to build more cores. The diagram makes the trade-off deliberately simple: replace one “fancy” core with two smaller ones.

If we leave the sine loop as one sequential stream, it uses one of the two simple cores and, under the diagram's assumptions, runs 25% slower than on the fancy core. We already know that different elements are independent; software must expose or establish that fact so a compiler or runtime can divide them safely. Compilers sometimes prove independence automatically, but an ordinary loop alone is no promise that its iterations can be split.

Expressing the parallelism

Option 1: threads. Create a second software thread and give it half the array. The main thread computes the other half, then waits for the helper to finish ("join"). With a C++ threading library that's a few lines. The programmer decides the split explicitly.

Option 2: declare independence. The lecture's imaginary forall means “run iterations i = 0 through N − 1 in any order.” A compiler or runtime can divide those iterations among cores. Real tools express related ideas through constructs such as OpenMP's parallel for; later lectures use ISPC and CUDA for other forms of data-parallel work.

With a fixed number of Taylor terms, eight sine iterations can advance through the same step together: each needs its own x[i] and running sum, but all can multiply their current numerators at once. SIMD—single instruction, multiple data—fetches and decodes one vector instruction and applies it across eight lanes. Each lane handles a different element, so the instruction-control cost is shared.

A C implementation can process eight array elements per loop trip, advancing i by eight. value and numer become vectors because their values differ by lane; the shared term count, denominator, and sign may remain scalar and be used by every lane. Intel AVX intrinsics, C functions that request particular vector instructions, include _mm256_mul_ps(a, b) for eight pairwise float multiplies. If N is not divisible by eight, the remaining elements need a shorter masked vector or a scalar cleanup loop.

Vector instructions can come from hand-written intrinsics, a language construct that expresses independent element work, or an auto-vectorizing compiler that proves a loop safe to group. On a CPU, those choices produce vector instructions in the compiled binary: explicit SIMD. The compiler still has to account for dependencies, alignment, and leftover elements; “same arithmetic” by itself is not enough.

The fixed-term sine loop gives eight lanes the same path. Other array loops contain a branch, so lane 0 might need one operation while lane 1 needs another. Imagine each lane holds one t = x[i] and runs this condition:

One vector instruction cannot perform two different operations in its lanes at the same instant. When both sides are needed, the execution follows them in turn with a mask marking which lanes are active for each side. CPU compilers can emit masked operations or compute and blend results; GPU hardware tracks active lanes within a group of threads. The explorer models the shared cost: click lanes to change their conditions, then watch how many lanes do useful work in each clock.

If only one lane takes a long branch, the other seven are idle during that branch. For example, with one lane per consecutive index, if (i % 8 == 0) { long computation } gives just one active lane in an eight-lane group for the long part: 1/8 of that vector unit's lane capacity during those instructions. Other parts of the program may still use all lanes.

Useful vocabulary

• Coherent execution: the grouped elements follow the same instruction path, so most SIMD lanes can stay active. Different CPU cores have separate instruction streams and need not follow the same path.
• Divergent execution: grouped elements need different paths, leaving some lanes masked while another path runs.

Explicit versus implicit SIMD

CPU vector instructions name a vector operation directly: explicit SIMD. On NVIDIA GPUs, programmers describe many logical scalar threads. Hardware groups neighboring threads into fixed 32-thread warps and schedules the warp's instructions across execution lanes. This is often called SIMT (single instruction, multiple threads) or implicit SIMD. A divergent warp must serve its different paths with only the appropriate lanes active; hardware does not gather arbitrary threads from across the chip just because they reach the same instruction.

Peak rates count what the hardware could do if instructions, independent data, and memory supply all line up. They are useful as ceilings; the program's achieved rate can be much lower. The next two calculations use the particular chips and operation assumptions shown in the lecture.

Worked example: peak throughput of a real CPU

The sine array has independent elements, but each element still has to load its input. If a later operation needs a value that has not arrived, that dependency chain stalls. A DRAM trip takes roughly 250 cycles on the example Kaby Lake machine. An out-of-order core may run other ready instructions or overlap several loads during that time; if it runs out of such work, its execution units go unused.

Technique 1: caches reduce the length of stalls

If a line is already in L1, a load may take about four cycles rather than a DRAM trip of hundreds. Reusing a recent line or a neighboring byte in that line makes hits likely. A first demand for a cold line cannot benefit from temporal reuse, though a prefetch or an earlier request for a nearby address might have brought it in. If the working set exceeds the cache, useful lines can be evicted before their next use.

Technique 2: prefetching hides stalls

The sine loop walks predictably through x[0], x[1], and onward. A prefetcher can notice such address patterns and request future cache lines before the program asks for them. If the line arrives in time, a later load hits. Prefetching overlaps data travel with earlier useful work; it does not make the memory device itself faster.

Now imagine x = some_function(); y = A[x];, where x becomes known only after a dependent computation and A is too large for cache. A prefetcher may have no reliable next address, and the load may miss. We need a way to use the core while that request is outstanding.

Think of starting a wash, then cooking dinner while the machine runs. A multithreaded core keeps several threads' register and PC contexts ready. Thread 1 can request data for one group of sine elements; while it waits, the core issues instructions from thread 2's independent group. By the time other threads have taken turns, thread 1's data may be ready. The memory request still takes just as long; the core has found useful work during the wait.

No free lunch: context storage

Each resident hardware thread needs registers for its live values. The register storage is finite: a workload that uses few registers per thread can keep more threads resident, while one with many live values may fit fewer. More resident threads offer more chances to hide a stall, but only when those threads have independent work and memory can serve their requests.

Interleaved versus simultaneous

Interleaved (temporal) multithreading: each clock, the core picks one thread and runs its instruction. That's the model in these diagrams. Simultaneous multithreading (SMT): each clock, the core can pick instructions from several threads and run them on different ALUs at once. Intel's Hyper-Threading, with 2 threads per core, is SMT. Either way, the core has the same number of ALUs. Multithreading doesn't add compute. It only helps keep the existing ALUs busy when some threads are waiting.

Count how many ready threads are needed to cover a memory wait. The explorer uses a deliberately simple one-issue core: one thread occupies it for k arithmetic clocks, then launches a memory request and waits L clocks before its next arithmetic burst. The request itself takes no separate issue slot in this model. Real loads do use hardware resources, so the formula below is a teaching estimate, not an exact sizing rule for a real chip.

One thread's cycle is k clocks of arithmetic plus L of waiting. If T threads stagger their bursts perfectly, together they can supply T × k busy clocks in that k + L interval. Utilization cannot exceed 100%, and the smallest whole-number T that covers the interval is:

threads needed = ⌈(k + L) / k⌉ = ⌈1 + L / k⌉     utilization = min(1, T·k / (k + L))

Put the pieces together in the lecture's fictitious chip. Sixteen cores give independent instruction streams somewhere to run. Eight SIMD lanes per core process groups of elements. Four contexts per core keep replacement threads ready when one waits. These counts play different roles: cores and lanes add arithmetic capacity; contexts help use that capacity during stalls.

A real CPU core: Intel Skylake / Kaby Lake

A GPU core: one NVIDIA V100 SM

A GPU pushes the same choices toward throughput. NVIDIA calls a major execution block a streaming multiprocessor (SM). An SM holds many warps, fixed groups of 32 logical threads that execute together. When one warp waits for memory, another ready warp can use the execution units.

The sine example gave each mechanism a job: independent array elements fed cores, fixed-term loops filled SIMD lanes, and extra ready threads could cover a load miss. The diagram collects those three requirements. They are useful checks, but satisfying them does not guarantee that memory can deliver data quickly enough.

A thought experiment: multiplying two big vectors

Now multiply two very large arrays: C[i] = A[i] × B[i]. Each element is independent, follows the same instructions, and offers plenty of tasks to keep threads resident. Yet every multiply needs two input values delivered and one output stored. Even an ideal schedule cannot multiply faster than memory can supply those bytes. Use the explorer to compare that supply with the chip's nominal compute rate.

Build a chip one choice at a time. Adding cores or SIMD lanes raises its peak arithmetic capacity; adding contexts gives each core more threads to turn to during stalls. Use the presets first, then change one control and watch which counter moves. The “work needed” counter describes enough elements to populate every context and lane, not a minimum for arithmetic saturation when one well-fed thread can keep a core busy.

Try the superscalar preset: it has one resident thread and two issue slots. The distinct-stream counter remains one, while two independent instructions from that one stream could issue together. Then add a second hardware context. The arithmetic peak stays the same, but a second thread can use a slot or cover a stall. That difference is central to reading the next simulator.

The closing diagrams combine issue slots, scalar and vector units, contexts, and cores. More execution units or cores can raise arithmetic peak, the operations the hardware could do each clock. More contexts do not add arithmetic units; they can raise achieved throughput by giving a stalled core another thread to run. The matrix and simulator let you test both questions: what capacity did the design add, and what work would use it?

The four dials, and what each one asks of the program

The permutation matrix

Rows are four simplified core designs; columns choose contexts per core. Set core count and, for two-issue designs, whether both instructions in a clock come from one thread or may come from different threads (SMT). Each card shows arithmetic peak and a full-residency count: enough streams to fill every context, with eight elements per stream for SIMD. Those counts are capacity markers, not universal minimums for peak arithmetic. Click a card to load a sample workload intended to exercise the design under the simulator's assumptions. For a 16-core card, the simulator demonstrates the same per-core design with four cores, while the matrix retains the 16-core peak. If the sample cannot cover a 12-clock memory wait with the available contexts, it sets modeled memory wait to zero; that is a simulation choice, not a claim that real cache hits take no time.

Try it: the saturation simulator

Pick a design, then change one software control at a time: thread count, independent instructions per stream, scalar/vector mix, active SIMD lanes, and modeled memory waiting. The simulator runs 120 clocks per core and reports achieved lane operations beside the chip's peak. Start with no modeled memory wait to see arithmetic limits; then raise latency and add contexts to see when extra threads recover otherwise empty slots.

Use “fraction of peak” to judge the actual modeled run. The checklist also marks whether every context and every possible SIMD work item is filled; those are full-occupancy checks, so they can show a shortfall even when a smaller number of compute-bound threads already keeps the arithmetic units busy.

The general recipe

For a chip with C cores, T contexts per core, issue width W, and SIMD width S, use the table as a set of separate checks. Some describe arithmetic peak; others describe filling all resident contexts or hiding a particular memory wait. They are not six universal minimum requirements for every program.

In row 5, k is useful arithmetic clocks between waits and L is the modeled wait in clocks, as in Section 11. The ceiling rounds up to a whole thread. A two-issue core or a mixed scalar/vector design needs additional checks: a thread may issue two operations together only when they are independent and suitable units are free. The matrix's latency percentages are planning heuristics; the discrete simulator gives the outcome for its particular scheduling rules.

Instruction stream

The sequence of instructions one thread executes. Its execution context holds that thread's registers and PC.

Superscalar

One core issuing several independent instructions from the same stream per clock, found automatically by hardware.

Multi-core

Several cores on one chip, each able to run an independent instruction stream.

SIMD

One instruction applied across multiple data lanes. CPU binaries may name vector operations explicitly; GPUs schedule fixed groups of scalar-looking threads together and mask lanes when their paths differ.

Coherent control flow

Many elements following the same instruction sequence, which SIMD needs. Its absence is divergence, handled by masking lanes.

Hardware multithreading

Several execution contexts resident on one core so it can switch threads to hide stalls. Interleaved: one thread per clock. Simultaneous (SMT): several threads in the same clock.

Latency vs throughput

Time for one task or request versus the number completed per unit time. More resident threads can improve throughput during waits; contention may increase an individual thread's time.

Stall

A period when dependent work cannot advance because an input has not arrived. Other independent work may still run.

Bandwidth

The rate at which data can move, measured in bytes per second; distinct from the latency of one request.

Based on Stanford CS149 (Fall 2023), Lecture 2, "A Modern Multi-Core Processor", by Kayvon Fatahalian and Kunle Olukotun. Explanations, diagrams and interactive models on this page are original study material. Processor figures (Kaby Lake latencies, i7-7700K, V100) are as presented in the lecture and in vendor documentation. For the course's own slides and videos, see the CS149 course site.