Why parallelism? Why efficiency?

Suppose you need to add a million numbers. One worker could add them all; several could each add a share and then combine their partial sums. A computer doing the latter is a parallel computer: a collection of processing elements (workers such as CPU cores) that cooperate on a problem. Cooperation is useful only if the workers save more time on arithmetic than they spend dividing work, exchanging results, or waiting for one another.

We measure the result with speedup. Run the same job with one processor and with P processors, then divide the two execution times:

If the job takes 60 seconds on one processor and 20 seconds on four, its speedup is 60 / 20 = 3×. Four processors could ideally finish in 15 seconds, a 4× speedup, if every part divided perfectly and cooperation were free. A second measure compares the observed gain with that ideal:

Here, efficiency is 3 / 4 = 75%: the observed speedup is three quarters of the ideal fourfold gain. That number tells us how far we are from ideal scaling, but not why. The next experiment separates two possible causes: uneven work and the cost of combining results.

The lecture makes processors out of students and asks them to add a pile of numbers. The explorer below models the same exercise. Split N numbers among P processors. Each processor adds its own share (one time unit per addition) and produces a partial sum. Those partial sums must become one final answer: passing one to another processor costs c time units per message, and adding two partial sums costs one more.

Start with the presets in order. Each changes one source of delay:

1. Demo 1: communication costs time. Two workers split the additions, but one must pass its result to the other. Moving closer reduces c, so the same arithmetic finishes sooner.
2. Demo 2: an uneven split leaves workers waiting. With four workers, the job finishes only when the last one finishes. Balancing the shares shortens that longest stretch.
3. Demo 3: too many workers can slow the job. Each worker gets fewer additions, but gathering more partial sums takes longer. A different combining plan can change the outcome.

Read the timeline from left to right: the job ends when its last required result arrives, not when the average processor finishes. Demo 2 shows this directly. Next compare Demo 3's two combining plans. If all P workers send to processor 0, it handles P − 1 messages in sequence. In a tree, pairs combine at the same time, then pairs of pairs combine. For 32 workers, that takes five communication rounds rather than 31 sequential messages to one receiver. The tree still communicates; it makes much of that work overlap.

A 2× speedup on 10 processors reaches only 20% of ideal scaling. That figure alone cannot tell us whether processors sat idle, spent time communicating, or did extra work; the timeline and the program can. The programmer chooses the work split and coordination. The hardware designer chooses capabilities under limits of chip area, power, and cost. Understanding both sides will matter once we ask why computers acquired so many parallel workers in the first place.

Our summing experiment showed that parallel work takes planning. For a long time, programmers could often avoid that planning. From the mid-1980s to the early 2000s, a program running on one processor typically gained speed as new processors arrived; single-thread performance grew at roughly a doubling every 18 months over that period. The cumulative gain was more than a thousandfold.

That changed the programmer's calculation. Rewriting a program for two processors might take substantial effort and, after communication costs, yield less than 2×. At the historical rate on the chart, a new processor could provide a comparable single-thread gain in about a year without rewriting the program. The benefit came from new hardware, not from free computation; for software authors it felt like a free lunch.

Two important sources of that single-thread gain were:

1. Instruction-level parallelism (ILP): a processor overlaps independent instructions from one program, even though the program was written as a sequence.
2. Higher clock frequency: more clock cycles each second give the processor more opportunities to advance work.

Neither source could keep delivering the same gains indefinitely. To see the limits, we first need a picture of the instructions a processor receives and the state they change.

Start with x = 1. Double it ten times and the result is 1024. In C, a short loop expresses that whole job; a compiler turns it into machine instructions. Each asks for a smaller operation, such as copying a value, adding, comparing, or jumping to another instruction. The figure writes those instructions in readable assembly notation; the processor receives their binary encoding.

You can read these instructions like a recipe: each line says what to do next, and later steps rely on values produced by earlier ones. The processor must preserve those relationships to get the right answer. As we will see, it can still overlap steps that do not rely on each other.

Follow one trip through the unoptimized loop. It tests the counter, loads x, doubles and stores it, increments the counter, and jumps back. The addl that doubles x is only one of nine instructions on that path; the rest make the loop work. An optimizing compiler could reduce this bookkeeping. The example makes clear why speed depends on more than the arithmetic operation alone.

To run the compiled program, a processor repeatedly finds an instruction, gets its input values, performs the operation, and records the result. The lecture's very simple processor groups the relevant hardware into three parts, drawn consistently in the diagrams:

• Fetch/decode uses the program counter to find the next instruction and determine what it asks for. The program counter is the address of the next instruction.
• The ALU (arithmetic and logic unit) performs operations such as addition. This model also needs a way to load from and store to memory, which we will meet later.
• The execution context includes registers, small storage locations that hold values the program is using. Together with memory and the program counter, those values form the program's state: the information needed to continue execution.

Step through add R0 ← R0, R1. It means: read R0 and R1, add their values, then put the answer in R0. The arrow points to the destination; R1 keeps its old value.

These are phases of an instruction, not necessarily four separate clock cycles. For the timing examples that follow, the lecture assumes a simplified processor that finishes one instruction per clock. Real processors overlap phases and handle several instructions at once. Lecture 2's Section 0 lets you follow whole programs phase by phase.

The simple processor finishes one instruction per clock. Consider a calculation that has more independent work: a = x*x + y*y + z*z. Put x, y, and z in registers R0, R1, and R2. In our simplified instruction set, the calculation takes five instructions:

At one instruction per clock, this takes five clocks. But the three multiplies have no dependencies on one another: each reads and writes a different register. A processor with more execution capacity can overlap them. Each add must wait for its own inputs: instruction 4 needs the results of 1 and 2, and instruction 5 needs the results of 3 and 4. An independent multiply can still share a clock with instruction 4 if its inputs are ready.

Those opportunities to overlap instructions from one program are called instruction-level parallelism (ILP). The arrows in the graph show dependencies. Even with unlimited execution units, the chain 1 → 4 → 5 (or 2 → 4 → 5) requires three successive clocks in this one-cycle-per-instruction model. Extra units cannot shorten a chain whose next instruction needs the previous one's answer.

Correct overlap preserves the result of sequential execution; it does not require every instruction to finish in source order. A later instruction may run early when it has all its inputs, but it must never read a value that an earlier instruction still needs to produce. Dependency checking enforces that rule. This is the parallelism a single processor core can find without asking the programmer to split the program into threads.

A processor that can start more than one instruction from a single stream in a clock is superscalar. It has multiple execution units and control logic that tracks which inputs are ready. The program is still written as one instruction stream; the hardware looks for safe overlap, as the previous explorer did.

The five-instruction example had room for three multiplies at once, then only one useful add at a time. Larger programs also contain dependencies and branches that limit how far the processor can look ahead. A wider core needs more execution units and more control logic, but it cannot create independent instructions the program does not have.

This is the first wall in the lecture's argument. Past some width, extra hardware costs chip area and power while yielding little single-thread speed. The limit depends on the workload, but every program has dependency chains. Lecture 2, Section 2 and Unit 3 show how a core looks for the independent work that remains.

A core can also finish work sooner by running its clock faster. Its clock marks cycles in which work can advance, so more cycles per second can mean more instructions completed per second. Clock rates rose for decades but largely leveled off in the mid-2000s, while transistor counts kept growing. The limiting question became how much power a chip could use and cool.

Two contributions matter in a simplified power model:

• Dynamic power is spent when circuits switch. For a fixed amount of switching per cycle, it grows roughly as switched capacitance × voltage² × frequency. Running more cycles each second raises this cost.
• Static power comes from leakage: a powered chip uses some energy even when circuits are not switching.

On a given chip design, reaching a higher clock often requires a higher supply voltage to keep switching reliable. Voltage appears squared in the dynamic-power term, so raising both voltage and frequency can make power climb much faster than clock rate. In the lecture's illustrative range, moving from about 2.4 to 4.8 GHz costs several times the power. Nearly all of that electrical energy ends up as heat that the cooling system must remove.

Power budgets vary greatly across devices. The table gives rough figures from the lecture, including thermal design power (TDP): a cooling-design rating, not a promise that a device always consumes that many watts. The i9-10900K's 95 W figure in the lecture is its configurable lower TDP; its default TDP is 125 W, used here.

The exact numbers are less important than the spread: a phone must work within a few watts, while a desktop chip may have over a hundred. Neither can raise clock speed without considering heat and battery or operating cost. Designers therefore look for more useful work within a fixed power budget.

Spend the power budget on more, slower cores

One option is to divide work between two slower cores. Each core may finish its own portion less quickly than a fast core would, but together they can finish more portions per second within the same budget. That total rate is throughput. Move the power-budget slider and compare the bars for total throughput with the line for one core's speed.

The bars rise only if the work divides across cores; the summing exercise already showed why that assumption can fail. A single thread uses one core and follows the line. In this fixed-budget model, giving the chip more cores lowers the clock available to each, so that single thread can even slow down. Multiple cores create an opportunity for more throughput, not an automatic speedup for every program.

Now the two limits fit together. Wider execution units found diminishing amounts of independent work in one instruction stream; higher clocks became too costly in power. The lecture captures this change in a stylized chart of processor trends, after Herb Sutter's 2005 article "The Free Lunch Is Over":

That shift changed where chip designers spent their growing transistor budgets:

• Single-thread speed still improves, but the old rate of improvement became much harder to sustain through higher clocks and wider issue alone.
• Designers increasingly use transistors for parallel units: more CPU cores and vector units that operate on several data values at once. They also build specialized units for tasks such as graphics, video, and neural networks.
• To benefit from those extra units, software must give them useful work. An unchanged single-thread program cannot use several cores simply because they are present.

The design response is visible across machines of very different sizes. Even a small Raspberry Pi has several CPU cores; a supercomputer joins hundreds of thousands of them with GPUs. The chart counts several kinds of parallel hardware, so compare the scale while keeping the units in mind:

What using it all is worth

The course's first programming assignment gives one striking, workload-specific result. Its optimized (-O3) single-threaded C baseline runs on a quad-core Intel CPU. A carefully rewritten version uses multiple cores, vector instructions, and hardware threads to run about 32–40× faster on that assignment. That gain is evidence of unused parallel capacity in the baseline, not a promise that every program can improve by the same factor.

So far, parallel efficiency has meant speedup divided by processor count. Power adds another question: how much useful work does the machine complete per joule of energy, or how much useful throughput per watt does it deliver? More cores are one answer when work divides well. Another is specialization: hardware designed for a recurring kind of work can often do it with less energy than a general-purpose CPU, by simplifying or sharing some of the CPU's control machinery.

The same principle reaches datacenters. Google's TPU (Tensor Processing Unit), for example, is designed around machine-learning operations and deployed in groups of chips. The lecture shows other neural-network accelerators as well. Their details differ, but each trades some generality for a chance to do a common workload more efficiently.

Before asking why data is slow to reach a core, start with the view a program uses. Its address space looks like an array of bytes, each with a numbered address. A byte holds eight bits, enough to represent a value from 0 to 255. Addresses here use hexadecimal: the 0x prefix marks base 16, so 0x10 is decimal 16. In the explorer, click that address to read the byte stored there.

The simple processor moves values with two kinds of instruction. A load copies data from a memory address into a register; a store copies a register's value to a memory address. In this teaching model, arithmetic uses register values. For example, st mem[R2] ← R0 would write R0's value to the address held in R2. Other instruction sets, including x86, can also use memory operands in arithmetic instructions.

A load starts a request; its data arrives later. The time between asking and receiving is memory access latency, often measured in clock cycles. At 4 GHz, one cycle is a quarter of a nanosecond: a four-cycle cache hit is about 1 ns, whereas a 248-cycle DRAM access is about 62 ns. The precise times vary by machine and where the data is found.

Latency can create a stall, a stretch when the processor cannot advance a needed instruction because its input is not ready. The next three-instruction example loads two values and adds them:

The add needs both loaded values, so it must wait for the slower of the two arrivals. The second load does not depend on the first, however: a processor that allows outstanding loads can send both requests before either returns. In a longer program, it might do other independent work while waiting as well.

The address-space view tells the programmer which value a load requests; it says nothing about how long the load takes. A cache keeps recently used data close to the core. If the requested data is present, the access is a hit and can finish quickly. Otherwise it is a miss, and the hardware must fetch the data from a slower level.

Caches are managed by hardware. A correctly synchronized program still reads and writes according to its memory model, even though caches can make different accesses take very different amounts of time. With multiple processors, keeping their cached views consistent requires extra machinery, a topic for later lectures.

The hardware transfers data in fixed-size, aligned cache lines. In the toy cache below, each line is four bytes: addresses 0x0–0x3 share one line, and 0x4–0x7 share the next. A miss on 0x0 brings in its whole line, so a following read of 0x1 can hit. The cache holds only two lines at once. When full, it uses least recently used (LRU) replacement: evict the line that has gone longest without an access.

Caches help when accesses have locality, a pattern in which future requests are likely to use data already brought close to the core. The examples show two forms:

• Spatial locality means using nearby addresses. In Example 1, the first access to 0x0 misses, but 0x1, 0x2, and 0x3 hit in the line that miss fetched.
• Temporal locality means using an address again soon. Repeated reads of 0x2 and 0x1 hit while their line remains cached.

In this model, a cold miss is the first request for a line that has not yet been brought into the cache. A capacity miss occurs when a previously held line was evicted because the cache could not hold all the lines the program kept using. Example 2 crosses four lines in a cache that holds two; on returning to 0x0, it misses again. Give the cache four lines and that final access hits.

Hits shorten the wait we saw in the load timeline; misses still pay to reach a slower level. A program that repeatedly uses a small collection of nearby data often gets many hits. Later lectures will also consider how fast a cache can deliver a stream of data, which is a bandwidth question rather than a single-load latency question.

A real processor does not have just one tiny cache. A load first looks near the core in level 1 (L1). On a miss, the hardware checks a larger L2, then a larger L3, and finally main memory (DRAM) if necessary. The requested line travels back toward the core, where later accesses may hit. Caches hold copies, so the data need not live in only one level. The program still names one address and receives the value prescribed by its memory model.

A program's working set is the data it keeps using over a stretch of execution. A small working set can fit near the core; a large or widely scattered one may send more loads to distant levels. The explorer estimates average load latency by counting what fraction of loads finish at each level. Read its sliders in sequence: “found in L1” applies to all loads; “of the rest, found in L2” applies only to L1 misses; and the L3 slider applies only to loads that missed both L1 and L2. For example, if 90% hit L1 and half of the remaining 10% hit L2, then 5% of all loads are served by L2.

• Single-thread speed no longer rises at its old pace. Instruction dependencies limit wider execution, and power limits faster clocks. Additional cores, vector lanes, and specialized units offer other paths to performance.
• Those paths need suitable work. A parallel program must divide the job, keep workers busy, and coordinate their results. Speedup and parallel efficiency reveal how well that plan scales.
• Data supply can decide the result. The same program-visible address space may be served by a nearby cache or distant DRAM. Locality helps keep loads short; misses can stall dependent instructions. The course's 32–40× assignment result illustrates what one carefully chosen workload can gain by using its machine's parallel resources.

Speedup

Time on 1 processor ÷ time on P processors. Ideal: P.

Parallel efficiency

Speedup ÷ P: the observed speedup as a fraction of ideal P-fold scaling.

Load imbalance

Some processors have more work than others; the rest sit idle waiting for them.

Communication

Moving data (such as partial results) between processors. It costs time, and can dominate a parallel computation.

Instruction

One simple operation for the processor to perform. A program is a list of them.

State

The information needed to continue a program, including register and memory values and the next instruction's address.

ILP

Instruction-level parallelism: opportunities to overlap independent instructions from one stream while preserving the intended result.

Superscalar

A processor that finds independent instructions in one stream automatically and executes several per clock.

Power wall

The constraint created when raising clock frequency, often with voltage, makes power and heat grow too steeply for continued easy clock-speed gains.

Specialization

Hardware designed around a particular workload, such as graphics or neural networks, to improve speed or energy use for that work.

Address space

Memory as an array of bytes, each identified by its address.

Latency

Time for the memory system to deliver requested data.

Stall

A delay while a needed instruction cannot advance because an input is not ready, often after a load.

Cache, cache line

On-chip storage holding copies of recently used memory, managed in fixed-size lines.

Spatial / temporal locality

Accessing addresses near recent ones / accessing the same addresses again soon. Both make caches effective.

Cold / capacity miss

A line's first requested access / a later miss after a previously held line was evicted for lack of space.

Based on Stanford CS149 (Fall 2023), Lecture 1, "Why Parallelism? Why Efficiency?", by Kayvon Fatahalian and Kunle Olukotun. The explanations, diagrams and interactive models on this page are original study material. Chart data (CPU trends, superscalar speedups, power, latencies and machine specifications) are as cited in the lecture. Charts are redrawn or stylized where noted. For the course's own slides and videos, see the CS149 course site.