Type-Safe Computer Vision · Part 3 of 3

Climbing the Pyramid

To find an object at any size, you shrink the image until the object fits your detector. Simple idea, beautiful theory. In practice, a staircase of traps: sizes that stop matching, two “scales” that aren’t the same thing, and coordinates that quietly drift half a pixel per step.

Thomas Stephan · 21 September 2026 · Series index

Why pyramids exist at all

Your detector looks for a face 40 pixels wide. The face in the photo is 400 pixels wide. Nobody retrains the detector. You shrink the image: half size, quarter size, eighth size, and run the same detector on every level. Stack the levels and you get the data structure every vision textbook opens with:

Each level is made by blurring the previous one a little (so shrinking doesn’t alias) and throwing away every second pixel. Two operations, blur and shrink, done together, over and over. Hold that thought: two operations. It becomes important in a minute.

Trap 1: the staircase doesn’t come back up

First trap, and a recurring question on every programming forum: take a 101-pixel-wide image down one level, then bring it back up.

Why does anyone go back up? Because the most useful pyramids are built from differences: subtract the blurry upsampled level from the sharper one above it and you isolate the detail that lives at that scale, the basis of seamless image blending, HDR fusion, and half of classical vision. And subtraction needs both images to be the same size. Let one odd number appear anywhere down the chain and the naive round trip breaks, loudly if you’re lucky, silently if you’re not. scikit-image shipped a broken version of this construction for years: its pyramid_laplacian cannot rebuild the original image, which its maintainers confirmed and which was still unfixed in September 2026. Why did nobody notice? A core developer answered that in one sentence: “somewhat bizarrely, we don’t have a reconstruction test in the test suite.”

The root cause is structural: shrink() and grow() are two independent, stateless functions. shrink() does not record the parent’s size, and grow() does not ask for it. The information “101 was odd” falls into the gap between two function calls.

Trap 2: there are two ladders, not one

Remember the two operations, blur and shrink? Most libraries fuse them into one function and call the result “scale.” But they are different axes, and you can move along each independently:

This distinction has a proper name. The blur axis is called scale-space: how much fine detail has been smoothed away, measured by the blur radius σ (sigma) in pixels of the original image. The shrink axis is just resolution: how many samples you kept. A GPU mipmap shrinks without caring about σ. A blob detector sweeps σ without shrinking at all. The classic Gaussian pyramid does both in lock-step, which is precisely why generations of developers learned they were one thing.

SIFT, the most famous algorithm in classical vision, finds keypoints you can recognize again in another photo. It needs both axes at once, as a grid: within each “octave” it keeps resolution fixed and steps the blur; between octaves it halves the resolution. The bookkeeping looks like this:

And how does OpenCV report which cell of that grid a keypoint came from? It packs the octave, the layer, and a sub-scale fraction into a single 32-bit integer. Users see keypoint.octave == 10420482, file a bug, and are pointed to an unexported internal function whose decode ritual involves masking the low byte and manually sign-extending it, because octave −1 is valid. The report has been open since 2015; the packing was reclassified from bug to feature and kept for backward compatibility. One commenter asked the question this whole series is about: “How should a typical user know that?”

Trap 3: the way back down is off by half a pixel

You found your object at level 3. Now: where is it in the original image? Everyone writes the same line: x_full = x_level * 8. And everyone ships a subtle lie, because that formula assumes a coarse pixel sits exactly where a full-resolution pixel sits, and depending on how the library shrinks, it usually doesn’t:

Whether that multiplication is honest depends on how the level was made. A coarse pixel sits at the middle of the samples it came from, so the question is whether those samples have a middle. An odd, symmetric footprint does, and the coarse pixel inherits that sample’s position, landing exactly on the fine grid. An even one does not, and the pixel lands on the boundary between two, half a pixel off. A 5-tap Gaussian is odd, which is why a proper Gaussian pyramid stays aligned; averaging 2×2 blocks is even, which is why it drifts, half a pixel at every step. And note that averaging a block is a kernel, an even-width one, so the developers most exposed to this are the ones who believe they are not filtering at all.

If half a pixel sounds like a rounding error, consider that whole framework ecosystems fought a multi-year war over exactly this: the infamous align_corners flag, whose default was wrong in TensorFlow 1 for years. The flag still means different things in different frameworks: with align_corners=False, coarse pixel k sits at 2k in TensorFlow 1 and at 2k + 0.5 in PyTorch. Same name, same value, half a pixel apart. Sub-pixel keypoints, optical flow, stereo: the entire point of these algorithms is precision below one pixel. A convention mismatch eats the whole error budget before your algorithm even starts.

Three traps, one pattern: the pyramid’s geometry (parent sizes, blur levels, grid offsets) lives in nobody’s data structure. It lives in comments, in conventions, and in rules chanted around a campfire on a foggy night, whose songs have faded by morning.

Give the geometry a home

None of these are logic bugs. The blur was correct, the subsampling was correct, the multiplication by 8 was multiplication by 8. What failed, every time, was unrecorded meaning: an odd parent size the down-step forgot, a σ nobody wrote down, a grid offset that existed only in the implementer’s head. Type bugs, the kind a type system can refuse to compile.

Here is how fovea answers all three, as of v0.5.0 (September 2026). The principle: a level’s geometry is a value it carries, not a convention you are expected to remember.

Trap 1, the staircase that would not come back up. A loose 51-pixel-wide image really is ambiguous, so the free up-step asks for its target and there is no overload that guesses. Inside a pyramid the parent size was recorded on the way down, so there is nothing left to ask for. Both calls exist, and the difference between them is the entire point:

let base: Image<Rgb8> = load("photo.png")?; // 101 × 68, an ordinary odd size
let pyr = Gaussian.build(&base, 3);        // 101×68, 51×34, 26×17

// An image on its own has lost the number 101, so the free function has
// to be told, and refuses to guess:
let back_up = pyr_up(&loose_level);
// error[E0061]: this function takes 2 arguments but 1 argument was supplied
//   = note: a 51-wide image could have come from 101 or from 102

// Inside the pyramid the parent is right there, so nothing is asked:
let back_up: Image<Rgb8> = pyr.expand(1).expect("level 1 has a parent");
// 101 × 68, exactly. The Option asks "is there a level above this one?",
// which is the only question left once the container was checked once.

Trap 2, the two ladders. Blur and resolution are separate capabilities, Decimated and ScaleLevel, and a level can hold one without the other. A Gaussian pyramid knows exactly where its samples sit, because it worked that out on the way down. It knows nothing about blur, because blur depends on a fact nobody gave it. So one ladder answers and the other is a compile error, and that asymmetry is the honest one:

let pyr = Gaussian.build(&base, 3);

// The resolution ladder: derived on the way down, kept by the level.
pyr.level(1).pixel_distance(); // 2.0, in base-image pixels
pyr.level(1).origin_offset();  // (0, 0): pyr_down keeps even samples

// The blur ladder is not derivable, so asking is a compile error:
let sigma = pyr.level(1).sigma();
// error[E0599]: no method named `sigma` found for reference `&PlacedImage<Rgb8>`
//   = note: `ScaleLevel` is implemented for `ScaledImage`, not `PlacedImage`

// It needs one fact only you have: how sharp was the input? Name it, and
// every level states its own blur.
let pyr = Gaussian.assuming_input_sigma(sigma!(0.5)).build(&base, 3);
pyr.level(1).sigma(); // 1.118

// SIFT needs both ladders at once, so its grid is its own type, indexed
// (octave, layer). No octave·(S+3)+layer by hand, no three values in one int.

That 1.118 deserves its own paragraph. One pyr_down applies a blur of σ = 1 measured in its parent’s pixels, so level 1 has σ = 1 only if the input was perfectly sharp. No photograph is; Lowe’s SIFT assumes σ = 0.5 for its input for exactly this reason. Under that assumption level 1 is √(0.5² + 1²) = 1.118, because variances add and σ does not. The gap widens as you climb: by level 3 the tempting extrapolation from the sampling distance is 13% short of the truth, silently. Writing σ = 1.0 for level 1 assumes a perfectly sharp input. That is why the method is called assuming_input_sigma: you cannot get a σ out of a pyramid without first saying what you assumed going in.

Trap 3, the way back down. The fix is to remove the multiplication altogether. A detector that runs on a level reports in the base frame:

// Detection on a level already reports base-frame positions: nothing to multiply.
let level = pyr.level(1);
let corners = fast_in_level(level, params, &Skip);
// found on the 51×34 level, reported in 101×68 base coordinates

// A point you lift yourself goes through one named method, in both directions:
let peak    = CoordinateF64::new(5.0, 3.0); // a sub-pixel peak, in level coordinates
let in_base = level.to_base(peak);          // origin_offset + pixel_distance·peak
                                            // gives (10.0, 6.0)
let back    = level.to_local(in_base);      // (5.0, 3.0) again: the exact inverse
// The half-pixel term is part of the contract of how this level was built,
// not something you had to be at the campfire for.

And because this is the same library from part 2: building any of it from a gamma-encoded image is still a compile error. A pyramid is blurs all the way down, so every level would repeat the 128-vs-188 mistake, so the type system demands linear light at the front door.

Why this matters more every year

Across three articles we met three bugs: swapped channels, gamma-blind averaging, and pyramid geometry lost between function calls. None of them was a wrong algorithm. All of them were wrong assumptions flowing silently between correct functions, and all of them are reproduced by the thousand in forums, tutorials, and now in the training data of every coding agent that writes vision code. The agents are fast, tireless, and trained on twenty years of examples that average sRGB bytes and multiply by 2^octave.

Vision systems are leaving the demo notebook for places where a wrong image is a lost measurement or a missed defect. We cannot review our way out at that scale, but we can make the mistakes unrepresentable. Write the meaning into the types: what order the channels are, whether the values are light, what geometry a level carries. Then every one of these classics (human-written or machine-written) stops being a debugging session and becomes a red squiggle.

The compiler was always willing to check our assumptions. We just had to write them down.