Branches, Indirect Calls, and Inlining

The update loop has two control-flow transitions:

Copy
for (const MarketUpdate& update : updates) {
    if (update.symbol[0] == 'A') {   // conditional branch
        apply_update(update);        // call
    }
}

In C++, both lines look like quick single-step operations. At the processor level, their runtime cost depends on whether the hardware can guess the next instruction address before the current one finishes, and whether the compiler can see inside apply_update.

Branch prediction and pipeline bubbles

Modern CPU cores pipeline their instructions. Instead of waiting for one instruction to complete before fetching the next, a core overlaps multiple instructions across fetch, decode, execute, and retire stages.

Sequential code feeds this pipeline smoothly: the processor simply loads instructions from the next address in memory. A conditional branch interrupts that stream. When the processor encounters if (update.symbol[0] == 'A'), it cannot know whether the next instruction is the first line of apply_update or the start of the next loop iteration until update.symbol[0] is loaded and compared.

Rather than letting execution units sit idle, the hardware predicts which way the branch will go based on past history. It then speculatively fetches and runs instructions along that guess.

When the branch outcome matches the guess, execution continues without a pause. When it guesses wrong, the core has to discard the speculative work, flush the pipeline, and restart fetching from the correct address. On a modern x86 core, that pipeline bubble costs roughly 15 to 20 clock cycles.

A correct guess keeps the pipeline full. A misprediction flushes speculative work and leaves an execution bubble.

This makes the cost of a branch depend heavily on your data. If 99% of your updates are for Apple, the predictor learns the pattern quickly and the check is practically free. If symbols are evenly scrambled between Apple and other tickers, the branch behaves like an unpredictable coin flip, and the misprediction penalty shows up on a large fraction of iterations.

Direct vs indirect calls

The function call on the next line raises a similar question about target addresses:

Copy
apply_update(update);

In a direct call, the target function is named in source code. The compiler writes the relative jump offset directly into the machine instruction (call rel32). The processor knows where it is jumping the moment it decodes the instruction, so instruction fetch continues down apply_update without needing to resolve a pointer first.

An indirect call jumps through a pointer or register instead:

Copy
handler(update);             // function pointer or std::function
strategy->on_update(update); // virtual method call

Here, the address is stored in memory. The core has to load the pointer before it can jump to it. While hardware branch target buffers (BTBs) attempt to predict indirect jump destinations, an indirect target that changes across calls can trigger pipeline flushes just like a mispredicted branch.

A direct call jumps to a fixed function. An indirect call has to load the destination address at runtime.

What indirect calls take away: inlining

The pointer load and BTB prediction are real costs, but for small hot functions, the bigger penalty is usually what the indirect call prevents.

When a compiler can see the target of a direct call, it can inline the function body into the caller. Inlining eliminates the call and ret instructions and skips setting up a stack frame. More importantly, it brings the function's code into the caller's optimization scope. Once the callee is inlined, the compiler can reuse registers, propagate constants from the surrounding loop, and eliminate redundant calculations.

Inlining replaces the call with the function body, allowing optimizations across the combined code.

An indirect call acts as an optimization barrier. Because the compiler cannot know which implementation will run at runtime, it must prepare for the worst: it has to assume the callee might read or modify any memory accessible to the program. That forces the compiler to spill local variables from registers to the stack, reload memory values it could otherwise have kept cached in registers, and abandon loop optimizations across the call site.

Writing around branch and call costs

When an unpredictable branch is unavoidable in hot code, you have two common ways to handle it.

First, you can sometimes make the computation branchless. If the branches only select values or perform simple math, compilers can use conditional move instructions (cmov on x86) instead of conditional jumps. A conditional move evaluates both paths and selects the result in a register based on CPU flags, so the instruction pipeline never has to guess or flush.

Copy
// Branching: prone to mispredictions if side alternates randomly
if (u.side == Side::BUY) {
    total_buy_qty += u.qty;
}

// Branchless style: computes without a conditional jump
total_buy_qty += (u.side == Side::BUY) * u.qty;

Branchless code is not an automatic win for every branch. If a branch is already 99% predictable, the branch instruction is often faster because the processor only executes the taken path, whereas branchless code always evaluates both sides. But when data is genuinely random, branchless selection avoids pipeline stalls.

Second, if your algorithm processes data in batches, sorting or partitioning the buffer by condition groups identical outcomes together. Scanning a run of 1,000 buys followed by 1,000 sells lets the branch predictor stay pegged at near 100% accuracy.

For function dispatch, keep hot-path handlers concrete whenever possible. Using templates or direct function calls lets the compiler inline the logic and keep intermediate state in registers. Reserve virtual dispatch and std::function for cold configuration, startup, or message routing outside the inner loop.

In interviews

• "Why can two O(n) loops with identical logic take very different times if one has an if statement?" → "Branch prediction. If the condition is predictable, the processor keeps speculative instructions flowing through the pipeline. If the condition is random, mispredictions cause the pipeline to flush and stall for 15 to 20 cycles on each miss."
• "What is the main cost of calling through a function pointer or std::function in a hot loop?" → "The indirect call is an optimization barrier. The CPU has to load the target pointer and predict the jump, but the bigger issue is that the compiler cannot inline the function, preventing register reuse and dead-code removal across the call."
• "When would you choose branchless code over a normal if?" → "When the branch outcome is truly unpredictable, like random market ticks. Branchless code evaluates both sides and uses conditional moves (cmov) to select the result without pipeline stalls. If the branch is predictable, though, a normal branch is often faster because it skips the untaken path."
• "What does inlining give you besides saving the call and ret instructions?" → "It brings the callee into the caller's optimization scope. The compiler can allocate local variables to existing registers, propagate loop constants into the function, and eliminate redundant memory loads across the call."